Compare commits
79 Commits
feature/un
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
|
9e7658389f | ||
|
|
280764fe9c | ||
|
|
8a780487d1 | ||
|
|
81dbef313d | ||
|
|
159a64a934 | ||
|
|
c13cd0ec1c | ||
|
|
d4bcdc933f | ||
|
|
d09e0f8283 | ||
|
|
397c716c98 | ||
|
|
47a7bfaaea | ||
|
|
be3a44bded | ||
| 3577f560ac | |||
|
|
e533dcb147 | ||
|
|
7382073910 | ||
|
|
c3fe0324ea | ||
|
|
f937afe1fd | ||
| df5308d900 | |||
|
|
ac992ca9f1 | ||
|
|
20864b4746 | ||
|
|
89eee66a15 | ||
|
|
e49c78aa90 | ||
|
|
ba0e00a358 | ||
|
|
901562129d | ||
|
|
fe127e59b6 | ||
|
|
9b86890bdc | ||
|
|
464c5ccdf6 | ||
|
|
9fdd6ac9a1 | ||
|
|
fef631a332 | ||
|
|
b4d011ca4e | ||
|
|
216680f6ab | ||
|
|
c3888ed257 | ||
|
|
3c96596c8b | ||
|
|
0b641aeb9b | ||
|
|
b4bd1c3cfd | ||
|
|
85f2e68e4d | ||
|
|
6be58473d0 | ||
|
|
98395dc8e9 | ||
|
|
27435bee11 | ||
|
|
98b62e4eef | ||
|
|
75c263ba7d | ||
|
|
61f91053f8 | ||
|
|
aa2a53157d | ||
|
|
f691e17c88 | ||
|
|
b94a4f54e7 | ||
|
|
0faffd0be0 | ||
|
|
9c0372694e | ||
|
|
d266e6e90c | ||
|
|
e9cba80f9b | ||
|
|
d23d4b9b97 | ||
|
|
8898b12ab0 | ||
|
|
90c9dd2622 | ||
|
|
8c50876ba0 | ||
|
|
d537a985b2 | ||
|
|
873039a33e | ||
|
|
330c6df546 | ||
|
|
8111a400f2 | ||
|
|
bf183110d3 | ||
|
|
1e81e33cc3 | ||
|
|
63be61b531 | ||
|
|
f41c34d34d | ||
|
|
c6d9c2a099 | ||
|
|
27f21796e9 | ||
|
|
8b0edd61aa | ||
|
|
2e138396e4 | ||
|
|
02d6e49ed6 | ||
|
|
584a1c3fc8 | ||
|
|
99735b4b7d | ||
|
|
dc1e5bc42f | ||
|
|
d6d3c7ac0b | ||
|
|
9f3cc3c2d1 | ||
|
|
7ba7ce1293 | ||
|
|
4ac976e1c4 | ||
|
|
13e68be5ee | ||
|
|
a2500607ee | ||
| 74aad393ce | |||
|
|
a114c6b9a2 | ||
| 140b4b5df5 | |||
|
|
83b46f4e76 | ||
| ac3dd118c4 |
2
.gitignore
vendored
2
.gitignore
vendored
@ -4,6 +4,7 @@
|
||||
#########################################################
|
||||
# pigsty-ca and other certs
|
||||
#########################################################
|
||||
|
||||
files/*.key
|
||||
files/*.crt
|
||||
files/pki/*
|
||||
@ -26,6 +27,7 @@ docker/data/
|
||||
#########################################################
|
||||
# tmp files
|
||||
#########################################################
|
||||
.env
|
||||
# IDE files
|
||||
.idea/
|
||||
.code/
|
||||
|
||||
2
.gitleaksignore
Normal file
2
.gitleaksignore
Normal file
@ -0,0 +1,2 @@
|
||||
66721ea54ac9cf407b0873d30dfdc0a1bc0116a2:dashboard-fresh/docs/LOGIN_API_GUIDE.md:generic-api-key:246
|
||||
66721ea54ac9cf407b0873d30dfdc0a1bc0116a2:dashboard-fresh/docs/LOGIN_API_GUIDE.md:generic-api-key:276
|
||||
28
KNOWN_ISSUES.md
Normal file
28
KNOWN_ISSUES.md
Normal file
@ -0,0 +1,28 @@
|
||||
# Known Issues - Observability Homepage
|
||||
|
||||
This document records known issues and design decisions for the consolidated Pigsty Observability Homepage.
|
||||
|
||||
## 1. Dashboard Merging & Grid Positioning
|
||||
- **Status**: Fixed
|
||||
- **Issue**: Merging multiple source dashboards (`pigsty.json`, `node.json`, `k8s.json`) into one `homepage.json` originally caused panels to stack vertically regardless of their horizontal layout.
|
||||
- **Resolution**: `merge_dashboards.py` was updated to preserve the relative vertical and horizontal positioning within each newly created section (Infra Overview, Node, K8S Cluster).
|
||||
|
||||
## 2. Variable Name Unification
|
||||
- **Status**: Fixed (Workaround)
|
||||
- **Issue**: The consolidated dashboard uses a unified variable set (`$hostname`, `$node`, etc.), but source queries in the Node dashboard expected `$name` and `$instance`.
|
||||
- **Resolution**: `merge_dashboards.py` performs a global regex replacement on the Node dashboard's JSON content before merging to align variable names.
|
||||
|
||||
## 3. External Links in Dashlists
|
||||
- **Status**: Manual Override
|
||||
- **Issue**: Grafana `dashlist` panels only show internal dashboards with specific tags. They do not support external URL links (like the Insight Workbench).
|
||||
- **Resolution**: The "Apps" dashlist panel was replaced with a `text` panel using HTML to provide a direct link to `https://observability.svc.plus/insight/`.
|
||||
|
||||
## 4. Root Path Redirection
|
||||
- **Status**: Fixed
|
||||
- **Issue**: Users visiting `observability.svc.plus/` were previously directed elsewhere (defaults in Caddyfile).
|
||||
- **Resolution**: Updated `Caddyfile` templates to redirect `/` (root) and `/zh` directly to `/grafana/`.
|
||||
|
||||
## 5. Panel UID Scaling
|
||||
- **Status**: Potential Issue
|
||||
- **Issue**: Panel IDs are re-assigned sequentially (1, 2, 3...) during merging. This might break internal dashboard persistence if panels are re-added or deleted frequently.
|
||||
- **Recommendation**: Avoid frequent re-merging if persistent panel links are required.
|
||||
700
README.md
700
README.md
@ -3,619 +3,213 @@
|
||||
[](LICENSE)
|
||||
[](https://svc.plus)
|
||||
|
||||
**Observability.svc.plus** is an advanced observability platform strictly following the **Apache 2.0** license.
|
||||
**Observability.svc.plus** is an observability solution strictly following the Apache 2.0 license.
|
||||
|
||||
> **Focus**: Monitoring & Observability (监控/可观测). Integrating **OpenTelemetry (OTel)**, with future plans to incorporate **DeepFlow Agent** and other open-source **NPM** (Network Performance Monitoring) probes.
|
||||
> **Focus**: Monitoring & Observability (监控/可观测). Integrating OpenTelemetry (OTel), VictoriaMetrics, and DeepFlow-based network observability without long-term raw-flow lock-in.
|
||||
|
||||
[Website](https://svc.plus/services) | [Public Demo](https://svc.plus/) | [Blog](https://svc.plus/blogs) | [Support](https://www.svc.plus/support)
|
||||
[Website](https://svc.plus/) | [Public Demo](https://svc.plus/services) | [Blog](https://svc.plus/blogs) | [Support](https://www.svc.plus/support)
|
||||
|
||||
[](https://observability.svc.plus)
|
||||
|
||||
## 🚀 快速开始
|
||||
## 1) 概述
|
||||
|
||||
### 一键安装 (默认)
|
||||
默认安装最新稳定版 , 默认使用当前主机名作为域名
|
||||
Observability.svc.plus provides a monitoring-focused stack for infrastructure and applications, centered on metrics, logs, and traces. It is designed for self-hosted, cloud-neutral operations with minimal vendor lock-in.
|
||||
|
||||
```bash
|
||||
curl -fsSL https://raw.githubusercontent.com/cloud-neutral-toolkit/observability.svc.plus/main/scripts/server-install.sh | bash
|
||||
## 2) 架构图
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
A["Client Nodes<br/>node_exporter / process_exporter / vector"] --> B["OTel / Ingest Gateway"]
|
||||
B --> C["Metrics Store<br/>VictoriaMetrics"]
|
||||
B --> D["Logs Store<br/>Loki-compatible pipeline"]
|
||||
B --> E["Traces / OTel pipeline"]
|
||||
C --> F["Grafana"]
|
||||
D --> F
|
||||
E --> F
|
||||
F --> G["Dashboard & Alerting"]
|
||||
```
|
||||
|
||||
### 指定版本与域名 (安装建议)
|
||||
## 3) Start
|
||||
|
||||
当前推荐按“混合部署到已有主机”的方式执行。
|
||||
|
||||
1. 先更新 DNS，把 `observability.svc.plus` 指到 `us-xhttp.svc.plus`
|
||||
2. 在 `us-xhttp.svc.plus` 上执行下面的 Server side 示例，部署中心端
|
||||
3. 再到其他已有主机执行下面的 Client side 示例，把采集数据回传到 `observability.svc.plus`
|
||||
|
||||
当前接入主机：
|
||||
|
||||
- `us-xhttp.svc.plus`：继续承载现有服务，同时承载 `observability.svc.plus`
|
||||
- `openclaw.svc.plus`：部署 agent，采集后上报到中心端
|
||||
- `jp-xhttp.svc.plus`：部署 agent，采集后上报到中心端
|
||||
|
||||
### Ansible (Recommended)
|
||||
|
||||
#### Server side
|
||||
|
||||
先导出 Cloudflare Token，然后在 `us-xhttp.svc.plus` 上执行服务端部署。`deploy_observability_service.yml` 会先把 Cloudflare 上的 `observability.svc.plus` 更新成指向 `us-xhttp.svc.plus` 的非代理记录，再等待公共 DNS 生效后继续部署，这样更容易保证 Caddy 首次自动签名成功。
|
||||
|
||||
```bash
|
||||
curl -fsSL https://raw.githubusercontent.com/cloud-neutral-toolkit/observability.svc.plus/main/scripts/server-install.sh \
|
||||
| bash -s -- observability.svc.plus
|
||||
export CLOUDFLARE_API_TOKEN=...
|
||||
ansible-playbook -i <your-inventory> deploy_observability_service.yml -l us-xhttp.svc.plus
|
||||
```
|
||||
|
||||
|
||||
## Features
|
||||
|
||||
- **Observability First**: SOTA monitoring for **PG** / **Infra** / **Node** based on **VictoriaMetrics**, **Grafana**, and **OpenTelemetry**.
|
||||
- **OTel Integration**: Native support for **OpenTelemetry**, facilitating unified trace, metric, and log ingestion.
|
||||
- **Future Ready**: Planned integration for **DeepFlow Agent** and other open-source **NPM** probes for deep network and application observability.
|
||||
- **Reliable Base**: Robust self-healing **HA** clusters, **PITR**, and secure infrastructure.
|
||||
- **Maintainable**: **One-Cmd Deploy**, **IaC** support, and easy customization.
|
||||
- **Controllable**: Self-sufficient Cloud Neutral FOSS. Run on **bare Linux**.
|
||||
|
||||
You can even use exotic [**PG kernel forks**](https://svc.plus/docs/pgsql/kernel) as an in-place replacement and wrap it as a full RDS service:
|
||||
|
||||
| Kernel | Key Feature | Description |
|
||||
|------------------------------------------------------------|:--------------------------------|------------------------------------------------|
|
||||
| [PostgreSQL](https://svc.plus/docs/pgsql/kernel/postgres) | **Extension Overwhelming** | Vanilla PostgreSQL with 444 extensions |
|
||||
| [Citus](https://svc.plus/docs/pgsql/kernel/citus) | **Horizontal Scaling** | Distributive PostgreSQL via native extension |
|
||||
| [WiltonDB](https://svc.plus/docs/pgsql/kernel/babelfish) | **SQL Server Migration** | Microsoft SQL Server wire-compatibility |
|
||||
| [IvorySQL](https://svc.plus/docs/pgsql/kernel/ivorysql) | **Oracle Migration** | Oracle Grammar and PL/SQL compatible |
|
||||
| [OpenHalo](https://svc.plus/docs/pgsql/kernel/openhalo) | **MySQL Migration** | MySQL wire-protocol compatibility |
|
||||
| [Percona](https://svc.plus/docs/pgsql/kernel/percona) | **Transparent Data Encryption** | Percona Distribution with pg_tde |
|
||||
| [FerretDB](https://svc.plus/docs/ferret) | **MongoDB Migration** | MongoDB wire-protocol compatibility |
|
||||
| [OrioleDB](https://svc.plus/docs/pgsql/kernel/orioledbdb) | **OLTP Optimization** | No bloat, No XID Wraparound, S3 Storage |
|
||||
| [PolarDB](https://svc.plus/docs/pgsql/kernel/polardb) | **Aurora flavor RAC** | RAC, China domestic compliance |
|
||||
| [Supabase](https://svc.plus/docs/app/supabase) | **Backend as Service** | BaaS based on PostgreSQL, Firebase alternative |
|
||||
|
||||
And gather the synergistic superpowers of all [**444+ PostgreSQL Extensions**](https://pgext.cloud/list) all together:
|
||||
|
||||
[](https://pgext.cloud)
|
||||
|
||||
|
||||
## Get Started
|
||||
|
||||
[](https://svc.plus/docs/pgsql)
|
||||
[](https://svc.plus/docs/node)
|
||||
[](https://svc.plus/docs/node)
|
||||
[](https://svc.plus/docs/ref/linux#el)
|
||||
[](https://svc.plus/docs/ref/linux#debian)
|
||||
[](https://svc.plus/docs/ref/linux#ubuntu)
|
||||
[](https://svc.plus/docs/setup/docker)
|
||||
|
||||
[**Prepare**](https://svc.plus/docs/deploy/prepare) a fresh `x86_64` / `aarch64` node runs any [**compatible**](https://svc.plus/docs/ref/linux) **Linux** OS Distros, then [**Install**](https://svc.plus/docs/setup/install#install) the platform with:
|
||||
如果希望给 `/ingest/*` 增加一层基础认证，可以在服务端部署时一起打开：
|
||||
|
||||
```bash
|
||||
curl -fsSL https://raw.githubusercontent.com/cloud-neutral-toolkit/observability.svc.plus/main/scripts/server-install.sh | bash
|
||||
export CLOUDFLARE_API_TOKEN=...
|
||||
ansible-playbook -i <your-inventory> deploy_observability_service.yml -l us-xhttp.svc.plus \
|
||||
-e observability_ingest_basic_auth_enabled=true \
|
||||
-e observability_ingest_basic_auth_user=ingest \
|
||||
-e observability_ingest_basic_auth_password='<strong-password>'
|
||||
```
|
||||
|
||||
Then [**configure**](https://svc.plus/docs/concept/iac/configure) and run the [**`deploy.yml`**](https://svc.plus/docs/setup/playbook) playbook with an [**admin user**](https://svc.plus/docs/deploy/admin) (**nopass** `ssh` & `sudo`):
|
||||
#### Client side (agent)
|
||||
|
||||
再到采集端主机执行 `node.yml` 的 push mode：
|
||||
|
||||
```bash
|
||||
./configure -g # generate config and random passwords
|
||||
./deploy.yml # deploy everything on current node
|
||||
ansible-playbook -i <your-inventory> node.yml \
|
||||
-l openclaw.svc.plus,jp-xhttp.svc.plus \
|
||||
-e node_monitor_mode=push \
|
||||
-e observability_endpoint=https://observability.svc.plus/
|
||||
```
|
||||
|
||||
Finally, you will get a [**singleton node ready**](https://svc.plus/docs/setup/install), with [**WebUI**](https://svc.plus/docs/setup/webui) on port `80/443` and [**Postgres**](https://svc.plus/docs/setup/pgsql) on port `5432`.
|
||||
|
||||
For dev/testing purposes, you can also run it inside [**Docker**](https://svc.plus/docs/setup/docker) containers: `cd docker; make launch`
|
||||
|
||||
--------
|
||||
|
||||
> [**Single-Node Setup**](https://svc.plus/docs/setup/install) | [**Production Deploy**](https://svc.plus/docs/deploy) | [**Offline Install**](https://svc.plus/docs/setup/offline) | [**Minimal Install**](https://svc.plus/docs/setup/slim) | [**Docker Install**](https://svc.plus/docs/setup/docker) | [**Run Supabase**](https://svc.plus/docs/app/supabase)
|
||||
|
||||
<details><summary>Install with the pig cli</summary><br>
|
||||
|
||||
Then you can launch pigsty with `pig sty` sub command:
|
||||
如果服务端已开启 ingest 基本认证，采集端也要带上同一组凭据：
|
||||
|
||||
```bash
|
||||
curl -fsSL https://repo.pigsty.io/pig | bash # install pig
|
||||
pig sty init # install latest pigsty src to ~/pigsty
|
||||
pig sty conf # auto-generate pigsty.yml config file
|
||||
pig sty deploy # run the deploy.yml playbook
|
||||
ansible-playbook -i <your-inventory> node.yml \
|
||||
-l openclaw.svc.plus,jp-xhttp.svc.plus \
|
||||
-e node_monitor_mode=push \
|
||||
-e observability_endpoint=https://observability.svc.plus/ \
|
||||
-e observability_ingest_basic_auth_enabled=true \
|
||||
-e observability_ingest_basic_auth_user=ingest \
|
||||
-e observability_ingest_basic_auth_password='<strong-password>'
|
||||
```
|
||||
|
||||
</details>
|
||||
> `node_monitor_mode=push` 会在远端主机上部署 `node_exporter + process_exporter + vector`，并把 metrics / logs 主动汇总到 `observability.svc.plus`。`vector` 固定归到采集端任务，服务端 `infra.yml` 不再默认部署它。
|
||||
>
|
||||
> 如果采集端与 Victoria 服务端同机，playbook 会自动把 metrics / logs 改走本机 `127.0.0.1` ingest；跨主机时默认走 `https://observability.svc.plus/` 并自动补全 `/ingest/metrics/api/v1/write` 和 `/ingest/logs/insert`。
|
||||
>
|
||||
> `observability_ingest_basic_auth_*` 只保护 `/ingest/*` 写入入口，不影响 Caddy 暴露的其他站点页面；服务端和采集端必须使用同一组认证信息。
|
||||
|
||||
### Script Installers
|
||||
|
||||
## 🚀 快速开始
|
||||
|
||||
### 一键安装 (默认)
|
||||
默认安装最新稳定版 , 默认使用当前主机名作为域名
|
||||
### Server side
|
||||
|
||||
```bash
|
||||
curl -fsSL https://raw.githubusercontent.com/cloud-neutral-toolkit/observability.svc.plus/main/scripts/server-install.sh | bash
|
||||
curl -fsSL "https://raw.githubusercontent.com/cloud-neutral-toolkit/observability.svc.plus/main/scripts/server-install.sh?$(date +%s)" | bash -s -- observability.svc.plus
|
||||
```
|
||||
|
||||
### 指定版本与域名 (安装建议)
|
||||
|
||||
```bash
|
||||
# bash -s -- <版本> <域名>
|
||||
curl -fsSL https://raw.githubusercontent.com/cloud-neutral-toolkit/observability.svc.plus/main/scripts/server-install.sh \
|
||||
| bash -s -- observability.svc.plus
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
|
||||
<details><summary>Clone src with git</summary><br>
|
||||
|
||||
You can also download the pigsty source with `git`, remember to check out a specific version tag, the `main` branch is for development.
|
||||
|
||||
```bash
|
||||
git clone https://github.com/pgsty/pigsty; cd pigsty; git checkout v4.0.0
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
|
||||
## 🛠️ Client Agent Installation
|
||||
|
||||
To install observability agents (Node Exporter, Process Exporter, Vector) on a client machine and send data to this platform:
|
||||
### Client side (agent)
|
||||
|
||||
```bash
|
||||
# bash -s -- --endpoint <YOUR_ENDPOINT>
|
||||
curl -fsSL https://raw.githubusercontent.com/cloud-neutral-toolkit/observability.svc.plus/main/scripts/agent-install.sh \
|
||||
| bash -s -- --endpoint https://infra.svc.plus/ingest/otlp
|
||||
| bash -s -- --endpoint https://observability.svc.plus/ingest/otlp
|
||||
```
|
||||
|
||||
> **Note**: The script automatically verifies the installation after setup.
|
||||
> **Note**
|
||||
> - `--endpoint` supports both:
|
||||
> - `https://observability.svc.plus`
|
||||
> - `https://observability.svc.plus/ingest/otlp`
|
||||
> - The installer auto-derives:
|
||||
> - metrics endpoint: `/ingest/metrics/api/v1/write`
|
||||
> - logs endpoint: `/ingest/logs/insert`
|
||||
> - The script automatically verifies installation after setup.
|
||||
|
||||
### Optional: DeepFlow Agent on Client
|
||||
|
||||
|
||||
|
||||
Integrated as a platform.
|
||||
|
||||
[](https://svc.plus/docs/infra) Nginx, Local Repo, DNSMasq, and the entire Victoria & Grafana observability stack.
|
||||
|
||||
The default [`deploy.yml`](deploy.yml) playbook will deploy `INFRA`, `NODE`, `ETCD` & `PGSQL` on the current node.
|
||||
Which gives you an out-of-the-box PostgreSQL singleton instance (`admin_ip:5432`) with everything ready.
|
||||
|
||||
The node can be used as an admin controller to deploy & monitor more nodes & clusters. For example, you can install these **6** **OPTIONAL** [extra modules](https://svc.plus/docs/ref/module#extra-modules) for advanced use cases:
|
||||
|
||||
[](https://svc.plus/docs/minio) S3-compatible object storage service; used as an optional central backup server for `PGSQL`.
|
||||
|
||||
[](https://svc.plus/docs/infra) Deploy Redis servers in standalone master-replica, sentinel, and native cluster mode.
|
||||
|
||||
[](https://svc.plus/docs/ferret) Native support for FerretDB — adding MongoDB wire protocol compatibility to Postgres!
|
||||
|
||||
[](https://svc.plus/docs/docker) Launch optional docker daemons to run other stateless parts besides Pigsty RDS.
|
||||
|
||||
[](https://svc.plus/docs/juice) JuiceFS can mount S3/MinIO, and even PostgreSQL as a filesystem shared by multi users.
|
||||
|
||||
[](https://svc.plus/docs/vibe) Vibe coding environment with VS Code Server, JupyterLab, Node.js, and Claude Code.
|
||||
|
||||
Of course, you can deploy different kinds of HA **PostgreSQL** clusters on multiple nodes, as much as you want.
|
||||
|
||||
|
||||
----------------
|
||||
|
||||
## PostgreSQL RDS
|
||||
|
||||
To deploy an [**additional**](https://svc.plus/docs/deploy) 3-node HA Postgres cluster `pg-test`. Add the cluster [**definition**](https://github.com/pgsty/pigsty/blob/main/conf/ha/full.yml#L66) to the [**config inventory**](https://svc.plus/docs/concept/iac/inventory):
|
||||
|
||||
```yaml
|
||||
pg-test:
|
||||
hosts:
|
||||
10.10.10.11: { pg_seq: 1, pg_role: primary }
|
||||
10.10.10.12: { pg_seq: 2, pg_role: replica }
|
||||
10.10.10.13: { pg_seq: 3, pg_role: offline }
|
||||
vars: { pg_cluster: pg-test }
|
||||
```
|
||||
|
||||
The default config file is [`pigsty.yml`](https://github.com/pgsty/pigsty/blob/main/pigsty.yml) under pigsty home, add the snippet above to the `all.children.pg-test`,
|
||||
Then, create the cluster with built-in playbooks in one command:
|
||||
If you have deployed DeepFlow with `deepflow.yml`, you can install `deepflow-agent` on client nodes via the same script:
|
||||
|
||||
```bash
|
||||
bin/pgsql-add pg-test # init pg-test cluster
|
||||
# example: endpoint exposed by caddy grpc ingress (deepflow_grpc_domain:443)
|
||||
curl -fsSL https://raw.githubusercontent.com/cloud-neutral-toolkit/observability.svc.plus/main/scripts/agent-install.sh \
|
||||
| bash -s -- \
|
||||
--endpoint https://observability.svc.plus/ingest/otlp \
|
||||
--deepflow-agent \
|
||||
--deepflow-grpc-endpoint deepflow-agent.svc.plus:443 \
|
||||
--deepflow-agent-download-url https://example.com/path/to/deepflow-agent
|
||||
```
|
||||
|
||||
<details><summary>Example: Complex PostgreSQL Customization</summary><br>
|
||||
> If `deepflow-agent` binary already exists on host, replace `--deepflow-agent-download-url` with `--deepflow-agent-bin /path/to/deepflow-agent`.
|
||||
|
||||
This config file provides a detailed example of a complex PostgreSQL cluster `pg-meta` with multiple
|
||||
[databases](https://svc.plus/docs/pgsql/config/db), [users](https://svc.plus/docs/pgsql/config/user), and [service](https://svc.plus/docs/pgsql/service) definition:
|
||||
## 🚀 DeepFlow Deployment (Server Side)
|
||||
|
||||
```yaml
|
||||
pg-meta:
|
||||
hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary , pg_offline_query: true } }
|
||||
vars:
|
||||
pg_cluster: pg-meta
|
||||
pg_databases: # define business databases on this cluster, array of database definition
|
||||
- name: meta # REQUIRED, `name` is the only mandatory field of a database definition
|
||||
baseline: cmdb.sql # optional, database sql baseline path, (relative path among ansible search path, e.g files/)
|
||||
pgbouncer: true # optional, add this database to pgbouncer database list? true by default
|
||||
schemas: [pigsty] # optional, additional schemas to be created, array of schema names
|
||||
extensions: # optional, additional extensions to be installed: array of `{name[,schema]}`
|
||||
- { name: postgis , schema: public }
|
||||
- { name: timescaledb }
|
||||
comment: pigsty meta database # optional, comment string for this database
|
||||
owner: postgres # optional, database owner, postgres by default
|
||||
template: template1 # optional, which template to use, template1 by default
|
||||
encoding: UTF8 # optional, database encoding, UTF8 by default. (MUST same as template database)
|
||||
locale: C # optional, database locale, C by default. (MUST same as template database)
|
||||
lc_collate: C # optional, database collate, C by default. (MUST same as template database)
|
||||
lc_ctype: C # optional, database ctype, C by default. (MUST same as template database)
|
||||
tablespace: pg_default # optional, default tablespace, 'pg_default' by default.
|
||||
allowconn: true # optional, allow connection, true by default. false will disable connect at all
|
||||
revokeconn: false # optional, revoke public connection privilege. false by default. (leave connect with grant option to owner)
|
||||
register_datasource: true # optional, register this database to grafana datasources? true by default
|
||||
connlimit: -1 # optional, database connection limit, default -1 disable limit
|
||||
pool_auth_user: dbuser_meta # optional, all connection to this pgbouncer database will be authenticated by this user
|
||||
pool_mode: transaction # optional, pgbouncer pool mode at database level, default transaction
|
||||
pool_size: 64 # optional, pgbouncer pool size at database level, default 64
|
||||
pool_size_reserve: 32 # optional, pgbouncer pool size reserve at database level, default 32
|
||||
pool_size_min: 0 # optional, pgbouncer pool size min at database level, default 0
|
||||
pool_max_db_conn: 100 # optional, max database connections at database level, default 100
|
||||
# shared parameters for CREATE DATABASE (shared with PostgreSQL)
|
||||
is_template: false # optional, mark this database as template? default false
|
||||
strategy: wal_log # optional (shared), PostgreSQL 15+, wal_log or file_copy
|
||||
locale_provider: libc # optional (shared), PostgreSQL 15+, icu or libc
|
||||
icu_locale: '' # optional (shared), PostgreSQL 15+, ICU locale
|
||||
icu_rules: '' # optional (shared), PostgreSQL 16+, ICU rules
|
||||
builtin_locale: '' # optional (shared), PostgreSQL 17+, builtin locale
|
||||
parameters: {} # optional, database-level parameters with `ALTER DATABASE SET`
|
||||
- { name: grafana ,owner: dbuser_grafana ,revokeconn: true ,comment: grafana primary database }
|
||||
- { name: bytebase ,owner: dbuser_bytebase ,revokeconn: true ,comment: bytebase primary database }
|
||||
- { name: kong ,owner: dbuser_kong ,revokeconn: true ,comment: kong the api gateway database }
|
||||
- { name: gitea ,owner: dbuser_gitea ,revokeconn: true ,comment: gitea meta database }
|
||||
- { name: wiki ,owner: dbuser_wiki ,revokeconn: true ,comment: wiki meta database }
|
||||
pg_users: # define business users/roles on this cluster, array of user definition
|
||||
- name: dbuser_meta # REQUIRED, `name` is the only mandatory field of a user definition
|
||||
password: DBUser.Meta # optional, password, can be a scram-sha-256 hash string or plain text
|
||||
login: true # optional, can log in, true by default (new biz ROLE should be false)
|
||||
superuser: false # optional, is superuser? false by default
|
||||
createdb: false # optional, can create database? false by default
|
||||
createrole: false # optional, can create role? false by default
|
||||
inherit: true # optional, can this role use inherited privileges? true by default
|
||||
replication: false # optional, can this role do replication? false by default
|
||||
bypassrls: false # optional, can this role bypass row level security? false by default
|
||||
pgbouncer: true # optional, add this user to pgbouncer user-list? false by default (production user should be true explicitly)
|
||||
connlimit: -1 # optional, user connection limit, default -1 disable limit
|
||||
expire_in: 3650 # optional, now + n days when this role is expired (OVERWRITE expire_at)
|
||||
expire_at: '2030-12-31' # optional, YYYY-MM-DD 'timestamp' when this role is expired (OVERWRITTEN by expire_in)
|
||||
comment: pigsty admin user # optional, comment string for this user/role
|
||||
roles: [dbrole_admin] # optional, belonged roles. default roles are: dbrole_{admin,readonly,readwrite,offline}
|
||||
parameters: {} # optional, role level parameters with `ALTER ROLE SET`
|
||||
pool_mode: transaction # optional, pgbouncer pool mode at user level, transaction by default
|
||||
pool_connlimit: -1 # optional, max database connections at user level, default -1 disable limit
|
||||
# shared parameters for CREATE ROLE (shared with PostgreSQL)
|
||||
# roles can also be defined in extended format for fine-grained control (PostgreSQL 16+)
|
||||
# roles:
|
||||
# - dbrole_readwrite # simple format: just grant the role
|
||||
# - name: dbrole_admin # extended format with options
|
||||
# state: grant # grant (default) | revoke/absent
|
||||
# admin: false # grant with admin option? (default false)
|
||||
# inherit: true # grant with inherit option? (PG16+, default null)
|
||||
# set: true # grant with set option? (PG16+, default null)
|
||||
- {name: dbuser_view ,password: DBUser.Viewer ,pgbouncer: true ,roles: [dbrole_readonly], comment: read-only viewer for meta database}
|
||||
- {name: dbuser_grafana ,password: DBUser.Grafana ,pgbouncer: true ,roles: [dbrole_admin] ,comment: admin user for grafana database }
|
||||
- {name: dbuser_bytebase ,password: DBUser.Bytebase ,pgbouncer: true ,roles: [dbrole_admin] ,comment: admin user for bytebase database }
|
||||
- {name: dbuser_kong ,password: DBUser.Kong ,pgbouncer: true ,roles: [dbrole_admin] ,comment: admin user for kong api gateway }
|
||||
- {name: dbuser_gitea ,password: DBUser.Gitea ,pgbouncer: true ,roles: [dbrole_admin] ,comment: admin user for gitea service }
|
||||
- {name: dbuser_wiki ,password: DBUser.Wiki ,pgbouncer: true ,roles: [dbrole_admin] ,comment: admin user for wiki.js service }
|
||||
pg_services: # extra services in addition to pg_default_services, array of service definition
|
||||
# standby service will route {ip|name}:5435 to sync replica's pgbouncer (5435->6432 standby)
|
||||
- name: standby # required, service name, the actual svc name will be prefixed with `pg_cluster`, e.g: pg-meta-standby
|
||||
port: 5435 # required, service exposed port (work as kubernetes service node port mode)
|
||||
ip: "*" # optional, service bind ip address, `*` for all ip by default
|
||||
selector: "[]" # required, service member selector, use JMESPath to filter inventory
|
||||
dest: default # optional, destination port, default|postgres|pgbouncer|<port_number>, 'default' by default
|
||||
check: /sync # optional, health check url path, / by default
|
||||
backup: "[? pg_role == `primary`]" # backup server selector
|
||||
maxconn: 3000 # optional, max allowed front-end connection
|
||||
balance: roundrobin # optional, haproxy load balance algorithm (roundrobin by default, other: leastconn)
|
||||
#options: 'inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100'
|
||||
pg_hba_rules:
|
||||
- {user: dbuser_view , db: all ,addr: infra ,auth: pwd ,title: 'allow grafana dashboard access cmdb from infra nodes'}
|
||||
pg_vip_enabled: true
|
||||
pg_vip_address: 10.10.10.2/24
|
||||
pg_vip_interface: eth1
|
||||
pg_crontab: # make a full backup 1 am everyday
|
||||
- '00 01 * * * /pg/bin/pg-backup full'
|
||||
This repo now provides dedicated DeepFlow roles:
|
||||
|
||||
```
|
||||
- `deepflow_mysql`
|
||||
- `deepflow_clickhouse_s3`
|
||||
- `deepflow_server`
|
||||
- `deepflow_connector`
|
||||
- `deepflow_agent`
|
||||
|
||||
[](https://pigsty.io/img/pigsty/home.jpg)
|
||||
|
||||
</details>
|
||||
|
||||
It will create a cluster with everything properly configured: [**High Availability**](https://svc.plus/docs/concept/ha) powered by patroni & etcd; [**Point-In-Time-Recovery**](https://svc.plus/docs/concept/pitr) powered by pgBackRest & optional MinIO / S3;
|
||||
auto-routed, pooled [**Services & Access**](https://svc.plus/docs/pgsql/service#default-service) pooled by pgBouncer and exposed by haproxy; and out-of-the-box [**Monitoring**](https://svc.plus/docs/pgsql/monitor/dashboard) & alerting powered by the **`INFRA`** module.
|
||||
|
||||
[](https://svc.plus/docs/concept/ha)
|
||||
|
||||
The cluster keeps serving as long as **ANY** instance survives, with excellent [fault-tolerance](https://svc.plus/docs/concept/ha/failure) performance:
|
||||
|
||||
> [**RPO = 0**](https://svc.plus/docs/concept/ha/rpo) on sync mode, **RPO < 1MB** on async mode; **RTO < 2s** on switchover, [**RTO < 30s**](https://svc.plus/docs/concept/ha/rto) on failover.
|
||||
|
||||
|
||||
|
||||
|
||||
----------------
|
||||
|
||||
## Customization
|
||||
|
||||
Pigsty is highly customizable, You can describe the entire database and infra deployment with **300+** [**parameters**](https://svc.plus/docs/concept/iac/inventory) in a single config file and materialize them with one command.
|
||||
There are many built-in configuration [**templates**](https://svc.plus/docs/concept/iac/template).
|
||||
|
||||
<details><summary>Example: Sandbox (4-node) with two PG cluster</summary><br>
|
||||
|
||||
The [`conf/full.yml`](https://github.com/pgsty/pigsty/blob/main/conf/full.yml) utilize four nodes to deploy two PostgreSQL clusters `pg-meta` and `pg-test`:
|
||||
|
||||
```yaml
|
||||
pg-meta:
|
||||
hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }
|
||||
vars:
|
||||
pg_cluster: pg-meta
|
||||
pg_users:
|
||||
- {name: dbuser_meta ,password: DBUser.Meta ,pgbouncer: true ,roles: [dbrole_admin] ,comment: pigsty admin user }
|
||||
- {name: dbuser_view ,password: DBUser.Viewer ,pgbouncer: true ,roles: [dbrole_readonly] ,comment: read-only viewer for meta database }
|
||||
pg_databases:
|
||||
- {name: meta ,baseline: cmdb.sql ,comment: pigsty meta database ,schemas: [pigsty]}
|
||||
pg_hba_rules:
|
||||
- {user: dbuser_view , db: all ,addr: infra ,auth: pwd ,title: 'allow grafana dashboard access cmdb from infra nodes'}
|
||||
pg_vip_enabled: true
|
||||
pg_vip_address: 10.10.10.2/24
|
||||
pg_vip_interface: eth1
|
||||
|
||||
# pgsql 3 node ha cluster: pg-test
|
||||
pg-test:
|
||||
hosts:
|
||||
10.10.10.11: { pg_seq: 1, pg_role: primary } # primary instance, leader of cluster
|
||||
10.10.10.12: { pg_seq: 2, pg_role: replica } # replica instance, follower of leader
|
||||
10.10.10.13: { pg_seq: 3, pg_role: replica, pg_offline_query: true } # replica with offline access
|
||||
vars:
|
||||
pg_cluster: pg-test # define pgsql cluster name
|
||||
pg_users: [{ name: test , password: test , pgbouncer: true , roles: [ dbrole_admin ] }]
|
||||
pg_databases: [{ name: test }]
|
||||
pg_vip_enabled: true
|
||||
pg_vip_address: 10.10.10.3/24
|
||||
pg_vip_interface: eth1
|
||||
```
|
||||
|
||||
You can even deploy PostgreSQL with different major versions and kernel forks in the same deployment:
|
||||
|
||||
[](https://svc.plus/docs/pgsql/kernel)
|
||||
|
||||
</details>
|
||||
|
||||
<details><summary>Example: Security Setup & Delayed Replica</summary><br>
|
||||
|
||||
The following [`conf/safe.yml`](https://github.com/pgsty/pigsty/blob/main/conf/ha/safe.yml) provision a 4-node [security](https://svc.plus/docs/deploy/security//) enhanced postgres cluster `pg-meta` with a delayed replica `pg-meta-delay`:
|
||||
|
||||
```yaml
|
||||
pg-meta: # 3 instance postgres cluster `pg-meta`
|
||||
hosts:
|
||||
10.10.10.10: { pg_seq: 1, pg_role: primary }
|
||||
10.10.10.11: { pg_seq: 2, pg_role: replica }
|
||||
10.10.10.12: { pg_seq: 3, pg_role: replica , pg_offline_query: true }
|
||||
vars:
|
||||
pg_cluster: pg-meta
|
||||
pg_conf: crit.yml
|
||||
pg_users:
|
||||
- { name: dbuser_meta , password: DBUser.Meta , pgbouncer: true , roles: [ dbrole_admin ] , comment: pigsty admin user }
|
||||
- { name: dbuser_view , password: DBUser.Viewer , pgbouncer: true , roles: [ dbrole_readonly ] , comment: read-only viewer for meta database }
|
||||
pg_databases:
|
||||
- {name: meta ,baseline: cmdb.sql ,comment: pigsty meta database ,schemas: [pigsty] ,extensions: [{name: postgis, schema: public}, {name: timescaledb}]}
|
||||
pg_default_service_dest: postgres
|
||||
pg_services:
|
||||
- { name: standby ,src_ip: "*" ,port: 5435 , dest: default ,selector: "[]" , backup: "[? pg_role == `primary`]" }
|
||||
pg_vip_enabled: true
|
||||
pg_vip_address: 10.10.10.2/24
|
||||
pg_vip_interface: eth1
|
||||
pg_listen: '${ip},${vip},${lo}'
|
||||
patroni_ssl_enabled: true
|
||||
pgbouncer_sslmode: require
|
||||
pgbackrest_method: minio
|
||||
pg_libs: 'timescaledb, $libdir/passwordcheck, pg_stat_statements, auto_explain' # add passwordcheck extension to enforce strong password
|
||||
pg_default_roles: # default roles and users in postgres cluster
|
||||
- { name: dbrole_readonly ,login: false ,comment: role for global read-only access }
|
||||
- { name: dbrole_offline ,login: false ,comment: role for restricted read-only access }
|
||||
- { name: dbrole_readwrite ,login: false ,roles: [dbrole_readonly] ,comment: role for global read-write access }
|
||||
- { name: dbrole_admin ,login: false ,roles: [pg_monitor, dbrole_readwrite] ,comment: role for object creation }
|
||||
- { name: postgres ,superuser: true ,expire_in: 7300 ,comment: system superuser }
|
||||
- { name: replicator ,replication: true ,expire_in: 7300 ,roles: [pg_monitor, dbrole_readonly] ,comment: system replicator }
|
||||
- { name: dbuser_dba ,superuser: true ,expire_in: 7300 ,roles: [dbrole_admin] ,pgbouncer: true ,pool_mode: session, pool_connlimit: 16 , comment: pgsql admin user }
|
||||
- { name: dbuser_monitor ,roles: [pg_monitor] ,expire_in: 7300 ,pgbouncer: true ,parameters: {log_min_duration_statement: 1000 } ,pool_mode: session ,pool_connlimit: 8 ,comment: pgsql monitor user }
|
||||
pg_default_hba_rules: # postgres host-based auth rules by default
|
||||
- {user: '${dbsu}' ,db: all ,addr: local ,auth: ident ,title: 'dbsu access via local os user ident' }
|
||||
- {user: '${dbsu}' ,db: replication ,addr: local ,auth: ident ,title: 'dbsu replication from local os ident' }
|
||||
- {user: '${repl}' ,db: replication ,addr: localhost ,auth: ssl ,title: 'replicator replication from localhost'}
|
||||
- {user: '${repl}' ,db: replication ,addr: intra ,auth: ssl ,title: 'replicator replication from intranet' }
|
||||
- {user: '${repl}' ,db: postgres ,addr: intra ,auth: ssl ,title: 'replicator postgres db from intranet' }
|
||||
- {user: '${monitor}' ,db: all ,addr: localhost ,auth: pwd ,title: 'monitor from localhost with password' }
|
||||
- {user: '${monitor}' ,db: all ,addr: infra ,auth: ssl ,title: 'monitor from infra host with password'}
|
||||
- {user: '${admin}' ,db: all ,addr: infra ,auth: ssl ,title: 'admin @ infra nodes with pwd & ssl' }
|
||||
- {user: '${admin}' ,db: all ,addr: world ,auth: cert ,title: 'admin @ everywhere with ssl & cert' }
|
||||
- {user: '+dbrole_readonly',db: all ,addr: localhost ,auth: ssl ,title: 'pgbouncer read/write via local socket'}
|
||||
- {user: '+dbrole_readonly',db: all ,addr: intra ,auth: ssl ,title: 'read/write biz user via password' }
|
||||
- {user: '+dbrole_offline' ,db: all ,addr: intra ,auth: ssl ,title: 'allow etl offline tasks from intranet'}
|
||||
pgb_default_hba_rules: # pgbouncer host-based authentication rules
|
||||
- {user: '${dbsu}' ,db: pgbouncer ,addr: local ,auth: peer ,title: 'dbsu local admin access with os ident'}
|
||||
- {user: 'all' ,db: all ,addr: localhost ,auth: pwd ,title: 'allow all user local access with pwd' }
|
||||
- {user: '${monitor}' ,db: pgbouncer ,addr: intra ,auth: ssl ,title: 'monitor access via intranet with pwd' }
|
||||
- {user: '${monitor}' ,db: all ,addr: world ,auth: deny ,title: 'reject all other monitor access addr' }
|
||||
- {user: '${admin}' ,db: all ,addr: intra ,auth: ssl ,title: 'admin access via intranet with pwd' }
|
||||
- {user: '${admin}' ,db: all ,addr: world ,auth: deny ,title: 'reject all other admin access addr' }
|
||||
- {user: 'all' ,db: all ,addr: intra ,auth: ssl ,title: 'allow all user intra access with pwd' }
|
||||
|
||||
# OPTIONAL delayed cluster for pg-meta
|
||||
pg-meta-delay: # delayed instance for pg-meta (1 hour ago)
|
||||
hosts: { 10.10.10.13: { pg_seq: 1, pg_role: primary, pg_upstream: 10.10.10.10, pg_delay: 1h } }
|
||||
vars: { pg_cluster: pg-meta-delay }
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
|
||||
<details><summary>Example: Horizontal Sharding with Citus</summary><br>
|
||||
|
||||
You can perform horizontal sharding on vanilla postgres with [**`CITUS`**](https://svc.plus/docs/pgsql/kernel/citus/).
|
||||
|
||||
```yaml
|
||||
# pg-citus: 10 node citus cluster (5 x primary-replica pair)
|
||||
pg-citus: # citus group
|
||||
hosts:
|
||||
10.10.10.50: { pg_group: 0, pg_cluster: pg-citus0 ,pg_vip_address: 10.10.10.60/24 ,pg_seq: 0, pg_role: primary }
|
||||
10.10.10.51: { pg_group: 0, pg_cluster: pg-citus0 ,pg_vip_address: 10.10.10.60/24 ,pg_seq: 1, pg_role: replica }
|
||||
10.10.10.52: { pg_group: 1, pg_cluster: pg-citus1 ,pg_vip_address: 10.10.10.61/24 ,pg_seq: 0, pg_role: primary }
|
||||
10.10.10.53: { pg_group: 1, pg_cluster: pg-citus1 ,pg_vip_address: 10.10.10.61/24 ,pg_seq: 1, pg_role: replica }
|
||||
10.10.10.54: { pg_group: 2, pg_cluster: pg-citus2 ,pg_vip_address: 10.10.10.62/24 ,pg_seq: 0, pg_role: primary }
|
||||
10.10.10.55: { pg_group: 2, pg_cluster: pg-citus2 ,pg_vip_address: 10.10.10.62/24 ,pg_seq: 1, pg_role: replica }
|
||||
10.10.10.56: { pg_group: 3, pg_cluster: pg-citus3 ,pg_vip_address: 10.10.10.63/24 ,pg_seq: 0, pg_role: primary }
|
||||
10.10.10.57: { pg_group: 3, pg_cluster: pg-citus3 ,pg_vip_address: 10.10.10.63/24 ,pg_seq: 1, pg_role: replica }
|
||||
10.10.10.58: { pg_group: 4, pg_cluster: pg-citus4 ,pg_vip_address: 10.10.10.64/24 ,pg_seq: 0, pg_role: primary }
|
||||
10.10.10.59: { pg_group: 4, pg_cluster: pg-citus4 ,pg_vip_address: 10.10.10.64/24 ,pg_seq: 1, pg_role: replica }
|
||||
vars:
|
||||
pg_mode: citus # pgsql cluster mode: citus
|
||||
pg_shard: pg-citus # citus shard name: pg-citus
|
||||
pg_primary_db: test # primary database used by citus
|
||||
pg_dbsu_password: DBUser.Postgres # all dbsu password access for citus cluster
|
||||
pg_vip_enabled: true
|
||||
pg_vip_interface: eth1
|
||||
pg_extensions: [ 'citus postgis timescaledb pgvector' ]
|
||||
pg_libs: 'citus, timescaledb, pg_stat_statements, auto_explain' # citus will be added by patroni automatically
|
||||
pg_users: [ { name: test ,password: test ,pgbouncer: true ,roles: [ dbrole_admin ] } ]
|
||||
pg_databases: [ { name: test ,owner: test ,extensions: [ { name: citus }, { name: postgis } ] } ]
|
||||
pg_hba_rules:
|
||||
- { user: 'all' ,db: all ,addr: 10.10.10.0/24 ,auth: trust ,title: 'trust citus cluster members' }
|
||||
- { user: 'all' ,db: all ,addr: 127.0.0.1/32 ,auth: ssl ,title: 'all user ssl access from localhost' }
|
||||
- { user: 'all' ,db: all ,addr: intra ,auth: ssl ,title: 'all user ssl access from intranet' }
|
||||
```
|
||||
|
||||
[](https://svc.plus/docs/pgsql/kernel/citus)
|
||||
|
||||
</details>
|
||||
|
||||
|
||||
You can deploy different kinds of PostgreSQL instance such as [`primary`](https://svc.plus/docs/pgsql/config/cluster#primary), [`replica`](https://svc.plus/docs/pgsql/config/cluster#replica), [`offline`](https://svc.plus/docs/pgsql/config/cluster#offline), [`delayed`](https://svc.plus/docs/pgsql/config/cluster#delayed), [`sync standby`](https://svc.plus/docs/pgsql/config/cluster#sync-standby), etc.,
|
||||
and customize with scene-optimize [**config templates**](https://svc.plus/docs/concept/iac/template) and all **444** [**extensions**](https://pgext.cloud/list) out-of-the-box.
|
||||
|
||||
You can define [**Users**](https://svc.plus/docs/pgsql/config/user), [**Databases**](https://svc.plus/docs/pgsql/config/db), [**Service**](https://svc.plus/docs/pgsql/service), [**HBAs**](https://svc.plus/docs/pgsql/config/hba) and other entities with code and provision them in one pass.
|
||||
|
||||
--------
|
||||
|
||||
You can also self-host postgres-centric software like [**`SUPABASE`**](https://svc.plus/docs/app/supabase), [**`Odoo`**](https://svc.plus/docs/app/odoo) & [**`Dify`**](https://svc.plus/docs/app/dify), Electric, GitLab, ... with Pigsty:
|
||||
|
||||
<details><summary>Example: Self-hosting Supabase</summary><br>
|
||||
|
||||
You can launch a [self-hosting supabase](https://svc.plus/docs/app/supabase) with MinIO and PostgreSQL with just several commands:
|
||||
Quick start:
|
||||
|
||||
```bash
|
||||
./configure -c supabase # use supabase config
|
||||
./deploy.yml # install pigsty
|
||||
./docker.yml # install docker compose
|
||||
./app.yml # launch supabase stateless part with docker
|
||||
./configure -c deepflow/deepflow
|
||||
vi pigsty.yml # adjust domain/password/ports
|
||||
./deploy.yml
|
||||
./docker.yml
|
||||
./deepflow.yml
|
||||
./infra.yml -t caddy # apply deepflow_grpc_domain ingress
|
||||
```
|
||||
|
||||
The [`conf/supabase.yml`](https://github.com/pgsty/pigsty/blob/main/conf/supabase.yml) just describe everything you need:
|
||||
Default inventory template: `conf/deepflow/deepflow.yml`
|
||||
|
||||
```yaml
|
||||
pg-meta:
|
||||
hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }
|
||||
vars:
|
||||
pg_cluster: pg-meta
|
||||
pg_users:
|
||||
# supabase roles: anon, authenticated, dashboard_user
|
||||
- { name: anon ,login: false }
|
||||
- { name: authenticated ,login: false }
|
||||
- { name: dashboard_user ,login: false ,replication: true ,createdb: true ,createrole: true }
|
||||
- { name: service_role ,login: false ,bypassrls: true }
|
||||
# supabase users: please use the same password
|
||||
- { name: supabase_admin ,password: 'DBUser.Supa' ,pgbouncer: true ,inherit: true ,roles: [ dbrole_admin ] ,superuser: true ,replication: true ,createdb: true ,createrole: true ,bypassrls: true }
|
||||
- { name: authenticator ,password: 'DBUser.Supa' ,pgbouncer: true ,inherit: false ,roles: [ dbrole_admin, authenticated ,anon ,service_role ] }
|
||||
- { name: supabase_auth_admin ,password: 'DBUser.Supa' ,pgbouncer: true ,inherit: false ,roles: [ dbrole_admin ] ,createrole: true }
|
||||
- { name: supabase_storage_admin ,password: 'DBUser.Supa' ,pgbouncer: true ,inherit: false ,roles: [ dbrole_admin, authenticated ,anon ,service_role ] ,createrole: true }
|
||||
- { name: supabase_functions_admin ,password: 'DBUser.Supa' ,pgbouncer: true ,inherit: false ,roles: [ dbrole_admin ] ,createrole: true }
|
||||
- { name: supabase_replication_admin ,password: 'DBUser.Supa' ,replication: true ,roles: [ dbrole_admin ]}
|
||||
- { name: supabase_read_only_user ,password: 'DBUser.Supa' ,bypassrls: true ,roles: [ dbrole_readonly, pg_read_all_data ] }
|
||||
pg_databases:
|
||||
- name: postgres
|
||||
baseline: supabase.sql
|
||||
owner: supabase_admin
|
||||
comment: supabase postgres database
|
||||
schemas: [ extensions ,auth ,realtime ,storage ,graphql_public ,supabase_functions ,_analytics ,_realtime ]
|
||||
extensions:
|
||||
- { name: pgcrypto ,schema: extensions } # cryptographic functions
|
||||
- { name: pg_net ,schema: extensions } # async HTTP
|
||||
- { name: pgjwt ,schema: extensions } # json web token API for postgres
|
||||
- { name: uuid-ossp ,schema: extensions } # generate universally unique identifiers (UUIDs)
|
||||
- { name: pgsodium } # pgsodium is a modern cryptography library for Postgres.
|
||||
- { name: supabase_vault } # Supabase Vault Extension
|
||||
- { name: pg_graphql } # pg_graphql: GraphQL support
|
||||
- { name: pg_jsonschema } # pg_jsonschema: Validate json schema
|
||||
- { name: wrappers } # wrappers: FDW collections
|
||||
- { name: http } # http: allows web page retrieval inside the database.
|
||||
- { name: pg_cron } # pg_cron: Job scheduler for PostgreSQL
|
||||
- { name: timescaledb } # timescaledb: Enables scalable inserts and complex queries for time-series data
|
||||
- { name: pg_tle } # pg_tle: Trusted Language Extensions for PostgreSQL
|
||||
- { name: vector } # pgvector: the vector similarity search
|
||||
- { name: pgmq } # pgmq: A lightweight message queue like AWS SQS and RSMQ
|
||||
# supabase required extensions
|
||||
pg_libs: 'timescaledb, plpgsql, plpgsql_check, pg_cron, pg_net, pg_stat_statements, auto_explain, pg_tle, plan_filter'
|
||||
pg_parameters:
|
||||
cron.database_name: postgres
|
||||
pgsodium.enable_event_trigger: off
|
||||
pg_hba_rules: # supabase hba rules, require access from docker network
|
||||
- { user: all ,db: postgres ,addr: intra ,auth: pwd ,title: 'allow supabase access from intranet' }
|
||||
- { user: all ,db: postgres ,addr: 172.17.0.0/16 ,auth: pwd ,title: 'allow access from local docker network' }
|
||||
pg_crontab: [ '00 01 * * * /pg/bin/pg-backup full' ] # make a full backup every 1am
|
||||
### Lightweight Topology
|
||||
|
||||
- `deepflow-server` stays containerized with Docker Compose
|
||||
- ClickHouse is kept as short-retention local storage
|
||||
- MinIO/S3 is optional in lightweight mode
|
||||
- `deepflow_connector` exports selected DeepFlow L4/L7 metrics to VictoriaMetrics
|
||||
- `deepflow_agent` supports `binary/systemd`, `docker`, and rendered `k8s` manifests
|
||||
- default `deepflow_agent_profile=lite` keeps `pcap` enabled and disables built-in `vector`
|
||||
|
||||
### Remote client example (openclaw.svc.plus)
|
||||
|
||||
```bash
|
||||
ssh root@openclaw.svc.plus \
|
||||
'curl -fsSL https://raw.githubusercontent.com/cloud-neutral-toolkit/observability.svc.plus/main/scripts/agent-install.sh \
|
||||
| bash -s -- --endpoint https://observability.svc.plus/ingest/otlp'
|
||||
```
|
||||
|
||||
|
||||
### Remote client example (jp-xhttp.svc.plus)
|
||||
|
||||
</details>
|
||||
```bash
|
||||
ssh root@jp-xhttp.svc.plus \
|
||||
'curl -fsSL https://raw.githubusercontent.com/cloud-neutral-toolkit/observability.svc.plus/main/scripts/agent-install.sh \
|
||||
| bash -s -- --endpoint https://observability.svc.plus/ingest/otlp'
|
||||
```
|
||||
|
||||
There are other pro, beta, or pilot modules, and there will be more coming in the future:
|
||||
### Optional SSH manager env example
|
||||
|
||||
[](https://svc.plus/docs/pgsql/kernel/babelfish)
|
||||
[](https://svc.plus/docs/pgsql/kernel/polardb)
|
||||
[](https://svc.plus/docs/pgsql/kernel/polardb-o)
|
||||
[](https://svc.plus/docs/pgsql/kernel/ivorysql)
|
||||
[](https://svc.plus/docs/pgsql/kernel/greenplum)
|
||||
[](https://svc.plus/docs/pgsql/kernel/cloudberry)
|
||||
[](https://svc.plus/docs/pgsql/kernel/openhalo)
|
||||
[](https://svc.plus/docs/pgsql/kernel/supabase)
|
||||
[](https://svc.plus/docs/pilot/kafka)
|
||||
[](https://svc.plus/docs/pilot/kafka)
|
||||
[](https://svc.plus/docs/pilot/duckdb)
|
||||
[](https://svc.plus/docs/pilot/tigerbeetle)
|
||||
[](https://svc.plus/docs/pilot/victoria)
|
||||
[](https://svc.plus/docs/pilot/kube)
|
||||
[](https://svc.plus/docs/pilot/consul)
|
||||
[](https://svc.plus/docs/vibe/)
|
||||
[](https://svc.plus/docs/pilot)
|
||||
```bash
|
||||
SSH_SERVER_CLAWBOT_HOST=openclaw.svc.plus
|
||||
SSH_SERVER_CLAWBOT_USER=root
|
||||
SSH_SERVER_CLAWBOT_KEYPATH=~/.ssh/id_rsa
|
||||
SSH_SERVER_CLAWBOT_PORT=22
|
||||
SSH_SERVER_CLAWBOT_DESCRIPTION=openclaw_server
|
||||
```
|
||||
|
||||
## 4) Features
|
||||
|
||||
----------------
|
||||
- **Observability First**: SOTA monitoring for PG / Infra / Node based on VictoriaMetrics, Grafana, and OpenTelemetry.
|
||||
- **OTel Integration**: Native support for OpenTelemetry, facilitating unified trace, metric, and log ingestion.
|
||||
- **DeepFlow Ready**: Lightweight DeepFlow server/agent deployment with short-lived flow storage and VictoriaMetrics archiving for high-value protocol metrics.
|
||||
- **Reliable Base**: Robust self-healing HA clusters, PITR, and secure infrastructure.
|
||||
- **Maintainable**: One-Cmd Deploy, IaC support, and easy customization.
|
||||
- **Controllable**: Self-sufficient Cloud Neutral FOSS. Run on bare Linux.
|
||||
|
||||
## Compatibility
|
||||
## 5) License & Upstream
|
||||
|
||||
We recommend using RockyLinux 10.0, Debian 13.2, and Ubuntu 24.04.2 for production use.
|
||||
- **License**: [Apache-2.0](LICENSE)
|
||||
- **Upstream references**:
|
||||
- [Pigsty](https://github.com/pgsty/pigsty)
|
||||
- [OpenTelemetry](https://opentelemetry.io/)
|
||||
- [VictoriaMetrics](https://victoriametrics.com/)
|
||||
- [Grafana](https://grafana.com/)
|
||||
|
||||
Pigsty runs on bare linux directly, and focuses on active maintained mainstream LTS [**Linux Distros**](https://svc.plus/docs/ref/linux):
|
||||
## 6) 致谢
|
||||
|
||||
| Code | Distro | `x86_64` | Status | `aarch64` | Status |
|
||||
|:--------:|-----------------------------------|:---------------------------------------------------:|:-------|:-----------------------------------------------------:|:-------|
|
||||
| **EL10** | RHEL 10 / Rocky10 / Alma10 / ... | [`el10.x86_64`](roles/node_id/vars/el10.x86_64.yml) | ✅📦 | [`el10.aarch64`](roles/node_id/vars/el10.aarch64.yml) | ✅📦 |
|
||||
| **EL9** | RHEL 9 / Rocky9 / Alma9 / ... | [`el9.x86_64`](roles/node_id/vars/el9.x86_64.yml) | ✅📦 | [`el9.aarch64`](roles/node_id/vars/el9.aarch64.yml) | ✅📦 |
|
||||
| **EL8** | RHEL 8 / Rocky8 / Alma8 / Anolis8 | [`el8.x86_64`](roles/node_id/vars/el8.x86_64.yml) | ✅📦 | [`el8.aarch64`](roles/node_id/vars/el8.aarch64.yml) | ✅📦 |
|
||||
| **U24** | Ubuntu 24.04 (noble) | [`u24.x86_64`](roles/node_id/vars/u24.x86_64.yml) | ✅📦 | [`u24.aarch64`](roles/node_id/vars/u24.aarch64.yml) | ✅📦 |
|
||||
| **U22** | Ubuntu 22.04 (jammy) | [`u22.x86_64`](roles/node_id/vars/u22.x86_64.yml) | ✅📦 | [`u22.aarch64`](roles/node_id/vars/u22.aarch64.yml) | ✅📦 |
|
||||
| **D13** | Debian 13 (trixie) | [`d13.x86_64`](roles/node_id/vars/d13.x86_64.yml) | ✅📦 | [`d13.aarch64`](roles/node_id/vars/d13.aarch64.yml) | ✅📦 |
|
||||
| **D12** | Debian 12 (bookworm) | [`d12.x86_64`](roles/node_id/vars/d12.x86_64.yml) | ✅📦 | [`d12.aarch64`](roles/node_id/vars/d12.aarch64.yml) | ✅📦 |
|
||||
|
||||
|
||||
|
||||
## Sponsors
|
||||
|
||||
Many thanks to our contributors and sponsors for making Pigsty possible.
|
||||
Special thanks to MiraclePlus for fund, to Cloudflare for hosting the Pigsty repo, and to Vercel for hosting the Pigsty website.
|
||||
|
||||
[](https://vercel.com/oss)
|
||||
|
||||
|
||||
## License
|
||||
|
||||
Pigsty is licensed under [**Apache-2.0**](LICENSE) (since v4.0), Check [**Docs**](https://svc.plus/docs/about/license) for details.
|
||||
|
||||
|
||||
## About
|
||||
|
||||
[](https://svc.plus/docs)
|
||||
[](https://github.com/pgsty/pigsty/discussions)
|
||||
[](https://t.me/joinchat/gV9zfZraNPM3YjFh)
|
||||
[](https://discord.gg/j5pG8qfKxU)
|
||||
[](https://pigsty.io/img/pigsty/pigsty-cc.jpg)
|
||||
[](https://qm.qq.com/q/vm8LIeUqGc)
|
||||
|
||||
[](https://vonng.com/)
|
||||
[](https://vonng.com/en/)
|
||||
[](mailto:rh@vonng.com)
|
||||
[-red?logo=c&color=steelblue)](https://github.com/Vonng)
|
||||
[](https://svc.plus/docs/about/license/)
|
||||
[](https://pigsty.cc/docs/about/service)
|
||||
感谢开源社区与所有贡献者，特别是 observability、database、DevOps 相关项目维护者与实践者。
|
||||
|
||||
@ -3,11 +3,11 @@ forks = 10
|
||||
nocows = 1
|
||||
timeout = 15
|
||||
pipelining = True
|
||||
inventory = pigsty.yml
|
||||
inventory = observability.yml
|
||||
host_key_checking = False
|
||||
command_warnings = False
|
||||
deprecation_warnings = False
|
||||
force_valid_group_names = ignore
|
||||
use_persistent_connections = True
|
||||
allow_world_readable_tmpfiles = False
|
||||
ansible_managed = 'ansible managed: %Y-%m-%d %H:%M:%S'
|
||||
ansible_managed = 'ansible managed: %Y-%m-%d %H:%M:%S'
|
||||
|
||||
@ -287,6 +287,15 @@ function fix_nopass_ssh(){
|
||||
if ! grep -q "${publicKey}" ~/.ssh/authorized_keys; then
|
||||
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
|
||||
fi
|
||||
# If root, ensure PermitRootLogin is allowed
|
||||
if [[ $(id -u) -eq 0 ]]; then
|
||||
if grep -q "PermitRootLogin" /etc/ssh/sshd_config; then
|
||||
sudo sed -i 's/^.*PermitRootLogin.*/PermitRootLogin prohibit-password/' /etc/ssh/sshd_config
|
||||
else
|
||||
echo "PermitRootLogin prohibit-password" | sudo tee -a /etc/ssh/sshd_config > /dev/null
|
||||
fi
|
||||
sudo systemctl reload ssh &>/dev/null || sudo systemctl reload sshd &>/dev/null
|
||||
fi
|
||||
return $(can_nopass_ssh)
|
||||
}
|
||||
|
||||
|
||||
115
conf/deepflow/deepflow.yml
Normal file
115
conf/deepflow/deepflow.yml
Normal file
@ -0,0 +1,115 @@
|
||||
---
|
||||
#==============================================================#
|
||||
# File : deepflow.yml
|
||||
# Desc : observability config for running DeepFlow stack
|
||||
# Ctime : 2026-02-04
|
||||
# Mtime : 2026-02-04
|
||||
# License : Apache-2.0 @ https://pigsty.io/docs/about/license/
|
||||
#==============================================================#
|
||||
|
||||
# how to use this template:
|
||||
#
|
||||
# curl -fsSL https://repo.pigsty.io/get | bash; cd ~/pigsty
|
||||
# ./bootstrap # prepare local repo & ansible
|
||||
# ./configure -c deepflow/deepflow # use this deepflow config template
|
||||
# vi pigsty.yml # IMPORTANT: CHANGE CREDENTIALS / DOMAIN
|
||||
# ./deploy.yml # install infra stack
|
||||
# ./docker.yml # install docker & docker-compose
|
||||
# ./deepflow.yml # install deepflow with compose + optional connector/agent
|
||||
|
||||
all:
|
||||
children:
|
||||
|
||||
deepflow:
|
||||
hosts: { 10.10.10.10: {} }
|
||||
vars:
|
||||
deepflow_enabled: true
|
||||
deepflow_mysql_enabled: true
|
||||
deepflow_clickhouse_s3_enabled: true
|
||||
deepflow_connector_enabled: true
|
||||
deepflow_agent_enabled: false
|
||||
|
||||
deepflow_deploy_profile: lite
|
||||
deepflow_storage_mode: short_ttl
|
||||
|
||||
deepflow_data: /data/deepflow
|
||||
|
||||
# role: deepflow_mysql
|
||||
deepflow_mysql_port: 13306
|
||||
deepflow_mysql_root_password: DeepFlow.Root.ChangeMe
|
||||
deepflow_mysql_user: deepflow
|
||||
deepflow_mysql_password: DeepFlow.MySQL.ChangeMe
|
||||
deepflow_mysql_database: deepflow
|
||||
|
||||
# role: deepflow_clickhouse_s3
|
||||
deepflow_clickhouse_http_port: 18123
|
||||
deepflow_clickhouse_tcp_port: 19000
|
||||
deepflow_clickhouse_retention_hours: 24
|
||||
deepflow_s3_enabled: false
|
||||
deepflow_minio_api_port: 19090
|
||||
deepflow_minio_console_port: 19091
|
||||
deepflow_s3_bucket: deepflow
|
||||
deepflow_s3_access_key: deepflow
|
||||
deepflow_s3_secret_key: DeepFlow.S3.ChangeMe
|
||||
deepflow_s3_region: us-east-1
|
||||
|
||||
# role: deepflow_server
|
||||
deepflow_server_grpc_port: 20035
|
||||
deepflow_server_http_port: 20417
|
||||
deepflow_app_port: 20880
|
||||
deepflow_clickhouse_addr: host.docker.internal:19000
|
||||
deepflow_s3_endpoint: http://host.docker.internal:19090
|
||||
deepflow_mysql_addr: host.docker.internal:13306
|
||||
deepflow_l4_log_ttl_hour: 24
|
||||
deepflow_l7_log_ttl_hour: 24
|
||||
deepflow_flow_metrics_ttl_hour: 24
|
||||
deepflow_metrics_ttl_hour: 24
|
||||
deepflow_prometheus_ttl_hour: 24
|
||||
|
||||
# role: deepflow_connector
|
||||
deepflow_connector_source_endpoint: http://127.0.0.1:20417/metrics
|
||||
deepflow_connector_remote_write_url: http://127.0.0.1:8428/api/v1/write
|
||||
|
||||
# role: deepflow_agent
|
||||
deepflow_agent_mode: binary
|
||||
deepflow_agent_profile: lite
|
||||
deepflow_agent_disable_pcap: false
|
||||
deepflow_agent_disable_vector: true
|
||||
deepflow_agent_grpc_endpoint: "{{ deepflow_grpc_domain }}:443"
|
||||
|
||||
infra: { hosts: { 10.10.10.10: { infra_seq: 1 } } }
|
||||
etcd: { hosts: { 10.10.10.10: { etcd_seq: 1 } }, vars: { etcd_cluster: etcd } }
|
||||
|
||||
vars:
|
||||
version: v4.0.0
|
||||
admin_ip: 10.10.10.10
|
||||
region: default
|
||||
node_tune: oltp
|
||||
pg_conf: oltp.yml
|
||||
docker_enabled: true
|
||||
|
||||
# Caddy gRPC ingress for deepflow-agent:
|
||||
caddy_enabled: true
|
||||
deepflow_grpc_enabled: true
|
||||
deepflow_grpc_domain: deepflow-agent.pigsty
|
||||
deepflow_grpc_upstream: 127.0.0.1:20035
|
||||
|
||||
infra_portal:
|
||||
home : { domain: svc.plus }
|
||||
deepflow : { domain: deepflow.pigsty ,endpoint: "10.10.10.10:20880" }
|
||||
|
||||
proxy_env:
|
||||
no_proxy: "localhost,127.0.0.1,10.0.0.0/8,192.168.0.0/16,*.pigsty,*.aliyun.com,mirrors.*,*.tsinghua.edu.cn"
|
||||
|
||||
repo_enabled: false
|
||||
node_repo_modules: node,infra,pgsql
|
||||
|
||||
grafana_admin_password: pigsty
|
||||
grafana_view_password: DBUser.Viewer
|
||||
pg_admin_password: DBUser.DBA
|
||||
pg_monitor_password: DBUser.Monitor
|
||||
pg_replication_password: DBUser.Replicator
|
||||
patroni_password: Patroni.API
|
||||
haproxy_admin_password: pigsty
|
||||
minio_secret_key: S3User.MinIO
|
||||
etcd_root_password: Etcd.Root
|
||||
8
configure
vendored
8
configure
vendored
@ -398,6 +398,14 @@ function check_ipaddr(){
|
||||
return 0
|
||||
fi
|
||||
|
||||
# multiple IP detected, try to find the "best" one (non-docker, non-loopback)
|
||||
local best_ip=$(hostname --all-ip-addresses | tr ' ' '\n' | grep -vE '^(127\.|172\.(1[6-9]|2[0-9]|3[01])\.|169\.254\.)' | head -n1)
|
||||
if [[ -n "${best_ip}" && ${interactive} != "true" ]]; then
|
||||
log_info "primary_ip = ${best_ip} (auto-selected from multiple)"
|
||||
PRIMARY_IP=${best_ip}
|
||||
return 0
|
||||
fi
|
||||
|
||||
# multiple IP detected
|
||||
log_warn "Multiple IP address candidates found:"
|
||||
list_ipaddr
|
||||
|
||||
28
deepflow.yml
Normal file
28
deepflow.yml
Normal file
@ -0,0 +1,28 @@
|
||||
#!/usr/bin/env ansible-playbook
|
||||
---
|
||||
#==============================================================#
|
||||
# File : deepflow.yml
|
||||
# Desc : deploy deepflow stack with three dedicated roles
|
||||
# Ctime : 2026-02-04
|
||||
# Mtime : 2026-02-04
|
||||
# Path : deepflow.yml
|
||||
# License : Apache-2.0 @ https://pigsty.io/docs/about/license/
|
||||
#==============================================================#
|
||||
|
||||
- name: DEEPFLOW STACK
|
||||
become: true
|
||||
hosts: all
|
||||
gather_facts: no
|
||||
|
||||
roles:
|
||||
- { role: node_id , tags: node-id, when: deepflow_enabled | default(true) | bool }
|
||||
- { role: deepflow_mysql , tags: deepflow_mysql, when: deepflow_mysql_enabled | default(true) | bool }
|
||||
- { role: deepflow_clickhouse_s3, tags: deepflow_clickhouse_s3, when: deepflow_clickhouse_s3_enabled | default(true) | bool }
|
||||
- { role: deepflow_server , tags: deepflow_server, when: deepflow_enabled | default(true) | bool }
|
||||
- { role: deepflow_connector , tags: deepflow_connector, when: deepflow_connector_enabled | default(false) | bool }
|
||||
- { role: deepflow_agent , tags: deepflow_agent, when: deepflow_agent_enabled | default(false) | bool }
|
||||
|
||||
# Usage:
|
||||
# 1. Define deepflow group in pigsty.yml
|
||||
# 2. Ensure docker is installed: ./docker.yml
|
||||
# 3. Run ./deepflow.yml -l <deepflow_group>
|
||||
11
deploy.yml
11
deploy.yml
@ -93,6 +93,17 @@
|
||||
roles: [ { role: infra } ]
|
||||
|
||||
|
||||
#---------------------------------------------------------------
|
||||
# init insight workbench
|
||||
#---------------------------------------------------------------
|
||||
- name: INSIGHT INIT
|
||||
become: true
|
||||
hosts: infra
|
||||
gather_facts: no
|
||||
tags: insight
|
||||
roles: [ { role: insight } ]
|
||||
|
||||
|
||||
#---------------------------------------------------------------
|
||||
# Node Monitor
|
||||
#---------------------------------------------------------------
|
||||
|
||||
147
deploy_observability_service.yml
Normal file
147
deploy_observability_service.yml
Normal file
@ -0,0 +1,147 @@
|
||||
---
|
||||
- name: Update Cloudflare DNS for observability.svc.plus
|
||||
hosts: localhost
|
||||
connection: local
|
||||
gather_facts: false
|
||||
vars:
|
||||
cloudflare_zone_name: svc.plus
|
||||
cloudflare_api_base: https://api.cloudflare.com/client/v4
|
||||
observability_domain: observability.svc.plus
|
||||
observability_dns_target: us-xhttp.svc.plus
|
||||
observability_dns_type: CNAME
|
||||
observability_dns_ttl: 1
|
||||
observability_dns_proxied: false
|
||||
dns_wait_retries: 30
|
||||
dns_wait_delay: 10
|
||||
tasks:
|
||||
- name: Validate Cloudflare token is present in environment
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- lookup('ansible.builtin.env', 'CLOUDFLARE_API_TOKEN') | length > 0
|
||||
fail_msg: "CLOUDFLARE_API_TOKEN must be exported before running this playbook."
|
||||
|
||||
- name: Resolve Cloudflare zone id
|
||||
ansible.builtin.uri:
|
||||
url: "{{ cloudflare_api_base }}/zones?name={{ cloudflare_zone_name }}"
|
||||
method: GET
|
||||
headers:
|
||||
Authorization: "Bearer {{ lookup('ansible.builtin.env', 'CLOUDFLARE_API_TOKEN') }}"
|
||||
Content-Type: application/json
|
||||
return_content: true
|
||||
register: cloudflare_zone_lookup
|
||||
|
||||
- name: Validate zone lookup result
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- cloudflare_zone_lookup.json.success
|
||||
- cloudflare_zone_lookup.json.result | length > 0
|
||||
fail_msg: "Unable to resolve Cloudflare zone id for {{ cloudflare_zone_name }}."
|
||||
|
||||
- name: Set Cloudflare zone id
|
||||
ansible.builtin.set_fact:
|
||||
cloudflare_zone_id: "{{ cloudflare_zone_lookup.json.result[0].id }}"
|
||||
|
||||
- name: Query existing observability DNS records
|
||||
ansible.builtin.uri:
|
||||
url: "{{ cloudflare_api_base }}/zones/{{ cloudflare_zone_id }}/dns_records?name={{ observability_domain }}"
|
||||
method: GET
|
||||
headers:
|
||||
Authorization: "Bearer {{ lookup('ansible.builtin.env', 'CLOUDFLARE_API_TOKEN') }}"
|
||||
Content-Type: application/json
|
||||
return_content: true
|
||||
register: observability_dns_records
|
||||
|
||||
- name: Remove conflicting observability DNS records with different type
|
||||
ansible.builtin.uri:
|
||||
url: "{{ cloudflare_api_base }}/zones/{{ cloudflare_zone_id }}/dns_records/{{ item.id }}"
|
||||
method: DELETE
|
||||
headers:
|
||||
Authorization: "Bearer {{ lookup('ansible.builtin.env', 'CLOUDFLARE_API_TOKEN') }}"
|
||||
Content-Type: application/json
|
||||
loop: "{{ observability_dns_records.json.result | default([]) }}"
|
||||
loop_control:
|
||||
label: "{{ item.type }} {{ item.name }}"
|
||||
when: item.type != observability_dns_type
|
||||
|
||||
- name: Create observability DNS record when missing
|
||||
ansible.builtin.uri:
|
||||
url: "{{ cloudflare_api_base }}/zones/{{ cloudflare_zone_id }}/dns_records"
|
||||
method: POST
|
||||
headers:
|
||||
Authorization: "Bearer {{ lookup('ansible.builtin.env', 'CLOUDFLARE_API_TOKEN') }}"
|
||||
Content-Type: application/json
|
||||
body_format: raw
|
||||
body: >-
|
||||
{{
|
||||
{
|
||||
'type': observability_dns_type,
|
||||
'name': observability_domain,
|
||||
'content': observability_dns_target,
|
||||
'ttl': (observability_dns_ttl | int),
|
||||
'proxied': (observability_dns_proxied | bool)
|
||||
} | to_json
|
||||
}}
|
||||
when: (observability_dns_records.json.result | selectattr('type', 'equalto', observability_dns_type) | list | length) == 0
|
||||
|
||||
- name: Update observability DNS record when target changes
|
||||
ansible.builtin.uri:
|
||||
url: "{{ cloudflare_api_base }}/zones/{{ cloudflare_zone_id }}/dns_records/{{ (observability_dns_records.json.result | selectattr('type', 'equalto', observability_dns_type) | list | first).id }}"
|
||||
method: PUT
|
||||
headers:
|
||||
Authorization: "Bearer {{ lookup('ansible.builtin.env', 'CLOUDFLARE_API_TOKEN') }}"
|
||||
Content-Type: application/json
|
||||
body_format: raw
|
||||
body: >-
|
||||
{{
|
||||
{
|
||||
'type': observability_dns_type,
|
||||
'name': observability_domain,
|
||||
'content': observability_dns_target,
|
||||
'ttl': (observability_dns_ttl | int),
|
||||
'proxied': (observability_dns_proxied | bool)
|
||||
} | to_json
|
||||
}}
|
||||
when:
|
||||
- (observability_dns_records.json.result | selectattr('type', 'equalto', observability_dns_type) | list | length) > 0
|
||||
- >
|
||||
((observability_dns_records.json.result | selectattr('type', 'equalto', observability_dns_type) | list | first).content != observability_dns_target)
|
||||
or
|
||||
(((observability_dns_records.json.result | selectattr('type', 'equalto', observability_dns_type) | list | first).proxied | default(false)) != observability_dns_proxied)
|
||||
|
||||
- name: Wait for public DNS to expose observability CNAME
|
||||
ansible.builtin.uri:
|
||||
url: "https://cloudflare-dns.com/dns-query?name={{ observability_domain }}&type=CNAME"
|
||||
method: GET
|
||||
headers:
|
||||
Accept: application/dns-json
|
||||
return_content: true
|
||||
register: observability_dns_public
|
||||
until:
|
||||
- observability_dns_public.status == 200
|
||||
- >
|
||||
(
|
||||
observability_dns_public.json.Status
|
||||
if (observability_dns_public.json is defined)
|
||||
else ((observability_dns_public.content | from_json).Status | default(1))
|
||||
) == 0
|
||||
- >
|
||||
(
|
||||
observability_dns_public.json.Answer
|
||||
if (observability_dns_public.json is defined)
|
||||
else ((observability_dns_public.content | from_json).Answer | default([]))
|
||||
) | selectattr('data', 'equalto', observability_dns_target ~ '.')
|
||||
| list | length > 0
|
||||
retries: "{{ dns_wait_retries }}"
|
||||
delay: "{{ dns_wait_delay }}"
|
||||
|
||||
- name: Show effective observability DNS target
|
||||
ansible.builtin.debug:
|
||||
msg: "{{ observability_domain }} -> {{ observability_dns_target }} proxied={{ observability_dns_proxied }}"
|
||||
|
||||
- import_playbook: infra.yml
|
||||
vars:
|
||||
infra_domain: observability.svc.plus
|
||||
infra_portal:
|
||||
home: { domain: observability.svc.plus }
|
||||
caddy_enabled: true
|
||||
nginx_enabled: false
|
||||
14
docs/DOC_COVERAGE.md
Normal file
14
docs/DOC_COVERAGE.md
Normal file
@ -0,0 +1,14 @@
|
||||
# Documentation Coverage Matrix
|
||||
|
||||
This matrix tracks the bilingual canonical documentation set for `observability.svc.plus` and maps it back to the current codebase and older docs.
|
||||
|
||||
该矩阵用于跟踪 `observability.svc.plus` 的双语规范文档，并将其与当前代码状态和历史文档对应起来。
|
||||
|
||||
| Category | EN | ZH | Current status | Existing references | Next check |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| Architecture | Yes | Yes | Seeded from current codebase; deeper legacy consolidation is still needed. | None yet; use the new canonical page as the starting point. | Keep diagrams and ownership notes synchronized with actual directories, services, and integration dependencies. |
|
||||
| Design | Yes | Yes | Seeded from current codebase; deeper legacy consolidation is still needed. | None yet; use the new canonical page as the starting point. | Promote one-off implementation notes into reusable design records when behavior, APIs, or deployment contracts change. |
|
||||
| Deployment | Yes | Yes | Seeded from current codebase; deeper legacy consolidation is still needed. | None yet; use the new canonical page as the starting point. | Verify deployment steps against current scripts, manifests, CI/CD flow, and environment contracts before each release. |
|
||||
| User Guide | Yes | Yes | Seeded from current codebase; deeper legacy consolidation is still needed. | None yet; use the new canonical page as the starting point. | Prefer workflow-oriented examples and keep screenshots or terminal snippets aligned with the latest UI or CLI behavior. |
|
||||
| Developer Guide | Yes | Yes | Seeded from current codebase; deeper legacy consolidation is still needed. | None yet; use the new canonical page as the starting point. | Keep setup and test commands tied to actual package scripts, Make targets, or language toolchains in this repository. |
|
||||
| Vibe Coding Reference | Yes | Yes | Seeded from current codebase; deeper legacy consolidation is still needed. | None yet; use the new canonical page as the starting point. | Review prompt templates and repo rules whenever the project adds new subsystems, protected areas, or mandatory verification steps. |
|
||||
31
docs/README.md
Normal file
31
docs/README.md
Normal file
@ -0,0 +1,31 @@
|
||||
# Observability Service Plus / 可观测性服务
|
||||
|
||||
This `docs/` directory now has a bilingual canonical layer for the current repository state.
|
||||
|
||||
本 `docs/` 目录现已补齐双语规范层，用于承接当前仓库状态下的核心文档。
|
||||
|
||||
## Quick Entry / 快速入口
|
||||
|
||||
- Coverage checklist / 覆盖检查矩阵: `docs/DOC_COVERAGE.md`
|
||||
- English index / 英文入口: `docs/en/README.md`
|
||||
- 中文入口 / Chinese index: `docs/zh/README.md`
|
||||
|
||||
## Canonical Bilingual Pages / 双语规范页
|
||||
|
||||
- `docs/en/architecture.md` / `docs/zh/architecture.md`
|
||||
- `docs/en/design.md` / `docs/zh/design.md`
|
||||
- `docs/en/deployment.md` / `docs/zh/deployment.md`
|
||||
- `docs/en/user-guide.md` / `docs/zh/user-guide.md`
|
||||
- `docs/en/developer-guide.md` / `docs/zh/developer-guide.md`
|
||||
- `docs/en/vibe-coding-reference.md` / `docs/zh/vibe-coding-reference.md`
|
||||
|
||||
## Current Repo Context / 当前仓库背景
|
||||
|
||||
- Root README: `Observability.svc.plus`
|
||||
- Previous docs index: `Documentation`
|
||||
- Manifest evidence / 构建清单: repository structure and scripts only
|
||||
- Active code and ops directories / 当前主要目录: `app/`, `api/`, `scripts/`
|
||||
|
||||
## Existing Docs To Reconcile / 需要继续归并的现有文档
|
||||
|
||||
- No pre-existing markdown docs were detected in this repository.
|
||||
23
docs/en/README.md
Normal file
23
docs/en/README.md
Normal file
@ -0,0 +1,23 @@
|
||||
# Observability Service Plus Documentation
|
||||
|
||||
This repository documents infrastructure orchestration and observability composition rather than a single application binary.
|
||||
|
||||
## Current state snapshot
|
||||
|
||||
- Root README title: `Observability.svc.plus`
|
||||
- Build/runtime evidence: repository structure and scripts only
|
||||
- Primary directories detected: `app/`, `api/`, `scripts/`
|
||||
- Existing docs count: 0
|
||||
|
||||
## Canonical pages
|
||||
|
||||
- [Architecture](architecture.md)
|
||||
- [Design](design.md)
|
||||
- [Deployment](deployment.md)
|
||||
- [User Guide](user-guide.md)
|
||||
- [Developer Guide](developer-guide.md)
|
||||
- [Vibe Coding Reference](vibe-coding-reference.md)
|
||||
|
||||
## Legacy docs to fold in
|
||||
|
||||
- No pre-existing markdown docs were detected in this repository.
|
||||
24
docs/en/architecture.md
Normal file
24
docs/en/architecture.md
Normal file
@ -0,0 +1,24 @@
|
||||
# Architecture
|
||||
|
||||
This repository documents infrastructure orchestration and observability composition rather than a single application binary.
|
||||
|
||||
Use this page as the canonical bilingual overview of system boundaries, major components, and repo ownership.
|
||||
|
||||
## Current code-aligned notes
|
||||
|
||||
- Documentation target: `observability.svc.plus`
|
||||
- Repo kind: `infra-observability`
|
||||
- Manifest and build evidence: repository structure and scripts only
|
||||
- Primary implementation and ops directories: `app/`, `api/`, `scripts/`
|
||||
- Package scripts snapshot: No package.json scripts were detected.
|
||||
|
||||
## Existing docs to reconcile
|
||||
|
||||
- No directly matching legacy docs were detected; this page is currently the canonical seed.
|
||||
|
||||
## What this page should cover next
|
||||
|
||||
- Describe the current implementation rather than an aspirational future-only design.
|
||||
- Keep terminology aligned with the repository root README, manifests, and actual directories.
|
||||
- Link deeper runbooks, specs, or subsystem notes from the legacy docs listed above.
|
||||
- Keep diagrams and ownership notes synchronized with actual directories, services, and integration dependencies.
|
||||
24
docs/en/deployment.md
Normal file
24
docs/en/deployment.md
Normal file
@ -0,0 +1,24 @@
|
||||
# Deployment
|
||||
|
||||
This repository documents infrastructure orchestration and observability composition rather than a single application binary.
|
||||
|
||||
Use this page to standardize deployment prerequisites, supported topologies, operational checks, and rollback notes.
|
||||
|
||||
## Current code-aligned notes
|
||||
|
||||
- Documentation target: `observability.svc.plus`
|
||||
- Repo kind: `infra-observability`
|
||||
- Manifest and build evidence: repository structure and scripts only
|
||||
- Primary implementation and ops directories: `app/`, `api/`, `scripts/`
|
||||
- Package scripts snapshot: No package.json scripts were detected.
|
||||
|
||||
## Existing docs to reconcile
|
||||
|
||||
- No directly matching legacy docs were detected; this page is currently the canonical seed.
|
||||
|
||||
## What this page should cover next
|
||||
|
||||
- Describe the current implementation rather than an aspirational future-only design.
|
||||
- Keep terminology aligned with the repository root README, manifests, and actual directories.
|
||||
- Link deeper runbooks, specs, or subsystem notes from the legacy docs listed above.
|
||||
- Verify deployment steps against current scripts, manifests, CI/CD flow, and environment contracts before each release.
|
||||
24
docs/en/design.md
Normal file
24
docs/en/design.md
Normal file
@ -0,0 +1,24 @@
|
||||
# Design
|
||||
|
||||
This repository documents infrastructure orchestration and observability composition rather than a single application binary.
|
||||
|
||||
Use this page to consolidate design decisions, ADR-style tradeoffs, and roadmap-sensitive implementation notes.
|
||||
|
||||
## Current code-aligned notes
|
||||
|
||||
- Documentation target: `observability.svc.plus`
|
||||
- Repo kind: `infra-observability`
|
||||
- Manifest and build evidence: repository structure and scripts only
|
||||
- Primary implementation and ops directories: `app/`, `api/`, `scripts/`
|
||||
- Package scripts snapshot: No package.json scripts were detected.
|
||||
|
||||
## Existing docs to reconcile
|
||||
|
||||
- No directly matching legacy docs were detected; this page is currently the canonical seed.
|
||||
|
||||
## What this page should cover next
|
||||
|
||||
- Describe the current implementation rather than an aspirational future-only design.
|
||||
- Keep terminology aligned with the repository root README, manifests, and actual directories.
|
||||
- Link deeper runbooks, specs, or subsystem notes from the legacy docs listed above.
|
||||
- Promote one-off implementation notes into reusable design records when behavior, APIs, or deployment contracts change.
|
||||
24
docs/en/developer-guide.md
Normal file
24
docs/en/developer-guide.md
Normal file
@ -0,0 +1,24 @@
|
||||
# Developer Guide
|
||||
|
||||
This repository documents infrastructure orchestration and observability composition rather than a single application binary.
|
||||
|
||||
Use this page to document local setup, project structure, test surfaces, and contribution conventions tied to the current codebase.
|
||||
|
||||
## Current code-aligned notes
|
||||
|
||||
- Documentation target: `observability.svc.plus`
|
||||
- Repo kind: `infra-observability`
|
||||
- Manifest and build evidence: repository structure and scripts only
|
||||
- Primary implementation and ops directories: `app/`, `api/`, `scripts/`
|
||||
- Package scripts snapshot: No package.json scripts were detected.
|
||||
|
||||
## Existing docs to reconcile
|
||||
|
||||
- No directly matching legacy docs were detected; this page is currently the canonical seed.
|
||||
|
||||
## What this page should cover next
|
||||
|
||||
- Describe the current implementation rather than an aspirational future-only design.
|
||||
- Keep terminology aligned with the repository root README, manifests, and actual directories.
|
||||
- Link deeper runbooks, specs, or subsystem notes from the legacy docs listed above.
|
||||
- Keep setup and test commands tied to actual package scripts, Make targets, or language toolchains in this repository.
|
||||
24
docs/en/user-guide.md
Normal file
24
docs/en/user-guide.md
Normal file
@ -0,0 +1,24 @@
|
||||
# User Guide
|
||||
|
||||
This repository documents infrastructure orchestration and observability composition rather than a single application binary.
|
||||
|
||||
Use this page to document primary user/operator tasks, everyday workflows, and navigation to existing how-to material.
|
||||
|
||||
## Current code-aligned notes
|
||||
|
||||
- Documentation target: `observability.svc.plus`
|
||||
- Repo kind: `infra-observability`
|
||||
- Manifest and build evidence: repository structure and scripts only
|
||||
- Primary implementation and ops directories: `app/`, `api/`, `scripts/`
|
||||
- Package scripts snapshot: No package.json scripts were detected.
|
||||
|
||||
## Existing docs to reconcile
|
||||
|
||||
- No directly matching legacy docs were detected; this page is currently the canonical seed.
|
||||
|
||||
## What this page should cover next
|
||||
|
||||
- Describe the current implementation rather than an aspirational future-only design.
|
||||
- Keep terminology aligned with the repository root README, manifests, and actual directories.
|
||||
- Link deeper runbooks, specs, or subsystem notes from the legacy docs listed above.
|
||||
- Prefer workflow-oriented examples and keep screenshots or terminal snippets aligned with the latest UI or CLI behavior.
|
||||
24
docs/en/vibe-coding-reference.md
Normal file
24
docs/en/vibe-coding-reference.md
Normal file
@ -0,0 +1,24 @@
|
||||
# Vibe Coding Reference
|
||||
|
||||
This repository documents infrastructure orchestration and observability composition rather than a single application binary.
|
||||
|
||||
Use this page to align AI-assisted coding prompts, repo boundaries, safe edit rules, and documentation update expectations.
|
||||
|
||||
## Current code-aligned notes
|
||||
|
||||
- Documentation target: `observability.svc.plus`
|
||||
- Repo kind: `infra-observability`
|
||||
- Manifest and build evidence: repository structure and scripts only
|
||||
- Primary implementation and ops directories: `app/`, `api/`, `scripts/`
|
||||
- Package scripts snapshot: No package.json scripts were detected.
|
||||
|
||||
## Existing docs to reconcile
|
||||
|
||||
- No directly matching legacy docs were detected; this page is currently the canonical seed.
|
||||
|
||||
## What this page should cover next
|
||||
|
||||
- Describe the current implementation rather than an aspirational future-only design.
|
||||
- Keep terminology aligned with the repository root README, manifests, and actual directories.
|
||||
- Link deeper runbooks, specs, or subsystem notes from the legacy docs listed above.
|
||||
- Review prompt templates and repo rules whenever the project adds new subsystems, protected areas, or mandatory verification steps.
|
||||
23
docs/zh/README.md
Normal file
23
docs/zh/README.md
Normal file
@ -0,0 +1,23 @@
|
||||
# 可观测性服务 文档
|
||||
|
||||
该仓库更偏向基础设施编排与可观测体系组合，而不是单一应用二进制。
|
||||
|
||||
## 当前状态快照
|
||||
|
||||
- 根 README 标题: `Observability.svc.plus`
|
||||
- 构建与运行时证据: repository structure and scripts only
|
||||
- 自动识别的主要目录: `app/`, `api/`, `scripts/`
|
||||
- 现有文档数量: 0
|
||||
|
||||
## 核心双语文档
|
||||
|
||||
- [架构](architecture.md)
|
||||
- [设计](design.md)
|
||||
- [部署](deployment.md)
|
||||
- [使用手册](user-guide.md)
|
||||
- [开发手册](developer-guide.md)
|
||||
- [Vibe Coding 参考](vibe-coding-reference.md)
|
||||
|
||||
## 待归并的历史文档
|
||||
|
||||
- No pre-existing markdown docs were detected in this repository.
|
||||
24
docs/zh/architecture.md
Normal file
24
docs/zh/architecture.md
Normal file
@ -0,0 +1,24 @@
|
||||
# 架构
|
||||
|
||||
该仓库更偏向基础设施编排与可观测体系组合，而不是单一应用二进制。
|
||||
|
||||
本页作为系统边界、核心组件与仓库职责的双语总览入口。
|
||||
|
||||
## 与当前代码对齐的说明
|
||||
|
||||
- 文档目标仓库: `observability.svc.plus`
|
||||
- 仓库类型: `infra-observability`
|
||||
- 构建与运行依据: repository structure and scripts only
|
||||
- 主要实现与运维目录: `app/`, `api/`, `scripts/`
|
||||
- `package.json` 脚本快照: No package.json scripts were detected.
|
||||
|
||||
## 需要继续归并的现有文档
|
||||
|
||||
- 尚未发现直接对应的历史文档，本页目前就是该类别的规范起点。
|
||||
|
||||
## 本页下一步应补充的内容
|
||||
|
||||
- 先描述当前已落地实现，再补充未来规划，避免只写愿景不写现状。
|
||||
- 术语需要与仓库根 README、构建清单和实际目录保持一致。
|
||||
- 将上方列出的历史 runbook、spec、子系统说明逐步链接并归并到本页。
|
||||
- 随着目录结构、服务关系和集成依赖变化，持续同步图示与职责说明。
|
||||
24
docs/zh/deployment.md
Normal file
24
docs/zh/deployment.md
Normal file
@ -0,0 +1,24 @@
|
||||
# 部署
|
||||
|
||||
该仓库更偏向基础设施编排与可观测体系组合，而不是单一应用二进制。
|
||||
|
||||
本页用于统一部署前提、支持的拓扑、运维检查项与回滚注意事项。
|
||||
|
||||
## 与当前代码对齐的说明
|
||||
|
||||
- 文档目标仓库: `observability.svc.plus`
|
||||
- 仓库类型: `infra-observability`
|
||||
- 构建与运行依据: repository structure and scripts only
|
||||
- 主要实现与运维目录: `app/`, `api/`, `scripts/`
|
||||
- `package.json` 脚本快照: No package.json scripts were detected.
|
||||
|
||||
## 需要继续归并的现有文档
|
||||
|
||||
- 尚未发现直接对应的历史文档，本页目前就是该类别的规范起点。
|
||||
|
||||
## 本页下一步应补充的内容
|
||||
|
||||
- 先描述当前已落地实现，再补充未来规划，避免只写愿景不写现状。
|
||||
- 术语需要与仓库根 README、构建清单和实际目录保持一致。
|
||||
- 将上方列出的历史 runbook、spec、子系统说明逐步链接并归并到本页。
|
||||
- 每次发布前，依据当前脚本、清单、CI/CD 流程和环境契约重新核对部署步骤。
|
||||
24
docs/zh/design.md
Normal file
24
docs/zh/design.md
Normal file
@ -0,0 +1,24 @@
|
||||
# 设计
|
||||
|
||||
该仓库更偏向基础设施编排与可观测体系组合，而不是单一应用二进制。
|
||||
|
||||
本页用于汇总设计决策、类似 ADR 的权衡记录，以及与路线图相关的实现说明。
|
||||
|
||||
## 与当前代码对齐的说明
|
||||
|
||||
- 文档目标仓库: `observability.svc.plus`
|
||||
- 仓库类型: `infra-observability`
|
||||
- 构建与运行依据: repository structure and scripts only
|
||||
- 主要实现与运维目录: `app/`, `api/`, `scripts/`
|
||||
- `package.json` 脚本快照: No package.json scripts were detected.
|
||||
|
||||
## 需要继续归并的现有文档
|
||||
|
||||
- 尚未发现直接对应的历史文档，本页目前就是该类别的规范起点。
|
||||
|
||||
## 本页下一步应补充的内容
|
||||
|
||||
- 先描述当前已落地实现，再补充未来规划，避免只写愿景不写现状。
|
||||
- 术语需要与仓库根 README、构建清单和实际目录保持一致。
|
||||
- 将上方列出的历史 runbook、spec、子系统说明逐步链接并归并到本页。
|
||||
- 当行为、API 或部署契约发生变化时，把一次性实现笔记提升为可复用设计记录。
|
||||
24
docs/zh/developer-guide.md
Normal file
24
docs/zh/developer-guide.md
Normal file
@ -0,0 +1,24 @@
|
||||
# 开发手册
|
||||
|
||||
该仓库更偏向基础设施编排与可观测体系组合，而不是单一应用二进制。
|
||||
|
||||
本页用于记录本地开发环境、项目结构、测试面与贴合当前代码库的贡献约定。
|
||||
|
||||
## 与当前代码对齐的说明
|
||||
|
||||
- 文档目标仓库: `observability.svc.plus`
|
||||
- 仓库类型: `infra-observability`
|
||||
- 构建与运行依据: repository structure and scripts only
|
||||
- 主要实现与运维目录: `app/`, `api/`, `scripts/`
|
||||
- `package.json` 脚本快照: No package.json scripts were detected.
|
||||
|
||||
## 需要继续归并的现有文档
|
||||
|
||||
- 尚未发现直接对应的历史文档，本页目前就是该类别的规范起点。
|
||||
|
||||
## 本页下一步应补充的内容
|
||||
|
||||
- 先描述当前已落地实现，再补充未来规划，避免只写愿景不写现状。
|
||||
- 术语需要与仓库根 README、构建清单和实际目录保持一致。
|
||||
- 将上方列出的历史 runbook、spec、子系统说明逐步链接并归并到本页。
|
||||
- 持续让环境搭建与测试命令对应真实存在的脚本、Make 目标或语言工具链。
|
||||
24
docs/zh/user-guide.md
Normal file
24
docs/zh/user-guide.md
Normal file
@ -0,0 +1,24 @@
|
||||
# 使用手册
|
||||
|
||||
该仓库更偏向基础设施编排与可观测体系组合，而不是单一应用二进制。
|
||||
|
||||
本页用于记录主要用户或运维角色的日常任务、常见流程，以及现有操作文档入口。
|
||||
|
||||
## 与当前代码对齐的说明
|
||||
|
||||
- 文档目标仓库: `observability.svc.plus`
|
||||
- 仓库类型: `infra-observability`
|
||||
- 构建与运行依据: repository structure and scripts only
|
||||
- 主要实现与运维目录: `app/`, `api/`, `scripts/`
|
||||
- `package.json` 脚本快照: No package.json scripts were detected.
|
||||
|
||||
## 需要继续归并的现有文档
|
||||
|
||||
- 尚未发现直接对应的历史文档，本页目前就是该类别的规范起点。
|
||||
|
||||
## 本页下一步应补充的内容
|
||||
|
||||
- 先描述当前已落地实现，再补充未来规划，避免只写愿景不写现状。
|
||||
- 术语需要与仓库根 README、构建清单和实际目录保持一致。
|
||||
- 将上方列出的历史 runbook、spec、子系统说明逐步链接并归并到本页。
|
||||
- 优先提供面向流程的示例，并确保截图或终端片段与最新 UI/CLI 行为一致。
|
||||
24
docs/zh/vibe-coding-reference.md
Normal file
24
docs/zh/vibe-coding-reference.md
Normal file
@ -0,0 +1,24 @@
|
||||
# Vibe Coding 参考
|
||||
|
||||
该仓库更偏向基础设施编排与可观测体系组合，而不是单一应用二进制。
|
||||
|
||||
本页用于统一 AI 辅助开发提示词、仓库边界、安全编辑规则与文档同步要求。
|
||||
|
||||
## 与当前代码对齐的说明
|
||||
|
||||
- 文档目标仓库: `observability.svc.plus`
|
||||
- 仓库类型: `infra-observability`
|
||||
- 构建与运行依据: repository structure and scripts only
|
||||
- 主要实现与运维目录: `app/`, `api/`, `scripts/`
|
||||
- `package.json` 脚本快照: No package.json scripts were detected.
|
||||
|
||||
## 需要继续归并的现有文档
|
||||
|
||||
- 尚未发现直接对应的历史文档，本页目前就是该类别的规范起点。
|
||||
|
||||
## 本页下一步应补充的内容
|
||||
|
||||
- 先描述当前已落地实现，再补充未来规划，避免只写愿景不写现状。
|
||||
- 术语需要与仓库根 README、构建清单和实际目录保持一致。
|
||||
- 将上方列出的历史 runbook、spec、子系统说明逐步链接并归并到本页。
|
||||
- 当项目新增子系统、受保护目录或强制验证步骤时，同步更新提示模板与仓库规则。
|
||||
5893
files/grafana/01-iaas-compute/node.json
Normal file
5893
files/grafana/01-iaas-compute/node.json
Normal file
File diff suppressed because it is too large
Load Diff
2670
files/grafana/11-paas-control-plane/pigsty.json
Normal file
2670
files/grafana/11-paas-control-plane/pigsty.json
Normal file
File diff suppressed because one or more lines are too long
6697
files/grafana/12-paas-cluster/k8s.json
Normal file
6697
files/grafana/12-paas-cluster/k8s.json
Normal file
File diff suppressed because it is too large
Load Diff
1883
files/grafana/24-bu-request/pglog-overview.json
Normal file
1883
files/grafana/24-bu-request/pglog-overview.json
Normal file
File diff suppressed because it is too large
Load Diff
@ -1,28 +1,31 @@
|
||||
# Grafana Dashboards
|
||||
|
||||
This directory contains Grafana dashboard definitions for Pigsty monitoring system.
|
||||
This directory contains Grafana dashboard definitions for the observability stack.
|
||||
|
||||
## Overview
|
||||
|
||||
Pigsty provides **57 built-in dashboards** organized by module:
|
||||
The repository currently provides **61 domain dashboards + 1 homepage dashboard**.
|
||||
Dashboards are organized by platform-engineering resource domains:
|
||||
|
||||
| Directory | Count | Description |
|
||||
|-----------------|-------|-------------------------------------------------------------------------|
|
||||
| [pgsql](pgsql/) | 29 | PostgreSQL cluster, instance, database, and query monitoring |
|
||||
| [infra](infra/) | 11 | Infrastructure components (VictoriaMetrics, Grafana, Nginx, etcd, etc.) |
|
||||
| [node](node/) | 8 | Host-level metrics (CPU, memory, disk, network, HAProxy, VIP) |
|
||||
| [redis](redis/) | 3 | Redis cluster and instance monitoring |
|
||||
| [app](app/) | 2 | Application dashboards (PostgreSQL logs analysis) |
|
||||
| [minio](minio/) | 2 | MinIO S3-compatible storage monitoring |
|
||||
| [mongo](mongo/) | 1 | MongoDB/FerretDB monitoring |
|
||||
| - | 1 | [pigsty.json](pigsty.json) - Main home dashboard |
|
||||
| Folder | Count | Description |
|
||||
|--------|-------|-------------|
|
||||
| [01-iaas-compute](01-iaas-compute/) | 5 | IAAS compute: node overview, cluster, instance, alert, compatibility summary |
|
||||
| [02-iaas-storage](02-iaas-storage/) | 4 | IAAS storage: disk, JuiceFS, MinIO overview and instance |
|
||||
| [03-iaas-network](03-iaas-network/) | 1 | IAAS network: VIP and node-network entry |
|
||||
| [11-paas-control-plane](11-paas-control-plane/) | 10 | PaaS control plane: Pigsty, Grafana, Victoria stack, Alertmanager, etcd, CMDB |
|
||||
| [12-paas-cluster](12-paas-cluster/) | 1 | PaaS cluster: Kubernetes overview |
|
||||
| [13-paas-db](13-paas-db/) | 29 | PaaS DB: PostgreSQL, PGRDS, PGCAT, Mongo/FerretDB |
|
||||
| [14-paas-cache](14-paas-cache/) | 3 | PaaS cache: Redis overview, cluster, instance |
|
||||
| [22-bu-proxy](22-bu-proxy/) | 2 | Business unit proxy: Nginx and HAProxy |
|
||||
| [24-bu-request](24-bu-request/) | 5 | Business unit request: logs, sessions, vector, request-side tooling |
|
||||
| - | 1 | [homepage.json](homepage.json) - Platform engineering entry dashboard |
|
||||
|
||||
|
||||
## Dashboard Catalog
|
||||
|
||||
### Home
|
||||
|
||||
- **[pigsty.json](pigsty.json)** - Pigsty home dashboard with global overview
|
||||
- **[homepage.json](homepage.json)** - Platform engineering entry dashboard with domain summaries and navigation
|
||||
|
||||
### PGSQL Dashboards
|
||||
|
||||
|
||||
@ -1,463 +0,0 @@
|
||||
{
|
||||
"annotations":{"list":[{"builtIn":1,"datasource":{"type":"datasource","uid":"grafana"},"enable":true,"hide":true,"iconColor":"rgba(0, 211, 255, 1)","name":"Annotations & Alerts","target":{"limit":100,"matchAny":false,"tags":[],"type":"dashboard"},"type":"dashboard"}]},
|
||||
"author":"Ruohang Feng (rh@vonng.com)",
|
||||
"description":"PostgreSQL CSVLog Sample Analysis",
|
||||
"editable":true,
|
||||
"fiscalYearStartMonth":0,
|
||||
"graphTooltip":0,
|
||||
"id":null,
|
||||
"license":"https://pigsty.io/docs/about/license/",
|
||||
"links":[
|
||||
{ "asDropdown":false,"icon":"bolt" ,"includeVars":false,"keepTime":false,"tags":[] ,"targetBlank":false,"title":"Auto Zoom","tooltip":"Adjust Time Range to Sample Log Events","type":"link" ,"url":"/d/pglog-overview?from=${ts_begin}&to=${ts_end}" },
|
||||
{ "asDropdown":true ,"icon":"external link","includeVars":true ,"keepTime":true ,"tags":["PGLOG"],"targetBlank":false,"title":"PGLOG" ,"tooltip":"" ,"type":"dashboards","url":"" }
|
||||
],
|
||||
"panels":[
|
||||
{
|
||||
"datasource":{"type":"postgres","uid":"ds-meta"},
|
||||
"fieldConfig":{
|
||||
"defaults":{
|
||||
"color":{"mode":"palette-classic"},
|
||||
"custom":{
|
||||
"axisBorderShow":false,
|
||||
"axisCenteredZero":false,
|
||||
"axisColorMode":"text",
|
||||
"axisLabel":"",
|
||||
"axisPlacement":"auto",
|
||||
"barAlignment":0,
|
||||
"barWidthFactor":0.6,
|
||||
"drawStyle":"bars",
|
||||
"fillOpacity":100,
|
||||
"gradientMode":"none",
|
||||
"hideFrom":{"legend":false,"tooltip":false,"viz":false},
|
||||
"insertNulls":false,
|
||||
"lineInterpolation":"linear",
|
||||
"lineWidth":1,
|
||||
"pointSize":5,
|
||||
"scaleDistribution":{"type":"linear"},
|
||||
"showPoints":"never",
|
||||
"spanNulls":true,
|
||||
"stacking":{"group":"A","mode":"normal"},
|
||||
"thresholdsStyle":{"mode":"off"}
|
||||
},
|
||||
"mappings":[],
|
||||
"thresholds":{"mode":"absolute","steps":[{"color":"#346f36cc"},{"color":"red","value":80}]},
|
||||
"unit":"none"
|
||||
},
|
||||
"overrides":[{"matcher":{"id":"byName","options":"ERROR"},"properties":[{"id":"color","value":{"fixedColor":"#cc4637d9","mode":"fixed"}}]},{"matcher":{"id":"byName","options":"FATAL"},"properties":[{"id":"color","value":{"fixedColor":"#b783af","mode":"fixed"}}]},{"matcher":{"id":"byName","options":"LOG"},"properties":[{"id":"color","value":{"fixedColor":"#346f36cc","mode":"fixed"}}]}]
|
||||
},
|
||||
"gridPos":{"h":5,"w":24,"x":0,"y":0},
|
||||
"id":17,
|
||||
"interval":"1m",
|
||||
"options":{"legend":{"calcs":["sum"],"displayMode":"list","placement":"bottom","showLegend":false},"tooltip":{"hideZeros":false,"mode":"multi","sort":"none"}},
|
||||
"pluginVersion":"12.0.0",
|
||||
"targets":[{"format":"time_series","group":[],"metricColumn":"none","rawQuery":true,"rawSql":"SELECT $__timeGroup(ts, $__interval) AS \"time\", count(*) AS value \nFROM ${log} WHERE $__timeFilter(ts)\nGROUP BY 1 ORDER BY 1;","refId":"A","select":[[{"params":["value"],"type":"column"}]],"timeColumn":"time","where":[{"name":"$__timeFilter","params":[],"type":"macro"}]}],
|
||||
"title":"Log Events",
|
||||
"type":"timeseries"
|
||||
},
|
||||
{
|
||||
"datasource":{"type":"postgres","uid":"ds-meta"},
|
||||
"fieldConfig":{
|
||||
"defaults":{
|
||||
"color":{"mode":"palette-classic"},
|
||||
"custom":{
|
||||
"axisBorderShow":false,
|
||||
"axisCenteredZero":false,
|
||||
"axisColorMode":"text",
|
||||
"axisLabel":"",
|
||||
"axisPlacement":"auto",
|
||||
"barAlignment":0,
|
||||
"barWidthFactor":0.6,
|
||||
"drawStyle":"bars",
|
||||
"fillOpacity":100,
|
||||
"gradientMode":"none",
|
||||
"hideFrom":{"legend":false,"tooltip":false,"viz":false},
|
||||
"insertNulls":false,
|
||||
"lineInterpolation":"linear",
|
||||
"lineWidth":1,
|
||||
"pointSize":5,
|
||||
"scaleDistribution":{"type":"linear"},
|
||||
"showPoints":"never",
|
||||
"spanNulls":true,
|
||||
"stacking":{"group":"A","mode":"normal"},
|
||||
"thresholdsStyle":{"mode":"off"}
|
||||
},
|
||||
"mappings":[],
|
||||
"thresholds":{"mode":"absolute","steps":[{"color":"#346f36cc"},{"color":"red","value":80}]},
|
||||
"unit":"none"
|
||||
},
|
||||
"overrides":[{"matcher":{"id":"byName","options":"ERROR"},"properties":[{"id":"color","value":{"fixedColor":"#cc4637d9","mode":"fixed"}}]},{"matcher":{"id":"byName","options":"FATAL"},"properties":[{"id":"color","value":{"fixedColor":"#b783af","mode":"fixed"}}]},{"matcher":{"id":"byName","options":"LOG"},"properties":[{"id":"color","value":{"fixedColor":"#346f36cc","mode":"fixed"}}]}]
|
||||
},
|
||||
"gridPos":{"h":6,"w":8,"x":0,"y":5},
|
||||
"id":8,
|
||||
"interval":"1s",
|
||||
"options":{"legend":{"calcs":["sum"],"displayMode":"table","placement":"right","showLegend":true},"tooltip":{"hideZeros":false,"mode":"single","sort":"none"}},
|
||||
"pluginVersion":"12.0.0",
|
||||
"targets":[{"format":"time_series","group":[],"metricColumn":"none","rawQuery":true,"rawSql":"SELECT $__timeGroup(ts, $__interval) AS \"time\", \n level AS metric, count(*) AS value \nFROM ${log}\nWHERE $__timeFilter(ts)\nGROUP BY 1, 2\nORDER BY 1, 2;","refId":"A","select":[[{"params":["value"],"type":"column"}]],"timeColumn":"time","where":[{"name":"$__timeFilter","params":[],"type":"macro"}]}],
|
||||
"title":"Log Events by Level",
|
||||
"type":"timeseries"
|
||||
},
|
||||
{
|
||||
"datasource":{"type":"postgres","uid":"ds-meta"},
|
||||
"fieldConfig":{
|
||||
"defaults":{
|
||||
"color":{"mode":"palette-classic"},
|
||||
"custom":{
|
||||
"axisBorderShow":false,
|
||||
"axisCenteredZero":false,
|
||||
"axisColorMode":"text",
|
||||
"axisLabel":"",
|
||||
"axisPlacement":"auto",
|
||||
"barAlignment":0,
|
||||
"barWidthFactor":0.6,
|
||||
"drawStyle":"bars",
|
||||
"fillOpacity":100,
|
||||
"gradientMode":"none",
|
||||
"hideFrom":{"legend":false,"tooltip":false,"viz":false},
|
||||
"insertNulls":false,
|
||||
"lineInterpolation":"linear",
|
||||
"lineWidth":1,
|
||||
"pointSize":5,
|
||||
"scaleDistribution":{"type":"linear"},
|
||||
"showPoints":"never",
|
||||
"spanNulls":true,
|
||||
"stacking":{"group":"A","mode":"normal"},
|
||||
"thresholdsStyle":{"mode":"off"}
|
||||
},
|
||||
"mappings":[],
|
||||
"thresholds":{"mode":"absolute","steps":[{"color":"#346f36cc"},{"color":"red","value":80}]},
|
||||
"unit":"none"
|
||||
},
|
||||
"overrides":[]
|
||||
},
|
||||
"gridPos":{"h":6,"w":8,"x":8,"y":5},
|
||||
"id":2,
|
||||
"interval":"1s",
|
||||
"options":{"legend":{"calcs":["sum"],"displayMode":"table","placement":"right","showLegend":true},"tooltip":{"hideZeros":false,"mode":"single","sort":"none"}},
|
||||
"pluginVersion":"12.0.0",
|
||||
"targets":[{"format":"time_series","group":[],"metricColumn":"none","rawQuery":true,"rawSql":"SELECT $__timeGroup(ts, $__interval) AS \"time\", \n code AS metric, count(*) AS value \nFROM ${log}\nWHERE $__timeFilter(ts)\nGROUP BY 1, 2\nORDER BY 1, 2;","refId":"A","select":[[{"params":["value"],"type":"column"}]],"timeColumn":"time","where":[{"name":"$__timeFilter","params":[],"type":"macro"}]}],
|
||||
"title":"Log Event by ErrCode",
|
||||
"type":"timeseries"
|
||||
},
|
||||
{
|
||||
"datasource":{"type":"postgres","uid":"ds-meta"},
|
||||
"fieldConfig":{
|
||||
"defaults":{
|
||||
"color":{"mode":"palette-classic"},
|
||||
"custom":{
|
||||
"axisBorderShow":false,
|
||||
"axisCenteredZero":false,
|
||||
"axisColorMode":"text",
|
||||
"axisLabel":"",
|
||||
"axisPlacement":"auto",
|
||||
"barAlignment":0,
|
||||
"barWidthFactor":0.6,
|
||||
"drawStyle":"bars",
|
||||
"fillOpacity":100,
|
||||
"gradientMode":"none",
|
||||
"hideFrom":{"legend":false,"tooltip":false,"viz":false},
|
||||
"insertNulls":false,
|
||||
"lineInterpolation":"linear",
|
||||
"lineWidth":1,
|
||||
"pointSize":5,
|
||||
"scaleDistribution":{"type":"linear"},
|
||||
"showPoints":"never",
|
||||
"spanNulls":true,
|
||||
"stacking":{"group":"A","mode":"normal"},
|
||||
"thresholdsStyle":{"mode":"off"}
|
||||
},
|
||||
"mappings":[],
|
||||
"thresholds":{"mode":"absolute","steps":[{"color":"#346f36cc"},{"color":"red","value":80}]},
|
||||
"unit":"none"
|
||||
},
|
||||
"overrides":[]
|
||||
},
|
||||
"gridPos":{"h":6,"w":8,"x":16,"y":5},
|
||||
"id":9,
|
||||
"interval":"1s",
|
||||
"options":{"legend":{"calcs":["sum"],"displayMode":"table","placement":"right","showLegend":true},"tooltip":{"hideZeros":false,"mode":"single","sort":"none"}},
|
||||
"pluginVersion":"12.0.0",
|
||||
"targets":[
|
||||
{
|
||||
"editorMode":"code",
|
||||
"format":"time_series",
|
||||
"group":[],
|
||||
"metricColumn":"none",
|
||||
"rawQuery":true,
|
||||
"rawSql":"SELECT $__timeGroup(ts, $__interval) AS \"time\", \n cmd_tag AS metric, count(*) AS value \nFROM ${log}\nWHERE $__timeFilter(ts)\nGROUP BY 1, 2\nORDER BY 1, 2;",
|
||||
"refId":"A",
|
||||
"select":[[{"params":["value"],"type":"column"}]],
|
||||
"sql":{"columns":[{"parameters":[],"type":"function"}],"groupBy":[{"property":{"type":"string"},"type":"groupBy"}],"limit":50},
|
||||
"timeColumn":"time",
|
||||
"where":[{"name":"$__timeFilter","params":[],"type":"macro"}]
|
||||
}
|
||||
],
|
||||
"title":"Count by CmdTag",
|
||||
"type":"timeseries"
|
||||
},
|
||||
{
|
||||
"datasource":{"type":"postgres","uid":"ds-meta"},
|
||||
"fieldConfig":{
|
||||
"defaults":{
|
||||
"color":{"mode":"palette-classic"},
|
||||
"custom":{
|
||||
"axisBorderShow":false,
|
||||
"axisCenteredZero":false,
|
||||
"axisColorMode":"text",
|
||||
"axisLabel":"",
|
||||
"axisPlacement":"auto",
|
||||
"barAlignment":0,
|
||||
"barWidthFactor":0.6,
|
||||
"drawStyle":"bars",
|
||||
"fillOpacity":100,
|
||||
"gradientMode":"none",
|
||||
"hideFrom":{"legend":false,"tooltip":false,"viz":false},
|
||||
"insertNulls":false,
|
||||
"lineInterpolation":"linear",
|
||||
"lineWidth":1,
|
||||
"pointSize":5,
|
||||
"scaleDistribution":{"type":"linear"},
|
||||
"showPoints":"never",
|
||||
"spanNulls":true,
|
||||
"stacking":{"group":"A","mode":"normal"},
|
||||
"thresholdsStyle":{"mode":"off"}
|
||||
},
|
||||
"mappings":[],
|
||||
"thresholds":{"mode":"absolute","steps":[{"color":"#346f36cc"},{"color":"red","value":80}]},
|
||||
"unit":"none"
|
||||
},
|
||||
"overrides":[{"matcher":{"id":"byName","options":"<null>"},"properties":[{"id":"color","value":{"fixedColor":"rgb(182, 182, 182)","mode":"fixed"}}]},{"matcher":{"id":"byName","options":"<n/a>"},"properties":[{"id":"color","value":{"fixedColor":"red","mode":"fixed"}}]},{"matcher":{"id":"byName","options":"<system>"},"properties":[{"id":"color","value":{"fixedColor":"dark-red","mode":"fixed"}}]}]
|
||||
},
|
||||
"gridPos":{"h":7,"w":8,"x":0,"y":11},
|
||||
"id":5,
|
||||
"interval":"1s",
|
||||
"options":{"legend":{"calcs":["sum"],"displayMode":"table","placement":"right","showLegend":true},"tooltip":{"hideZeros":false,"mode":"single","sort":"none"}},
|
||||
"pluginVersion":"12.0.0",
|
||||
"targets":[
|
||||
{
|
||||
"editorMode":"code",
|
||||
"format":"time_series",
|
||||
"group":[],
|
||||
"metricColumn":"none",
|
||||
"rawQuery":true,
|
||||
"rawSql":"SELECT $__timeGroup(ts, $__interval) AS \"time\", \n CASE WHEN datname IS NULL OR datname = '' THEN '<system>' ELSE datname END AS metric, count(*) AS value \nFROM ${log}\nWHERE $__timeFilter(ts)\nGROUP BY 1, 2\nORDER BY 1, 2;",
|
||||
"refId":"A",
|
||||
"select":[[{"params":["value"],"type":"column"}]],
|
||||
"sql":{"columns":[{"parameters":[],"type":"function"}],"groupBy":[{"property":{"type":"string"},"type":"groupBy"}],"limit":50},
|
||||
"timeColumn":"time",
|
||||
"where":[{"name":"$__timeFilter","params":[],"type":"macro"}]
|
||||
}
|
||||
],
|
||||
"title":"Log Events by Database",
|
||||
"type":"timeseries"
|
||||
},
|
||||
{
|
||||
"datasource":{"type":"postgres","uid":"ds-meta"},
|
||||
"fieldConfig":{
|
||||
"defaults":{
|
||||
"color":{"mode":"palette-classic"},
|
||||
"custom":{
|
||||
"axisBorderShow":false,
|
||||
"axisCenteredZero":false,
|
||||
"axisColorMode":"text",
|
||||
"axisLabel":"",
|
||||
"axisPlacement":"auto",
|
||||
"barAlignment":0,
|
||||
"barWidthFactor":0.6,
|
||||
"drawStyle":"bars",
|
||||
"fillOpacity":100,
|
||||
"gradientMode":"none",
|
||||
"hideFrom":{"legend":false,"tooltip":false,"viz":false},
|
||||
"insertNulls":false,
|
||||
"lineInterpolation":"linear",
|
||||
"lineWidth":1,
|
||||
"pointSize":5,
|
||||
"scaleDistribution":{"type":"linear"},
|
||||
"showPoints":"never",
|
||||
"spanNulls":true,
|
||||
"stacking":{"group":"A","mode":"normal"},
|
||||
"thresholdsStyle":{"mode":"off"}
|
||||
},
|
||||
"mappings":[],
|
||||
"thresholds":{"mode":"absolute","steps":[{"color":"#346f36cc"},{"color":"red","value":80}]},
|
||||
"unit":"none"
|
||||
},
|
||||
"overrides":[{"matcher":{"id":"byName","options":"<null>"},"properties":[{"id":"color","value":{"fixedColor":"rgb(182, 182, 182)","mode":"fixed"}}]}]
|
||||
},
|
||||
"gridPos":{"h":7,"w":8,"x":8,"y":11},
|
||||
"id":6,
|
||||
"interval":"1s",
|
||||
"options":{"legend":{"calcs":["sum"],"displayMode":"table","placement":"right","showLegend":true},"tooltip":{"hideZeros":false,"mode":"single","sort":"none"}},
|
||||
"pluginVersion":"12.0.0",
|
||||
"targets":[
|
||||
{
|
||||
"format":"time_series",
|
||||
"group":[],
|
||||
"metricColumn":"none",
|
||||
"rawQuery":true,
|
||||
"rawSql":"SELECT $__timeGroup(ts, $__interval) AS \"time\", \n coalesce(username, '<null>') AS metric, count(*) AS value \nFROM ${log}\nWHERE $__timeFilter(ts)\nGROUP BY 1, 2\nORDER BY 1, 2;",
|
||||
"refId":"A",
|
||||
"select":[[{"params":["value"],"type":"column"}]],
|
||||
"timeColumn":"time",
|
||||
"where":[{"name":"$__timeFilter","params":[],"type":"macro"}]
|
||||
}
|
||||
],
|
||||
"title":"Log Events by Username",
|
||||
"type":"timeseries"
|
||||
},
|
||||
{
|
||||
"datasource":{"type":"postgres","uid":"ds-meta"},
|
||||
"fieldConfig":{
|
||||
"defaults":{"color":{"mode":"thresholds"},"custom":{"align":"center","cellOptions":{"type":"auto"},"inspect":false},"mappings":[],"thresholds":{"mode":"absolute","steps":[{"color":"#346f36cc"},{"color":"red","value":80}]},"unit":"none"},
|
||||
"overrides":[{"matcher":{"id":"byName","options":"App"},"properties":[{"id":"custom.width"}]},{"matcher":{"id":"byName","options":"Count"},"properties":[{"id":"custom.width","value":80}]}]
|
||||
},
|
||||
"gridPos":{"h":7,"w":4,"x":16,"y":11},
|
||||
"id":12,
|
||||
"options":{"cellHeight":"sm","footer":{"countRows":false,"fields":"","reducer":["sum"],"show":false},"showHeader":true,"sortBy":[{"desc":true,"displayName":"Count"}]},
|
||||
"pluginVersion":"12.0.0",
|
||||
"targets":[
|
||||
{
|
||||
"editorMode":"code",
|
||||
"format":"table",
|
||||
"group":[],
|
||||
"metricColumn":"none",
|
||||
"rawQuery":true,
|
||||
"rawSql":"SELECT conn, count(*)\nFROM ${log}\nWHERE $__timeFilter(ts)\nGROUP BY conn ORDER BY 1;",
|
||||
"refId":"A",
|
||||
"select":[[{"params":["value"],"type":"column"}]],
|
||||
"sql":{"columns":[{"parameters":[],"type":"function"}],"groupBy":[{"property":{"type":"string"},"type":"groupBy"}],"limit":50},
|
||||
"timeColumn":"time",
|
||||
"where":[{"name":"$__timeFilter","params":[],"type":"macro"}]
|
||||
}
|
||||
],
|
||||
"title":"Connection",
|
||||
"transformations":[{"id":"organize","options":{"excludeByName":{},"indexByName":{},"renameByName":{"app":"App","conn":"Connection","count":"Count"}}}],
|
||||
"type":"table"
|
||||
},
|
||||
{
|
||||
"datasource":{"type":"postgres","uid":"ds-meta"},
|
||||
"fieldConfig":{
|
||||
"defaults":{"color":{"mode":"thresholds"},"custom":{"align":"center","cellOptions":{"type":"auto"},"inspect":false},"mappings":[],"thresholds":{"mode":"absolute","steps":[{"color":"#346f36cc"},{"color":"red","value":80}]},"unit":"none"},
|
||||
"overrides":[{"matcher":{"id":"byName","options":"App"},"properties":[{"id":"custom.width"}]},{"matcher":{"id":"byName","options":"Count"},"properties":[{"id":"custom.width","value":80}]}]
|
||||
},
|
||||
"gridPos":{"h":7,"w":4,"x":20,"y":11},
|
||||
"id":7,
|
||||
"options":{"cellHeight":"sm","footer":{"countRows":false,"fields":"","reducer":["sum"],"show":false},"showHeader":true,"sortBy":[{"desc":true,"displayName":"Count"}]},
|
||||
"pluginVersion":"12.0.0",
|
||||
"targets":[
|
||||
{
|
||||
"editorMode":"code",
|
||||
"format":"table",
|
||||
"group":[],
|
||||
"metricColumn":"none",
|
||||
"rawQuery":true,
|
||||
"rawSql":"SELECT appname AS app, count(*) AS count \nFROM ${log}\nWHERE $__timeFilter(ts) GROUP BY 1 ORDER BY 1;",
|
||||
"refId":"A",
|
||||
"select":[[{"params":["value"],"type":"column"}]],
|
||||
"sql":{"columns":[{"parameters":[],"type":"function"}],"groupBy":[{"property":{"type":"string"},"type":"groupBy"}],"limit":50},
|
||||
"timeColumn":"time",
|
||||
"where":[{"name":"$__timeFilter","params":[],"type":"macro"}]
|
||||
}
|
||||
],
|
||||
"title":"Application",
|
||||
"transformations":[{"id":"organize","options":{"excludeByName":{},"indexByName":{},"renameByName":{"app":"App","count":"Count"}}}],
|
||||
"type":"table"
|
||||
},
|
||||
{
|
||||
"datasource":{"type":"postgres","uid":"ds-meta"},
|
||||
"fieldConfig":{
|
||||
"defaults":{"color":{"mode":"thresholds"},"custom":{"align":"center","cellOptions":{"type":"auto"},"filterable":false,"inspect":false},"mappings":[],"thresholds":{"mode":"absolute","steps":[{"color":"#346f36cc"}]}},
|
||||
"overrides":[
|
||||
{"matcher":{"id":"byName","options":"Time"},"properties":[{"id":"custom.width","value":190}]},{"matcher":{"id":"byName","options":"Conn"},"properties":[{"id":"custom.width","value":140},{"id":"custom.filterable","value":true}]},{"matcher":{"id":"byName","options":"PID"},"properties":[{"id":"custom.width","value":75},{"id":"custom.filterable","value":true}]},
|
||||
{"matcher":{"id":"byName","options":"Line"},"properties":[{"id":"custom.width","value":30}]},{"matcher":{"id":"byName","options":"Session"},"properties":[{"id":"custom.width","value":150},{"id":"links","value":[{"title":"PGLOG Session for ${__data.fields.Session}","url":"/d/pglog-session?var-sid=${__data.fields.Session}&${__url_time_range}"}]},{"id":"custom.filterable","value":true}]},
|
||||
{"matcher":{"id":"byName","options":"CMD"},"properties":[{"id":"custom.width","value":100},{"id":"custom.cellOptions","value":{"type":"json-view"}},{"id":"custom.filterable","value":true}]},{"matcher":{"id":"byName","options":"Session Start"},"properties":[{"id":"custom.width","value":200}]},{"matcher":{"id":"byName","options":"VXID"},"properties":[{"id":"custom.width","value":110}]},
|
||||
{"matcher":{"id":"byName","options":"TXID"},"properties":[{"id":"custom.width","value":70}]},
|
||||
{
|
||||
"matcher":{"id":"byName","options":"Level"},
|
||||
"properties":[
|
||||
{ "id":"custom.width" ,"value":80 },
|
||||
{ "id":"mappings" ,"value":[{"options":{"DEBUG":{"color":"rgba(128, 128, 128, 0.5)","index":7},"ERROR":{"color":"#cc4637d9","index":4},"FATAL":{"color":"#b783af","index":5},"INFO":{"color":"#346f36cc","index":1},"LOG":{"color":"#3e668f","index":0},"NOTICE":{"color":"#5b9cd5","index":2},"PANIC":{"color":"text","index":6},"WARNING":{"color":"#f79f64","index":3}},"type":"value"}] },
|
||||
{ "id":"custom.cellOptions","value":{"mode":"basic","type":"color-background"} },
|
||||
{ "id":"custom.filterable" ,"value":true }
|
||||
]
|
||||
},
|
||||
{"matcher":{"id":"byName","options":"Code"},"properties":[{"id":"custom.width","value":80},{"id":"unit","value":"string"},{"id":"custom.cellOptions","value":{"type":"color-text"}},{"id":"mappings","value":[{"options":{"00000":{"color":"#346f36cc","index":0}},"type":"value"}]},{"id":"thresholds","value":{"mode":"absolute","steps":[{"color":"#cc4637d9"}]}},{"id":"custom.filterable","value":true}]},
|
||||
{"matcher":{"id":"byName","options":"Appname"},"properties":[{"id":"custom.width","value":180},{"id":"custom.filterable","value":true}]},{"matcher":{"id":"byName","options":"Backend"},"properties":[{"id":"custom.width","value":112}]},
|
||||
{"matcher":{"id":"byName","options":"Query"},"properties":[{"id":"custom.minWidth","value":250},{"id":"custom.align","value":"left"},{"id":"custom.inspect","value":true}]},{"matcher":{"id":"byName","options":"Message"},"properties":[{"id":"custom.cellOptions","value":{"type":"auto"}},{"id":"custom.align","value":"left"},{"id":"custom.inspect","value":true},{"id":"custom.minWidth","value":300}]},
|
||||
{"matcher":{"id":"byName","options":"Username"},"properties":[{"id":"custom.width","value":120},{"id":"custom.filterable","value":true}]},{"matcher":{"id":"byName","options":"Database"},"properties":[{"id":"custom.width","value":100},{"id":"custom.filterable","value":true}]},
|
||||
{"matcher":{"id":"byRegexp","options":"/Detail|Hint|IQ|IQP|Ctx|QueryP|Location/"},"properties":[{"id":"custom.width","value":100},{"id":"custom.inspect","value":true}]}
|
||||
]
|
||||
},
|
||||
"gridPos":{"h":15,"w":24,"x":0,"y":18},
|
||||
"id":11,
|
||||
"options":{"cellHeight":"sm","footer":{"countRows":false,"enablePagination":true,"fields":"","reducer":["sum"],"show":false},"showHeader":true,"sortBy":[{"desc":false,"displayName":"Time"}]},
|
||||
"pluginVersion":"12.0.0",
|
||||
"targets":[
|
||||
{
|
||||
"editorMode":"code",
|
||||
"format":"table",
|
||||
"group":[],
|
||||
"metricColumn":"none",
|
||||
"rawQuery":true,
|
||||
"rawSql":"SELECT * FROM ${log} WHERE $__timeFilter(ts) ORDER BY ts DESC LIMIT 1000;",
|
||||
"refId":"A",
|
||||
"select":[[{"params":["value"],"type":"column"}]],
|
||||
"sql":{"columns":[{"parameters":[],"type":"function"}],"groupBy":[{"property":{"type":"string"},"type":"groupBy"}],"limit":50},
|
||||
"timeColumn":"time",
|
||||
"where":[{"name":"$__timeFilter","params":[],"type":"macro"}]
|
||||
}
|
||||
],
|
||||
"title":"",
|
||||
"transformations":[
|
||||
{
|
||||
"id":"organize",
|
||||
"options":{
|
||||
"excludeByName":{},
|
||||
"includeByName":{},
|
||||
"indexByName":{"appname":13,"cmd_tag":7,"code":12,"conn":4,"context":20,"datname":2,"detail":15,"hint":17,"iq":18,"iqp":19,"level":11,"location":22,"msg":14,"pid":3,"q":16,"qp":21,"sid":5,"sln":6,"stime":8,"ts":0,"txid":10,"username":1,"vxid":9},
|
||||
"renameByName":{"appname":"Appname","backend":"Backend","cmd_tag":"CMD","code":"Code","conn":"Conn","context":"Ctx","datname":"Database","detail":"Detail","hint":"Hint","iq":"IQ","iqp":"IQP","level":"Level","location":"Location","msg":"Message","pid":"PID","q":"Query","qp":"QueryP","sid":"Session","sln":"Line","stime":"Session Start","ts":"Time","txid":"TXID","username":"Username","vxid":"VXID"}
|
||||
}
|
||||
}
|
||||
],
|
||||
"type":"table"
|
||||
},
|
||||
{
|
||||
"collapsed":true,
|
||||
"gridPos":{"h":1,"w":24,"x":0,"y":33},
|
||||
"id":21,
|
||||
"panels":[
|
||||
{
|
||||
"fieldConfig":{"defaults":{},"overrides":[]},
|
||||
"gridPos":{"h":4,"w":24,"x":0,"y":34},
|
||||
"id":16,
|
||||
"options":{
|
||||
"code":{"language":"plaintext","showLineNumbers":false,"showMiniMap":false},
|
||||
"content":"Run on meta node as admin user to fetch and pour log into sample table: `$log`\n```\ncatlog | pglog # get local (metadb) today's log\ncatlog node-1 | pglog # get certain node today's log\ncatlog node-1 '2022-03-20' | pglog # get node-1's csvlog @ 2022-03-20\n# pg12, 13, 14 have different csvlog schema, use pglog12, pglog13 instead\n```",
|
||||
"mode":"markdown"
|
||||
},
|
||||
"pluginVersion":"12.0.0",
|
||||
"title":"",
|
||||
"type":"text"
|
||||
}
|
||||
],
|
||||
"title":"How to use PGLOG Analysis ?",
|
||||
"type":"row"
|
||||
}
|
||||
],
|
||||
"preload":false,
|
||||
"refresh":"",
|
||||
"schemaVersion":41,
|
||||
"tags":["APP","PGLOG","Overview"],
|
||||
"templating":{
|
||||
"list":[
|
||||
{ "description":"Schema qualified csv log table" , "hide":2,"label":"Log Table","skipUrlSync":true, "name":"log" ,"query":"pglog.sample" , "type":"constant" },
|
||||
{ "current":{},"datasource":{"type":"postgres","uid":"ds-meta"},"description":"First log timestamp of this sample","definition":"SELECT min(ts) - '5min'::INTERVAL FROM ${log};","hide":2,"label":"TimeBegin", "includeAll":false,"name":"ts_begin","query":"SELECT min(ts) - '5min'::INTERVAL FROM ${log};","options":[],"type":"query" ,"refresh":1,"regex":"" },
|
||||
{ "current":{},"datasource":{"type":"postgres","uid":"ds-meta"},"description":"Last log timestamp of this sample" ,"definition":"SELECT max(ts) + '5min'::INTERVAL FROM ${log};","hide":2,"label":"Time End" , "includeAll":false,"name":"ts_end" ,"query":"SELECT max(ts) + '5min'::INTERVAL FROM ${log};","options":[],"type":"query" ,"refresh":1,"regex":"" }
|
||||
]
|
||||
},
|
||||
"time":{"from":"now-24h","to":"now"},
|
||||
"timepicker":{},
|
||||
"timezone":"browser",
|
||||
"title":"PGLOG Overview",
|
||||
"uid":"pglog-overview",
|
||||
"version":1
|
||||
}
|
||||
@ -10,11 +10,51 @@
|
||||
#==============================================================#
|
||||
import os, sys, json, requests
|
||||
|
||||
|
||||
def env_flag(name, default):
|
||||
value = os.environ.get(name)
|
||||
if value is None:
|
||||
return default
|
||||
return value.lower() in ('1', 'true', 'yes', 'on')
|
||||
|
||||
# grafana access info
|
||||
ENDPOINT = os.environ.get("GRAFANA_ENDPOINT", 'http://i.pigsty/ui')
|
||||
USERNAME = os.environ.get("GRAFANA_USERNAME", 'admin')
|
||||
PASSWORD = os.environ.get("GRAFANA_PASSWORD", 'pigsty')
|
||||
CREATE_FOLDERS = True
|
||||
CREATE_FOLDERS = env_flag('GRAFANA_CREATE_FOLDERS', True)
|
||||
SKIP_SUBFOLDERS = env_flag('GRAFANA_SKIP_SUBFOLDERS', False)
|
||||
|
||||
FOLDER_TITLES = {
|
||||
'01-iaas-compute': 'IAAS / 计算',
|
||||
'02-iaas-storage': 'IAAS / 存储',
|
||||
'03-iaas-network': 'IAAS / 网络',
|
||||
'11-paas-control-plane': 'PaaS / 平台控制面',
|
||||
'12-paas-cluster': 'PaaS / 集群',
|
||||
'13-paas-db': 'PaaS / DB',
|
||||
'14-paas-cache': 'PaaS / 缓存',
|
||||
'15-paas-queue': 'PaaS / 队列',
|
||||
'21-bu-dns': '业务单元 / DNS',
|
||||
'22-bu-proxy': '业务单元 / 代理',
|
||||
'23-bu-gateway': '业务单元 / 网关',
|
||||
'24-bu-request': '业务单元 / 请求',
|
||||
'25-bu-throughput': '业务单元 / 吞吐',
|
||||
}
|
||||
|
||||
FOLDER_TAGS = {
|
||||
'01-iaas-compute': ['IAAS', 'IAAS-COMPUTE'],
|
||||
'02-iaas-storage': ['IAAS', 'IAAS-STORAGE'],
|
||||
'03-iaas-network': ['IAAS', 'IAAS-NETWORK'],
|
||||
'11-paas-control-plane': ['PAAS', 'PAAS-CONTROL-PLANE'],
|
||||
'12-paas-cluster': ['PAAS', 'PAAS-CLUSTER'],
|
||||
'13-paas-db': ['PAAS', 'PAAS-DB'],
|
||||
'14-paas-cache': ['PAAS', 'PAAS-CACHE'],
|
||||
'15-paas-queue': ['PAAS', 'PAAS-QUEUE'],
|
||||
'21-bu-dns': ['BU', 'BU-DNS'],
|
||||
'22-bu-proxy': ['BU', 'BU-PROXY'],
|
||||
'23-bu-gateway': ['BU', 'BU-GATEWAY'],
|
||||
'24-bu-request': ['BU', 'BU-REQUEST'],
|
||||
'25-bu-throughput': ['BU', 'BU-THROUGHPUT'],
|
||||
}
|
||||
|
||||
METADB_PASSWORD = 'DBUser.Viewer'
|
||||
DEFAULT_DATASOURCES = {
|
||||
@ -118,7 +158,7 @@ def add_folder(uid, title=""):
|
||||
if not CREATE_FOLDERS:
|
||||
return
|
||||
if title == "":
|
||||
title = uid.upper()
|
||||
title = resolve_folder_title(uid)
|
||||
post('folders', {"uid": uid, "title": title})
|
||||
return put('folders/%s' % uid, {"title": title, "overwrite": True})
|
||||
|
||||
@ -212,6 +252,30 @@ def load_dashboard(path, substitute=False):
|
||||
else:
|
||||
return json.load(open(path))
|
||||
|
||||
|
||||
def resolve_folder_title(uid):
|
||||
return FOLDER_TITLES.get(uid, uid.upper())
|
||||
|
||||
|
||||
def enrich_dashboard(dashboard, folder=None):
|
||||
if not folder:
|
||||
return dashboard
|
||||
extra_tags = FOLDER_TAGS.get(folder, [])
|
||||
if not extra_tags:
|
||||
return dashboard
|
||||
existing_tags = dashboard.get("tags", [])
|
||||
if not isinstance(existing_tags, list):
|
||||
existing_tags = []
|
||||
merged_tags = []
|
||||
seen = set()
|
||||
for tag in existing_tags + extra_tags:
|
||||
if not tag or tag in seen:
|
||||
continue
|
||||
seen.add(tag)
|
||||
merged_tags.append(tag)
|
||||
dashboard["tags"] = merged_tags
|
||||
return dashboard
|
||||
|
||||
# json serializer: use compact_json if available, fallback to standard json
|
||||
try:
|
||||
from compact_json import Formatter
|
||||
@ -271,24 +335,25 @@ def init_all(dashboard_dir):
|
||||
if os.path.isfile(abs_path) and f.endswith('.json') and not f.startswith('.'):
|
||||
print("init dashboard : %s" % f)
|
||||
add_dashboard(load_dashboard(abs_path, True))
|
||||
if os.path.isdir(abs_path):
|
||||
if os.path.isdir(abs_path) and not SKIP_SUBFOLDERS:
|
||||
folders.append((f, abs_path)) # folder name, abs path
|
||||
|
||||
home_uid = "pigsty"
|
||||
star_dashboard_by_uid(home_uid) # home dashboards will be loaded above if exists
|
||||
update_org_preference(home_uid, "light")
|
||||
update_user_preference(home_uid, "light")
|
||||
home_uid = "home"
|
||||
if home_uid:
|
||||
star_dashboard_by_uid(home_uid) # home dashboards will be loaded above if exists
|
||||
update_org_preference(home_uid, "light")
|
||||
update_user_preference(home_uid, "light")
|
||||
|
||||
# load other second-layer dashboards
|
||||
for folder_name, folder_path in folders:
|
||||
print("init folder %s" % folder_name)
|
||||
add_folder(folder_name, folder_name.upper())
|
||||
add_folder(folder_name, resolve_folder_title(folder_name))
|
||||
|
||||
for f in os.listdir(folder_path):
|
||||
abs_path = os.path.join(dashboard_dir, folder_name, f)
|
||||
if os.path.isfile(abs_path) and f.endswith('.json') and not f.startswith('.'):
|
||||
print("init dashboard: %s / %s" % (folder_name, f))
|
||||
add_dashboard(load_dashboard(abs_path, True), folder_name)
|
||||
add_dashboard(enrich_dashboard(load_dashboard(abs_path, True), folder_name), folder_name)
|
||||
|
||||
|
||||
def load_all(dashboard_dir):
|
||||
@ -299,18 +364,18 @@ def load_all(dashboard_dir):
|
||||
if os.path.isfile(abs_path) and f.endswith('.json') and not f.startswith('.'):
|
||||
print("load dashboard : %s" % f)
|
||||
add_dashboard(load_dashboard(abs_path))
|
||||
if os.path.isdir(abs_path):
|
||||
if os.path.isdir(abs_path) and not SKIP_SUBFOLDERS:
|
||||
folders.append((f, abs_path)) # folder name, abs path
|
||||
|
||||
for folder_name, folder_path in folders:
|
||||
print("add folder %s" % folder_name)
|
||||
add_folder(folder_name, folder_name.upper())
|
||||
add_folder(folder_name, resolve_folder_title(folder_name))
|
||||
|
||||
for f in os.listdir(folder_path):
|
||||
abs_path = os.path.join(dashboard_dir, folder_name, f)
|
||||
if os.path.isfile(abs_path) and f.endswith('.json') and not f.startswith('.'):
|
||||
print("load dashboard: %s / %s" % (folder_name, f))
|
||||
add_dashboard(load_dashboard(abs_path), folder_name)
|
||||
add_dashboard(enrich_dashboard(load_dashboard(abs_path), folder_name), folder_name)
|
||||
|
||||
|
||||
def dump_all(dashboard_dir):
|
||||
|
||||
2400
files/grafana/homepage.json
Normal file
2400
files/grafana/homepage.json
Normal file
File diff suppressed because it is too large
Load Diff
File diff suppressed because one or more lines are too long
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in New Issue
Block a user