From 8de8380cc6c51a98eb5dc3ccb808dd5e06b8fb93 Mon Sep 17 00:00:00 2001 From: Wu Sheng Date: Sat, 29 Aug 2026 22:57:05 +0800 Subject: [PATCH 1/4] Upgrade to SkyWalking 11.0.0, Horizon UI 1.0.0 and BanyanDB 0.11.0 Chart 5.0.0. OAP 11 accepts BanyanDB server API 0.11 only, by string equality with no lenient fallback, so the three versions move together. ## The admin host OAP 11 introduces admin-server on 17128 and serves /status/* and /debugging/* there exclusively, so oap.ports.admin is required. It is genuinely new: there is no v10.5 release, admin-server does not exist in v10.4.0, and in 10.4 port 17128 is the AI-pipeline URI-recognition server. The chart described the port as "OAP 10.5+", pointing operators of the last v10 release at the wrong service. The UI ConfigMap compounded it by emitting adminUrl with `default 17128`, so setting oap.ports.admin to null removed the Service port but still told Horizon to talk to 17128. adminUrl is now emitted only when the port is exposed. Running against OAP 10.4.0 -- the last v10 release -- needs oap.ports.admin=null and ui.config.templates.mode=readonly. ## Horizon configuration is env-var first Horizon 1.0.0's image ships a fully env-tokenized /app/horizon.yaml and the chart was mounting a ConfigMap of literals over it. Horizon expands ${...} over the raw file TEXT before parsing, so a literal makes its HORIZON_* env var silently inert, and an omitted field falls back to a schema default that consults process.env for only ~18 of the image's ~65 vars. The chart was disabling most of the image's configuration surface with nothing to signal it. Of the 12 fields the ConfigMap wrote, two carried anything the image cannot know. ui.config now defaults to {} and the chart writes only what it computes -- the OAP query/admin URLs, zipkinUrl when that port is exposed, publicUrl from the ingress host, and server.port from ui.service.internalPort -- each as a ${HORIZON_*:default} token so env still overrides. ## Documentation Moved into docs/ and published on the website, organised by what an operator is doing rather than by artifact. The website hosts a repo by cloning it and reading docs/menu.yml; this project was listed under its old name skywalking-kubernetes with no repoUrl, so none of it was reachable. Root README goes 421 -> 69 lines and stays a hub because four pages in the main SkyWalking docs link to this repository root. The three parameter tables move to docs/reference/ as their canonical home, ending the copy-drift between root and chart READMEs. docs/changes/changes.md gives release notices a home other than values.yaml comments. ## E2E Every assertion now goes Horizon BFF -> OAP -> storage. The old suite queried OAP's GraphQL with swctl, which tests OAP rather than the chart -- it would pass with a completely broken UI and touched no value the chart computes. Five cells: Horizon 1.0.0 across OAP 11.0.0 and 10.4.0, each on Elasticsearch and BanyanDB (0.11 and 0.10.3 respectively), plus BanyanDB cluster mode. The 10.4 cells are not a reduced smoke test: none of the endpoints 404 on 10.4, they answer 200 and report degradation in the body, so the admin and template assertions are inverted rather than dropped and the data-plane ones are identical. That makes them a real test of the admin=null + readonly path. Three traps found in Horizon's handlers, each of which would let a naive suite pass while proving nothing: the layer is MESH not GENERAL, because the fixture feeds OAP through Envoy ALS; /api/layer/:key/services returns reachable:true with an empty list even when OAP is down; and there is no built-in admin/admin. The five SWCK jobs become two. They were a strict subset of apache/skywalking-swck's own e2e suite, so they could only report bugs this repository cannot fix -- which is what happened when SWCK 0.10.0's generated horizon.yaml turned out to be invalid under Horizon 1.0.0's strict schema. What remains is what only this repository can test: e2e-operator-chart installs chart/operator and has the injector mutate a pod, exercising the chart's own cert.yaml, webhook.yaml, 63-char truncation and RBAC, none of which SWCK's kustomize deployment touches; e2e-adapter-chart checks the APIService. ## Correction Horizon's BFF does not refuse to start without users. It boots, serves the login page, and answers /api/auth/health with 200 -- this chart's readiness probe -- so the pod reports Ready and nobody can log in. Four places promised a CrashLoopBackOff. That is worse than a crash, and the docs now say so. --- .github/workflows/e2e.ci.yaml | 30 +- README.md | 342 ++---------------- chart/adapter/README.md | 39 +- chart/operator/README.md | 37 +- chart/skywalking/Chart.yaml | 2 +- chart/skywalking/README.md | 226 +----------- chart/skywalking/templates/NOTES.txt | 25 +- chart/skywalking/templates/ui-configmap.yaml | 46 ++- chart/skywalking/templates/ui-deployment.yaml | 6 + chart/skywalking/values-my-es.yaml | 4 +- chart/skywalking/values.yaml | 137 ++++--- docs/README.md | 69 ++++ docs/changes/changes.md | 51 +++ docs/contributing/e2e-tests.md | 183 ++++++++++ docs/contributing/release.md | 213 +++++++++++ docs/evaluate/requirements.md | 137 +++++++ docs/evaluate/version-compatibility.md | 125 +++++++ docs/evaluate/what-this-chart-deploys.md | 157 ++++++++ docs/expose/oap-endpoints.md | 158 ++++++++ docs/expose/tls.md | 279 ++++++++++++++ docs/expose/ui-service-and-ingress.md | 127 +++++++ docs/install/chart-sources.md | 157 ++++++++ docs/install/quick-start.md | 123 +++++++ docs/menu.yml | 100 +++++ docs/operate/oap-configuration.md | 206 +++++++++++ docs/operate/oap-init-job.md | 194 ++++++++++ docs/operate/satellite.md | 139 +++++++ docs/operate/scaling.md | 234 ++++++++++++ docs/operate/swck-adapter.md | 142 ++++++++ docs/operate/swck-operator.md | 173 +++++++++ docs/reference/adapter-chart-values.md | 40 ++ docs/reference/operator-chart-values.md | 38 ++ docs/reference/skywalking-chart-values.md | 244 +++++++++++++ docs/storage/banyandb.md | 200 ++++++++++ docs/storage/choose-a-backend.md | 106 ++++++ docs/storage/elasticsearch.md | 234 ++++++++++++ docs/storage/postgresql.md | 151 ++++++++ docs/troubleshooting/install-and-startup.md | 325 +++++++++++++++++ docs/troubleshooting/ui-and-login.md | 277 ++++++++++++++ docs/ui/configure.md | 219 +++++++++++ docs/ui/horizon-ui.md | 118 ++++++ docs/ui/logins.md | 220 +++++++++++ docs/upgrade/upgrading.md | 212 +++++++++++ test/e2e/e2e-adapter-chart.yaml | 54 +++ test/e2e/e2e-banyandb-cluster.yaml | 242 ------------- test/e2e/e2e-banyandb-standalone.yaml | 239 ------------ test/e2e/e2e-elasticsearch.yaml | 238 ------------ test/e2e/e2e-oap10-banyandb.yaml | 143 ++++++++ test/e2e/e2e-oap10-elasticsearch.yaml | 141 ++++++++ test/e2e/e2e-oap11-banyandb-cluster.yaml | 148 ++++++++ test/e2e/e2e-oap11-banyandb-standalone.yaml | 148 ++++++++ test/e2e/e2e-oap11-elasticsearch.yaml | 146 ++++++++ test/e2e/e2e-operator-chart.yaml | 94 +++++ test/e2e/env | 42 ++- ...pendency-services-instance-productpage.yml | 42 --- .../dependency-services-productpage.yml | 85 ----- .../expected/dependency-services-reviews.yml | 66 ---- .../{endpoint.yaml => horizon-admin-live.yml} | 7 +- ...plicas.yaml => horizon-admin-readonly.yml} | 6 +- ...ce-endpoint-reviews.yml => horizon-me.yml} | 7 +- ...ductpage.yml => horizon-mesh-services.yml} | 10 +- test/e2e/expected/horizon-oap-info.yml | 17 + test/e2e/expected/horizon-service-cpm.yml | 18 + test/e2e/expected/horizon-templates-live.yml | 18 + .../expected/horizon-templates-readonly.yml | 18 + test/e2e/expected/horizon-zipkin.yml | 16 + test/e2e/expected/metrics-has-value.yml | 34 -- test/e2e/expected/metrics.yaml | 25 -- test/e2e/expected/service-apdex.yaml | 25 -- test/e2e/expected/service-instance.yml | 40 -- test/e2e/expected/service.yml | 59 --- test/e2e/expected/swck-adapter-apiservice.yml | 16 + test/e2e/expected/swck-agent-configmap.yml | 19 + test/e2e/expected/swck-crds.yml | 20 + test/e2e/expected/swck-demo-service.yaml | 26 -- test/e2e/expected/swck-injected.yml | 16 + .../hpa-metrics.yaml => inject-probe.yaml} | 27 +- test/e2e/kind.yaml | 30 -- test/e2e/script/horizon.sh | 53 +++ test/e2e/setup-e2e-shell/install-swctl.sh | 39 -- test/e2e/swck/deploy-elasticsearch.yaml | 53 --- test/e2e/swck/oap-agent-adapter-hpa.yaml | 127 ------- test/e2e/swck/oap-agent-banyandb.yaml | 118 ------ test/e2e/swck/oap-ui-agent-elasticsearch.yaml | 108 ------ .../swck/oap-ui-agent-oapserverconfig.yaml | 148 -------- test/e2e/swck/oap-ui-agent-satellite.yaml | 121 ------- .../skywalking-components-elasticsearch.yaml | 49 --- .../swck/skywalking-components-satellite.yaml | 67 ---- test/e2e/swck/skywalking-components.yaml | 49 --- 89 files changed, 6703 insertions(+), 2756 deletions(-) create mode 100644 docs/README.md create mode 100644 docs/changes/changes.md create mode 100644 docs/contributing/e2e-tests.md create mode 100644 docs/contributing/release.md create mode 100644 docs/evaluate/requirements.md create mode 100644 docs/evaluate/version-compatibility.md create mode 100644 docs/evaluate/what-this-chart-deploys.md create mode 100644 docs/expose/oap-endpoints.md create mode 100644 docs/expose/tls.md create mode 100644 docs/expose/ui-service-and-ingress.md create mode 100644 docs/install/chart-sources.md create mode 100644 docs/install/quick-start.md create mode 100644 docs/menu.yml create mode 100644 docs/operate/oap-configuration.md create mode 100644 docs/operate/oap-init-job.md create mode 100644 docs/operate/satellite.md create mode 100644 docs/operate/scaling.md create mode 100644 docs/operate/swck-adapter.md create mode 100644 docs/operate/swck-operator.md create mode 100644 docs/reference/adapter-chart-values.md create mode 100644 docs/reference/operator-chart-values.md create mode 100644 docs/reference/skywalking-chart-values.md create mode 100644 docs/storage/banyandb.md create mode 100644 docs/storage/choose-a-backend.md create mode 100644 docs/storage/elasticsearch.md create mode 100644 docs/storage/postgresql.md create mode 100644 docs/troubleshooting/install-and-startup.md create mode 100644 docs/troubleshooting/ui-and-login.md create mode 100644 docs/ui/configure.md create mode 100644 docs/ui/horizon-ui.md create mode 100644 docs/ui/logins.md create mode 100644 docs/upgrade/upgrading.md create mode 100644 test/e2e/e2e-adapter-chart.yaml delete mode 100644 test/e2e/e2e-banyandb-cluster.yaml delete mode 100644 test/e2e/e2e-banyandb-standalone.yaml delete mode 100644 test/e2e/e2e-elasticsearch.yaml create mode 100644 test/e2e/e2e-oap10-banyandb.yaml create mode 100644 test/e2e/e2e-oap10-elasticsearch.yaml create mode 100644 test/e2e/e2e-oap11-banyandb-cluster.yaml create mode 100644 test/e2e/e2e-oap11-banyandb-standalone.yaml create mode 100644 test/e2e/e2e-oap11-elasticsearch.yaml create mode 100644 test/e2e/e2e-operator-chart.yaml delete mode 100644 test/e2e/expected/dependency-services-instance-productpage.yml delete mode 100644 test/e2e/expected/dependency-services-productpage.yml delete mode 100644 test/e2e/expected/dependency-services-reviews.yml rename test/e2e/expected/{endpoint.yaml => horizon-admin-live.yml} (86%) rename test/e2e/expected/{replicas.yaml => horizon-admin-readonly.yml} (93%) rename test/e2e/expected/{service-endpoint-reviews.yml => horizon-me.yml} (86%) rename test/e2e/expected/{service-endpoint-productpage.yml => horizon-mesh-services.yml} (86%) create mode 100644 test/e2e/expected/horizon-oap-info.yml create mode 100644 test/e2e/expected/horizon-service-cpm.yml create mode 100644 test/e2e/expected/horizon-templates-live.yml create mode 100644 test/e2e/expected/horizon-templates-readonly.yml create mode 100644 test/e2e/expected/horizon-zipkin.yml delete mode 100644 test/e2e/expected/metrics-has-value.yml delete mode 100644 test/e2e/expected/metrics.yaml delete mode 100644 test/e2e/expected/service-apdex.yaml delete mode 100644 test/e2e/expected/service-instance.yml delete mode 100644 test/e2e/expected/service.yml create mode 100644 test/e2e/expected/swck-adapter-apiservice.yml create mode 100644 test/e2e/expected/swck-agent-configmap.yml create mode 100644 test/e2e/expected/swck-crds.yml delete mode 100644 test/e2e/expected/swck-demo-service.yaml create mode 100644 test/e2e/expected/swck-injected.yml rename test/e2e/{expected/hpa-metrics.yaml => inject-probe.yaml} (57%) delete mode 100644 test/e2e/kind.yaml create mode 100755 test/e2e/script/horizon.sh delete mode 100644 test/e2e/setup-e2e-shell/install-swctl.sh delete mode 100644 test/e2e/swck/deploy-elasticsearch.yaml delete mode 100644 test/e2e/swck/oap-agent-adapter-hpa.yaml delete mode 100644 test/e2e/swck/oap-agent-banyandb.yaml delete mode 100644 test/e2e/swck/oap-ui-agent-elasticsearch.yaml delete mode 100644 test/e2e/swck/oap-ui-agent-oapserverconfig.yaml delete mode 100644 test/e2e/swck/oap-ui-agent-satellite.yaml delete mode 100644 test/e2e/swck/skywalking-components-elasticsearch.yaml delete mode 100644 test/e2e/swck/skywalking-components-satellite.yaml delete mode 100644 test/e2e/swck/skywalking-components.yaml diff --git a/.github/workflows/e2e.ci.yaml b/.github/workflows/e2e.ci.yaml index f4c8f53..a2dbb23 100644 --- a/.github/workflows/e2e.ci.yaml +++ b/.github/workflows/e2e.ci.yaml @@ -35,22 +35,20 @@ jobs: strategy: matrix: test: - - name: Run Skywalking E2E Test (Elasticsearch as database) - config: test/e2e/e2e-elasticsearch.yaml - - name: Run Skywalking E2E Test (standalone BanyanDB as database) - config: test/e2e/e2e-banyandb-standalone.yaml - - name: Run Skywalking E2E Test (BanyanDB cluster as database) - config: test/e2e/e2e-banyandb-cluster.yaml - - name: Run SWCK oap-agent-adapter-hpa Test - config: test/e2e/swck/oap-agent-adapter-hpa.yaml - - name: Run SWCK oap-agent-banyandb Test - config: test/e2e/swck/oap-agent-banyandb.yaml - - name: Run SWCK oap-ui-agent-elasticsearch Test - config: test/e2e/swck/oap-ui-agent-elasticsearch.yaml - - name: Run SWCK oap-ui-agent-oapserverconfig Test - config: test/e2e/swck/oap-ui-agent-oapserverconfig.yaml - - name: Run SWCK oap-ui-agent-satellite Test - config: test/e2e/swck/oap-ui-agent-satellite.yaml + - name: Horizon + OAP 11 + Elasticsearch + config: test/e2e/e2e-oap11-elasticsearch.yaml + - name: Horizon + OAP 11 + BanyanDB (standalone) + config: test/e2e/e2e-oap11-banyandb-standalone.yaml + - name: Horizon + OAP 11 + BanyanDB (cluster) + config: test/e2e/e2e-oap11-banyandb-cluster.yaml + - name: Horizon + OAP 10.4 + Elasticsearch + config: test/e2e/e2e-oap10-elasticsearch.yaml + - name: Horizon + OAP 10.4 + BanyanDB 0.10 + config: test/e2e/e2e-oap10-banyandb.yaml + - name: Operator chart (agent injection) + config: test/e2e/e2e-operator-chart.yaml + - name: Adapter chart (APIService) + config: test/e2e/e2e-adapter-chart.yaml name: ${{ matrix.test.name }} steps: - uses: actions/checkout@v2 diff --git a/README.md b/README.md index 6698a29..4d356d2 100644 --- a/README.md +++ b/README.md @@ -9,333 +9,53 @@ Apache SkyWalking Kubernetes Helm SkyWalking Kubernetes Helm repository provides ways to install and configure SkyWalking in a Kubernetes cluster. The scripts are written in Helm 3. -# Chart Detailed Configuration +# Documentation -Chart detailed configuration can be found at [Chart Readme](./chart/skywalking/README.md) +**Full documentation: [skywalking.apache.org/docs/skywalking-helm/next/readme/](https://skywalking.apache.org/docs/skywalking-helm/next/readme/)** +— or read it in this repository under [`docs/`](docs/README.md). -There are required values that you must set explicitly when deploying SkyWalking. +| | | +|---|---| +| Install | [Quick Start](docs/install/quick-start.md) · [Where to Get the Chart](docs/install/chart-sources.md) | +| Storage | [Pick a Backend](docs/storage/choose-a-backend.md) · [Elasticsearch](docs/storage/elasticsearch.md) · [BanyanDB](docs/storage/banyandb.md) · [PostgreSQL](docs/storage/postgresql.md) | +| Web UI | [Horizon UI](docs/ui/horizon-ui.md) · [Set Up Logins](docs/ui/logins.md) · [Configure Horizon](docs/ui/configure.md) | +| Operate | [OAP Init Job](docs/operate/oap-init-job.md) · [Configure OAP](docs/operate/oap-configuration.md) · [Scaling](docs/operate/scaling.md) · [Satellite](docs/operate/satellite.md) | +| Upgrade | [Upgrade](docs/upgrade/upgrading.md) · [Version Compatibility](docs/evaluate/version-compatibility.md) | +| Trouble | [Install and Startup](docs/troubleshooting/install-and-startup.md) · [UI and Login](docs/troubleshooting/ui-and-login.md) | +| Values | [skywalking](docs/reference/skywalking-chart-values.md) · [adapter](docs/reference/adapter-chart-values.md) · [operator](docs/reference/operator-chart-values.md) | -| name | description | example | -| ---- | ----------- | ------- | -| `oap.image.tag` | the OAP docker image tag | `10.4.0` | -| `oap.storageType` | the storage type of the OAP | `elasticsearch`, `postgresql`, `banyandb`, etc. | -| `ui.image.tag` | the Horizon UI docker image tag | `horizon-0.6.0` | - -You can set these required values via command line (e.g. `--set oap.image.tag=10.4.0 --set oap.storageType=elasticsearch`), -or edit them in a separate file(e.g. [`values.yaml`](chart/skywalking/values.yaml), [`values-my-es.yaml`](chart/skywalking/values-my-es.yaml)) -and use `-f ` or `--values=` to set them. - -## Web UI (Horizon UI) - -The web UI shipped by this chart is [Apache SkyWalking Horizon UI](https://github.com/apache/skywalking-horizon-ui), -which replaces the legacy `skywalking-booster-ui`. Compared to booster-ui: - -- The container bundles a Node-based BFF in front of the SPA. It connects to OAP on **two** ports: the GraphQL query port (`12800`, `oap.ports.rest`) and the admin REST port (`17128`, `oap.ports.admin`, available on OAP 10.5+). -- The container exposes **port 8081** (was 8080) and **does not pass-through `/graphql`** to OAP. Callers that previously talked to the UI's GraphQL endpoint (e.g. `swctl --base-url=http:///graphql`) must now talk to the OAP service directly (`http://:12800/graphql`). -- The BFF requires **authentication**. There is no built-in `admin/admin` fallback — `ui.config.auth.local.users` ships **empty**, and the BFF refuses to start until you provide at least one user. -- The full `horizon.yaml` schema (server, oap, auth, rbac, session, audit, debugLog) is owned upstream: - - Canonical commented example: [horizon.example.yaml](https://github.com/apache/skywalking-horizon-ui/blob/main/horizon.example.yaml) (also shipped inside the image at `/app/horizon.example.yaml`) - - Per-section reference: [docs/setup/horizon-yaml.md](https://github.com/apache/skywalking-horizon-ui/blob/main/docs/setup/horizon-yaml.md) - - Anything you set under `ui.config:` in your Helm values is rendered verbatim into `horizon.yaml`, so the upstream docs apply 1:1. -- Release images are published to Docker Hub as `apache/skywalking-ui:horizon-x.y.z`. Pre-release / dev images live at `ghcr.io/apache/skywalking-horizon-ui` (tags: SHA, `vX.Y.Z`, `main`). +# Required values -### Quick demo install (publicly-known credentials) - -For a first-run / trusted-network demo, paste the snippet below into a values file. It seeds two users — **`admin/admin`** (admin role) and **`skywalking/skywalking`** (viewer + maintainer) — using `argon2id` hashes of those plaintext passwords. - -> ⚠ The hashes below are publicly known in this repo. Use only on trusted networks; rotate before exposing the UI externally. - -```yaml -# demo-values.yaml -ui: - config: - auth: - backend: local - local: - users: - - username: admin # password: admin - passwordHash: "$argon2id$v=19$m=65536,t=3,p=4$eemqy1r72oSXR58y8VpRqw$Bn/dULrmJTHEi3263KfgWDEwQmUsqNLi3xwyv/DekHM" - roles: [admin] - - username: skywalking # password: skywalking - passwordHash: "$argon2id$v=19$m=65536,t=3,p=4$Zqj8HhQDqm8d5c2MipHYZw$BsaCnu4bdd4uadIldx3wwYLsdo47Thxb7Lv1MXpWG2Q" - roles: [viewer, maintainer] -``` - -```shell -helm install "${SKYWALKING_RELEASE_NAME}" \ - oci://registry-1.docker.io/apache/skywalking-helm \ - --version "${SKYWALKING_RELEASE_VERSION}" \ - -n "${SKYWALKING_RELEASE_NAMESPACE}" \ - --set oap.image.tag= \ - --set oap.storageType=elasticsearch \ - --set ui.image.tag=horizon- \ - -f demo-values.yaml -``` +Three values have no default and must be set explicitly on every install. -Then port-forward and log in as `admin/admin`: - -```shell -kubectl port-forward -n "${SKYWALKING_RELEASE_NAMESPACE}" \ - svc/${SKYWALKING_RELEASE_NAME}-skywalking-helm-ui 8080:80 -open http://127.0.0.1:8080 -``` - -### Production: hash via Kubernetes Secret - -For anything beyond a demo, swap the publicly-known hash for one you generated yourself and feed it through a Secret + `${VAR}` interpolation: - -```shell -HASH=$(cd skywalking-horizon-ui && pnpm --filter bff cli:hash 'your-strong-password' | tail -1) - -kubectl create secret generic horizon-admin \ - -n "${SKYWALKING_RELEASE_NAMESPACE}" \ - --from-literal=HORIZON_ADMIN_HASH="$HASH" - -cat > my-values.yaml <<'EOF' -ui: - envFromSecret: horizon-admin - config: - auth: - local: - users: - - username: admin - passwordHash: "${HORIZON_ADMIN_HASH}" - roles: [admin] -EOF - -helm install "${SKYWALKING_RELEASE_NAME}" \ - oci://registry-1.docker.io/apache/skywalking-helm \ - --version "${SKYWALKING_RELEASE_VERSION}" \ - -n "${SKYWALKING_RELEASE_NAMESPACE}" \ - --set oap.image.tag= \ - --set oap.storageType=elasticsearch \ - --set ui.image.tag=horizon- \ - -f my-values.yaml -``` +| name | description | example | +| ---- | ----------- | ------- | +| `oap.image.tag` | the OAP docker image tag | `11.0.0` | +| `oap.storageType` | the storage type of the OAP | `elasticsearch`, `postgresql`, `banyandb` | +| `ui.image.tag` | the Horizon UI docker image tag | `horizon-1.0.0` | -Full `horizon.yaml` reference: https://github.com/apache/skywalking-horizon-ui/blob/main/docs/setup/horizon-yaml.md +Set them on the command line, or put them in a values file and pass `-f my-values.yaml`. # Install -Let's set some variables for convenient use later. - ```shell -export SKYWALKING_RELEASE_VERSION=4.9.0 # change the release version according to your need -export SKYWALKING_RELEASE_NAME=skywalking # change the release name according to your scenario -export SKYWALKING_RELEASE_NAMESPACE=default # change the namespace to where you want to install SkyWalking -``` - -## Install released version using Docker Helm repository (>= 4.3.0) - -```shell -helm install "${SKYWALKING_RELEASE_NAME}" \ - oci://registry-1.docker.io/apache/skywalking-helm \ - --version "${SKYWALKING_RELEASE_VERSION}" \ - -n "${SKYWALKING_RELEASE_NAMESPACE}" \ - --set oap.image.tag=10.4.0 \ - --set oap.storageType=elasticsearch \ - --set ui.image.tag=horizon-0.6.0 -``` - -To use BanyanDB as storage solution, you can try - -```shell -helm install "${SKYWALKING_RELEASE_NAME}" \ - oci://registry-1.docker.io/apache/skywalking-helm \ - --version "${SKYWALKING_RELEASE_VERSION}" \ - -n "${SKYWALKING_RELEASE_NAMESPACE}" \ - --set oap.image.tag=10.4.0 \ +helm install skywalking oci://registry-1.docker.io/apache/skywalking-helm \ + --version 5.0.0 -n skywalking --create-namespace \ + --set oap.image.tag=11.0.0 \ --set oap.storageType=banyandb \ - --set ui.image.tag=horizon-0.6.0 \ + --set ui.image.tag=horizon-1.0.0 \ --set elasticsearch.enabled=false \ --set banyandb.enabled=true \ - --set banyandb.image.tag=0.10.1 -``` - -BanyanDB can be configured through various parameters. A comprehensive list of these parameters can be found in the configuration section of [BanyanDB Helm](https://github.com/apache/skywalking-banyandb-helm?tab=readme-ov-file#configuration) repository. These parameters allow you to customize aspects such as replication, resource allocation, persistence, and more to suit your specific deployment needs. Remember to prepend 'banyandb.' to all parameter names when applying the settings. For example, `banyandb.image.tag` can be used to specify the version of BanyanDB. - - -## Install released version using Apache Jfrog Helm repository (<= 4.3.0) - -```shell -export REPO=skywalking -helm repo add ${REPO} https://apache.jfrog.io/artifactory/skywalking-helm -``` - -## Install development version of SkyWalking using master branch - -This is needed **only** when you want to install SkyWalking from master branch. - -```shell script -export REPO=chart -git clone https://github.com/apache/skywalking-helm -cd skywalking-helm -helm repo add elastic https://helm.elastic.co -helm dep up ${REPO}/skywalking -``` - -## Install development version of SWCK Adapter using master branch - -This is needed **only** when you want to install [SWCK Adapter](https://github.com/apache/skywalking-swck/tree/master/adapter) from master branch. - -SWCK Adapter chart detailed configuration can be found at [Adapter Chart Readme](./chart/adapter/README.md). - -You can install the Adapter with the default configuration as follows. - -```shell script -export REPO=chart -git clone https://github.com/apache/skywalking-helm -cd skywalking-helm -helm -n skywalking-custom-metrics-system install adapter ${REPO}/adapter --create-namespace -``` - -## Install development version of SWCK Operator using master branch - -This is needed **only** when you want to install [SWCK Operator](https://github.com/apache/skywalking-swck/tree/master/operator) from master branch. - -Before installing Operator, you have to install [cert-manager](https://cert-manager.io/) at first. - -```shell script -kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.9.1/cert-manager.yaml -``` - -SWCK Operator chart detailed configuration can be found at [Operator Chart Readme](./chart/operator/README.md). - -You can install the Operator with the default configuration as follows. - -```shell script -export REPO=chart -git clone https://github.com/apache/skywalking-helm -cd skywalking-helm -helm -n skywalking-swck-system install operator ${REPO}/operator + --set banyandb.image.tag=0.11.0 ``` -## Install a specific version of SkyWalking - -In theory, you can deploy all versions of SkyWalking that are >= 6.0.0-GA, by specifying the desired `oap.image.tag`/`ui.image.tag`. - -Please note that some configurations that are added in the later versions of SkyWalking may not work in earlier versions, and thus if you -specify those configurations, they may take no effect. - -here are some examples. - -- Deploy SkyWalking 10.4.0 - -```shell script -helm install "${SKYWALKING_RELEASE_NAME}" ${REPO}/skywalking -n "${SKYWALKING_RELEASE_NAMESPACE}" \ - --set oap.image.tag=10.4.0 \ - --set oap.storageType=elasticsearch \ - --set ui.image.tag=horizon-0.6.0 \ - --set eck-operator.installCRDs=false -``` +The default storage backend is Elasticsearch, which needs its CRDs installed first — see +[Quick Start](docs/install/quick-start.md) for that path and for the other chart sources +(development snapshots, building from source). -Elasticsearch is deployed via [ECK (Elastic Cloud on Kubernetes)](https://github.com/elastic/cloud-on-k8s). -When `elasticsearch.enabled=true` (the default), the chart deploys both the ECK operator and an Elasticsearch 8.18.8 cluster. -Because Elasticsearch CRDs must exist before the chart can be installed, you need to install them first: - -```shell -helm dep up chart/skywalking -tar xzf chart/skywalking/charts/eck-operator-3.3.1.tgz -C /tmp eck-operator/charts/eck-operator-crds -helm install eck-crds /tmp/eck-operator/charts/eck-operator-crds -n "${SKYWALKING_RELEASE_NAMESPACE}" --create-namespace -``` - -Then install the chart with `--set eck-operator.installCRDs=false` to avoid duplicating the CRDs. - -To use an existing external Elasticsearch instead, disable the embedded deployment (no CRD pre-install needed): - -```yaml -elasticsearch: - enabled: false - config: - host: elasticsearch-es-http - port: - http: 9200 - user: "xxx" # [optional] - password: "xxx" # [optional] -``` - -The same goes for PostgreSQL and BanyanDB. - -## Install development version using ghcr.io Helm repository - -If you are willing to help testing the latest codes that are not released yet, we provided a snapshot -Helm repository on ghcr.io for convenient use, replace the full commit hash in the version option to -deploy the revision that you want to test. - -```shell -helm -n istio-system install skywalking \ - oci://ghcr.io/apache/skywalking-helm/skywalking-helm \ - --version "0.0.0-b670c41d94a82ddefcf466d54bab5c492d88d772" \ - -n "${SKYWALKING_RELEASE_NAMESPACE}" \ - --set oap.image.tag=10.4.0 \ - --set oap.storageType=elasticsearch \ - --set ui.image.tag=horizon-0.6.0 -``` - -## Install development version using source codes - -This is needed **only** when you want to install source codes. - -```shell script -helm install "${SKYWALKING_RELEASE_NAME}" ${REPO}/skywalking -n "${SKYWALKING_RELEASE_NAMESPACE}" -``` - -## Install a specific version of SkyWalking with an existing database - -If you want to use an existing Elasticsearch cluster as storage solution, modify the connection information in file [`values-my-es.yaml`](chart/skywalking/values-my-es.yaml). - -```shell script -helm install "${SKYWALKING_RELEASE_NAME}" ${REPO}/skywalking -n "${SKYWALKING_RELEASE_NAMESPACE}" \ - -f ./skywalking/values-my-es.yaml -``` - -## Install SkyWalking with Satellite - -Enable the satellite as gateway, and set the satellite image tag. - -```shell script -helm install "${SKYWALKING_RELEASE_NAME}" ${REPO}/skywalking -n "${SKYWALKING_RELEASE_NAMESPACE}" \ - --set satellite.enabled=true \ - --set satellite.image.tag=v0.4.0 -``` - -After satellite have been installed, you should replace the `oap` address to the `satellite` address, the address from agent or `istio`, such as `skywalking-satellite.istio-system:11800`. - -## Customization - -- Override configuration files - -You can override the configuration files for OAP or Satellite by adding configuration section `oap.config` and `satellite.config`, -check [the examples](chart/skywalking/values.yaml), search keyword `config: {}`. - -- Pass environment variables to OAP - -The SkyWalking OAP exposes many configurations that can be specified by environment variables, as listed in [the main repo](https://github.com/apache/skywalking/blob/master/docs/en/setup/backend/configuration-vocabulary.md). -You can set those environment variables by `--set oap.env.=`, such as `--set oap.env.SW_ENVOY_METRIC_ALS_HTTP_ANALYSIS=k8s-mesh`. - -> The environment variables take priority over the overrode configuration files. - -## OAP init job - -The OAP storage schema (Elasticsearch indices / SQL tables / BanyanDB groups) is created by a -one-shot `*-oap-init-*` Job that runs OAP in `-Dmode=init`. The main OAP Deployment runs in -`-Dmode=no-init` and blocks (its `12800` port stays closed, so it is not Ready) until that schema -exists. The init Job is a **normal release resource** that runs in the main install/upgrade phase, -so `helm upgrade --install --wait` works: the Job creates the schema while OAP waits for it. To get -Helm to surface init-Job failures directly (instead of only seeing OAP fail to become Ready), add -`--wait-for-jobs` alongside `--wait`. - -The Job name carries a hash of the chart values, so any `helm upgrade` that changes a value -re-creates the Job and re-runs init automatically (Helm prunes the previous one). - -To **force a rerun** without changing any value — delete the Job and re-run `helm upgrade`; Helm -recreates the (now missing) Job and init runs again: - -```shell -kubectl delete job -n "${SKYWALKING_RELEASE_NAMESPACE}" -l release=$SKYWALKING_RELEASE_NAME -helm upgrade "$SKYWALKING_RELEASE_NAME" -n "${SKYWALKING_RELEASE_NAMESPACE}" --reuse-values -``` +**A fresh install has no login.** Horizon UI ships no default credentials and does not fail closed: +the pod reports Ready and nobody can sign in until you configure users. See +[Set Up Logins](docs/ui/logins.md). # Contact Us * Submit an [issue](https://github.com/apache/skywalking/issues) diff --git a/chart/adapter/README.md b/chart/adapter/README.md index da8647f..2e47ef3 100644 --- a/chart/adapter/README.md +++ b/chart/adapter/README.md @@ -35,39 +35,10 @@ The command removes all the adapter components associated with the chart and del ## Configuration -The following table lists the configurable parameters of the adapter chart and their default values. - -| Parameter | Description | Default | -|--------------------------------------------------------------|--------------------------------------------------------------------------------------------------|--------------------------------------| -| `fullnameOverride` | Override fullname | `nil` | -| `.namespace` | Namespace of adapter deployment | `skywalking-custom-metrics-system` | -| `.replicas` | The replicas of adapter | `1` | -| `.serviceAccountName` | The service account name of adapter | `skywalking-custom-metrics-apiserver` | -| `.image.repository` | Adapter container image name | `docker.io/apache/skywalking-swck` | -| `.image.pullPolicy` | Adapter container image pull policy | `IfNotPresent` | -| `.image.tag` | Adapter container image tag | `0.10.0` | -| `.service.port` | The port for the adapter service | `6.1.0` | -| `.oap.service.name` | The service name of OAP | `skywalking-system-oap` | -| `.oap.service.namespace` | The service namespace of OAP | `skywalking-system` | -| `.oap.service.port` | The service port of OAP | `12800` | -| `.resources.limits.cpu` | The limits of cpu in the adapter | `100m` | -| `.resources.limits.memory` | The limits of memory in the adapter | `200Mi` | -| `.resources.requests.cpu` | The requests of cpu in the adapter | `100m` | -| `.resources.requests.memory` | The requests of memory in the adapter | `200Mi` | -| `.affinity` | The affinity policy of adapter | `{}` | - -Specify each parameter using the `--set key=value[,key=value]` argument to `helm install`. For example, - -**Note** You could refer to the [helm install](https://helm.sh/docs/helm/helm_install/) for more command information. - -```console -$ helm install myrelease adapter --set fullnameOverride=newadapter -``` - -Alternatively, a YAML file that specifies the values for the above parameters can be provided while installing the chart. For example, +Every value this chart accepts is documented in the values reference: -```console -$ helm install my-release adapter -f values.yaml -``` +- on the website: [adapter Chart Values](https://skywalking.apache.org/docs/skywalking-helm/next/reference/adapter-chart-values/) +- in this repository: [`docs/reference/adapter-chart-values.md`](../../docs/reference/adapter-chart-values.md) -> **Tip**: You can use the default [values.yaml](values.yaml) +Full documentation for the charts in this repository lives at +[skywalking.apache.org/docs/skywalking-helm](https://skywalking.apache.org/docs/skywalking-helm/next/readme/). diff --git a/chart/operator/README.md b/chart/operator/README.md index 0caf090..4948e07 100644 --- a/chart/operator/README.md +++ b/chart/operator/README.md @@ -35,37 +35,10 @@ The command removes all the operator components associated with the chart and de ## Configuration -The following table lists the configurable parameters of the operator chart and their default values. - -| Parameter | Description | Default | -|------------------------------|------------------------------------------------------------------------------------------------------------------------------|--------------------------------------| -| `fullnameOverride` | Override fullname | `nil` | -| `.replicaCount` | The replicas of operator | `1` | -| `.serviceAccountName` | The service account name of operator | `skywalking-swck-controller-manager` | -| `.image.repository` | Operator container image name | `docker.io/apache/skywalking-swck` | -| `.image.pullPolicy` | Operator container image pull policy | `IfNotPresent` | -| `.image.tag` | Operator container image tag | `0.10.0` | -| `.metrics.service.port` | The port for the operator metrics service | `8443` | -| `.webhook.service.port` | The port for the operator web hook service | `9443` | -| `.resources.limits.cpu` | The limits of cpu in the operator | `200m` | -| `.resources.limits.memory` | The limits of memory in the operator | `300Mi` | -| `.resources.requests.cpu` | The requests of cpu in the operator | `200m` | -| `.resources.requests.memory` | The requests of memory in the operator | `300Mi` | -| `.affinity` | The affinity policy of operator | `{}` | -| `cert-manager.enabled` | Whether to install demo cert-manager. DO NOT use this in production, this is for quick start. | `false` | - -Specify each parameter using the `--set key=value[,key=value]` argument to `helm install`. For example, - -**Note** You could refer to the [helm install](https://helm.sh/docs/helm/helm_install/) for more command information. - -```console -$ helm install myrelease operator --set fullnameOverride=newoperator -``` - -Alternatively, a YAML file that specifies the values for the above parameters can be provided while installing the chart. For example, +Every value this chart accepts is documented in the values reference: -```console -$ helm install my-release operator -f values.yaml -``` +- on the website: [operator Chart Values](https://skywalking.apache.org/docs/skywalking-helm/next/reference/operator-chart-values/) +- in this repository: [`docs/reference/operator-chart-values.md`](../../docs/reference/operator-chart-values.md) -> **Tip**: You can use the default [values.yaml](values.yaml) +Full documentation for the charts in this repository lives at +[skywalking.apache.org/docs/skywalking-helm](https://skywalking.apache.org/docs/skywalking-helm/next/readme/). diff --git a/chart/skywalking/Chart.yaml b/chart/skywalking/Chart.yaml index cf8932f..371d5e2 100644 --- a/chart/skywalking/Chart.yaml +++ b/chart/skywalking/Chart.yaml @@ -16,7 +16,7 @@ apiVersion: v2 name: skywalking-helm home: https://skywalking.apache.org -version: 4.9.0 +version: 5.0.0 description: Helm Chart for Apache SkyWalking icon: https://raw.githubusercontent.com/apache/skywalking-kubernetes/master/logo/sw-logo-for-chart.jpg sources: diff --git a/chart/skywalking/README.md b/chart/skywalking/README.md index fd05b8a..d91134e 100644 --- a/chart/skywalking/README.md +++ b/chart/skywalking/README.md @@ -39,226 +39,10 @@ The command removes all the Kubernetes components associated with the chart and ## Configuration -The following table lists the configurable parameters of the Skywalking chart and their default values. +Every value this chart accepts is documented in the values reference: -| Parameter | Description | Default | -|----------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------| -| `nameOverride` | Override name | `nil` | -| `serviceAccounts.oap.create` | Create of the OAP service account | `true` | -| `serviceAccounts.oap.name` | Name of the OAP service account to use custom service account when `serviceAccounts.oap.create` is set to false | `` | -| `imagePullSecrets` | Image pull secrets | `[]` | -| `oap.name` | OAP deployment name | `oap` | -| `oap.dynamicConfig.enabled` | Enable oap dynamic configuration through k8s configmap | `false` | -| `oap.dynamicConfig.period` | Sync period in seconds | `60` | -| `oap.dynamicConfig.config` | Oap dynamic configuration [documentation](https://github.com/apache/skywalking/blob/master/docs/en/setup/backend/dynamic-config.md) | `{}` | -| `oap.image.repository` | OAP container image name | `skywalking.docker.scarf.sh/apache/skywalking-oap-server` | -| `oap.image.tag` | OAP container image tag | `6.1.0` | -| `oap.image.pullPolicy` | OAP container image pull policy | `IfNotPresent` | -| `oap.ports.grpc` | OAP grpc port for tracing or metric | `11800` | -| `oap.ports.rest` | OAP http port for the GraphQL query protocol (used by the UI and by `swctl`) | `12800` | -| `oap.ports.admin` | OAP admin REST port (runtime-rule, DSL debugging, inspect, status). Required by Horizon UI's admin features; available on OAP 10.5+ | `17128` | -| `oap.ports.zipkin-receiver` | OAP http port for Zipkin receiver(not exposed by default) | `9411` | -| `oap.ports.zipkin-query` | OAP http port for querying Zipkin traces and UI(not exposed by default) | `9412` | -| `oap.replicas` | OAP k8s deployment replicas | `2` | -| `oap.service.type` | OAP svc type | `ClusterIP` | -| `oap.service.annotations` | OAP svc annotations | `{}` | -| `oap.javaOpts` | Parameters to be added to `JAVA_OPTS`environment variable for OAP | `-Xms2g -Xmx2g` | -| `oap.antiAffinity` | OAP anti-affinity policy | `soft` | -| `oap.nodeAffinity` | OAP node affinity policy | `{}` | -| `oap.nodeSelector` | OAP labels for master pod assignment | `{}` | -| `oap.tolerations` | OAP tolerations | `[]` | -| `oap.resources` | OAP node resources requests & limits | `{} - cpu limit must be an integer` | -| `oap.startupProbe` | Configuration fields for the [startupProbe](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/). The default budget (`failureThreshold` * `periodSeconds` = 300s) is large enough for OAP to wait in no-init mode while the OAP init Job creates the storage schema. | `tcpSocket.port: 12800`
`failureThreshold: 30`
`periodSeconds: 10` -| `oap.livenessProbe` | Configuration fields for the [livenessProbe](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/) | `tcpSocket.port: 12800`
`initialDelaySeconds: 5`
`periodSeconds: 10` -| `oap.readinessProbe` | Configuration fields for the [readinessProbe](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/) | `tcpSocket.port: 12800`
`initialDelaySeconds: 5`
`periodSeconds: 10` -| `oap.env` | OAP environment variables | `[]` | -| `oap.securityContext` | Allows you to set the [securityContext](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-pod) for the pod | `fsGroup: 1000`
`runAsUser: 1000` | -| `oap.sidecars` | Extra sidecar containers to run in the OAP pod (appended to the pod's `containers` list, rendered through `tpl`) | `[]` | -| `ui.enabled` | Deploy the Horizon UI. Set `false` to skip the UI Deployment, Service, Ingress, ConfigMap, and PVC entirely (useful when an external UI talks to OAP directly) | `true` | -| `ui.name` | Web UI deployment name | `ui` | -| `ui.replicas` | Web UI k8s deployment replicas. Keep at `1` unless your ingress provides sticky sessions — the Horizon BFF holds the session table in memory | `1` | -| `ui.image.repository` | Horizon UI container image. Release images: Docker Hub `apache/skywalking-ui` tagged `horizon-x.y.z`. Dev images: `ghcr.io/apache/skywalking-horizon-ui` | `skywalking.docker.scarf.sh/apache/skywalking-ui` | -| `ui.image.tag` | Horizon UI image tag (required), e.g. `horizon-0.6.0` | `null` | -| `ui.image.pullPolicy` | Web UI container image pull policy | `IfNotPresent` | -| `ui.nodeAffinity` | Web UI node affinity policy | `{}` | -| `ui.nodeSelector` | Web UI labels for pod assignment | `{}` | -| `ui.tolerations` | Web UI tolerations | `[]` | -| `ui.ingress.enabled` | Create Ingress for Web UI | `false` | -| `ui.ingress.annotations` | Associate annotations to the Ingress | `{}` | -| `ui.ingress.path` | Associate path with the Ingress | `/` | -| `ui.ingress.hosts` | Associate hosts with the Ingress | `[]` | -| `ui.ingress.tls` | Associate TLS with the Ingress | `[]` | -| `ui.service.type` | Web UI svc type | `ClusterIP` | -| `ui.service.externalPort` | external port for the service | `80` | -| `ui.service.internalPort` | internal port for the service (Horizon BFF binds 8081) | `8081` | -| `ui.service.externalIPs` | external IP addresses | `nil` | -| `ui.service.loadBalancerIP` | Load Balancer IP address | `nil` | -| `ui.service.annotations` | Kubernetes service annotations | `{}` | -| `ui.service.loadBalancerSourceRanges` | Limit load balancer source IPs to list of CIDRs (where available)) | `[]` | -| `ui.securityContext` | Pod [securityContext](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-pod). The image runs as the non-root `horizon` user; `fsGroup` makes mounted PVCs group-writable for that user | `fsGroup: 101` | -| `ui.livenessProbe` | TCP liveness probe | `tcpSocket.port: 8081` | -| `ui.readinessProbe` | HTTP readiness probe; verifies the BFF is up and the auth backend is healthy. `/api/auth/health` is the only unauthenticated BFF health endpoint | `httpGet.path: /api/auth/health, port: 8081` | -| `ui.persistence.enabled` | Mount a PVC at `/data` for audit log / setup / alarm state / wire debug log. When `false`, state lands in the container's writable layer and is lost on pod restart | `false` | -| `ui.persistence.existingClaim` | Use a pre-created PVC; if empty, the chart creates one | `""` | -| `ui.persistence.storageClass` | Storage class for the chart-managed PVC | `""` | -| `ui.persistence.accessModes` | PVC access modes | `[ReadWriteOnce]` | -| `ui.persistence.size` | PVC size | `1Gi` | -| `ui.config` | `horizon.yaml` content (deep-merged onto chart defaults that point `oap.queryUrl`/`adminUrl`/`zipkinUrl` at the in-cluster OAP; `zipkinUrl` is derived from `oap.ports.zipkin-query` and only resolves to a usable URL when that port is set). See the upstream [`horizon.example.yaml`](https://github.com/apache/skywalking-horizon-ui/blob/main/horizon.example.yaml) and [horizon.yaml reference](https://github.com/apache/skywalking-horizon-ui/blob/main/docs/setup/horizon-yaml.md). `auth.local.users` defaults to empty (BFF refuses to start); see the root README "Web UI" section for a demo snippet and the production Secret pattern | see `values.yaml` | -| `ui.envFromSecret` | Reference a Secret whose keys are exposed as env vars in the BFF container, for use with `${VAR}` interpolation in `ui.config` (e.g. admin password hash) | `""` | -| `ui.extraEnv` | Extra env vars passed to the BFF container | `[]` | -| `oapInit.nodeAffinity` | OAP init job node affinity policy | `{}` | -| `oapInit.nodeSelector` | OAP init job labels for master pod assignment | `{}` | -| `oapInit.tolerations` | OAP init job tolerations | `[]` | -| `oapInit.extraPodLabels` | OAP init job metadata labels | `[]` | -| `oapInit.ttlSecondsAfterFinished` | Seconds after which the finished OAP init Job (and its Pod) is auto-deleted by the Kubernetes TTL-after-finished controller. Empty keeps the Job. Leave empty with GitOps tools (Argo CD/Flux), which would recreate it after deletion. | `""` | -| `satellite.name` | Satellite deployment name | `satellite` | -| `satellite.replicas` | Satellite k8s deployment replicas | `1` | -| `satellite.enabled` | Is enable Satellite | `false` | -| `satellite.image.repository` | Satellite container image name | `skywalking.docker.scarf.sh/apache/skywalking-satellite` | -| `satellite.image.tag` | Satellite container image tag | `v0.4.0` | -| `satellite.image.pullPolicy` | Satellite container image pull policy | `IfNotPresent` | -| `satellite.antiAffinity` | Satellite anti-affinity policy | `soft` | -| `satellite.nodeAffinity` | Satellite node affinity policy | `{}` | -| `satellite.nodeSelector` | Satellite labels for pod assignment | `{}` | -| `satellite.tolerations` | Satellite tolerations | `[]` | -| `satellite.service.type` | Satellite svc type | `ClusterIP` | -| `satellite.ports.grpc` | Satellite grpc port for tracing, metrics, logs, events | `11800` | -| `satellite.ports.prometheus` | Satellite http port for Prometheus monitoring | `1234` | -| `satellite.resources` | Satellite node resources requests & limits | `{} - cpu limit must be an integer` | -| `satellite.podAnnotations` | Configurable [annotations](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/) applied to all Satellite pods | `{}` | -| `satellite.env` | Satellite environment variables | `[]` | -| `satellite.securityContext` | Allows you to set the [securityContext](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-pod) for the pod | `fsGroup: 1000`
`runAsUser: 1000` | +- on the website: [skywalking Chart Values](https://skywalking.apache.org/docs/skywalking-helm/next/reference/skywalking-chart-values/) +- in this repository: [`docs/reference/skywalking-chart-values.md`](../../docs/reference/skywalking-chart-values.md) -### Elasticsearch (ECK) - -Elasticsearch is deployed via [ECK (Elastic Cloud on Kubernetes)](https://github.com/elastic/cloud-on-k8s). -The chart includes the ECK operator and an `eck-elasticsearch` subchart, both controlled by `elasticsearch.enabled`. -Because Elasticsearch CRDs must exist before the ES custom resource can be created, the ECK operator CRDs need to be installed separately before deploying the chart. See the main [README](../../README.md) for installation steps. - -#### Top-level parameters - -| Parameter | Description | Default | -|---|---|---| -| `elasticsearch.enabled` | Deploy the ECK operator and an ECK-managed Elasticsearch cluster | `true` | -| `elasticsearch.version` | Elasticsearch version to deploy | `8.18.8` | -| `elasticsearch.fullnameOverride` | Override the Elasticsearch resource name. The ECK service will be `{name}-es-http` | `""` | -| `elasticsearch.labels` | Labels applied to the Elasticsearch resource | `{}` | -| `elasticsearch.annotations` | Annotations applied to the Elasticsearch resource | `{}` | -| `elasticsearch.http` | HTTP layer settings. TLS is disabled by default for OAP connectivity | `tls.selfSignedCertificate.disabled: true` | -| `elasticsearch.secureSettings` | [Secure settings](https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-es-secure-settings.html) to inject from Kubernetes secrets | `[]` | -| `elasticsearch.updateStrategy` | [Update strategy](https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-update-strategy.html) controlling simultaneous changes | `{}` | -| `elasticsearch.volumeClaimDeletePolicy` | Policy for PVC deletion on scale-down or cluster deletion | `""` | -| `elasticsearch.ingress.enabled` | Enable ingress to expose Elasticsearch externally | `false` | - -#### External Elasticsearch (when `elasticsearch.enabled` is `false`) - -| Parameter | Description | Default | -|---|---|---| -| `elasticsearch.config.host` | Elasticsearch host | `elasticsearch` | -| `elasticsearch.config.port.http` | Elasticsearch HTTP port | `9200` | -| `elasticsearch.config.user` | Elasticsearch user (optional) | `""` | -| `elasticsearch.config.password` | Elasticsearch password (optional) | `""` | - -#### Node sets (`elasticsearch.nodeSets[]`) - -ECK [node sets](https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-node-configuration.html) define the topology of the Elasticsearch cluster. Each entry in the list creates a group of Elasticsearch nodes. - -| Parameter | Description | Default | -|---|---|---| -| `nodeSets[].name` | Name of the node set | `default` | -| `nodeSets[].count` | Number of Elasticsearch nodes in this set | `3` | -| `nodeSets[].config` | Elasticsearch configuration (e.g. `node.store.allow_mmap`, `node.roles`) | `node.store.allow_mmap: false` | -| `nodeSets[].volumeClaimTemplates` | Persistent storage for Elasticsearch data | `[]` (ECK default: EmptyDir) | - -#### Pod template (`elasticsearch.nodeSets[].podTemplate`) - -The pod template follows standard Kubernetes Pod spec nested under `podTemplate.spec`. This controls scheduling, resources, init containers, etc. - -| Parameter | Description | Default | -|---|---|---| -| `podTemplate.metadata.annotations` | Pod annotations (e.g. `iam.amazonaws.com/role`) | `{}` | -| `podTemplate.metadata.labels` | Extra pod labels | `{}` | -| `podTemplate.spec.affinity` | Pod [affinity](https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-advanced-node-scheduling.html) rules | not set | -| `podTemplate.spec.nodeSelector` | Node selector for pod assignment | not set | -| `podTemplate.spec.tolerations` | Pod tolerations | not set | -| `podTemplate.spec.imagePullSecrets` | Image pull secrets | not set | -| `podTemplate.spec.priorityClassName` | Priority class name | not set | -| `podTemplate.spec.terminationGracePeriodSeconds` | Grace period for pod termination | not set | -| `podTemplate.spec.initContainers` | Init containers (e.g. sysctl `vm.max_map_count`) | not set | -| `podTemplate.spec.containers[].resources` | Container resource requests & limits | `requests: 100m cpu, 2Gi mem`
`limits: 2Gi mem` | -| `podTemplate.spec.containers[].env` | Environment variables (e.g. `ES_JAVA_OPTS`) | not set | -| `podTemplate.spec.containers[].securityContext` | Container-level security context | not set (ECK managed) | - -Specify each parameter using the `--set key=value[,key=value]` argument to `helm install`. For example, - -```console -$ helm install myrelease skywalking --set nameOverride=newSkywalking -``` - -Alternatively, a YAML file that specifies the values for the above parameters can be provided while installing the -chart. For example, - -```console -$ helm install my-release skywalking -f values.yaml -``` - -> **Tip**: You can use the default [values.yaml](values.yaml) - -### RBAC Configuration - -Roles and RoleBindings resources will be created automatically for `OAP` . - -> **Tip**: You can refer to the default `oap-role.yaml` file in [templates](templates/) to customize your own. - -### Ingress TLS - -If your cluster allows automatic create/retrieve of TLS certificates ( -e.g. [kube-lego](https://github.com/jetstack/kube-lego)), please refer to the documentation for that mechanism. - -To manually configure TLS, first create/retrieve a key & certificate pair for the address(skywalking ui) you wish to -protect. Then create a TLS secret in the namespace: - -```console -kubectl create secret tls skywalking-tls --cert=path/to/tls.cert --key=path/to/tls.key -``` - -Include the secret's name, along with the desired hostnames, in the skywalking-ui Ingress TLS section of your -custom `values.yaml` file: - -```yaml -ui: - ingress: - ## If true, Skywalking ui server Ingress will be created - ## - enabled: true - - ## Skywalking ui server Ingress hostnames - ## Must be provided if Ingress is enabled - ## - hosts: - - skywalking - - ## Skywalking ui server Ingress TLS configuration - ## Secrets must be manually created in the namespace - ## - tls: - - secretName: skywalking - hosts: - - skywalking -``` - -### Envoy ALS - -Envoy ALS(access log service) provides fully logs about RPC routed, including HTTP and TCP. - -If you want to open envoy ALS, you can do this by modifying values.yaml. default open. - -```yaml -serviceAccounts: - oap: - create: true -``` - -When envoy als ,will give ServiceAccount clusterrole permission. -More envoy als ,please refer -to https://github.com/apache/skywalking/blob/master/docs/en/setup/envoy/als_setting.md#observe-service-mesh-through-als +Full documentation for the charts in this repository lives at +[skywalking.apache.org/docs/skywalking-helm](https://skywalking.apache.org/docs/skywalking-helm/next/readme/). diff --git a/chart/skywalking/templates/NOTES.txt b/chart/skywalking/templates/NOTES.txt index 8d02cad..fbc3037 100644 --- a/chart/skywalking/templates/NOTES.txt +++ b/chart/skywalking/templates/NOTES.txt @@ -53,18 +53,29 @@ Get the UI URL by running these commands: NOTE on Horizon UI auth: -The web UI is Apache SkyWalking Horizon UI. Its BFF refuses to start when -no users are configured — there is no built-in "admin/admin" fallback. +The web UI is Apache SkyWalking Horizon UI. There is no built-in +"admin/admin" fallback, and no login is configured by default. -If your `ui.config.auth.local.users` list is empty, the UI pod will -CrashLoopBackOff. See the README "Web UI" section for: +The BFF does NOT fail closed: with no users it still boots, serves the login +page, and answers the readiness probe — so the pod reports Ready and nobody +can log in. Configure users before you rely on it. See the README "Web UI" +section for: - a copy-pastable demo snippet (publicly-known admin/admin hash) - the production Secret + ${VAR} interpolation pattern -Full horizon.yaml config reference and upstream example: +Running against OAP 10.x? Set: + + --set ui.config.templates.mode=readonly + +Horizon reads dashboard templates from OAP 11's /ui-management REST API by +default; OAP 10 does not serve it. Everything else works either way. + +Full horizon.yaml config reference: https://github.com/apache/skywalking-horizon-ui/blob/main/docs/setup/horizon-yaml.md - https://github.com/apache/skywalking-horizon-ui/blob/main/horizon.example.yaml - (also shipped inside the image at /app/horizon.example.yaml) + +Horizon 1.0.0 ships its commented example AS the shipped config — the image's +/app/horizon.yaml is fully env-tokenized. This chart replaces that file with +the ConfigMap rendered from `ui.config`. {{- end }} diff --git a/chart/skywalking/templates/ui-configmap.yaml b/chart/skywalking/templates/ui-configmap.yaml index 99918fd..f11f781 100644 --- a/chart/skywalking/templates/ui-configmap.yaml +++ b/chart/skywalking/templates/ui-configmap.yaml @@ -15,23 +15,47 @@ {{- if .Values.ui.enabled }} {{- /* -horizon.yaml is built by deep-merging: - 1. chart-managed defaults that point at the in-cluster OAP service - 2. whatever the operator put in `ui.config` -The merge order means user values win on every field except oap.queryUrl / -oap.adminUrl, where we always default to the in-cluster service if the -operator left them unset. zipkinUrl is only injected when the Zipkin query -port is enabled — Horizon's BFF rejects an empty/non-URL value at startup. +horizon.yaml is built by deep-merging chart-computed values with `ui.config`. + +The chart-computed values are emitted as `${VAR:default}` TOKENS, not literals, +and that is load-bearing. Horizon expands `${...}` over the raw file TEXT before +parsing it (apps/bff/src/config/loader.ts), so a literal-valued file silently +makes the matching HORIZON_* env var inert -- and for a field the file omits, +env only applies to the handful whose schema default reads process.env. Writing +tokens keeps the in-cluster value as the default AND leaves every field +overridable by env, which is how the image is meant to be configured. + +Only values the image cannot know belong here. Anything equal to Horizon's own +default is left out so upstream owns it. */ -}} +{{- $oap := include "skywalking.oap.fullname" . }} {{- $oapDefaults := dict - "queryUrl" (printf "http://%s:%v" (include "skywalking.oap.fullname" .) .Values.oap.ports.rest) - "adminUrl" (printf "http://%s:%v" (include "skywalking.oap.fullname" .) (default 17128 .Values.oap.ports.admin)) + "queryUrl" (printf "${HORIZON_OAP_QUERY_URL:http://%s:%v}" $oap .Values.oap.ports.rest) }} +{{- /* The admin host arrived in OAP 11. On a 10.x release 17128 is the +AI-pipeline URI-recognition server, so advertising an adminUrl there would point +Horizon at the wrong service -- only emit it when the port is actually exposed. */ -}} +{{- if .Values.oap.ports.admin }} +{{- $_ := set $oapDefaults "adminUrl" (printf "${HORIZON_OAP_ADMIN_URL:http://%s:%v}" $oap .Values.oap.ports.admin) }} +{{- end }} +{{- /* zipkinUrl only when the Zipkin query port is exposed -- the schema +requires a valid URL, so an empty value fails at boot. */ -}} {{- $zipkinQueryPort := index .Values.oap.ports "zipkin-query" }} {{- if $zipkinQueryPort }} -{{- $_ := set $oapDefaults "zipkinUrl" (printf "http://%s:%v/zipkin" (include "skywalking.oap.fullname" .) $zipkinQueryPort) }} +{{- $_ := set $oapDefaults "zipkinUrl" (printf "${HORIZON_OAP_ZIPKIN_URL:http://%s:%v/zipkin}" $oap $zipkinQueryPort) }} +{{- end }} +{{- /* server.port is chart-derived rather than a token: the container port and +the probes both come from ui.service.internalPort, so the BFF has to bind the +same one. server.host is omitted -- the image's own ENV already sets it. */ -}} +{{- $serverDefaults := dict "port" (.Values.ui.service.internalPort | int) }} +{{- /* Horizon derives its public base URL per-request when publicUrl is blank, +which is wrong behind an ingress that rewrites Host: SSO callbacks and the OAuth +issuer would be built from the internal address. */ -}} +{{- if and .Values.ui.ingress.enabled .Values.ui.ingress.hosts }} +{{- $scheme := ternary "https" "http" (not (empty .Values.ui.ingress.tls)) }} +{{- $_ := set $serverDefaults "publicUrl" (printf "${HORIZON_PUBLIC_URL:%s://%s}" $scheme (first .Values.ui.ingress.hosts)) }} {{- end }} -{{- $defaults := dict "oap" $oapDefaults }} +{{- $defaults := dict "oap" $oapDefaults "server" $serverDefaults }} {{- $userConfig := default (dict) .Values.ui.config }} {{- $merged := mergeOverwrite (deepCopy $defaults) (deepCopy $userConfig) }} apiVersion: v1 diff --git a/chart/skywalking/templates/ui-deployment.yaml b/chart/skywalking/templates/ui-deployment.yaml index eb4c6c7..f60da57 100644 --- a/chart/skywalking/templates/ui-deployment.yaml +++ b/chart/skywalking/templates/ui-deployment.yaml @@ -109,6 +109,9 @@ spec: readOnly: true - name: horizon-data mountPath: /data +{{- with .Values.ui.extraVolumeMounts }} +{{ toYaml . | indent 8 }} +{{- end }} volumes: - name: horizon-config configMap: @@ -123,4 +126,7 @@ spec: {{- else }} emptyDir: {} {{- end }} +{{- with .Values.ui.extraVolumes }} +{{ toYaml . | indent 6 }} +{{- end }} {{- end }} diff --git a/chart/skywalking/values-my-es.yaml b/chart/skywalking/values-my-es.yaml index aea448c..049dd9f 100644 --- a/chart/skywalking/values-my-es.yaml +++ b/chart/skywalking/values-my-es.yaml @@ -19,12 +19,12 @@ oap: image: - tag: 10.0.0 + tag: 11.0.0 storageType: elasticsearch ui: image: - tag: 10.0.0 + tag: horizon-1.0.0 elasticsearch: enabled: false diff --git a/chart/skywalking/values.yaml b/chart/skywalking/values.yaml index ff37eb3..da120dc 100644 --- a/chart/skywalking/values.yaml +++ b/chart/skywalking/values.yaml @@ -41,10 +41,11 @@ oap: # zabbix: 10051 grpc: 11800 rest: 12800 - # admin REST exposed by OAP 10.5+. Horizon UI's BFF talks to this for - # runtime-rule, DSL debugging, inspect, status, and config dump. Set to - # null (or remove) if the deployed OAP version predates 10.5 — Horizon - # admin features will be unavailable in that case. + # admin REST host: /status/*, /debugging/*, inspect, dsl-debugging, + # runtime-rule, and the dashboard template store Horizon uses when + # `ui.config.templates.mode` is `live`. Introduced in OAP 11 -- set this to + # null on any 10.x release, where 17128 is the AI-pipeline URI-recognition + # server instead, so pointing Horizon at it reaches the wrong service. admin: 17128 # zipkin-receiver: 9411 # zipkin-query: 9412 @@ -122,10 +123,6 @@ oap: # # # - # ui-initialized-templates: - # general: - # general-service.json: | - # [{"id":"General-Service" ... }] # When 'dynamicConfig.enabled' set to true, enable oap dynamic configuration through k8s configmap， # Note: The default configmap data is empty, please refer to the detailed documentation (https://github.com/apache/skywalking/blob/master/docs/en/setup/backend/dynamic-config.md) # Sync period in seconds. Defaults to 60 seconds. @@ -164,11 +161,20 @@ ui: replicas: 1 image: # Horizon UI release images ship to Docker Hub `apache/skywalking-ui` - # tagged `horizon-` (e.g. `horizon-0.6.0`). Dev / pre-release - # images live at `ghcr.io/apache/skywalking-horizon-ui` and are tagged - # by SHA / `vX.Y.Z` / `main`. + # tagged `horizon-` (e.g. `horizon-1.0.0`). Dev / pre-release + # images live at `ghcr.io/apache/skywalking-horizon-ui`, tagged by full + # commit SHA, `` and `main`. + # + # Horizon releases independently of OAP — there is no 1:1 version mapping. + # Horizon 1.0.0 works against OAP 10.4.0 -- the last 10.x release -- as + # well as OAP 11.x, so pin it whichever OAP release you run; on 10.4 also + # set `ui.config.templates.mode: readonly` and `oap.ports.admin: null`, + # neither of which 10.x serves. + # + # This must be a `horizon-*` tag. The legacy booster UI is not supported — + # see the README. repository: skywalking.docker.scarf.sh/apache/skywalking-ui - tag: null # Must be set explicitly, e.g. horizon-0.6.0 + tag: null # Must be set explicitly, e.g. horizon-1.0.0 pullPolicy: IfNotPresent # podAnnotations: # example: oap-foo @@ -212,9 +218,10 @@ ui: annotations: {} ## Limit load balancer source ips to list of CIDRs (where available) # loadBalancerSourceRanges: [] - # The image runs as a non-root alpine system user `horizon`. PVCs mounted - # at /data and (optionally) /app/bundled_templates must be group-writable - # by that user; fsGroup defaults to 101 (alpine's typical `nobody` GID). + # The image runs as a non-root alpine system user `horizon`. Any volume + # mounted into the container (/data, and anything added via + # `ui.extraVolumeMounts`) must be group-writable by that user; fsGroup + # defaults to 101 (alpine's typical `nobody` GID). securityContext: fsGroup: 101 # Persistent state directory used by the BFF for audit log, setup state, @@ -244,49 +251,45 @@ ui: initialDelaySeconds: 10 periodSeconds: 10 failureThreshold: 6 - # `config` is rendered verbatim into the `horizon.yaml` ConfigMap mounted - # at /app/horizon.yaml. Schema and field reference: + # Extra `horizon.yaml` content, deep-merged over the values the chart computes + # (the in-cluster OAP URLs, the ingress-derived publicUrl, and the BFF port). + # Schema reference: # https://github.com/apache/skywalking-horizon-ui/blob/main/docs/setup/horizon-yaml.md - # See also /app/horizon.example.yaml inside the image for the upstream - # commented example. # - # Chart-managed defaults wire `oap.queryUrl` / `oap.adminUrl` to the - # in-cluster OAP service. Anything you set here merges over the defaults - # via Helm's mergeOverwrite semantics. + # EMPTY BY DESIGN. Horizon's image ships a fully env-tokenized + # /app/horizon.yaml, and the chart's ConfigMap replaces it -- so every field + # written here as a literal makes its `HORIZON_*` env var inert. Prefer + # `ui.extraEnv` / `ui.envFromSecret` for anything the chart does not compute, + # and set a field here only when you want to pin it regardless of env. # - # `auth.local.users` is intentionally empty by default — Horizon's BFF - # refuses to start without it, which is the upstream's "no default - # admin/admin" stance. See the README "Web UI" section for two paths: - # a copy-pastable demo snippet (publicly-known admin/admin hash) or the - # production Secret + `${VAR}` interpolation pattern. - config: - server: - host: 0.0.0.0 - port: 8081 - oap: - # queryUrl / adminUrl default to the in-cluster OAP service when unset. - # Override here if you point the UI at an external OAP cluster. - # queryUrl: http://skywalking-oap:12800 - # adminUrl: http://skywalking-oap:17128 - timeoutMs: 15000 - # auth: - # username: skywalking - # password: "${HORIZON_OAP_PW}" - auth: - backend: local - local: - users: [] - # - username: admin - # passwordHash: "${HORIZON_ADMIN_HASH}" - # roles: [admin] - rbac: - enabled: true - session: - ttlMinutes: 60 - cookieName: horizon_sid - # Flip to true when serving over HTTPS (browser refuses Secure - # cookies over plain HTTP). - cookieSecure: false + # Some examples, with the env var that does the same job: + # + # templates.mode HORIZON_TEMPLATES_MODE set `readonly` on OAP 10.x + # (and oap.ports.admin: null) + # session.cookieSecure HORIZON_SESSION_COOKIE_SECURE true when serving HTTPS + # server.trustProxy HORIZON_TRUST_PROXY hop count or ingress CIDR + # auth.local.users HORIZON_AUTH_LOCAL_USERS JSON array, see below + # oap.auth HORIZON_OAP_AUTH JSON, OAP basic-auth + # + # NO LOGIN IS CONFIGURED BY DEFAULT, and the BFF does not refuse to start + # without one -- it boots, logs an error, serves the login page, and answers + # the readiness probe with 200, so the pod goes Ready and nobody can log in. + # Supply users before you rely on the deployment. Either put them in a Secret + # as a JSON array and reference it from `ui.envFromSecret`: + # + # HORIZON_AUTH_LOCAL_USERS=[{"username":"admin","passwordHash":"$argon2id$...","roles":["admin"]}] + # + # or write them here with a `${VAR}` token so the hash still comes from a + # Secret rather than this file (tokens written here DO expand): + # + # config: + # auth: + # local: + # users: + # - username: admin + # passwordHash: "${HORIZON_ADMIN_HASH}" + # roles: [admin] + config: {} # Optional: pass extra env vars into the BFF container. Use this to # provide values referenced by `${VAR}` interpolation in `config` # (e.g. admin password hash, OAP basic-auth password) without baking @@ -304,6 +307,30 @@ ui: # ${VAR} interpolation. # envFromSecret: horizon-secrets envFromSecret: "" + # Extra volumes / mounts for the BFF container. Two things in Horizon 1.0.0 + # are configured by PATH rather than by value and therefore need one: + # + # auth.tokensFile — API tokens for scripts / CI / MCP clients + # sourceMaps.bootMountDir — durable .map files for the Browser Errors tab + # (the image sets it to /app/sourcemaps; runtime + # uploads without a volume are in-memory only and + # are lost on pod restart) + # + # extraVolumes: + # - name: horizon-tokens + # secret: + # secretName: horizon-tokens + # - name: horizon-sourcemaps + # persistentVolumeClaim: + # claimName: horizon-sourcemaps + # extraVolumeMounts: + # - name: horizon-tokens + # mountPath: /app/tokens + # readOnly: true + # - name: horizon-sourcemaps + # mountPath: /app/sourcemaps + extraVolumes: [] + extraVolumeMounts: [] oapInit: diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 0000000..fa97a1b --- /dev/null +++ b/docs/README.md @@ -0,0 +1,69 @@ +# Apache SkyWalking Helm Chart + +Helm 3 charts for running Apache SkyWalking on Kubernetes: the OAP backend, the Horizon UI web +console, an optional Satellite gateway, and a storage backend the chart can deploy for you. + +This repository ships three charts: + +| chart | what it installs | +|---|---| +| `chart/skywalking` | OAP, Horizon UI, optional Satellite, optional storage (Elasticsearch, PostgreSQL, BanyanDB) | +| `chart/operator` | the [SWCK](https://github.com/apache/skywalking-swck) operator | +| `chart/adapter` | the SWCK custom-metrics adapter, for autoscaling on SkyWalking metrics | + +## Start here + +```shell +helm install skywalking oci://registry-1.docker.io/apache/skywalking-helm \ + --version 5.0.0 -n skywalking --create-namespace \ + --set oap.image.tag=11.0.0 \ + --set oap.storageType=banyandb \ + --set ui.image.tag=horizon-1.0.0 \ + --set elasticsearch.enabled=false \ + --set banyandb.enabled=true \ + --set banyandb.image.tag=0.11.0 +``` + +Those three `--set` values have no defaults and must be given on every install. Read +[Quick Start](install/quick-start.md) for the full path, including the Elasticsearch CRD +prerequisite if you use the default storage backend. + +**A fresh install has no login.** Horizon UI ships no default credentials and does not fail closed — +the pod reports Ready and nobody can sign in until you configure users. See +[Set Up Logins](ui/logins.md). + +## Find your way + +- **Deciding whether this fits** — [What This Chart Deploys](evaluate/what-this-chart-deploys.md), + [Requirements](evaluate/requirements.md), + [Version Compatibility](evaluate/version-compatibility.md) +- **Installing** — [Quick Start](install/quick-start.md), + [Where to Get the Chart](install/chart-sources.md) +- **Storage** — [Pick a Storage Backend](storage/choose-a-backend.md), then + [Elasticsearch](storage/elasticsearch.md), [BanyanDB](storage/banyandb.md) or + [PostgreSQL](storage/postgresql.md) +- **The web UI** — [Horizon UI in This Chart](ui/horizon-ui.md), + [Set Up Logins](ui/logins.md), [Configure Horizon](ui/configure.md) +- **Exposing it** — [UI Service and Ingress](expose/ui-service-and-ingress.md), + [TLS](expose/tls.md), [OAP Endpoints for Agents](expose/oap-endpoints.md) +- **Running it** — [The OAP Init Job](operate/oap-init-job.md), + [Configure OAP](operate/oap-configuration.md), + [Scaling and the OAP Cluster](operate/scaling.md), + [Satellite Gateway](operate/satellite.md), + [SWCK Operator](operate/swck-operator.md), + [SWCK Metrics Adapter](operate/swck-adapter.md) +- **Upgrading** — [Upgrade](upgrade/upgrading.md) +- **When it breaks** — [Install and Startup Failures](troubleshooting/install-and-startup.md), + [UI and Login Problems](troubleshooting/ui-and-login.md) +- **Every value** — [skywalking](reference/skywalking-chart-values.md), + [adapter](reference/adapter-chart-values.md), + [operator](reference/operator-chart-values.md) +- **Contributing** — [Run the E2E Tests](contributing/e2e-tests.md), + [Package and Publish a Release](contributing/release.md) + +## Related documentation + +- [SkyWalking backend setup](https://skywalking.apache.org/docs/main/latest/en/setup/backend/backend-setup/) +- [OAP configuration vocabulary](https://skywalking.apache.org/docs/main/latest/en/setup/backend/configuration-vocabulary/) +- [Horizon UI `horizon.yaml` reference](https://github.com/apache/skywalking-horizon-ui/blob/main/docs/setup/horizon-yaml.md) +- [BanyanDB Helm chart](https://github.com/apache/skywalking-banyandb-helm) diff --git a/docs/changes/changes.md b/docs/changes/changes.md new file mode 100644 index 0000000..b27cac0 --- /dev/null +++ b/docs/changes/changes.md @@ -0,0 +1,51 @@ +# Changelog + +## 5.0.0 + +Targets SkyWalking OAP 11.0.0, Horizon UI 1.0.0 and BanyanDB 0.11.0. See +[Upgrade](../upgrade/upgrading.md) for the migration steps. + +### Breaking changes + +- **OAP 11 requires BanyanDB 0.11.x.** OAP pins the BanyanDB server API versions it accepts + (`SW_STORAGE_BANYANDB_COMPATIBLE_SERVER_API_VERSIONS`, `0.11` in 11.0.0) and checks them with + string equality, so pairing OAP 11 with BanyanDB 0.10.x makes OAP refuse to start. The three + versions move together — see [Version Compatibility](../evaluate/version-compatibility.md). +- **`oap.ports.admin` is required.** OAP 11 enables every admin feature module by default and + serves `/status/*` and `/debugging/*` on the admin port only; they are no longer mirrored on + `oap.ports.rest`. Horizon UI reads status, inspect, DSL debugging and the dashboard template + store from it. +- **The legacy booster UI is no longer supported.** OAP 11 deleted `apm-webapp` and the + `skywalking-booster-ui` submodule along with the `docker.ui` build target, so + `apache/skywalking-ui` publishes no `11.x` tag — only `horizon-*` tags. Replace + `ui.image.tag=` with `ui.image.tag=horizon-1.0.0`. +- **`oap.config.ui-initialized-templates` does nothing.** OAP 11 removed the on-disk dashboard + seed files and `UITemplateInitializer`, along with the sidebar menu storage, the + `UIConfigurationManagement` GraphQL mutations and `SW_ENABLE_UPDATE_UI_TEMPLATE`. Horizon UI + ships its own dashboard library and manages templates over the admin REST port. +- **`ui.config` is empty by default.** The chart now writes only the values it computes, as + `${HORIZON_*:default}` tokens, and Horizon is configured by environment variable. A literal + written into `ui.config` makes that field's `HORIZON_*` variable inert — see + [Configure Horizon](../ui/configure.md). +- **The UI no longer proxies `/graphql`.** Callers that talked to the UI's GraphQL endpoint + (for example `swctl --base-url=http:///graphql`) must target the OAP service directly on + `oap.ports.rest`. + +### Features + +- `ui.extraVolumes` / `ui.extraVolumeMounts`, for the two Horizon settings that take a filesystem + path: `auth.tokensFile` and `sourceMaps.bootMountDir`. +- `server.publicUrl` is derived from the first `ui.ingress.hosts` entry when an ingress is enabled, + so single sign-on callbacks and the OAuth issuer are built from the address operators actually + reach — see [UI Service and Ingress](../expose/ui-service-and-ingress.md). +- `server.port` is derived from `ui.service.internalPort`, so the BFF binds the port the container + exposes. +- Documentation moved into `docs/` and is published at + [skywalking.apache.org/docs/skywalking-helm](https://skywalking.apache.org/docs/skywalking-helm/next/readme/). + +### Corrections + +- Horizon UI does **not** refuse to start without configured users. It boots, serves the login + page, and answers `/api/auth/health` with 200 — which is this chart's readiness probe — so the + pod reports Ready and nobody can sign in. Earlier documentation claimed a `CrashLoopBackOff`. + See [Set Up Logins](../ui/logins.md). diff --git a/docs/contributing/e2e-tests.md b/docs/contributing/e2e-tests.md new file mode 100644 index 0000000..29f5b8f --- /dev/null +++ b/docs/contributing/e2e-tests.md @@ -0,0 +1,183 @@ +# Run the E2E Tests + +Every change to this chart is validated by an end-to-end suite that installs the chart into a +throwaway [kind](https://kind.sigs.k8s.io/) cluster, pushes real traffic through it, and queries +the OAP with `swctl`. This page explains how that suite is put together and how to run one test +on your own machine. + +## The framework + +The tests are driven by [skywalking-infra-e2e](https://github.com/apache/skywalking-infra-e2e). +Each config in the CI matrix — the three files at the top of `test/e2e/` and the five under +`test/e2e/swck/` — is a complete test: it declares the cluster to create, the setup steps to run, +the traffic to generate, and the assertions to make. The phases are: + +| Phase | What it does here | +| --- | --- | +| `setup` | Creates the kind cluster from `file:`, loads `init-system-environment:` (the `env` file) into the shell environment, then runs the `steps:` in order — install tooling, install Istio, `helm install` the chart, deploy the demo app, start traffic. `kind.expose-ports` port-forwards the named services out of the cluster. | +| `trigger` | Optional HTTP loop. The three chart tests generate load with a `wrk` Deployment instead (`test/e2e/traffic-gen.yaml`), so they have no `trigger:` block; the SWCK tests use one against `/hello`. | +| `verify` | Runs each `query:` (mostly `swctl ... /graphql`) and matches the output against an `expected/` template, retrying on the declared `retry:` schedule. | +| `cleanup` | Deletes the kind cluster. The SWCK tests set `cleanup.on: always`. The three chart tests declare no `cleanup:`, so infra-e2e's own default applies — `always` when `CI=true` (GitHub Actions always tears down), `success` otherwise, so a locally *failed* run leaves the cluster up for debugging. | + +`expected/*.yml` are templates, not literals — they use matchers such as `{{- contains .results }}` +and `{{ notEmpty .value }}`, so a test asserts "this metric exists and has a value", not an exact +number. + +## The test files + +The three top-level tests are the same Istio + bookinfo scenario with a different storage backend. +Each installs Istio (demo profile) with Envoy ALS pointed at +`skywalking-satellite.istio-system:11800`, installs the chart with Satellite enabled, deploys +[bookinfo](https://istio.io/latest/docs/examples/bookinfo/), runs `wrk` against the ingress +gateway, and then asserts on the service list, endpoints, service/instance/relation metrics and +Envoy sidecar metrics. + +| File | Storage flags | Notes | +| --- | --- | --- | +| `test/e2e/e2e-elasticsearch.yaml` | `oap.storageType=elasticsearch` (the ECK subchart, left on by the chart's `elasticsearch.enabled: true`; `oap.storageType` itself has no default) | Pre-installs the ECK CRDs out of `eck-operator-3.3.1.tgz` and passes `eck-operator.installCRDs=false`. | +| `test/e2e/e2e-banyandb-standalone.yaml` | `oap.storageType=banyandb`, `banyandb.enabled=true`, `banyandb.standalone.enabled=true`, `banyandb.cluster.enabled=false`, `banyandb.auth.enabled=true`, `elasticsearch.enabled=false` | Covers the BanyanDB standalone path with auth on. | +| `test/e2e/e2e-banyandb-cluster.yaml` | as above but `standalone.enabled=false`, `cluster.enabled=true`, and *no* `auth.enabled`, plus a `256Mi` FODC agent memory request/limit (`banyandb.cluster.fodc.agent.resources.*`) | Installs with `--timeout 10m` because the cluster takes longer to become ready. | + +All three run on `test/e2e/kind28.yaml` — one control-plane and three workers on +`kindest/node:v1.28.15` — install into namespace `istio-system` with +`fullnameOverride=skywalking`, and expose `service/skywalking-oap:12800` so `swctl` can reach the +GraphQL endpoint as `${service_skywalking_oap_host}:${service_skywalking_oap_12800}`. + +See [Elasticsearch](../storage/elasticsearch.md) and [BanyanDB](../storage/banyandb.md) for what +those flags mean outside the tests. + +### The SWCK set + +`test/e2e/swck/` tests the [operator](../operate/swck-operator.md) and adapter charts rather than +the `skywalking` chart. They run on `test/e2e/kind.yaml` (`kindest/node:v1.21.14`), install +cert-manager and `chart/operator`, deploy the CRs from `skywalking-components*.yaml` with the image +placeholders substituted from `env`, inject the Java agent into the SWCK demo app, and verify +`service ls` / `service_cpm` / `endpoint_cpm`. + +| File | What it covers | +| --- | --- | +| `oap-agent-banyandb.yaml` | `OAPServer` + `UI` CRs against a standalone BanyanDB installed from the OCI chart. | +| `oap-ui-agent-elasticsearch.yaml` | Same CRs against a plain Elasticsearch StatefulSet (`deploy-elasticsearch.yaml`, ES 8.18.8, security disabled). | +| `oap-ui-agent-satellite.yaml` | Adds the `Satellite` CR; the agent reports to `skywalking-system-satellite` instead of the OAP. | +| `oap-ui-agent-oapserverconfig.yaml` | `OAPServerConfig` (static config, waits for the OAP pod to be recreated) and `OAPServerDynamicConfig`. | +| `oap-agent-adapter-hpa.yaml` | Installs `chart/adapter` as a custom-metrics API server and asserts an HPA scaled the demo Deployment to 3 replicas. | + +The shared `skywalking-components*.yaml` manifests use `OAP_IMAGE_PLACEHOLDER`, +`UI_IMAGE_PLACEHOLDER`, `SATELLITE_IMAGE_PLACEHOLDER` and `PLACEHOLDER` (the CR `version` field, +replaced with `latest`), substituted by `sed` in the setup step. + +## `test/e2e/env` — the image pin file + +Every test loads this file via `init-system-environment`, so it is the single place where the +versions under test are pinned. Changing a version for CI means editing this file. + +| Variable | Pins | +| --- | --- | +| `OAP_REPO` / `OAP_TAG` | `docker.io/apache/skywalking-oap-server` : `11.0.0` | +| `UI_REPO` / `UI_TAG` | `docker.io/apache/skywalking-ui` : `horizon-1.0.0` | +| `SWCK_OPERATOR_REPO` / `SWCK_OPERATOR_TAG` | `docker.io/apache/skywalking-swck` : `0.10.0`, matching the default in `chart/operator/values.yaml` | +| `SATELLITE_REPO` / `SATELLITE_TAG` | `ghcr.io/apache/skywalking-satellite/skywalking-satellite` at a commit tag | +| `BANYANDB_REPO` / `BANYANDB_TAG` | `ghcr.io/apache/skywalking-banyandb` at commit `3b83e18…`, the v0.11.0 commit. The file records why: the Docker Hub release image for 0.11.0 has not been pushed, so CI uses the GHCR dev image built from that tag. | +| `SW_CTL_COMMIT` | The `skywalking-cli` commit that `setup-e2e-shell/install-swctl.sh` builds `swctl` from, and re-installs if the local binary reports a different version. | + +OAP 11.0.0, Horizon UI 1.0.0 and BanyanDB 0.11.0 have to move together — see +[Version Compatibility](../evaluate/version-compatibility.md). The BanyanDB *chart* version is not +pinned here: the four SWCK tests that install BanyanDB read it out of +`chart/skywalking/Chart.yaml` with `yq`. + +## `test/e2e/values.yaml` — the shared overlay + +All three chart tests pass `-f test/e2e/values.yaml` after their `--set` flags. It holds only what +the tests need on top of the chart defaults: + +- `oap.config."metadata-service-mapping.yaml"` — names mesh services `e2e::` so the + `verify` queries can look up `e2e::productpage` and `e2e::reviews`. +- `ui.config.auth` — defines two local users (`admin`/`admin`, `skywalking`/`skywalking`). The + chart ships `ui.config: {}` and Horizon has no `admin/admin` fallback, so without this block + nobody could log in to a port-forwarded run. See [Set Up Logins](../ui/logins.md). +- `elasticsearch.nodeSets` — a single 2Gi ES node with `node.store.allow_mmap: false` and relaxed + disk watermarks, so ES stays green on a kind node with little free disk. + +## Run one locally + +You need Docker and Go — the `e2e` binary embeds kind as a library, so no separate `kind` binary is +required. The setup steps install `yq`, `swctl`, `kubectl`, `istioctl` and `helm` into +`/usr/local/bin` and install Istio into the cluster, so run this on a machine you don't mind +changing. + +Build the `e2e` CLI once: + +```shell +git clone https://github.com/apache/skywalking-infra-e2e.git +cd skywalking-infra-e2e +make install DESTDIR=/usr/local/bin +``` + +Then, **from the root of this repo** (the config files reference `chart/skywalking` and +`test/e2e/setup-e2e-shell/…` relative to the working directory): + +```shell +export ISTIO_VERSION=1.24.0 +e2e run -c test/e2e/e2e-banyandb-standalone.yaml +``` + +`ISTIO_VERSION` is set by the CI workflow, not by `test/e2e/env`, and both `install-istioctl.sh` +and the bookinfo manifest URLs read it — export it yourself when running locally. + +To iterate without re-creating the cluster, run the phases separately: + +```shell +e2e setup -c test/e2e/e2e-banyandb-standalone.yaml +e2e verify -c test/e2e/e2e-banyandb-standalone.yaml # repeat as you debug +e2e cleanup -c test/e2e/e2e-banyandb-standalone.yaml # deletes the kind cluster +``` + +While the cluster is up: + +```shell +kubectl -n istio-system get pods +kubectl -n istio-system logs deploy/skywalking-oap +kubectl -n istio-system port-forward svc/skywalking-ui 8080:80 # then log in as admin/admin +``` + +## How CI runs them + +`.github/workflows/e2e.ci.yaml` defines a single job, `als`, with a matrix of eight entries — one +per test file — each with a 60-minute timeout: + +```yaml +strategy: + matrix: + test: + - name: Run Skywalking E2E Test (Elasticsearch as database) + config: test/e2e/e2e-elasticsearch.yaml + # …and the two BanyanDB tests plus the five SWCK tests +``` + +Each entry logs in to `ghcr.io`, sets up Go 1.24, and hands its config file to the +`apache/skywalking-infra-e2e` action, pinned to SHA `8c21e43e…`. The +[ASF Actions policy](https://infra.apache.org/github-actions-policy.html) requires the SHA pin on +the third-party `docker/login-action`; `apache/*` actions are allowed unpinned, so this pin is only +for reproducibility: + +```yaml +- uses: apache/skywalking-infra-e2e@8c21e43e241a32a54bdf8eeceb9099eb27e5e9b4 + with: + e2e-file: $GITHUB_WORKSPACE/${{ matrix.test.config }} +``` + +On failure the job dumps disk usage and the local Docker images, then uploads +`$SW_INFRA_E2E_LOG_DIR` as the `logs` artifact — start there when a CI run fails but a local run +passes. A trailing `build` job depends on `als` and only runs `echo`, so one job name aggregates +the whole matrix. + +The workflow runs on every `pull_request` and on pushes to `master`. The `paths-ignore: ['**.md']` +filter applies only to the push trigger, so a docs-only pull request still runs the full matrix. + +## Adding a test + +1. Copy the closest existing config; keep image references as `$OAP_REPO`/`$OAP_TAG` style + variables so `test/e2e/env` stays the only place versions are pinned. +2. Reuse `test/e2e/values.yaml` and the `expected/` templates where you can. +3. Add a matrix entry in `.github/workflows/e2e.ci.yaml` — a config file that is not in the matrix + never runs. diff --git a/docs/contributing/release.md b/docs/contributing/release.md new file mode 100644 index 0000000..3bb3fd7 --- /dev/null +++ b/docs/contributing/release.md @@ -0,0 +1,213 @@ +# Package and Publish a Release + +Everything a release does lives in the repository `Makefile`. This page walks through each target — +what it produces, what it depends on, and what must be true before a tag is cut. + +## The version is read, never passed + +The `Makefile` derives all names from `chart/skywalking/Chart.yaml`: + +```makefile +CHART_DIR = chart/skywalking +VERSION = $(shell cat ${CHART_DIR}/Chart.yaml | grep '^version: ' | awk '{print $$2}') +CHART_NAME = $(shell cat ${CHART_DIR}/Chart.yaml | grep '^name: ' | awk '{print $$2}') +RELEASE_SRC = ${CHART_NAME}-${VERSION}-src +``` + +`make release VERSION=…` does not do what it looks like: a command-line assignment overrides the +make variable but not the file `helm package` writes — that name comes from `Chart.yaml` — so the +signing and push steps go looking for a tarball that does not exist. Bumping a release means editing +the `version:` line in `chart/skywalking/Chart.yaml` and committing it — that is exactly what the +"Ready to release" commits do. Note that `CHART_NAME` comes from the chart's `name:` field (`skywalking-helm`), not from the +directory name (`skywalking`), so at chart version `5.0.0` the artifacts are: + +| | | +| --- | --- | +| chart package | `skywalking-helm-5.0.0.tgz` | +| source tarball | `skywalking-helm-5.0.0-src.tgz` | + +The `adapter` and `operator` charts carry their own `version:` (`0.10.0` each) in their own +`Chart.yaml` and are **not** touched by any Makefile target. Only `chart/skywalking` is packaged, +signed and published here. Separately, `.github/workflows/publish-helm.yaml` pushes all three charts +to `oci://ghcr.io/apache/skywalking-helm` on every push to `master`, rewriting `version:` to +`0.0.0-` first — those are snapshots, not releases. + +`TMPDIR` defaults to `/tmp` and is the only variable declared with `?=`, i.e. the only one meant to +be overridden (`make release TMPDIR=/var/tmp`). The recipe shell is `/bin/bash -eo pipefail`. + +## Targets at a glance + +| Target | Depends on | Produces | +| --- | --- | --- | +| `prepare` | — | `NOTICE` + `LICENSE` copied into `chart/skywalking/` | +| `package` | `prepare` | `skywalking-helm-.tgz` | +| `clean` | — | nothing; deletes artifacts and resolved dependencies | +| `release-src` | `clean` | `skywalking-helm--src.tgz` | +| `release` | `release-src`, `package` | both tarballs, each with `.asc` and `.sha512` | +| `publish` | `package` | the chart pushed to Docker Hub as an OCI artifact | + +## `prepare` — the NOTICE/LICENSE copy dance + +```makefile +prepare: + cp -R NOTICE ${CHART_DIR}/NOTICE + cp -R LICENSE ${CHART_DIR}/LICENSE +``` + +`NOTICE` and `LICENSE` live at the repository root, but an Apache release artifact has to carry them +*inside* the distributed tarball. `helm package` only picks up files under the chart directory, so +they are copied in immediately before packaging and deleted immediately after — the working tree +never keeps a second copy, and neither file is tracked under `chart/skywalking/`. + +If a `make package` run dies partway through, those two copies are left behind. `make clean` removes +them. + +## `package` + +```makefile +package: prepare + helm dep up ${CHART_DIR} + helm package ${CHART_DIR} + rm -rf ${CHART_DIR}/NOTICE + rm -rf ${CHART_DIR}/LICENSE +``` + +`helm dep up` resolves every entry in the `dependencies:` block of `Chart.yaml` — `eck-operator`, +`eck-elasticsearch`, `postgresql`, `skywalking-banyandb-helm` — downloading each into +`chart/skywalking/charts/` and writing `chart/skywalking/Chart.lock`. Both are gitignored. `helm +package` then rolls the chart directory, its vendored subcharts, and the just-copied `NOTICE` and +`LICENSE` into `skywalking-helm-.tgz` in the repository root, and the two copies are +removed. + +The subchart `.tgz` files are baked into the package. Whatever `helm dep up` resolved at package time +is what users get — which is the reason for the RC rule below. + +## `clean` + +Deletes, in one `rm` invocation: + +- `$(TMPDIR)/skywalking-helm--src.tgz` +- `bin/` +- `chart/skywalking/NOTICE`, `chart/skywalking/LICENSE` +- `chart/skywalking/Chart.lock`, `chart/skywalking/charts/` +- `skywalking-helm-.tgz` and its `.asc` / `.sha512` +- `skywalking-helm--src.tgz` and its `.asc` / `.sha512` + +The recipe is written as backslash-continued lines with no `&&` between them, so the shell sees a +single `rm` command whose later `rm` and `-rf` tokens land in the argument list. **Release from +Linux.** GNU `rm` permutes arguments, so the trailing `-rf` still applies (and `-f` swallows the +stray `rm` operands) and the target does what it reads like. BSD `rm` — macOS — stops option +parsing at the first operand, so `-r` never takes effect: it fails with `rm: bin/: is a directory`, +leaves `bin/` and `chart/skywalking/charts/` in place, and exits `1`, which aborts `make clean`, +`make release-src` and `make release`. Do not "fix" one line in isolation without re-reading the +whole recipe. + +Because `clean` wipes `charts/` and `Chart.lock`, the next `package` re-resolves dependencies from +scratch. + +## `release-src` + +```makefile +release-src: clean + tar -zcvf $(TMPDIR)/$(RELEASE_SRC).tgz \ + --exclude bin --exclude .git --exclude .idea \ + --exclude .gitignore --exclude .DS_Store --exclude .github \ + . && \ + mv $(TMPDIR)/$(RELEASE_SRC).tgz . +``` + +It archives the whole working directory — hence the `clean` prerequisite, which guarantees no +`charts/`, `Chart.lock` or previous `.tgz` sneaks into the source release. The tar is built in +`$(TMPDIR)` and only then moved into the repository root, so the archive never contains itself. + +`tar .` takes the *working tree*, not `HEAD`. Run it from a pristine checkout of the release commit: +any untracked scratch file that is not in the exclude list ships inside the ASF source release. + +## `release` + +```makefile +release: release-src package + gpg --batch --yes --armor --detach-sig $(RELEASE_SRC).tgz + shasum -a 512 $(RELEASE_SRC).tgz > $(RELEASE_SRC).tgz.sha512 + gpg --batch --yes --armor --detach-sig $(CHART_NAME)-$(VERSION).tgz + shasum -a 512 $(CHART_NAME)-$(VERSION).tgz > $(CHART_NAME)-$(VERSION).tgz.sha512 +``` + +The full sequence, in order: `clean` → `release-src` → `prepare` → `package` → sign and checksum. +**Both** artifacts are signed and checksummed — the source tarball and the chart package — leaving +six files in the repository root: + +```text +skywalking-helm-5.0.0-src.tgz +skywalking-helm-5.0.0-src.tgz.asc +skywalking-helm-5.0.0-src.tgz.sha512 +skywalking-helm-5.0.0.tgz +skywalking-helm-5.0.0.tgz.asc +skywalking-helm-5.0.0.tgz.sha512 +``` + +Signing is armored and detached, and `--batch --yes` means GPG will not prompt: the signing key must +already be the default secret key in the release manager's keyring, with the passphrase available +through the agent. Verify before uploading: + +```shell +gpg --verify skywalking-helm-5.0.0.tgz.asc skywalking-helm-5.0.0.tgz +shasum -a 512 -c skywalking-helm-5.0.0.tgz.sha512 +``` + +Do not run this target with `make -j`. `release-src` starts with `clean`, and `package` writes into +the same tree; only serial execution keeps the ordering the recipe assumes. + +## `publish` + +```makefile +publish: package + helm push ${CHART_NAME}-${VERSION}.tgz oci://registry-1.docker.io/apache +``` + +Re-packages (so it is safe to run standalone) and pushes the chart to Docker Hub as an OCI artifact. +The pushed name comes from the chart's own metadata, so it lands at +`oci://registry-1.docker.io/apache/skywalking-helm` — the address users install from, documented in +[Where to Get the Chart](../install/chart-sources.md). `helm registry login registry-1.docker.io` +first; the push needs write access to the `apache` Docker Hub organization. + +This is the last step, run only after the vote passes. + +## Release-time rule: no RC dependencies + +`chart/skywalking/Chart.yaml` is allowed to pin a release candidate of a dependency **while +developing**. As of writing it does: + +```yaml +- name: skywalking-banyandb-helm + alias: banyandb + version: 0.7.0-rc1 + repository: oci://registry-1.docker.io/apache + condition: banyandb.enabled +``` + +Every `-rc*` or otherwise non-GA `version:` in the `dependencies:` block must be moved to a released +version before the release tag is cut. `helm dep up` bakes the resolved subchart into the package, so +an RC pin means the released chart ships a release candidate to every user, and an ASF release cannot +depend on artifacts that were never themselves released. Check with: + +```shell +grep '^ version:' chart/skywalking/Chart.yaml +``` + +(The dependency versions are the indented `version:` lines; the chart's own `version:` starts at +column 0. Do not anchor the search to the `- name:` line above it — two entries carry an `alias:` +in between, including the BanyanDB one.) + +Any hit containing `rc`, `alpha`, `beta` or `SNAPSHOT` blocks the release. + +## Checklist before tagging + +1. `version:` in `chart/skywalking/Chart.yaml` bumped to the release version. +2. No RC-pinned entry in that file's `dependencies:` block. +3. `docs/changes/changes.md` has a section for the version. +4. Image tags quoted across the docs match the release (`oap.image.tag`, `ui.image.tag`, BanyanDB) — + see [Version Compatibility](../evaluate/version-compatibility.md). +5. [E2E tests](e2e-tests.md) green on the release commit. +6. `make release` on Linux from a clean checkout; verify the four signature and checksum files. +7. After the vote: `make publish`. diff --git a/docs/evaluate/requirements.md b/docs/evaluate/requirements.md new file mode 100644 index 0000000..e92d0ac --- /dev/null +++ b/docs/evaluate/requirements.md @@ -0,0 +1,137 @@ +# Requirements + +What your cluster and workstation need before `helm install`, and the one pre-install step the +default (Elasticsearch) path requires. + +## Kubernetes and Helm + +| Requirement | Version | Where it comes from | +|---|---|---| +| Kubernetes | 1.21+ | `eck-operator` 3.3.1 and `eck-elasticsearch` 0.18.1 both declare `kubeVersion: '>=1.21.0-0'`. Helm does **not** enforce a *subchart's* `kubeVersion`, so an older cluster still installs — it just runs an unsupported ECK. | +| Kubernetes (adapter / operator charts) | 1.24.0+ | Stated by the `chart/adapter` and `chart/operator` READMEs. | +| Helm | 3.8+ (4.x also works) | All charts here are `apiVersion: v2`, and `helm dep up` pulls the BanyanDB dependency from an OCI registry (`oci://registry-1.docker.io/apache`), which needs Helm's non-experimental OCI support (3.8.0). | +| `kubectl` | matching your cluster | Used for the CRD pre-install and for all troubleshooting. | + +CI exercises the chart on a four-node [kind](https://kind.sigs.k8s.io/) cluster running +`kindest/node:v1.28.15` (`test/e2e/kind28.yaml`), so 1.28 is the version with the most coverage. +The SWCK suites under `test/e2e/swck/` still run on `test/e2e/kind.yaml` (`kindest/node:v1.21.14`). + +The chart adapts to older Ingress APIs (`ui-ingress.yaml` falls back from +`networking.k8s.io/v1` to `v1beta1` to `extensions/v1beta1`), so ingress itself is not what sets +the minimum. + +## Cluster permissions + +Installing needs more than namespace-scoped rights: + +- **CRDs are cluster-scoped.** The ECK CRDs (see below) can only be installed by a user with + cluster-admin-level permissions. +- **The chart creates a `ClusterRole` and `ClusterRoleBinding`** for OAP (`oap-clusterrole.yaml`, + `oap-clusterrolebinding.yaml`), granting `get`/`watch`/`list` on pods, pod logs, endpoints, + services, nodes, namespaces, configmaps, `extensions` deployments/replicasets and Istio + `networking.istio.io` `serviceentries` — this is what backs Kubernetes-based service discovery + and mesh analysis. Both templates are gated on `serviceAccounts.oap.create`, so set + `serviceAccounts.oap.create=false` and supply your own `serviceAccounts.oap.name` if you cannot + create cluster-scoped RBAC. +- **The SWCK adapter chart registers an `APIService`** (`v1beta1.external.metrics.k8s.io`), which + requires the API aggregation layer. + +## Storage volumes + +Only Elasticsearch and a BanyanDB *cluster* claim persistent volumes out of the box. The other +paths render `emptyDir` — fine for a trial, data loss on every pod restart anywhere else: + +| Component | Claims a PV by default? | Note | +|---|---|---| +| Elasticsearch (ECK) | Yes | `elasticsearch.nodeSets[0].count: 3`, and because `volumeClaimTemplates` is left commented out ECK applies its own default — a 1Gi `elasticsearch-data` PVC per node on the default StorageClass. Uncomment `elasticsearch.nodeSets[].volumeClaimTemplates` in `values.yaml` to size it (the example asks for 30Gi). See [ECK volume claim settings](https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-volume-claim-templates.html). | +| BanyanDB standalone (`banyandb.standalone.enabled: true`, the default) | **No** | The subchart ships `storage.standalone.enabled: false`. Set `banyandb.storage.standalone.enabled=true` to claim the 200Gi `standalone-data` volume. | +| BanyanDB cluster (`banyandb.cluster.enabled=true`) | Yes | `banyandb.storage.data.enabled: true` — five data PVCs (50Gi/50Gi/5Gi/50Gi/5Gi) plus 10Gi for liaison, all with `storageClass: null`, i.e. the cluster default StorageClass. | +| PostgreSQL | **No** | This chart overrides the Bitnami `postgresql` 12.1.2 defaults with `postgresql.primary.persistence.enabled: false` and `postgresql.readReplicas.persistence.enabled: false`; the data directory is an `emptyDir`. It is a demo backend, as `values.yaml` says. | +| Horizon UI | No | `ui.persistence.enabled: false` (1Gi, `ReadWriteOnce` once enabled). Turn it on to keep BFF audit log, setup state and alarm state across restarts. | + +`oap.resources`, `ui.resources` and `satellite.resources` are all `{}` by default — no requests or +limits are set, so plan capacity yourself. For a laptop-sized cluster, drop the Elasticsearch node +count to 1 the way `test/e2e/values.yaml` does. + +## Install the ECK CRDs first + +**When you need this:** `elasticsearch.enabled=true`, which is the **default**. The +`eck-operator-crds` chart ships the CRDs as ordinary templates, so without this step they would be +created in the same install that also renders the `Elasticsearch` custom resource depending on +them. Install them as their own release first: + +```shell +helm repo add elastic https://helm.elastic.co +helm dep up chart/skywalking +tar xzf chart/skywalking/charts/eck-operator-3.3.1.tgz -C /tmp eck-operator/charts/eck-operator-crds +helm install eck-crds /tmp/eck-operator/charts/eck-operator-crds \ + -n "${SKYWALKING_RELEASE_NAMESPACE}" --create-namespace +``` + +Then install the chart with `eck-operator.installCRDs=false` so the operator does not try to create +the same cluster-scoped CRDs again: + +```shell +helm install skywalking chart/skywalking -n "${SKYWALKING_RELEASE_NAMESPACE}" \ + --set oap.image.tag=11.0.0 \ + --set oap.storageType=elasticsearch \ + --set ui.image.tag=horizon-1.0.0 \ + --set eck-operator.installCRDs=false +``` + +Pass `installCRDs=false` for the same reason if the ECK CRDs are already present because someone +else installed ECK in the cluster — they are global resources, and a second chart owning them would +overwrite them on upgrade and delete them on uninstall. + +The `eck-crds` release is independent of the SkyWalking release: `helm uninstall skywalking` leaves +the CRDs in place, and uninstalling `eck-crds` deletes every ECK-managed Elasticsearch in the +cluster. + +**When you do NOT need this:** any install with `elasticsearch.enabled=false` — no ECK operator, no +CRDs. Note that `elasticsearch.enabled` is the *only* switch: the dependency condition is +`elasticsearch.enabled`, not `oap.storageType`, so `--set oap.storageType=banyandb` on its own +still deploys ECK and an Elasticsearch cluster nobody uses. Turn it off explicitly: + +```yaml +elasticsearch: + enabled: false + # Only read when oap.storageType is elasticsearch — i.e. an external cluster. + config: + host: elasticsearch-es-http + port: + http: 9200 +``` + +## Required values + +Three values have no default and must be set on every install: + +| Value | Example | +|---|---| +| `oap.image.tag` | `11.0.0` | +| `oap.storageType` | `elasticsearch`, `postgresql`, `banyandb` | +| `ui.image.tag` | `horizon-1.0.0` | + +With `banyandb.enabled=true` the BanyanDB subchart adds a fourth — `banyandb.image.tag` (e.g. +`0.11.0`) — or rendering fails with `banyandb.image.tag is required`. + +## Extra prerequisites for the SWCK charts + +`chart/operator` needs [cert-manager](https://cert-manager.io/) for its admission webhook +certificates (`cert-manager.io/v1` `Certificate` and `Issuer`, plus +`cert-manager.io/inject-ca-from` on the webhook configurations). Install it yourself: + +```shell +kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.9.1/cert-manager.yaml +``` + +The chart also carries a cert-manager 1.9.1 subchart behind `cert-manager.enabled` (default +`false`) — that one is for quick starts only, not production. + +## Next + +- [Quick Start](../install/quick-start.md) +- [Where to Get the Chart](../install/chart-sources.md) +- [Version Compatibility](version-compatibility.md) +- [Elasticsearch](../storage/elasticsearch.md) · [BanyanDB](../storage/banyandb.md) +- [SWCK Operator](../operate/swck-operator.md) · [SWCK Metrics Adapter](../operate/swck-adapter.md) diff --git a/docs/evaluate/version-compatibility.md b/docs/evaluate/version-compatibility.md new file mode 100644 index 0000000..9ede702 --- /dev/null +++ b/docs/evaluate/version-compatibility.md @@ -0,0 +1,125 @@ +# Version Compatibility + +Which OAP, UI, storage and operator versions go together for chart **5.0.0**. Only two pairings are actually constrained — OAP↔BanyanDB (hard, OAP refuses to start) and OAP↔Horizon UI (soft, two settings) — everything else is a free choice within the ranges below. + +## The tested set + +This is the combination `test/e2e/` installs on every CI run, so it is the set the chart is known to work with. + +| Component | Version | Where you set it | +|---|---|---| +| SkyWalking OAP | `11.0.0` | `oap.image.tag` — **required**, no default | +| Horizon UI | `horizon-1.0.0` | `ui.image.tag` — **required**, no default | +| BanyanDB | `0.11.0` | `banyandb.image.tag` — **required** when `banyandb.enabled=true` | +| `skywalking-banyandb-helm` subchart | `0.7.0-rc1` | `chart/skywalking/Chart.yaml` dependency | +| Elasticsearch (ECK-managed) | `8.18.8` | `elasticsearch.version` | +| `eck-operator` / `eck-elasticsearch` charts | `3.3.1` / `0.18.1` | `chart/skywalking/Chart.yaml` dependencies | +| PostgreSQL (Bitnami chart `12.1.2`, demo only) | appVersion `15.1.0` | `chart/skywalking/Chart.yaml` dependency | +| Satellite | optional, `satellite.enabled=false` by default (CI enables it with a GHCR commit build, not a release tag) | `satellite.image.tag` | +| SWCK operator / adapter charts | `0.10.0` (image `apache/skywalking-swck:0.10.0`) | `chart/operator`, `chart/adapter` | +| cert-manager (operator chart dependency) | `1.9.1` | `chart/operator/Chart.yaml` | +| Kubernetes | `v1.28.15` for the main chart e2e (`test/e2e/kind28.yaml`); the SWCK e2e still runs `v1.21.14` (`test/e2e/kind.yaml`) | your cluster | + +```shell +helm install skywalking oci://registry-1.docker.io/apache/skywalking-helm \ + --version 5.0.0 \ + --set oap.image.tag=11.0.0 \ + --set oap.storageType=banyandb \ + --set ui.image.tag=horizon-1.0.0 \ + --set elasticsearch.enabled=false \ + --set banyandb.enabled=true \ + --set banyandb.image.tag=0.11.0 +``` + +> CI currently pins the BanyanDB image to a GHCR build of the exact `v0.11.0` commit rather than `docker.io/apache/skywalking-banyandb:0.11.0`, because the Docker Hub push for that tag has not been dispatched yet. See the comments in `test/e2e/env`. If `0.11.0` is not pullable from Docker Hub for you either, use the same GHCR pin. + +## OAP and BanyanDB are locked together + +OAP ships the list of BanyanDB **server API** versions it accepts in `config/bydb.yml`: + +```yaml +compatibleServerApiVersions: ${SW_STORAGE_BANYANDB_COMPATIBLE_SERVER_API_VERSIONS:"0.11"} +``` + +If the server advertises anything else, OAP does not start: + +``` +ERROR [] - ... Incompatible BanyanDB server API version: 0.x. But accepted versions: 0.y +org.apache.skywalking.oap.server.library.module.ModuleStartException: Incompatible BanyanDB server API version... +``` + +| OAP | Accepted server API | BanyanDB release | +|---|---|---| +| `11.0.0` | `0.11` | `0.11.x` | +| `10.4.0` | `0.10` | `0.10.x` | + +So **OAP 11 requires BanyanDB 0.11.x**. Pairing it with 0.10.x fails at boot, and there is no forward or backward slack — the value is a single API version, not a range. The API-version-to-release mapping is published upstream at [BanyanDB API versions](https://skywalking.apache.org/docs/skywalking-banyandb/latest/installation/versions/). + +Because of this, `oap.image.tag` and `banyandb.image.tag` must move in the same `helm upgrade`. Overriding `SW_STORAGE_BANYANDB_COMPATIBLE_SERVER_API_VERSIONS` via `oap.env` to force an unlisted pairing is not supported — see [BanyanDB](../storage/banyandb.md). + +## OAP and Horizon UI + +Horizon UI releases on its own cadence; there is no 1:1 version mapping with OAP. Pin `ui.image.tag=horizon-1.0.0` whichever OAP you run. + +| OAP | Horizon `1.0.0` | Extra configuration | +|---|---|---| +| `11.x` | Native — full feature set | None — the chart's defaults are correct | +| `10.3` – `10.4` | Partial — data plane only | `ui.config.templates.mode: readonly` **and** `oap.ports.admin: null` | +| `< 10.3` | Partial, with query gaps | As above; Horizon sends `queryTrace(..., duration)` (OAP 10.3+) and `findEndpoint(..., duration)` (OAP 10.2+) with no fallback | + +Against **OAP 10.x**, Horizon cannot read dashboard templates from OAP (the `/ui-management/templates*` admin REST API is an OAP 11 addition), so it must fall back to the templates bundled in its image: + +```yaml +ui: + config: + templates: + mode: readonly +``` + +Dashboards, traces, logs, topology, alarms and profiling all work over the query port. OAP 10 has no admin server at all — `admin-server` (`SW_ADMIN_SERVER_PORT`, default `17128`) and the modules that mount on it (`ui-management`, `receiver-runtime-rule`, `dsl-debugging`, `inspect`) first appear in OAP 11 — so Inspect, DSL Management, Live Debugger, the Alarm Rule editor and Cluster Status → Admin do not appear. Horizon probes each feature at boot rather than checking a version number. + +So on any 10.x OAP, also drop the admin port from the chart's Service and Deployment: + +```yaml +oap: + ports: + admin: null +``` + +With the port unset the chart omits `adminUrl` from `horizon.yaml` entirely, so Horizon simply never probes an admin host. Leaving the port at its default would be worse than useless on 10.x: 17128 there is the AI-pipeline URI-recognition server, not an admin API. See [Horizon UI in This Chart](../ui/horizon-ui.md). + +## The legacy booster UI is not an option + +`skywalking-booster-ui` (and `skywalking-rocketbot-ui` before it) is **not supported by this chart or by SkyWalking**. OAP 11.0.0 deleted `apm-webapp/` and the `skywalking-ui` submodule from the distribution, along with the `docker.ui` build target. + +- The last booster image published to `apache/skywalking-ui` is `10.4.0`. **There is no `11.x` tag and there will not be one** — every new tag in that repository is `horizon-x.y.z`. +- The OAP surfaces booster relied on are gone too: the `ui-initialized-templates` seed files, sidebar menu storage, the `UIConfigurationManagement` GraphQL mutations, and `SW_ENABLE_UPDATE_UI_TEMPLATE`. + +`ui.image.tag` must be a `horizon-*` tag. If you are upgrading from a chart release that set `ui.image.tag=`, see [Upgrading](../upgrade/upgrading.md) — Horizon also requires configured users, with no `admin/admin` fallback. + +## Storage backend version ranges + +The chart does not constrain these; OAP does. + +| Backend | Supported by OAP 11 | What the chart deploys | +|---|---|---| +| BanyanDB | `0.11.x` only (see above) | subchart, `banyandb.image.tag` required | +| Elasticsearch | 7.x, 8.x, 9.x | ECK-managed `8.18.8` (`elasticsearch.version`) | +| OpenSearch | 1.x, 2.x, 3.x (upstream tests 1.3.10, 2.4.0, 2.8.0, 3.0.0) | not deployed — connect as external ES | +| PostgreSQL | 8.2 or newer (JDBC driver 42.3.2) | Bitnami subchart, PG `15.1.0`, **demo only** | + +Details: [Elasticsearch](../storage/elasticsearch.md), [BanyanDB](../storage/banyandb.md), [PostgreSQL](../storage/postgresql.md), and the upstream [OAP storage docs](https://skywalking.apache.org/docs/main/latest/en/setup/backend/backend-storage/). + +## Kubernetes, Helm and SWCK + +- **Helm 3 or newer.** Chart `5.0.0` is `apiVersion: v2`, which rules out Helm 2; the chart README lists Helm 3 as the prerequisite. +- The main chart e2e runs on **kind `v1.28.15`** (`test/e2e/kind28.yaml`); the SWCK e2e tests still run kind `v1.21.14` (`test/e2e/kind.yaml`). The chart declares no `kubeVersion` constraint. +- The **SWCK operator and adapter** charts version independently of the main chart, both at `0.10.0`. Operator `0.10.0` is the release that added Horizon UI support, so pair it with `horizon-*` UI images. It needs [cert-manager](https://cert-manager.io/) installed first (the chart bundles `1.9.1` as an optional dependency). See [SWCK Operator](../operate/swck-operator.md). + +## Before you change a version + +1. Move `oap.image.tag` and `banyandb.image.tag` together, in one `helm upgrade`. +2. Keep `ui.image.tag` on a `horizon-*` tag; it can lag or lead OAP. +3. Elasticsearch and PostgreSQL upgrades are subchart concerns — follow ECK / Bitnami procedures, not this chart. + +If OAP crash-loops after a version change, [Install and Startup Failures](../troubleshooting/install-and-startup.md) starts with the incompatibility errors. diff --git a/docs/evaluate/what-this-chart-deploys.md b/docs/evaluate/what-this-chart-deploys.md new file mode 100644 index 0000000..747255b --- /dev/null +++ b/docs/evaluate/what-this-chart-deploys.md @@ -0,0 +1,157 @@ +# What This Chart Deploys + +What a `helm install` of `chart/skywalking` actually creates in your cluster, which pieces are +optional, and which value key turns each one on or off. + +## The three charts in this repository + +| chart | installs | relationship | +|---|---|---| +| `chart/skywalking` | OAP backend, Horizon UI, optional Satellite, optional storage backend | the main chart — everything below is about this one | +| `chart/operator` | the [SWCK](https://github.com/apache/skywalking-swck) operator (CRDs, controller, webhooks) | **separate install**, separate release | +| `chart/adapter` | the SWCK custom-metrics adapter for HPA | **separate install**, separate release | + +The `skywalking` chart does not depend on, install, or require either of the other two. Install +them on their own when you want them — see [SWCK Operator](../operate/swck-operator.md) and +[SWCK Metrics Adapter](../operate/swck-adapter.md). + +## A default install + +Three values have no defaults and must be supplied every time: `oap.image.tag`, +`oap.storageType`, `ui.image.tag`. + +```shell +helm install skywalking oci://registry-1.docker.io/apache/skywalking-helm \ + --version 5.0.0 -n skywalking --create-namespace \ + --set oap.image.tag=11.0.0 \ + --set oap.storageType=elasticsearch \ + --set ui.image.tag=horizon-1.0.0 +``` + +With everything else left at its default (`elasticsearch.enabled=true`), that release contains: + +| kind | name (release `skywalking`) | notes | +|---|---|---| +| Deployment | `skywalking-skywalking-helm-oap` | `oap.replicas: 2`, runs `-Dmode=no-init` | +| Job | `skywalking-skywalking-helm-oap-init-` | one-shot, runs `-Dmode=init` to create the storage schema | +| Service | `skywalking-skywalking-helm-oap` | ClusterIP; ports `11800`, `12800`, `17128` | +| Deployment | `skywalking-skywalking-helm-ui` | `ui.replicas: 1`, `strategy: Recreate` | +| Service | `skywalking-skywalking-helm-ui` | ClusterIP, `80` → container `8081` | +| ConfigMap | `skywalking-skywalking-helm-ui` | the `horizon.yaml` mounted at `/app/horizon.yaml` | +| ServiceAccount | `skywalking-skywalking-helm-oap` | used by both the OAP Deployment and the init Job | +| Role + RoleBinding | `skywalking-skywalking-helm` | `get/watch/list` on pods, configmaps | +| ClusterRole + ClusterRoleBinding | `skywalking-skywalking-helm` | `get/watch/list` on pods, pods/log, endpoints, services, nodes, namespaces, configmaps, deployments, replicasets, Istio `serviceentries` | +| ECK operator | `elastic-operator` StatefulSet, webhook Service/Secret, CRDs, RBAC | from the `eck-operator` subchart | +| Elasticsearch | `skywalking-elasticsearch` | ECK custom resource, version `8.18.8`, one nodeSet of `3` | + +Resource names are `-` (`skywalking-helm`) plus a component suffix, unless you +set `nameOverride` / `fullnameOverride`. + +> The `eck-operator` subchart carries the ECK CRDs (`eck-operator.installCRDs`, default `true`), +> but they are ordinary templates — Helm cannot apply a CRD and an `Elasticsearch` custom resource +> of that kind in the same release. On a cluster that has never run ECK, install the CRDs first and +> then add `--set eck-operator.installCRDs=false`. Steps in +> [Elasticsearch](../storage/elasticsearch.md). + +## Components and their switches + +| component | value key | default | what it renders | +|---|---|---|---| +| OAP Deployment + Service | — | always | `oap-deployment.yaml`, `oap-svc.yaml` | +| OAP init Job | — | always | `oap-init.job.yaml`, a normal release resource (not a Helm hook) | +| OAP ServiceAccount + RBAC | `serviceAccounts.oap.create` | `true` | ServiceAccount, Role, RoleBinding, ClusterRole, ClusterRoleBinding | +| OAP config-override ConfigMap | `oap.config` | `{}` (nothing) | `-oap-cm-override`, mounted into `/skywalking/config` | +| OAP dynamic-config ConfigMap | `oap.dynamicConfig.enabled` | `false` | `skywalking-dynamic-config` + `SW_CONFIGURATION=k8s-configmap` | +| Horizon UI Deployment, Service, ConfigMap | `ui.enabled` | `true` | all UI resources; `false` deploys OAP only | +| UI Ingress | `ui.ingress.enabled` | `false` | `ui-ingress.yaml` | +| UI PersistentVolumeClaim | `ui.persistence.enabled` | `false` | `-ui-data` for `/data`; otherwise an `emptyDir` | +| Satellite | `satellite.enabled` | `false` | Deployment, Service, ServiceAccount, Role, RoleBinding | +| Satellite config override | `satellite.config` | `{}` (nothing) | `-satellite-cm-override` | +| Elasticsearch (ECK operator + cluster) | `elasticsearch.enabled` | `true` | `eck-operator` and `eck-elasticsearch` subcharts | +| PostgreSQL | `postgresql.enabled` | `false` | Bitnami `postgresql` subchart (demo only, no persistence by default) | +| BanyanDB | `banyandb.enabled` | `false` | `skywalking-banyandb-helm` subchart; also needs `banyandb.image.tag` | + +`elasticsearch.enabled` and `oap.storageType` are independent. If you switch `oap.storageType` to +`banyandb` or `postgresql`, also set `elasticsearch.enabled=false` — otherwise the chart still +deploys an Elasticsearch cluster that nothing uses. + +## What OAP gets out of the box + +- **Init container.** Every OAP pod and the init Job start with a `wait-for-storage` init container + that blocks until the backend answers: `busybox:1.30` + `nc` for Elasticsearch, `postgres:13` + + `pg_isready` for PostgreSQL, `curlimages/curl` against `/api/healthz` for BanyanDB. +- **Two modes.** The Deployment runs `-Dmode=no-init` and keeps port `12800` closed until the + schema exists; the Job runs `-Dmode=init` and creates it. See + [The OAP Init Job](../operate/oap-init-job.md). +- **Cluster coordination.** `SW_CLUSTER=kubernetes` with `SW_CLUSTER_K8S_NAMESPACE` and a label + selector — that is what the ServiceAccount and RBAC above are for. +- **JVM.** `oap.javaOpts` defaults to `-Xmx2g -Xms2g`; `oap.resources` is empty (no requests or + limits) by default. +- **Storage env.** `SW_STORAGE` plus the backend-specific variables are derived from + `oap.storageType` and whether the backend is embedded or external. +- **Probes.** Liveness/readiness are TCP on `12800`; the startup probe allows `30 × 10s` so a cold + start can wait for the init Job. + +## Ports + +| service | port | value key | purpose | +|---|---|---|---| +| OAP | `11800` | `oap.ports.grpc` | agent / Satellite gRPC ingest | +| OAP | `12800` | `oap.ports.rest` | HTTP ingest + GraphQL query | +| OAP | `17128` | `oap.ports.admin` | admin REST (`/status/*`, `/debugging/*`, template store); required by Horizon | +| UI | `80` → `8081` | `ui.service.externalPort` / `ui.service.internalPort` | Horizon BFF | +| Satellite | `11800` | `satellite.ports.grpc` | agent gRPC ingest | +| Satellite | `1234` | `satellite.ports.prometheus` | Satellite self-telemetry | + +`oap.ports` is a free-form map: add `zipkin-receiver`, `zipkin-query`, `promql`, `logql`, `traceql` +or `metrics` entries and they are opened on both the container and the Service. Setting +`zipkin-receiver` / `zipkin-query` additionally sets the matching `SW_RECEIVER_ZIPKIN*` / +`SW_QUERY_ZIPKIN*` environment variables, and `zipkin-query` makes the chart wire Horizon's +`oap.zipkinUrl`. + +## Optional: Satellite + +A gateway that fronts OAP for agent traffic. Off by default; it needs its own image tag. + +```shell +helm install skywalking oci://registry-1.docker.io/apache/skywalking-helm \ + --version 5.0.0 -n skywalking \ + --set oap.image.tag=11.0.0 \ + --set oap.storageType=elasticsearch \ + --set ui.image.tag=horizon-1.0.0 \ + --set satellite.enabled=true \ + --set satellite.image.tag=v0.4.0 +``` + +Agents then point at the Satellite Service instead of the OAP one. See +[Satellite Gateway](../operate/satellite.md). + +## Optional: an embedded storage backend + +| backend | enable | what lands in the release | +|---|---|---| +| Elasticsearch | `elasticsearch.enabled=true` (default) | ECK operator + an `Elasticsearch` CR (`8.18.8`, 3 nodes) | +| BanyanDB | `banyandb.enabled=true` + `banyandb.image.tag` | StatefulSet, ServiceAccount, a `-grpc` ClusterIP Service (`17912`) and a `-http` Service (`17913`, type `LoadBalancer` by default) | +| PostgreSQL | `postgresql.enabled=true` | Bitnami PostgreSQL — demo defaults, persistence disabled | + +Set the corresponding `*.enabled` to `false` and fill in `*.config.*` to point OAP at an existing +external instance instead. Details per backend: +[Elasticsearch](../storage/elasticsearch.md), [BanyanDB](../storage/banyandb.md), +[PostgreSQL](../storage/postgresql.md), or start at +[Pick a Storage Backend](../storage/choose-a-backend.md). + +## What this chart does *not* deploy + +- **Any login for the UI.** Horizon ships no default credentials and does not fail closed — the pod + goes Ready and nobody can sign in. Configure users: [Set Up Logins](../ui/logins.md). +- **Agents or instrumentation.** You install those with your applications. +- **An ingress controller, cert-manager, or storage class.** See + [Requirements](requirements.md). +- **The SWCK operator or metrics adapter** — separate charts in this repo, installed separately. +- **Storage you sized yourself.** The chart's ES nodeSet ships without `volumeClaimTemplates`, so + ECK falls back to its own default claim (1Gi on the default StorageClass) — enough to start, not + enough to keep. The PostgreSQL subchart is set to `primary.persistence.enabled: false`, so that + one really is ephemeral. + +Every value in the table above, with its full description, is in the +[skywalking chart values reference](../reference/skywalking-chart-values.md). diff --git a/docs/expose/oap-endpoints.md b/docs/expose/oap-endpoints.md new file mode 100644 index 0000000..da72050 --- /dev/null +++ b/docs/expose/oap-endpoints.md @@ -0,0 +1,158 @@ +# OAP Endpoints for Agents + +Which OAP ports this chart opens, how to open more (Zipkin, PromQL, LogQL, TraceQL, self-telemetry), and what agents, `swctl` and Grafana should point at. + +## One map drives everything + +`chart/skywalking/templates/oap-svc.yaml` turns every key in `oap.ports` into a Service port: + +```yaml + ports: + {{- range $key, $value := .Values.oap.ports }} + - port: {{ $value }} + name: {{ $key }} + {{- end }} +``` + +`oap-deployment.yaml` ranges over the same map for `containerPort`. So **adding a key to `oap.ports` is how you expose a port** — there is no separate switch. The Service has no `targetPort`, so the Service port and the container port are always the same number. + +The Service is named `-oap`, type `ClusterIP` by default. The chart is named `skywalking-helm`, so `helm install skywalking ...` renders `skywalking-skywalking-helm-oap`; add `--set fullnameOverride=skywalking` (what the e2e tests do) to get a plain `skywalking-oap`: + +```shell +helm dep up chart/skywalking +helm template skywalking chart/skywalking \ + --set oap.image.tag=11.0.0 \ + --set oap.storageType=banyandb \ + --set ui.image.tag=horizon-1.0.0 \ + -s templates/oap-svc.yaml +``` + +## The ports + +| Key | Default | Serves | In `oap.ports` by default | Needs OAP env too | +|---|---|---|---|---| +| `grpc` | `11800` | Native agent protocol (traces, JVM/meters, profiling, logs, events). Every other gRPC receiver — OTLP, Envoy ALS, Rover eBPF, Satellite forwarding — shares it, because `SW_RECEIVER_GRPC_PORT` defaults to `0` (fall back to the core gRPC server) | yes | no | +| `rest` | `12800` | GraphQL query API at `/graphql` (Horizon's `oap.queryUrl`, `swctl`). HTTP receivers share it for the same reason (`SW_RECEIVER_SHARING_REST_PORT:0`). All three pod probes `tcpSocket` this port | yes | no | +| `admin` | `17128` | admin-server: `/status/*`, `/debugging/*`, inspect, dsl-debugging, runtime-rule, and the `ui-management` template store Horizon uses when `ui.config.templates.mode` is `live`. Wired into Horizon as `oap.adminUrl` | yes | no | +| `zipkin-receiver` | `9411` | Zipkin span ingestion, `POST /api/v2/spans` | no | no — the chart sets `SW_RECEIVER_ZIPKIN=default` and `SW_RECEIVER_ZIPKIN_REST_PORT` for you | +| `zipkin-query` | `9412` | Zipkin query API under `/zipkin` (Zipkin-Lens compatible). Also makes the chart write Horizon's `oap.zipkinUrl` | no | no — the chart sets `SW_QUERY_ZIPKIN=default` and `SW_QUERY_ZIPKIN_REST_PORT` | +| `promql` | `9090` | Prometheus-compatible query API (Grafana datasource) | no | no — the `promql` module is on by default in OAP 11 | +| `logql` | `3100` | Loki-compatible log query API | no | no — the `logql` module is on by default in OAP 11 | +| `traceql` | `3200` | Tempo-compatible trace query API | no | **yes** — `SW_TRACEQL` defaults to empty (module off) | +| `metrics` | `1234` | OAP self-telemetry, Prometheus exposition for scraping | no | no — `SW_TELEMETRY` defaults to `prometheus` on `1234` | +| `zabbix` | `10051` | Zabbix receiver (the example line in `values.yaml`) | no | **yes** — `SW_RECEIVER_ZABBIX` defaults to empty | + +"In `oap.ports` by default" means the chart publishes the Service/container port. The last column is about OAP itself: a port that is published but whose module is off answers nothing. + +`admin: 17128` is required for OAP 11 and is what Horizon's admin calls go to. Set it to `null` on any OAP 10.x release: `admin-server` first appears in OAP 11, and on 10.x port 17128 is the AI-pipeline URI-recognition server. + +## Opening more ports + +Uncomment the lines already present in `values.yaml`, or pass them on the command line: + +```shell +helm install skywalking oci://registry-1.docker.io/apache/skywalking-helm \ + --version 5.0.0 \ + --set oap.image.tag=11.0.0 \ + --set oap.storageType=banyandb \ + --set ui.image.tag=horizon-1.0.0 \ + --set oap.ports.zipkin-receiver=9411 \ + --set oap.ports.zipkin-query=9412 +``` + +For a module that is off in OAP, add the port *and* the env — `oap.env` values are rendered quoted, so booleans are safe: + +```yaml +oap: + ports: + grpc: 11800 + rest: 12800 + admin: 17128 + traceql: 3200 + metrics: 1234 + env: + SW_TRACEQL: default + SW_TRACEQL_ENABLE_DATASOURCE_SKYWALKING: true +``` + +Port names are Kubernetes port names: lowercase, `[a-z0-9-]`, 15 characters max (`zipkin-receiver` is exactly 15). + +### Do not renumber `grpc` / `rest` / `admin` on their own + +The chart feeds `oap.ports` to Kubernetes, not to OAP — for these three it never sets the matching `SW_*` variable. OAP keeps binding `SW_CORE_GRPC_PORT:11800`, `SW_CORE_REST_PORT:12800` and `SW_ADMIN_SERVER_PORT:17128`, so changing only the value gives you a Service pointing at a closed port. If you must move one, change all of it: + +```yaml +oap: + ports: + rest: 12801 + env: + SW_CORE_REST_PORT: 12801 + livenessProbe: + tcpSocket: { port: 12801 } + initialDelaySeconds: 5 + periodSeconds: 10 + readinessProbe: + tcpSocket: { port: 12801 } + initialDelaySeconds: 5 + periodSeconds: 10 + startupProbe: + tcpSocket: { port: 12801 } + failureThreshold: 30 + periodSeconds: 10 +``` + +The default probes hardcode `12800`; leaving them behind makes the pod restart-loop. Horizon needs no change either way — the UI ConfigMap renders `oap.queryUrl` / `oap.adminUrl` from `oap.ports.rest` / `oap.ports.admin`. `zipkin-receiver` and `zipkin-query` have no such problem — the chart derives their env from the value. + +## What to point at what + +Assuming release `skywalking` in namespace `skywalking`, installed with `--set fullnameOverride=skywalking` so the Service is `skywalking-oap`. Without that override the name is `skywalking-skywalking-helm-oap` — substitute it everywhere below. + +| Client | Address | +|---|---| +| Java / Go / Python / Node.js / Rust / nginx-lua agents | `skywalking-oap.skywalking.svc:11800` (gRPC, no scheme) | +| OTLP, Envoy ALS, Rover eBPF | the same `:11800` | +| Browser and other HTTP reporters | `http://skywalking-oap.skywalking.svc:12800` | +| `swctl` | `--base-url=http://skywalking-oap.skywalking.svc:12800/graphql` | +| Horizon UI | wired by the chart: query `:12800`, admin `:17128` | +| Zipkin senders | `http://skywalking-oap.skywalking.svc:9411/api/v2/spans` | +| Grafana (Prometheus / Loki datasource) | `http://skywalking-oap.skywalking.svc:9090`, `:3100` | +| Prometheus scrape of OAP itself | `skywalking-oap.skywalking.svc:1234` | + +Java agent example: + +```shell +-Dskywalking.collector.backend_service=skywalking-oap.skywalking.svc:11800 +``` + +Horizon does **not** proxy `/graphql` to OAP, so anything that used to query through the UI must talk to the OAP Service directly on `12800`. + +If [Satellite](../operate/satellite.md) is enabled, agents point at the Satellite Service on `11800` instead and Satellite forwards to OAP. + +## Reaching OAP from outside the cluster + +The chart ships no Ingress for OAP — only the UI has one (see [UI Service and Ingress](ui-service-and-ingress.md)). Options: + +```shell +# one laptop, temporary +kubectl port-forward -n skywalking svc/skywalking-oap 12800:12800 +``` + +```yaml +# agents outside the cluster +oap: + service: + type: LoadBalancer + annotations: + service.beta.kubernetes.io/aws-load-balancer-type: nlb +``` + +`oap.service.type` accepts any Service type (`ClusterIP`, `NodePort`, `LoadBalancer`) and `oap.service.annotations` is passed straight through to the Service metadata. The agent protocol on `11800` is gRPC over HTTP/2, so an L7 HTTP ingress will not do — use an L4 / NLB Service or a gRPC-aware gateway. + +A public OAP Service publishes **every** key in `oap.ports`, and `admin: 17128` is one of them by default — it serves `/status/*`, `/debugging/*`, inspect and the runtime-rule write APIs. Front the public address with something that forwards only `11800` / `12800` rather than dropping `admin` from `oap.ports`: the UI ConfigMap still renders `oap.adminUrl` as `:17128`, so removing the port breaks Horizon. See [TLS](tls.md) before putting any of these on the internet. + +## Related + +- [Configure OAP](../operate/oap-configuration.md) — `oap.env`, `oap.config`, dynamic configuration +- [Satellite Gateway](../operate/satellite.md) +- [skywalking Chart values](../reference/skywalking-chart-values.md) +- [OAP Configuration Vocabulary](https://skywalking.apache.org/docs/main/latest/en/setup/backend/configuration-vocabulary/) — every `SW_*` variable and its default diff --git a/docs/expose/tls.md b/docs/expose/tls.md new file mode 100644 index 0000000..135f7c7 --- /dev/null +++ b/docs/expose/tls.md @@ -0,0 +1,279 @@ +# TLS + +How to serve the Horizon UI over HTTPS: create the certificate Secret, wire it into +`ui.ingress.tls`, and tell Horizon it is now behind TLS so session cookies are marked +`Secure`. + +## What terminates TLS + +The chart does not serve HTTPS itself. The Horizon BFF container listens on plain HTTP +(`ui.service.internalPort`, default `8081`) and the Service forwards +`ui.service.externalPort` (default `80`) to it. TLS terminates at your **ingress +controller**, using a Kubernetes TLS Secret that you reference from `ui.ingress.tls`. + +A cloud load balancer can terminate it instead — `ui.service.type: LoadBalancer` plus a +cert annotation in `ui.service.annotations` (values.yaml shows the AWS ACM one). That path +skips the Ingress, so the chart derives no `server.publicUrl` for you and you set it in +`ui.config.server.publicUrl` yourself; the rest of this page still applies. + +So "enable TLS" is two steps that must both happen: + +1. Give the ingress a certificate (`ui.ingress.tls`). +2. Tell Horizon it is being reached over `https` (`ui.config.session.cookieSecure`, and + usually `ui.config.server.trustProxy`). + +Step 2 is not automatic. Skipping it still leaves a working UI — the session cookie is +`SameSite=strict`, so the browser keeps sending it over HTTPS — but the cookie carries no +`Secure` flag and would go out in clear on any plain-HTTP request to the same host. See +[Mark session cookies Secure](#mark-session-cookies-secure). + +## Create the TLS Secret + +The Secret must live in the release namespace and be of type `kubernetes.io/tls`. + +```shell +kubectl create secret tls skywalking-tls \ + -n "${SKYWALKING_RELEASE_NAMESPACE}" \ + --cert=path/to/tls.crt \ + --key=path/to/tls.key +``` + +If you run [cert-manager](https://cert-manager.io/docs/), do not create the Secret by +hand — name it in `ui.ingress.tls[].secretName` and let the issuer fill it in. The chart +renders `ui.ingress.annotations` onto the Ingress verbatim, so the usual annotations work: + +```yaml +ui: + ingress: + enabled: true + annotations: + cert-manager.io/cluster-issuer: letsencrypt-prod + hosts: + - skywalking.example.com + tls: + - secretName: skywalking-tls + hosts: + - skywalking.example.com +``` + +## Wire `ui.ingress.tls` + +`ui.ingress.tls` is passed through to the Ingress `spec.tls` list unchanged, so it takes +the standard Kubernetes shape — a list of `{secretName, hosts}` entries. Default is `[]`. + +```yaml +ui: + ingress: + enabled: true + hosts: + - skywalking.example.com + tls: + - secretName: skywalking-tls + hosts: + - skywalking.example.com +``` + +Rendering that gives: + +```yaml +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: my-release-skywalking-helm-ui +spec: + rules: + - host: skywalking.example.com + http: + paths: + - path: / + backend: + service: + name: my-release-skywalking-helm-ui + port: + number: 80 + pathType: Prefix + tls: + - hosts: + - skywalking.example.com + secretName: skywalking-tls +``` + +The hostnames in `ui.ingress.tls[].hosts` should match the entries in `ui.ingress.hosts`, +or the controller will not find a certificate for the request's `Host`. + +## What setting `ui.ingress.tls` changes on its own + +One thing, and it is in the ConfigMap rather than the Ingress: + +| | | +|---|---| +| `server.publicUrl` | written only when `ui.ingress.enabled` **and** `ui.ingress.hosts` is non-empty; derived from the **first** `ui.ingress.hosts` entry, with scheme `https` when `ui.ingress.tls` is non-empty and `http` otherwise | + +Horizon uses `publicUrl` to build SSO callbacks and as its OAuth issuer, so an `http://` +issuer on an HTTPS deployment breaks logins. Setting `ui.ingress.tls` flips it for you: + +```yaml + server: + port: 8081 + publicUrl: ${HORIZON_PUBLIC_URL:https://skywalking.example.com} +``` + +Everything else about TLS you set yourself. + +## Mark session cookies Secure + +`session.cookieSecure` tells the BFF to set the `Secure` attribute on the session cookie. +Horizon's default is `false`, and the BFF logs a warning at boot when it is `false` outside +development. + +Set it to `true` once you serve over HTTPS: + +```yaml +ui: + config: + session: + cookieSecure: true +``` + +**Set it in `ui.config`, not as an env var.** The `HORIZON_SESSION_COOKIE_SECURE` +environment variable is inert under this chart. The chart mounts its own generated +`horizon.yaml` over `/app/horizon.yaml`, and that file omits the `session:` block +entirely. Horizon expands `${...}` over the raw *text* of the config file, so with no +`session:` block there is no token to expand and the schema default (`false`) wins — +`ui.extraEnv` will not change it. This is the general rule for the chart's ConfigMap and +is covered in [Configure Horizon](../ui/configure.md). + +If you still want the value overridable by env — for example to keep one values file for +both an HTTP dev cluster and an HTTPS production one — write the token yourself, quoted so +Helm keeps it a string: + +```yaml +ui: + config: + session: + cookieSecure: "${HORIZON_SESSION_COOKIE_SECURE:true}" +``` + +which renders into `horizon.yaml` as an expandable token that still defaults to `true`: + +```yaml + session: + cookieSecure: ${HORIZON_SESSION_COOKIE_SECURE:true} +``` + +### Why it matters + +Browsers refuse to store or send a `Secure` cookie over plain HTTP. The two failure modes +are symmetric: + +| Setting | Served over | Result | +|---|---|---| +| `cookieSecure: true` | `http://` | Browser drops the cookie. Login "succeeds" and immediately bounces back to the login page. | +| `cookieSecure: false` | `https://` | Login works, but the session cookie has no `Secure` flag and would be sent in clear on any HTTP request to the same host. | + +So flip `cookieSecure` in the same change that adds `ui.ingress.tls`, and flip it back if +you drop to plain HTTP. A `kubectl port-forward` to `http://localhost` is the usual +exception — most browsers treat localhost as a secure origin and keep the cookie — but if +a port-forward login bounces you straight back to the login page, `cookieSecure` is the +first thing to check. + +## Record the real client address + +Behind an ingress, every request appears to come from the ingress. To make the login audit +record the actual client, set `server.trustProxy`: + +```yaml +ui: + config: + server: + trustProxy: 1 # one proxy in front; or an address / CIDR +``` + +Use a hop count (`1` = a single proxy in front of the BFF) or the ingress address/CIDR. +`trustProxy: true` is **refused at boot** — it would trust the whole `X-Forwarded-For` +header, letting any caller choose the address that gets recorded. A `/0` block is refused +for the same reason, and a hostname is refused because the underlying server accepts only +addresses. + +`server.trustProxy` is read once when the HTTP server is constructed, so it takes effect +only on pod restart. + +## Full example + +```yaml +# tls-values.yaml +ui: + ingress: + enabled: true + annotations: + cert-manager.io/cluster-issuer: letsencrypt-prod + hosts: + - skywalking.example.com + tls: + - secretName: skywalking-tls + hosts: + - skywalking.example.com + config: + session: + cookieSecure: true + server: + trustProxy: 1 +``` + +```shell +helm upgrade --install "${SKYWALKING_RELEASE_NAME}" \ + oci://registry-1.docker.io/apache/skywalking-helm \ + --version 5.0.0 \ + -n "${SKYWALKING_RELEASE_NAMESPACE}" \ + --set oap.image.tag=11.0.0 \ + --set oap.storageType=elasticsearch \ + --set ui.image.tag=horizon-1.0.0 \ + --set eck-operator.installCRDs=false \ + -f tls-values.yaml +``` + +`helm upgrade` does not carry `--set` values over from the previous release, so repeat +every flag the original install used — the storage ones included. The flags above match +the Elasticsearch install in [Quick Start](../install/quick-start.md); a BanyanDB or +PostgreSQL release has its own set (see [Pick a Storage Backend](../storage/choose-a-backend.md)). + +Changing `ui.config` changes the UI ConfigMap, and the Deployment carries a +`checksum/config` annotation over it, so the UI pod is recreated on upgrade. Horizon's BFF +keeps its session table in memory, so everyone logged in is logged out by that restart. + +## Verify + +```shell +# The Ingress advertises the secret +kubectl get ingress -n "${SKYWALKING_RELEASE_NAMESPACE}" \ + -o jsonpath='{.items[*].spec.tls}' + +# The Secret exists and is a TLS secret +kubectl get secret skywalking-tls -n "${SKYWALKING_RELEASE_NAMESPACE}" \ + -o jsonpath='{.type}' + +# horizon.yaml has the https publicUrl and Secure cookies +kubectl get configmap "${SKYWALKING_RELEASE_NAME}-skywalking-helm-ui" \ + -n "${SKYWALKING_RELEASE_NAMESPACE}" \ + -o jsonpath='{.data.horizon\.yaml}' + +# The certificate served on the wire +curl -vI https://skywalking.example.com 2>&1 | grep -i 'subject\|issuer' +``` + +## What this page does not cover + +- **TLS on the OAP endpoints** (gRPC `11800`, REST `12800`) for agents. Those are plain + Services, not an Ingress — see [OAP Endpoints for Agents](oap-endpoints.md). +- **TLS between OAP and storage.** Elasticsearch HTTP TLS is disabled by default in this + chart (`elasticsearch.http.tls.selfSignedCertificate.disabled: true`) so OAP can connect + without trusting the self-signed certificate — see + [Elasticsearch](../storage/elasticsearch.md). + +## Related + +- [UI Service and Ingress](ui-service-and-ingress.md) — service types, hosts, paths +- [Configure Horizon](../ui/configure.md) — how `ui.config` and `HORIZON_*` env vars interact +- [Set Up Logins](../ui/logins.md) — local users, SSO, and the `publicUrl` an SSO callback needs +- [UI and Login Problems](../troubleshooting/ui-and-login.md) +- [`horizon.yaml` reference](https://github.com/apache/skywalking-horizon-ui/blob/main/docs/setup/horizon-yaml.md) diff --git a/docs/expose/ui-service-and-ingress.md b/docs/expose/ui-service-and-ingress.md new file mode 100644 index 0000000..ea3fb1d --- /dev/null +++ b/docs/expose/ui-service-and-ingress.md @@ -0,0 +1,127 @@ +# UI Service and Ingress + +How to reach the Horizon UI that this chart deploys: the Service it creates, the four ways to get traffic to it, and what changes in Horizon's own config once you put an Ingress in front. + +## The Service + +`templates/ui-svc.yaml` creates one Service named `--ui`, selecting the UI pod. It maps `ui.service.externalPort` → `ui.service.internalPort`: + +```yaml +ui: + service: + type: ClusterIP + externalPort: 80 # the port on the Service + internalPort: 8081 # the port the Horizon BFF binds +``` + +`internalPort` is load-bearing in three places at once — the Service `targetPort`, the container's `containerPort` (named `page`), and `server.port` in the generated `horizon.yaml`. Change it and all three move together; the liveness/readiness probes in `values.yaml` are hardcoded to `8081`, so change those too if you move it. + +| Value | Default | Notes | +|---|---|---| +| `ui.service.type` | `ClusterIP` | `ClusterIP`, `NodePort` or `LoadBalancer` | +| `ui.service.externalPort` | `80` | Service port; also the Ingress backend port | +| `ui.service.internalPort` | `8081` | Horizon BFF listen port | +| `ui.service.clusterIP` | unset | Only applied when `type: ClusterIP` | +| `ui.service.nodePort` | unset | Only applied when `type: NodePort`; auto-allocated if unset | +| `ui.service.loadBalancerIP` | unset | Rendered whenever set — the template does not check the type | +| `ui.service.loadBalancerSourceRanges` | unset | List of CIDRs | +| `ui.service.externalIPs` | unset | List of IPs | +| `ui.service.annotations` | `{}` | e.g. cloud load-balancer / TLS-cert annotations | +| `ui.service.portName` | unset | The Service port is unnamed unless you set this | + +Setting `ui.enabled: false` skips the Deployment, Service, Ingress, ConfigMap and PVC entirely — nothing on this page applies then. + +## Reaching the UI without an Ingress + +```shell +# ClusterIP (default) — port-forward +kubectl port-forward svc/-skywalking-helm-ui 8080:80 -n +# then open http://127.0.0.1:8080 + +# NodePort +kubectl get svc -skywalking-helm-ui -n \ + -o jsonpath='{.spec.ports[0].nodePort}' + +# LoadBalancer +kubectl get svc -skywalking-helm-ui -n \ + -o jsonpath='{.status.loadBalancer.ingress[0].ip}' +``` + +`helm status ` prints the same commands, filled in for your release. + +Note that the UI does **not** proxy `/graphql` to OAP. Tools like `swctl` must talk to the OAP Service directly — see [OAP Endpoints for Agents](oap-endpoints.md). + +## The Ingress + +```yaml +ui: + ingress: + enabled: true + annotations: + kubernetes.io/ingress.class: nginx + hosts: + - skywalking.example.com + tls: + - secretName: skywalking-tls + hosts: + - skywalking.example.com +``` + +Renders one Ingress named `--ui`, one rule per `hosts` entry, backed by the UI Service on `ui.service.externalPort`. + +Three things to know: + +- **The path comes from the host entry, not from `ui.ingress.path`.** Each entry is split on `/`: `skywalking.example.com` gives path `/`, and `skywalking.example.com/ui` gives host `skywalking.example.com` with path `/ui` (`pathType: Prefix`). `ui.ingress.path` is only used in the post-install `helm status` output — it does not reach the Ingress object. +- **There is no `ingressClassName` field.** Select a controller with the `kubernetes.io/ingress.class` annotation under `ui.ingress.annotations`. +- **The API version is picked from the cluster** — `networking.k8s.io/v1`, else `v1beta1`, else `extensions/v1beta1`. Offline `helm template` has no cluster to ask and falls back to `extensions/v1beta1`; add `--api-versions networking.k8s.io/v1/Ingress` when rendering locally to see what a modern cluster gets. + +## Enabling the Ingress rewrites `server.publicUrl` + +When `ui.ingress.enabled` is true **and** `ui.ingress.hosts` is non-empty, `templates/ui-configmap.yaml` adds `server.publicUrl` to the generated `horizon.yaml`, built from the **first** host entry — scheme `https` when `ui.ingress.tls` is non-empty, `http` otherwise: + +```yaml +# rendered horizon.yaml, with hosts: [skywalking.example.com] and a tls entry +server: + port: 8081 + publicUrl: ${HORIZON_PUBLIC_URL:https://skywalking.example.com} +``` + +Why the chart bothers: with `publicUrl` blank, Horizon derives its public base URL **per request**, from whatever `Host` and scheme reached the BFF. Behind an ingress that terminates TLS and rewrites `Host`, that is the internal address — so SSO redirect/callback URLs and the OAuth issuer Horizon advertises point at a hostname the browser and the identity provider cannot use. Pinning it once at deploy time makes those stable. + +Details worth knowing: + +- The host entry is used **verbatim**, path included: `hosts: [skywalking.example.com/ui]` yields `publicUrl: http://skywalking.example.com/ui`. +- The scheme is decided by `ui.ingress.tls` alone. TLS terminated further out (a cloud LB in front of an ingress with no `tls:` block) still renders `http://` — set the value yourself in that case. +- It is written as a `${HORIZON_PUBLIC_URL:...}` token, so the derived value is only a default: `ui.extraEnv`/`ui.envFromSecret` can still override it at runtime. +- To pin it in the chart instead, set `ui.config.server.publicUrl` — a literal there wins over the derived default, and makes `HORIZON_PUBLIC_URL` inert. See [Configure Horizon](../ui/configure.md). + +Also set `ui.config.session.cookieSecure: true` when you serve the UI over HTTPS, so the session cookie is not sent in the clear. The `HORIZON_SESSION_COOKIE_SECURE` env var will not do it — the chart's `horizon.yaml` omits the `session:` block, so there is no token to expand. See [TLS](tls.md). + +## `server.trustProxy` and the client address + +Horizon's login audit records the client address of each attempt. Behind an ingress every request arrives from the proxy, so without `trustProxy` the audit records the ingress address for everyone. + +```yaml +ui: + config: + server: + trustProxy: 1 # one proxy hop in front of the BFF +``` + +Accepted forms are a **hop count** (`1` = one proxy in front, `2` = a proxy behind a proxy) or the ingress **address / CIDR** (e.g. `10.0.0.0/8`, comma-separated for several). + +`true` is **refused at boot** — the pod will not start. Blanket trust would let any caller set `X-Forwarded-For` and choose the address written to the audit log, so Horizon requires you to name how far the trust extends. Pick the hop count or CIDR that matches your actual topology; too large a hop count has the same problem as `true`. + +Set it in `ui.config`, not via `ui.extraEnv`. `HORIZON_TRUST_PROXY` is inert under this chart: the generated `horizon.yaml` has no `server.trustProxy` line, so there is no `${...}` token for Horizon to expand and the schema default (`false`) wins. Write `trustProxy: "${HORIZON_TRUST_PROXY:1}"` under `ui.config.server` if you want it env-overridable. `server.trustProxy` is read once when the HTTP server is constructed, so it only takes effect on a pod restart — the Deployment's `checksum/config` annotation makes `helm upgrade` do that for you. + +## Replicas and sticky sessions + +The Horizon BFF holds its session table **in memory**, and the Deployment uses `strategy: Recreate` for that reason. `ui.replicas` defaults to `1`; raising it without sticky routing on your ingress breaks logins on requests that land on the other pod. Keep it at `1` unless your ingress controller is configured for session affinity. + +## Related + +- [Horizon UI in This Chart](../ui/horizon-ui.md) +- [Set Up Logins](../ui/logins.md) +- [TLS](tls.md) +- [UI and Login Problems](../troubleshooting/ui-and-login.md) +- [horizon.yaml reference](https://github.com/apache/skywalking-horizon-ui/blob/main/docs/setup/horizon-yaml.md) diff --git a/docs/install/chart-sources.md b/docs/install/chart-sources.md new file mode 100644 index 0000000..9d502fe --- /dev/null +++ b/docs/install/chart-sources.md @@ -0,0 +1,157 @@ +# Where to Get the Chart + +Four places serve this chart; three are current and one is frozen. This page tells you which +address to point `helm install` at, and what each one actually carries. + +## Source matrix + +| Source | Address | Carries | Use when | +| --- | --- | --- | --- | +| Docker Hub (OCI) | `oci://registry-1.docker.io/apache/skywalking-helm` | Released `skywalking-helm` chart, `4.3.0` and newer (current: `5.0.0`) | Default. Any normal install. | +| Apache JFrog (legacy) | `https://apache.jfrog.io/artifactory/skywalking-helm` | Released charts `4.3.0` and older only — frozen, no new releases | You are pinned to an old chart and cannot move yet. | +| ghcr.io (OCI) | `oci://ghcr.io/apache/skywalking-helm/` | Snapshot of every commit on `master`, versioned `0.0.0-` | Testing an unreleased fix. | +| Source tree | `git clone` + `helm dep up chart/skywalking` | Your working copy, including the `adapter` and `operator` charts | You are editing the chart, or installing SWCK. | + +Chart version and SkyWalking version are separate things. Whichever source you use, the three +required values are always yours to set: + +| value | current | +| --- | --- | +| `oap.image.tag` | `11.0.0` | +| `oap.storageType` | `elasticsearch`, `banyandb`, or `postgresql` | +| `ui.image.tag` | `horizon-1.0.0` | + +## Released chart, Docker Hub OCI registry (>= 4.3.0) + +The chart is pushed to Docker Hub as an OCI artifact (`make publish` runs +`helm push … oci://registry-1.docker.io/apache`). There is no `helm repo add` step — an OCI +reference is the chart. + +```shell +export SKYWALKING_RELEASE_VERSION=5.0.0 +export SKYWALKING_RELEASE_NAME=skywalking +export SKYWALKING_RELEASE_NAMESPACE=default + +helm install "${SKYWALKING_RELEASE_NAME}" \ + oci://registry-1.docker.io/apache/skywalking-helm \ + --version "${SKYWALKING_RELEASE_VERSION}" \ + -n "${SKYWALKING_RELEASE_NAMESPACE}" \ + --set oap.image.tag=11.0.0 \ + --set oap.storageType=elasticsearch \ + --set ui.image.tag=horizon-1.0.0 \ + --set eck-operator.installCRDs=false +``` + +An OCI registry cannot be added with `helm repo add`, so `helm search repo` will not list the +available chart versions — pick and pin `--version` deliberately. To inspect or vendor a version +before installing: + +```shell +helm show values oci://registry-1.docker.io/apache/skywalking-helm --version 5.0.0 +helm pull oci://registry-1.docker.io/apache/skywalking-helm --version 5.0.0 +``` + +With the default `elasticsearch.enabled=true`, the ECK CRDs must already exist in the cluster before +this install runs — the release contains an `Elasticsearch` custom resource, so the CRD cannot be +created by the same release. Install the `eck-operator-crds` subchart first, then install with +`--set eck-operator.installCRDs=false` (its default is `true`) so the CRDs are not duplicated. Both +steps are in [Elasticsearch](../storage/elasticsearch.md). + +## Apache JFrog Helm repository (<= 4.3.0) + +The classic (non-OCI) repository. It holds releases up to `4.3.0` and receives nothing new; it is +listed here only so old runbooks resolve. + +```shell +helm repo add skywalking https://apache.jfrog.io/artifactory/skywalking-helm +helm repo update +helm search repo skywalking/skywalking --versions +``` + +The chart was named `skywalking` back then (it was renamed to `skywalking-helm` after `4.3.0`), so +the repo-qualified name is `skywalking/skywalking`. + +Anything `4.3.0` or newer comes from the Docker Hub OCI address above. + +## Development snapshots, ghcr.io + +Every push to `master` publishes all three charts to ghcr.io from +`.github/workflows/publish-helm.yaml`, versioned `0.0.0-`. Replace the +SHA with the revision you want to test. + +| chart in repo | ghcr.io reference | +| --- | --- | +| `chart/skywalking` | `oci://ghcr.io/apache/skywalking-helm/skywalking-helm` | +| `chart/adapter` | `oci://ghcr.io/apache/skywalking-helm/skywalking-helm-swck-adapter` | +| `chart/operator` | `oci://ghcr.io/apache/skywalking-helm/skywalking-helm-swck-operator` | + +```shell +helm install "${SKYWALKING_RELEASE_NAME}" \ + oci://ghcr.io/apache/skywalking-helm/skywalking-helm \ + --version "0.0.0-b670c41d94a82ddefcf466d54bab5c492d88d772" \ + -n "${SKYWALKING_RELEASE_NAMESPACE}" \ + --set oap.image.tag=11.0.0 \ + --set oap.storageType=elasticsearch \ + --set ui.image.tag=horizon-1.0.0 \ + --set eck-operator.installCRDs=false +``` + +These are snapshots of unreleased code. Use them to verify a fix, not to run production. + +## From source + +Clone, resolve the subcharts, then install from the local path. + +```shell +git clone https://github.com/apache/skywalking-helm +cd skywalking-helm +helm dep up chart/skywalking + +helm install "${SKYWALKING_RELEASE_NAME}" chart/skywalking \ + -n "${SKYWALKING_RELEASE_NAMESPACE}" \ + --set oap.image.tag=11.0.0 \ + --set oap.storageType=elasticsearch \ + --set ui.image.tag=horizon-1.0.0 \ + --set eck-operator.installCRDs=false +``` + +`helm dep up` is required before the first install and after any `Chart.yaml` change; it downloads +into `chart/skywalking/charts/` and writes `Chart.lock`. Each dependency declares its repository as +a URL, so no `helm repo add` is needed. What it pulls (these are **chart** versions, not image +versions — the BanyanDB server version is `banyandb.image.tag`, currently `0.11.0`): + +| dependency | version | repository | condition | +| --- | --- | --- | --- | +| `eck-operator` | 3.3.1 | `https://helm.elastic.co/` | `elasticsearch.enabled` | +| `eck-elasticsearch` (alias `elasticsearch`) | 0.18.1 | `https://helm.elastic.co/` | `elasticsearch.enabled` | +| `postgresql` | 12.1.2 | `https://raw.githubusercontent.com/bitnami/charts/archive-full-index/bitnami` | `postgresql.enabled` | +| `skywalking-banyandb-helm` (alias `banyandb`) | 0.7.0-rc1 | `oci://registry-1.docker.io/apache` | `banyandb.enabled` | + +`make package` does the same `helm dep up` plus `helm package`, and drops +`skywalking-helm-5.0.0.tgz` in the repo root. `make clean` removes the pulled subcharts, +`Chart.lock` and the packaged tarball. See +[Package and Publish a Release](../contributing/release.md). + +## The other two charts + +`chart/adapter` and `chart/operator` (both `0.10.0`) are **not** published to the Docker Hub or +JFrog addresses — only to ghcr.io snapshots, or installed from a clone. + +```shell +# SWCK metrics adapter — no subchart dependencies +helm -n skywalking-custom-metrics-system install adapter chart/adapter --create-namespace + +# SWCK operator — declares cert-manager 1.9.1, disabled by default +helm dep up chart/operator +helm -n skywalking-swck-system install operator chart/operator --create-namespace +``` + +The operator's `cert-manager` dependency ships `enabled: false`, so install +[cert-manager](https://cert-manager.io/) yourself first, or set `cert-manager.enabled=true`. +Details in [SWCK Operator](../operate/swck-operator.md) and +[SWCK Metrics Adapter](../operate/swck-adapter.md). + +## Next + +- [Quick Start](quick-start.md) — a working install, end to end. +- [Version Compatibility](../evaluate/version-compatibility.md) — which OAP, UI and BanyanDB versions go together. diff --git a/docs/install/quick-start.md b/docs/install/quick-start.md new file mode 100644 index 0000000..166b079 --- /dev/null +++ b/docs/install/quick-start.md @@ -0,0 +1,123 @@ +# Quick Start + +The shortest path from an empty Kubernetes cluster to a running SkyWalking install: OAP 11.0.0 +with Elasticsearch storage and Horizon UI 1.0.0, reachable in your browser. + +## Before you start + +- A Kubernetes cluster and a working `kubectl` context. +- Helm 3.8 or newer (the OCI install below needs a Helm that can pull `oci://` charts). +- `oap.storageType` has no default and must be set, but `elasticsearch.enabled` defaults to `true`, + so the chart deploys Elasticsearch through [ECK](https://github.com/elastic/cloud-on-k8s) as a + **3-node** cluster (`elasticsearch.nodeSets[0].count: 3`, `elasticsearch.version: 8.18.8`). + See [Requirements](../evaluate/requirements.md) for sizing, and + [Pick a Storage Backend](../storage/choose-a-backend.md) if you would rather start with BanyanDB. + +## 1. Set the release variables + +```shell +export SKYWALKING_RELEASE_VERSION=5.0.0 +export SKYWALKING_RELEASE_NAME=skywalking +export SKYWALKING_RELEASE_NAMESPACE=default +``` + +## 2. Install the ECK CRDs + +Helm renders the chart's `Elasticsearch` custom resource during install, so the ECK CRDs must +already exist in the cluster. Install them as their own release — the version matches the +`eck-operator` dependency pinned in `chart/skywalking/Chart.yaml`: + +```shell +helm install eck-crds eck-operator-crds \ + --repo https://helm.elastic.co --version 3.3.1 \ + -n "${SKYWALKING_RELEASE_NAMESPACE}" --create-namespace +``` + +Then pass `--set eck-operator.installCRDs=false` when installing SkyWalking so the two releases do +not both own the CRDs. + +> Skip this step entirely if you set `elasticsearch.enabled=false` — using an external +> Elasticsearch, BanyanDB, or PostgreSQL needs no CRDs. + +## 3. Install the chart + +```shell +helm install "${SKYWALKING_RELEASE_NAME}" \ + oci://registry-1.docker.io/apache/skywalking-helm \ + --version "${SKYWALKING_RELEASE_VERSION}" \ + -n "${SKYWALKING_RELEASE_NAMESPACE}" --create-namespace \ + --set oap.image.tag=11.0.0 \ + --set oap.storageType=elasticsearch \ + --set ui.image.tag=horizon-1.0.0 \ + --set eck-operator.installCRDs=false +``` + +The three values that have no default and must always be set: + +| value | this install | notes | +| --- | --- | --- | +| `oap.image.tag` | `11.0.0` | OAP server image tag | +| `oap.storageType` | `elasticsearch` | also `banyandb`, `postgresql` | +| `ui.image.tag` | `horizon-1.0.0` | must be a `horizon-*` tag | + +Other chart sources — Apache JFrog, ghcr.io snapshots, a local clone — are covered in +[Where to Get the Chart](../install/chart-sources.md). + +## 4. Wait for it to come up + +```shell +kubectl get pods -n "${SKYWALKING_RELEASE_NAMESPACE}" -w +``` + +A one-shot `*-oap-init-*` Job creates the storage schema; the OAP Deployment runs in `-Dmode=no-init` +and stays un-Ready until that Job finishes. Both run in the main install phase, so you can add +`--wait --wait-for-jobs` to the `helm install` above and let Helm block instead (the extra +`--wait-for-jobs` makes Helm surface an init-Job failure directly). To watch the Job: + +```shell +kubectl get job -n "${SKYWALKING_RELEASE_NAMESPACE}" -l release="${SKYWALKING_RELEASE_NAME}" +kubectl logs -n "${SKYWALKING_RELEASE_NAMESPACE}" job/ -f +``` + +Details in [The OAP Init Job](../operate/oap-init-job.md); failures in +[Install and Startup Failures](../troubleshooting/install-and-startup.md). + +## 5. Reach the UI + +The UI Service is `ClusterIP` on port `80` (targeting the BFF's port `8081`). Its name is +`-skywalking-helm-ui`, because the chart is named `skywalking-helm`: + +```shell +kubectl port-forward -n "${SKYWALKING_RELEASE_NAMESPACE}" \ + svc/${SKYWALKING_RELEASE_NAME}-skywalking-helm-ui 8080:80 +open http://127.0.0.1:8080 +``` + +> Prefer shorter resource names? Add `--set fullnameOverride=skywalking` at install time and the +> Service becomes `skywalking-ui`. + +For a `NodePort`, `LoadBalancer` or Ingress instead of port-forwarding, see +[UI Service and Ingress](../expose/ui-service-and-ingress.md). + +## 6. Create a login — the install has none + +Horizon UI has **no** built-in `admin/admin` account, and this chart configures no users. The BFF +does not fail closed: it boots, serves the login page, and passes its readiness probe, so the pod +reports Ready and nobody can log in. + +Go to [Set Up Logins](../ui/logins.md) for a copy-pastable demo user and the production +Secret-backed pattern. Do this before you rely on the deployment. + +## Uninstall + +```shell +helm uninstall "${SKYWALKING_RELEASE_NAME}" -n "${SKYWALKING_RELEASE_NAMESPACE}" +helm uninstall eck-crds -n "${SKYWALKING_RELEASE_NAMESPACE}" +``` + +## Next steps + +- [Set Up Logins](../ui/logins.md) — required before anyone can use the UI. +- [OAP Endpoints for Agents](../expose/oap-endpoints.md) — point agents at gRPC `11800` / HTTP `12800`. +- [Pick a Storage Backend](../storage/choose-a-backend.md) — Elasticsearch vs BanyanDB vs PostgreSQL. +- [Configure OAP](../operate/oap-configuration.md) — environment variables and config overrides. diff --git a/docs/menu.yml b/docs/menu.yml new file mode 100644 index 0000000..2364657 --- /dev/null +++ b/docs/menu.yml @@ -0,0 +1,100 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +catalog: + - name: "Welcome" + path: "/readme" + - name: "Evaluate" + catalog: + - name: "What This Chart Deploys" + path: "/evaluate/what-this-chart-deploys" + - name: "Requirements" + path: "/evaluate/requirements" + - name: "Version Compatibility" + path: "/evaluate/version-compatibility" + - name: "Install" + catalog: + - name: "Quick Start" + path: "/install/quick-start" + - name: "Where to Get the Chart" + path: "/install/chart-sources" + - name: "Choose Storage" + catalog: + - name: "Pick a Storage Backend" + path: "/storage/choose-a-backend" + - name: "Elasticsearch" + path: "/storage/elasticsearch" + - name: "BanyanDB" + path: "/storage/banyandb" + - name: "PostgreSQL" + path: "/storage/postgresql" + - name: "Configure the UI" + catalog: + - name: "Horizon UI in This Chart" + path: "/ui/horizon-ui" + - name: "Set Up Logins" + path: "/ui/logins" + - name: "Configure Horizon" + path: "/ui/configure" + - name: "Expose It" + catalog: + - name: "UI Service and Ingress" + path: "/expose/ui-service-and-ingress" + - name: "TLS" + path: "/expose/tls" + - name: "OAP Endpoints for Agents" + path: "/expose/oap-endpoints" + - name: "Operate" + catalog: + - name: "The OAP Init Job" + path: "/operate/oap-init-job" + - name: "Configure OAP" + path: "/operate/oap-configuration" + - name: "Scaling and the OAP Cluster" + path: "/operate/scaling" + - name: "Satellite Gateway" + path: "/operate/satellite" + - name: "SWCK Operator" + path: "/operate/swck-operator" + - name: "SWCK Metrics Adapter" + path: "/operate/swck-adapter" + - name: "Upgrade" + path: "/upgrade/upgrading" + - name: "Troubleshoot" + catalog: + - name: "Install and Startup Failures" + path: "/troubleshooting/install-and-startup" + - name: "UI and Login Problems" + path: "/troubleshooting/ui-and-login" + - name: "Values Reference" + catalog: + - name: "skywalking Chart" + path: "/reference/skywalking-chart-values" + - name: "adapter Chart" + path: "/reference/adapter-chart-values" + - name: "operator Chart" + path: "/reference/operator-chart-values" + - name: "OAP Configuration Vocabulary" + path: "https://skywalking.apache.org/docs/main/latest/en/setup/backend/configuration-vocabulary/" + - name: "horizon.yaml Reference" + path: "https://github.com/apache/skywalking-horizon-ui/blob/main/docs/setup/horizon-yaml.md" + - name: "Contributing" + catalog: + - name: "Run the E2E Tests" + path: "/contributing/e2e-tests" + - name: "Package and Publish a Release" + path: "/contributing/release" + - name: "Changelog" + path: "/changes/changes" diff --git a/docs/operate/oap-configuration.md b/docs/operate/oap-configuration.md new file mode 100644 index 0000000..57e73e2 --- /dev/null +++ b/docs/operate/oap-configuration.md @@ -0,0 +1,206 @@ +# Configure OAP + +How to change OAP backend behaviour from the chart: environment variables (`oap.env`), overridden +configuration files under `/skywalking/config` (`oap.config`), and runtime dynamic configuration +backed by a ConfigMap (`oap.dynamicConfig`). + +## Three levers + +| Value | What it changes | Applies to | Takes effect | +|---|---|---|---| +| `oap.env` | Environment variables on the OAP container | OAP Deployment **and** the OAP init Job | On pod restart (Helm rolls the Deployment because the pod spec changed) | +| `oap.config` | Files dropped into `/skywalking/config` | OAP Deployment **and** the OAP init Job | On pod restart — see the caveat below, a config-only change does **not** roll the Deployment | +| `oap.dynamicConfig` | Runtime rules OAP re-reads from a ConfigMap | OAP Deployment only | Within `oap.dynamicConfig.period` seconds, no restart | + +Environment variables win over the shipped configuration files: OAP's `application.yml` resolves +almost every setting as `${SW_SOMETHING:default}`, so setting `SW_SOMETHING` overrides the default +baked into the image. That only holds while the placeholder survives — if you replace a file through +`oap.config` and write a literal where the shipped file had `${SW_SOMETHING:...}`, the literal wins +and the variable is ignored. + +## Environment variables (`oap.env`) + +`oap.env` is a plain map of name to value. Every entry is appended to the container `env` list (the +chart quotes the value, so numbers and booleans are safe to write unquoted). + +```shell +helm upgrade --install skywalking oci://registry-1.docker.io/apache/skywalking-helm \ + --version 5.0.0 -n skywalking --create-namespace \ + --set oap.image.tag=11.0.0 \ + --set oap.storageType=banyandb \ + --set ui.image.tag=horizon-1.0.0 \ + --set elasticsearch.enabled=false \ + --set banyandb.enabled=true \ + --set banyandb.image.tag=0.11.0 \ + --set oap.env.SW_ENVOY_METRIC_ALS_HTTP_ANALYSIS=k8s-mesh \ + --set oap.env.SW_ENVOY_METRIC_ALS_TCP_ANALYSIS=k8s-mesh +``` + +Or in a values file, which is easier once you have more than one or a value contains commas: + +```yaml +oap: + env: + SW_ENVOY_METRIC_ALS_HTTP_ANALYSIS: k8s-mesh + SW_ENVOY_METRIC_ALS_TCP_ANALYSIS: k8s-mesh + K8S_SERVICE_NAME_RULE: 'e2e::${service.metadata.name}' + SW_CORE_RECORD_DATA_TTL: 3 +``` + +The full list of variables OAP understands is upstream: +[Configuration Vocabulary](https://skywalking.apache.org/docs/main/latest/en/setup/backend/configuration-vocabulary/). + +### Variables the chart already manages + +Do not set these through `oap.env` — `oap.env` is rendered *after* them, producing a duplicate entry +in the same `env` list instead of a clean override. Use the dedicated value instead. + +| Variable(s) | Set by the chart from | Use this instead | +|---|---|---| +| `JAVA_OPTS` | `oap.javaOpts` plus `-Dmode=no-init` (`-Dmode=init` in the init Job) | `oap.javaOpts` (default `-Xmx2g -Xms2g`) | +| `SW_STORAGE`, `SW_STORAGE_ES_CLUSTER_NODES`, `SW_ES_USER`, `SW_ES_PASSWORD`, `SW_JDBC_URL`, `SW_DATA_SOURCE_USER`, `SW_DATA_SOURCE_PASSWORD`, `SW_STORAGE_BANYANDB_TARGETS`, `SW_STORAGE_BANYANDB_USER`, `SW_STORAGE_BANYANDB_PASSWORD` | `oap.storageType` and the storage backend's own values | [Pick a Storage Backend](../storage/choose-a-backend.md) | +| `SW_CLUSTER`, `SW_CLUSTER_K8S_NAMESPACE`, `SW_CLUSTER_K8S_LABEL` | Fixed to `kubernetes` plus the release namespace and label selector | Nothing — the chart always runs OAP in Kubernetes cluster mode | +| `SW_RECEIVER_ZIPKIN`, `SW_RECEIVER_ZIPKIN_REST_PORT`, `SW_QUERY_ZIPKIN`, `SW_QUERY_ZIPKIN_REST_PORT` | Rendered only when `oap.ports.zipkin-receiver` / `oap.ports.zipkin-query` are set | `oap.ports` | +| `SW_CONFIGURATION`, `SW_CONFIG_CONFIGMAP_PERIOD` | Rendered when `oap.dynamicConfig.enabled` is `true` | `oap.dynamicConfig` | +| `SKYWALKING_COLLECTOR_UID` | The pod UID, via the downward API | Nothing | + +`JAVA_OPTS` is the one that bites: a second entry can shadow the chart's `-Dmode=no-init`, which is +what makes the Deployment leave schema creation to the init Job. Put JVM flags in `oap.javaOpts`. + +`oap.env` is also applied to the one-shot OAP init Job, so schema-affecting variables (storage TTL, +index settings) reach the process that creates the schema. See [The OAP Init Job](oap-init-job.md). + +## Configuration file overrides (`oap.config`) + +`oap.config` drops arbitrary files into `/skywalking/config` inside the OAP container — the image's +own config directory. Every top-level key becomes a file name; a **nested map becomes a +subdirectory**, up to three levels deep. + +```yaml +oap: + config: + log4j2.xml: | + + + + metadata-service-mapping.yaml: | + serviceName: e2e::${LABELS."service.istio.io/canonical-name"} + serviceInstanceName: ${NAME} + oal: + core.oal: | + service_resp_time = from(Service.latency).longAvg(); + service_sla = from(Service.*).percent(status == true); + service_cpm = from(Service.*).cpm(); + otel-rules: + k8s: + k8s-cluster.yaml: | + # three levels also work +``` + +That renders to these mounts in the OAP container: + +| `oap.config` path | Mounted at | +|---|---| +| `log4j2.xml` | `/skywalking/config/log4j2.xml` | +| `metadata-service-mapping.yaml` | `/skywalking/config/metadata-service-mapping.yaml` | +| `oal` → `core.oal` | `/skywalking/config/oal/core.oal` | +| `otel-rules` → `k8s` → `k8s-cluster.yaml` | `/skywalking/config/otel-rules/k8s/k8s-cluster.yaml` | + +### How it works + +The chart flattens the nested map into a single ConfigMap named `{fullname}-oap-cm-override` (for a +release `skywalking`, `skywalking-skywalking-helm-oap-cm-override`), joining path segments with `-`: +`oal` → `core.oal` becomes the ConfigMap key `oal-core.oal`. Each key is then mounted back at its +real path with `subPath`, so only that one file is placed and the rest of the image's +`/skywalking/config` directory stays intact. + +```shell +kubectl get cm -n skywalking -l component=oap +kubectl exec -n skywalking deploy/skywalking-skywalking-helm-oap -- ls /skywalking/config/oal +``` + +Two consequences worth knowing: + +- **A config-only change does not restart OAP.** The Deployment's pod template does not embed a + checksum of this ConfigMap, and `subPath` mounts do not track ConfigMap updates. After a + `helm upgrade` that only touches `oap.config`, roll the pods yourself: + ```shell + kubectl rollout restart -n skywalking deploy/skywalking-skywalking-helm-oap + ``` + (The init Job *is* re-created, because its name hashes the chart values.) +- **Avoid `-` collisions in the flattened key space.** A top-level key literally named `oal-core.oal` + and a nested `oal` → `core.oal` produce the same ConfigMap key. + +Files most commonly overridden here: `log4j2.xml` (log level and appenders), `oal/*.oal` +([OAL scripts](https://skywalking.apache.org/docs/main/latest/en/concepts-and-designs/oal/)) and +`metadata-service-mapping.yaml` (Kubernetes-to-service naming for the Envoy/mesh receivers). + +Secrets — TLS material, keystores — should go through `oap.secretMounts` rather than `oap.config`, +which is a plain ConfigMap. + +## Dynamic configuration (`oap.dynamicConfig`) + +Dynamic configuration is OAP's runtime-rule mechanism: alarm rules, sampling policies, thresholds +that you want to change without restarting the backend. Setting `oap.dynamicConfig.enabled=true` +wires OAP to its `k8s-configmap` configuration provider. + +```yaml +oap: + dynamicConfig: + enabled: true + period: 60 + config: + agent-analyzer.default.slowDBAccessThreshold: default:200,mongodb:50 + alarm.default.alarm-settings: | + rules: + service_resp_time_rule: + metrics-name: service_resp_time + op: ">" + threshold: 1000 + period: 10 + count: 3 + silence-period: 5 + message: Response time of service {name} is more than 1000ms in 3 minutes of last 10 minutes. +``` + +What the chart does when `enabled` is `true`: + +| | | +|---|---| +| ConfigMap | Creates one named exactly `skywalking-dynamic-config`, with `data` taken verbatim from `oap.dynamicConfig.config` and labels `app`/`release`/`component` | +| Env | Sets `SW_CONFIGURATION=k8s-configmap` and `SW_CONFIG_CONFIGMAP_PERIOD` to `oap.dynamicConfig.period` (seconds, default `60`) on the OAP Deployment | +| Discovery | OAP finds the ConfigMap in `SW_CLUSTER_K8S_NAMESPACE` (the release namespace) using the label selector `SW_CLUSTER_K8S_LABEL`, which the chart sets to `app=,release=,component=oap` — the same labels it puts on the ConfigMap | +| RBAC | The chart's Role already grants `get`/`watch`/`list` on `configmaps` when `serviceAccounts.oap.create` is `true` | + +Keys are `..`; values may be a scalar or a multi-line block. The +catalogue of supported keys is upstream: +[Dynamic Configuration](https://skywalking.apache.org/docs/main/latest/en/setup/backend/dynamic-config/), +and the provider the chart wires up is +[Dynamic Configuration ConfigMap](https://skywalking.apache.org/docs/main/latest/en/setup/backend/dynamic-config-configmap/). + +Things to watch: + +- **The default `config` is `{}`.** Turning `enabled` on by itself creates an empty ConfigMap and + changes no behaviour — OAP keeps using its static defaults until you add keys. +- **The ConfigMap name is fixed**, not release-prefixed. Two SkyWalking releases in the same + namespace would fight over `skywalking-dynamic-config`; give them separate namespaces. +- **Helm owns the content.** Editing the ConfigMap with `kubectl edit` works until the next + `helm upgrade`, which resets it to `oap.dynamicConfig.config`. Keep the values file as the source + of truth. +- **The init Job does not get these variables** — dynamic configuration applies to the running OAP + Deployment only. + +Verify it took: + +```shell +kubectl get cm skywalking-dynamic-config -n skywalking -o yaml +kubectl logs -n skywalking deploy/skywalking-skywalking-helm-oap | grep -i configmap +``` + +## Related + +- [The OAP Init Job](oap-init-job.md) — which changes re-run schema creation +- [Scaling and the OAP Cluster](scaling.md) +- [OAP Endpoints for Agents](../expose/oap-endpoints.md) — `oap.ports` and the Service +- [skywalking Chart Values](../reference/skywalking-chart-values.md) — every `oap.*` value +- [OAP Configuration Vocabulary](https://skywalking.apache.org/docs/main/latest/en/setup/backend/configuration-vocabulary/) diff --git a/docs/operate/oap-init-job.md b/docs/operate/oap-init-job.md new file mode 100644 index 0000000..7dd6a12 --- /dev/null +++ b/docs/operate/oap-init-job.md @@ -0,0 +1,194 @@ +# The OAP Init Job + +The chart creates the storage schema with a one-shot `*-oap-init-*` Job, and the main OAP +Deployment refuses to serve until that Job has finished. This page explains that handshake, why +the Job is deliberately not a Helm hook, and how to watch, rerun, or clean it up. + +## The handshake + +Two OAP containers from the same image, started with different modes: + +| Workload | `JAVA_OPTS` | Behaviour | +| --- | --- | --- | +| `*-oap-init-*` Job | ` -Dmode=init` | Creates the storage schema (Elasticsearch indices / SQL tables / BanyanDB groups), then exits. `restartPolicy: Never`. | +| `*-oap` Deployment | `-Dmode=no-init ` | Never touches the schema. Blocks with port `12800` closed until the schema exists, so the pod is not Ready. | + +Order of events on `helm install`: + +1. Both the Job pod and the OAP pods run the same `wait-for-storage` init container and block + until the storage backend answers. For Elasticsearch and BanyanDB it gives up after 60 attempts + 5s apart (at least 5 minutes) and fails; for PostgreSQL it retries `pg_isready` every 3s + indefinitely. +2. The Job's OAP boots in `init` mode, writes the schema and exits `Completed`. +3. The OAP pods, still blocked in `no-init` mode, see the schema, open `12800`, and pass their + startup/readiness probes. +4. Helm's `--wait` resolves, because the Job and the Deployment run in the same phase. + +Because the OAP pods' readiness depends on the Job, a failing Job shows up as OAP pods stuck in +`0/1 Running` — see [Watching and debugging](#watching-and-debugging) below. + +## Why it is not a Helm hook + +The Job is a normal release resource, not a `post-install`/`post-upgrade` hook. A hook would +deadlock `helm upgrade --install --wait`: + +- Helm waits for every release resource to become Ready **before** it runs `post-*` hooks. +- The OAP Deployment never becomes Ready until the schema exists. +- The schema is created by the hook, which never runs. Helm waits until timeout. + +As a main-phase resource the Job runs alongside the Deployment, and the two unblock each other. + +To have Helm surface init-Job failures directly — instead of only reporting that OAP never became +Ready — add `--wait-for-jobs` alongside `--wait`: + +```shell +helm upgrade --install skywalking oci://registry-1.docker.io/apache/skywalking-helm \ + --version 5.0.0 -n skywalking --create-namespace \ + --set oap.image.tag=11.0.0 \ + --set oap.storageType=banyandb \ + --set ui.image.tag=horizon-1.0.0 \ + --set elasticsearch.enabled=false \ + --set banyandb.enabled=true \ + --set banyandb.image.tag=0.11.0 \ + --wait --wait-for-jobs +``` + +## The value-hashed Job name + +A Job's `spec.template` is immutable. With a stable name, any `helm upgrade` that changes the pod +template (a new image tag, a new env var) would fail with `field is immutable`. So the name carries +a short hash of the rendered values: + +```gotemplate +{{ printf "%s-init-%s" (include "skywalking.oap.fullname" . | trunc 40 | trimSuffix "-") + (.Values | toYaml | sha256sum | trunc 8) }} +``` + +For a release named `skywalking` that renders to, for example: + +```text +skywalking-skywalking-helm-oap-init-39aeb14b +``` + +Consequences worth knowing: + +- Any value change produces a new Job name, so `helm upgrade` creates a fresh Job and re-runs init; + Helm prunes the previous one because it is no longer in the manifest. +- The hash covers the **whole** values tree, including subchart values — an unrelated change + (a UI setting, a BanyanDB replica count) also re-runs init. +- The OAP name is truncated to 40 characters before the suffix, so the Job name stays inside the + 63-character DNS limit. + +## Configuring the Job + +The Job reuses the OAP values for everything that must match the server: `oap.image.*`, +`oap.javaOpts`, `oap.env`, `oap.resources`, `oap.securityContext`, the OAP service account, the +storage env vars derived from `oap.storageType`, and the same `oap.config` / `oap.secretMounts` +volumes. Only these `oapInit.*` keys are Job-specific: + +| Value | Default | Purpose | +| --- | --- | --- | +| `oapInit.nodeAffinity` | `{}` | Node affinity for the Job pod | +| `oapInit.nodeSelector` | `{}` | Node selector for the Job pod | +| `oapInit.tolerations` | `[]` | Tolerations for the Job pod | +| `oapInit.extraPodLabels` | `{}` | Extra pod labels, e.g. `sidecar.istio.io/inject: "false"` | +| `oapInit.ttlSecondsAfterFinished` | `""` | Auto-delete the finished Job via the TTL-after-finished controller | + +A service mesh sidecar is the common reason to set `extraPodLabels` — an injected sidecar keeps the +pod alive after the OAP container exits, so the Job never reports `Complete`: + +```yaml +oapInit: + extraPodLabels: + sidecar.istio.io/inject: "false" +``` + +### ttlSecondsAfterFinished and the GitOps caveat + +`oapInit.ttlSecondsAfterFinished` is empty by default, so completed Jobs and their pods stay in the +namespace where you can still read their logs. Set it to tidy them up: + +```shell +--set oapInit.ttlSecondsAfterFinished=600 +``` + +**Leave it empty when the release is managed by a GitOps tool (Argo CD, Flux).** The TTL controller +deletes the Job, the GitOps reconcile sees a resource missing from the desired state, recreates it, +and init runs again on every sync loop. The value-hashed name already makes upgrades work without +TTL — the setting is only cosmetic cleanup. + +## The startup probe budget + +Because `no-init` OAP keeps `12800` closed while it waits, the chart's default startup probe is +generous — from `chart/skywalking/values.yaml`: + +```yaml +oap: + startupProbe: {} + # Boot budget defaults to 30 (failureThreshold) * 10 (periodSeconds) = 300 seconds. + # In no-init mode OAP keeps port 12800 closed until the OAP init Job has created the storage + # schema, so the budget must be large enough to cover storage startup + schema creation; + # otherwise the pod is restarted while it is legitimately waiting for the init Job. +``` + +That 300-second budget has to cover storage startup **plus** schema creation. A cold Elasticsearch +cluster, a slow PVC, or a first-time BanyanDB group creation can exceed it, and the pod is then +restarted while it was legitimately waiting. Raise it rather than reducing the wait: + +```yaml +oap: + startupProbe: + tcpSocket: + port: 12800 + failureThreshold: 90 # 90 * 10s = 15 minutes + periodSeconds: 10 +``` + +Setting `oap.startupProbe` replaces the default block entirely, so repeat `tcpSocket.port: 12800`. +The liveness and readiness probes default to the same TCP check on `12800` with +`initialDelaySeconds: 5`, `periodSeconds: 10`. + +## Watching and debugging + +```shell +export NS=skywalking +export RELEASE=skywalking + +# The chart renders exactly one Job, whatever the storage type +kubectl get jobs -n "$NS" -l release="$RELEASE" +JOB=$(kubectl get jobs -n "$NS" -l release="$RELEASE" -o jsonpath='{.items[0].metadata.name}') + +# Logs (while running, and after completion until the TTL removes it) +kubectl logs -n "$NS" "job/$JOB" --tail=200 + +# Block until it finishes +kubectl wait --for=condition=complete -n "$NS" "job/$JOB" --timeout=10m +``` + +Common outcomes: + +| Symptom | Cause | +| --- | --- | +| Job pod stuck in `Init:0/1` | `wait-for-storage` cannot reach the backend — wrong host/port, backend not up | +| Job pod `Error` / `BackoffLimitExceeded` | OAP could not create the schema: bad credentials, an incompatible or half-started backend. Read the logs | +| OAP container exited, pod still `Running`, Job never `Complete` | An injected mesh sidecar keeps the pod alive; set `oapInit.extraPodLabels` as above | +| OAP pods `0/1 Running`, restarting every ~5 min | Init never completed, and the startup probe budget expired | + +## Forcing a rerun + +Every `helm upgrade` with a changed value already reruns init. To rerun it without changing any +value, delete the Job and upgrade — Helm recreates the now-missing resource: + +```shell +kubectl delete job -n skywalking -l release=skywalking +helm upgrade skywalking oci://registry-1.docker.io/apache/skywalking-helm \ + --version 5.0.0 -n skywalking --reuse-values +``` + +## See also + +- [Configure OAP](oap-configuration.md) — the `oap.env`, `oap.javaOpts` and config-override values the Job inherits +- [Pick a Backend](../storage/choose-a-backend.md) — what "the schema" means per storage type +- [Install and Startup troubleshooting](../troubleshooting/install-and-startup.md) +- [Upgrade](../upgrade/upgrading.md) +- [skywalking chart values](../reference/skywalking-chart-values.md) diff --git a/docs/operate/satellite.md b/docs/operate/satellite.md new file mode 100644 index 0000000..b24bd5d --- /dev/null +++ b/docs/operate/satellite.md @@ -0,0 +1,139 @@ +# Satellite Gateway + +[SkyWalking Satellite](https://skywalking.apache.org/docs/skywalking-satellite/latest/readme/) is an +optional lightweight gateway that sits in front of OAP: agents and Envoy send to Satellite, Satellite +buffers and load-balances into the OAP pods. This page covers turning it on in the chart, the ports it +exposes, repointing agents at it, and changing its settings. + +## Enable it + +Satellite is off by default (`satellite.enabled: false`) and its image tag has no default — set both: + +```shell +helm upgrade --install "${SKYWALKING_RELEASE_NAME}" \ + oci://registry-1.docker.io/apache/skywalking-helm \ + --version 5.0.0 \ + -n "${SKYWALKING_RELEASE_NAMESPACE}" \ + --set oap.image.tag=11.0.0 \ + --set oap.storageType=elasticsearch \ + --set ui.image.tag=horizon-1.0.0 \ + --set eck-operator.installCRDs=false \ + --set satellite.enabled=true \ + --set satellite.image.tag=v1.3.0 +``` + +The storage flags above are just the [Quick Start](../install/quick-start.md) ones — only the two +`satellite.*` flags are new. Keep whatever storage flags your release already uses; Satellite does +not care which backend OAP writes to. + +`satellite.image.tag` is `required` in the template — leaving it unset fails the render with +`satellite.image.tag is required`. Pick a tag published for +`apache/skywalking-satellite` (default repository: `skywalking.docker.scarf.sh/apache/skywalking-satellite`); +the e2e tests instead run a CI build from `ghcr.io/apache/skywalking-satellite/skywalking-satellite` +(see `test/e2e/env`). + +Enabling it renders five resources, all named `{fullname}-satellite`: + +| Resource | Purpose | +| --- | --- | +| `Deployment` | `satellite.replicas` pods (default `1`) | +| `Service` | `satellite.service.type`, default `ClusterIP` | +| `ServiceAccount` | named `{fullname}-satellite`; `serviceAccounts.satellite` renames it | +| `Role` + `RoleBinding` | `get`/`watch`/`list` on `pods`, for OAP discovery | + +`{fullname}` is `{release}-skywalking-helm` unless you set `fullnameOverride`. With +`--set fullnameOverride=skywalking` (what the e2e tests use) the Service is simply +`skywalking-satellite`. + +## Ports + +Container ports and Service ports are both generated from the `satellite.ports` map, so the key +becomes the port name: + +| `satellite.ports.*` | Default | Purpose | +| --- | --- | --- | +| `grpc` | `11800` | agent/Envoy traffic in — this is the address agents use | +| `prometheus` | `1234` | Satellite's own Prometheus metrics | +| `pprof` | commented out (`6060`) | Go pprof, uncomment only for debugging | + +Add any extra port by adding a key to the map; nothing else needs changing. + +> The readiness probe is a TCP check against `oap.ports.grpc`, **not** `satellite.ports.grpc`. If you +> move Satellite's gRPC port without moving OAP's, the probe targets a port Satellite is not listening +> on and the pod never becomes Ready. + +## Point agents at Satellite + +Once Satellite is up, replace the OAP backend address in your agents and mesh config with the +Satellite Service on port `11800` — the port number is the same, only the host changes: + +```properties +# Java agent — host is the Satellite Service, i.e. {fullname}-satellite. +collector.backend_service=${SW_AGENT_COLLECTOR_BACKEND_SERVICES:skywalking-satellite.:11800} +``` + +```shell +# Istio: send Envoy access logs to Satellite instead of OAP +istioctl install -y \ + --set meshConfig.defaultConfig.envoyAccessLogService.address=skywalking-satellite.istio-system:11800 +``` + +Everything else — the UI, the OAP REST/admin endpoints, `swctl` — keeps talking to the OAP Service +directly. Satellite only fronts the gRPC ingestion path. See +[OAP Endpoints for Agents](../expose/oap-endpoints.md) for the endpoints Satellite does not front. + +## How Satellite finds OAP + +The chart wires Kubernetes-native discovery, so Satellite talks to OAP **pods** rather than the OAP +Service. These env vars are set by the template and should not be overridden: + +| Env | Value | +| --- | --- | +| `SATELLITE_GRPC_CLIENT_FINDER` | `kubernetes` | +| `SATELLITE_GRPC_CLIENT_KUBERNETES_NAMESPACE` | the release namespace | +| `SATELLITE_GRPC_CLIENT_KUBERNETES_KIND` | `pod` | +| `SATELLITE_GRPC_CLIENT_KUBERNETES_SELECTOR_LABEL` | `app={release},release={release},component=oap` — the `app` label is the release name, not the chart name | +| `SATELLITE_GRPC_CLIENT_KUBERNETES_EXTRA_PORT` | `oap.ports.grpc` (`11800`) | + +That is why the chart also creates the Role granting `pods` read access, and the RoleBinding that +grants it to whatever `skywalking.serviceAccountName.satellite` resolves to — so renaming the account +with `serviceAccounts.satellite` keeps the permission. The chart always *creates* that +ServiceAccount, so `serviceAccounts.satellite` is a rename, not a way to adopt a ServiceAccount that +already exists (Helm would fail on the conflict). + +## Change Satellite settings + +Entries in the `satellite.env` map are rendered after the built-ins above, so they can override +anything the shipped config reads from the environment — which is almost every setting, because +`satellite_config.yaml` writes each one as an `${ENV:default}` placeholder (`logger.level` from +`SATELLITE_LOGGER_LEVEL`, `telemetry.prometheus.address` from +`SATELLITE_TELEMETRY_PROMETHEUS_ADDRESS`, and so on): + +```shell +--set satellite.env.SATELLITE_LOGGER_LEVEL=debug +``` + +The full placeholder list is in +[Override settings](https://skywalking.apache.org/docs/skywalking-satellite/latest/en/setup/configuration/override-settings/) +upstream. + +### The `satellite.config` map + +`satellite.config` is a map of file name to file content. Each key is written into the +`{fullname}-satellite-cm-override` ConfigMap and mounted with `subPath` at +`/skywalking/config/`; a nested map becomes a directory, so `config..` mounts at +`/skywalking/config//`. + +> The Satellite image loads its configuration from `/skywalking/configs/satellite_config.yaml` — +> `configs`, plural — and the chart passes no `--config` argument to change that. A +> `satellite.config` key named `satellite_config.yaml` therefore lands at +> `/skywalking/config/satellite_config.yaml`, *beside* the shipped file rather than over it, and has +> no effect. Do not use `satellite.config` to rewrite the pipeline; use `satellite.env`, or bake a +> custom image. + +## Related + +- [Configure OAP](oap-configuration.md) — the same `env` map on the OAP side, where `oap.config` + *does* mount over the shipped files (the OAP image reads `/skywalking/config`, singular) +- [Scaling and the OAP Cluster](scaling.md) +- [Chart values reference](../reference/skywalking-chart-values.md) diff --git a/docs/operate/scaling.md b/docs/operate/scaling.md new file mode 100644 index 0000000..3b5c8a9 --- /dev/null +++ b/docs/operate/scaling.md @@ -0,0 +1,234 @@ +# Scaling and the OAP Cluster + +How this chart forms an OAP cluster on Kubernetes, which values control replica count, pod spreading +and sizing, and why the UI must stay at a single replica. + +## How OAP replicas find each other + +The chart runs OAP in Kubernetes cluster mode. `SW_CLUSTER` is hardwired to `kubernetes` in +`chart/skywalking/templates/oap-deployment.yaml` — the chart exposes no dedicated value for +switching to `standalone`, `zookeeper`, `etcd`, etc. Four environment variables drive it, all +rendered by the chart: + +| Env var | Rendered from | Example (`helm install skywalking ... -n sw`) | +|---|---|---| +| `SW_CLUSTER` | hardwired | `kubernetes` | +| `SW_CLUSTER_K8S_NAMESPACE` | `.Release.Namespace` | `sw` | +| `SW_CLUSTER_K8S_LABEL` | `skywalking.oap.labels` helper | `app=skywalking,release=skywalking,component=oap` | +| `SKYWALKING_COLLECTOR_UID` | downward API, `metadata.uid` | the pod's own UID | + +The label selector is built from the release name and `oap.name`, and it matches exactly the labels +the OAP Deployment puts on its pods. Each OAP pod watches pods in its own namespace carrying those +labels, treats every Ready peer as a cluster member at *pod IP + the core gRPC port*, and uses +`SKYWALKING_COLLECTOR_UID` to recognise which of those pods is itself. + +Two consequences worth knowing: + +- **Cluster membership is namespace-scoped.** Two releases in different namespaces never join each + other. Two releases in the *same* namespace do not join either, because `release=` is part of the + selector. +- **The peer port is OAP's configured core gRPC port, not `oap.ports.grpc`.** `oap.ports.grpc` + only declares the container port and the Service port; the chart does not set `SW_CORE_GRPC_PORT`. + If you change `oap.ports.grpc` away from `11800`, set the matching backend property too: + +```shell +--set oap.ports.grpc=21800 \ +--set oap.env.SW_CORE_GRPC_PORT=21800 +``` + +## The RBAC that makes it work + +Watching pods requires permission, so `serviceAccounts.oap.create: true` (the default) creates a +ServiceAccount, a namespaced Role + RoleBinding, and a ClusterRole + ClusterRoleBinding, all bound to +the OAP pods: + +| Template | Scope | Grants (`get`, `watch`, `list` only) | +|---|---|---| +| `oap-role.yaml` | namespace | `pods`, `configmaps` | +| `oap-clusterrole.yaml` | cluster | `pods`, `pods/log`, `endpoints`, `services`, `nodes`, `namespaces`, `configmaps`; `extensions` `deployments`/`replicasets`; `networking.istio.io` `serviceentries` | + +The Role is the part cluster formation depends on (`pods` for peer discovery, `configmaps` for +`oap.dynamicConfig.enabled`). The ClusterRole covers OAP's cluster-wide Kubernetes features — +Kubernetes monitoring and service-mesh metadata. Everything is read-only; nothing is granted `create`, +`update` or `delete`. + +If you set `serviceAccounts.oap.create: false`, **none of those five objects are rendered** — the +`{{- if .Values.serviceAccounts.oap.create }}` guard wraps the ServiceAccount, both roles and both +bindings. You must then supply `serviceAccounts.oap.name` and grant that account at least the +namespaced `pods` read permission yourself, or OAP starts and never discovers a peer. + +## Scaling OAP + +`oap.replicas` defaults to `2`. It maps straight to the Deployment's `spec.replicas`. + +```shell +helm install skywalking \ + oci://registry-1.docker.io/apache/skywalking-helm \ + --version 5.0.0 \ + -n skywalking --create-namespace \ + --set oap.image.tag=11.0.0 \ + --set oap.storageType=banyandb \ + --set ui.image.tag=horizon-1.0.0 \ + --set elasticsearch.enabled=false \ + --set banyandb.enabled=true \ + --set banyandb.image.tag=0.11.0 \ + --set oap.replicas=3 +``` + +or afterwards: + +```shell +# the Deployment carries the same labels the cluster selector uses +kubectl scale deployment -n \ + -l app=,release=,component=oap --replicas=3 +``` + +(A `kubectl scale` is transient — the next `helm upgrade` resets it to `oap.replicas`.) + +Notes: + +- Scaling is safe at any time. New pods join by label, and existing pods pick them up from the pod + informer once they report Ready. +- **Scaling via `helm upgrade` re-runs schema init.** The init Job's name carries a hash of *all* + chart values, `oap.replicas` included, so changing the replica count creates a fresh Job and runs + `-Dmode=init` again. That is harmless — init is idempotent and the running OAP pods are untouched + — but expect a new `*-oap-init-*` Job on every scale. See [The OAP Init Job](oap-init-job.md). + A `kubectl scale` bypasses Helm entirely and creates no Job. +- The E2E tests run with `--set oap.replicas=1`; a single replica is a valid configuration and still + uses Kubernetes cluster mode (it simply discovers a one-member cluster). +- The chart ships **no HorizontalPodAutoscaler and no PodDisruptionBudget**. Add your own if you + need them. + +## Spreading pods across nodes + +`oap.antiAffinity` accepts `soft` (default) or `hard`, and controls what goes into the pod's +`affinity.podAntiAffinity`. Both forms select on the same three labels (`app`, `release`, +`component`) with `topologyKey: kubernetes.io/hostname`. + +`soft` renders a preference: + +```yaml +affinity: + podAntiAffinity: + preferredDuringSchedulingIgnoredDuringExecution: + - weight: 1 + podAffinityTerm: + topologyKey: kubernetes.io/hostname + labelSelector: + matchLabels: + app: "skywalking" + release: "skywalking" + component: "oap" +``` + +`hard` renders a requirement: + +```yaml +affinity: + podAntiAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + - topologyKey: "kubernetes.io/hostname" + labelSelector: + matchLabels: + app: "skywalking" + release: "skywalking" + component: "oap" +``` + +With `hard`, at most one OAP pod runs per node — so `oap.replicas` greater than the number of +schedulable nodes leaves the surplus pods `Pending` forever. Any other value (including `""`) renders +no `podAntiAffinity` at all; the `if`/`else if` chain matches only these two strings. + +The remaining scheduling knobs are passed through verbatim: + +| Value | Default | Rendered as | +|---|---|---| +| `oap.nodeAffinity` | `{}` | `affinity.nodeAffinity` | +| `oap.nodeSelector` | `{}` | `spec.nodeSelector` | +| `oap.tolerations` | `[]` | `spec.tolerations` | + +The same three exist for `ui.*`, `oapInit.*` and `satellite.*`; `satellite.antiAffinity` behaves +identically to the OAP one. + +## Sizing a replica + +`oap.resources` is `{}` by default — the container ships with no requests and no limits. The +`values.yaml` example uses whole CPU cores, and the chart's values reference notes that the CPU limit +must be an integer: + +```yaml +oap: + resources: + limits: + cpu: 8 + memory: 8Gi + requests: + cpu: 8 + memory: 4Gi +``` + +`oap.javaOpts` defaults to `-Xmx2g -Xms2g` and is **appended to a fixed prefix**, not substituted for +it. The rendered variable is: + +```yaml +- name: JAVA_OPTS + value: "-Dmode=no-init -Xmx2g -Xms2g" +``` + +`-Dmode=no-init` is what keeps the OAP pods out of schema initialisation (the init Job owns that). +Because your value replaces the whole default string, always carry the heap flags forward: + +```shell +--set oap.javaOpts="-Xmx6g -Xms6g -XX:+UseG1GC" +``` + +Keep the heap comfortably below the memory limit — the container also needs off-heap and metaspace +room, or the kubelet OOM-kills the pod mid-startup. Increasing `oap.replicas` is generally the better +lever than a very large single heap. + +## Why the UI stays at one replica + +`ui.replicas` defaults to `1` and should stay there unless you add sticky sessions. Horizon UI's BFF +holds the active session table **in process memory**. With two replicas behind a plain round-robin +Service, roughly every other request lands on a pod that has never seen your session, so logins break +and the UI bounces users back to the login page. + +For the same reason the UI Deployment uses: + +```yaml +strategy: + type: Recreate +``` + +instead of the default `RollingUpdate` — Recreate tears the old pod down before starting the new one, +so there is never a window with two pods serving traffic from disjoint session tables. The trade-off +is a short outage on every UI upgrade or config change, which is the intended behaviour here. Note +that the UI pod also rolls whenever `horizon.yaml` changes, via a `checksum/config` annotation over +the rendered ConfigMap. + +Two more things tie the UI to a single pod: + +- `ui.persistence.enabled` is `false` by default, so `/data` is an `emptyDir`. Turning it on + provisions one PVC with `accessModes: [ReadWriteOnce]` for the audit log, setup state, alarm state + and wire debug log — a second replica on another node cannot attach it. +- `ui.livenessProbe`/`ui.readinessProbe` target the BFF on `8081`; readiness uses + `/api/auth/health`, the only unauthenticated health endpoint. + +If you genuinely need more than one UI pod, you must supply sticky sessions at the ingress +(for example `nginx.ingress.kubernetes.io/affinity: cookie` on an ingress-nginx controller via +`ui.ingress.annotations`), switch `ui.persistence` to a `ReadWriteMany` class or disable it, and +accept that a rolling update is now your responsibility to make safe. Scaling the UI is not a +supported configuration of this chart; scale OAP instead — that is where the query load actually +lands. See [Horizon UI in This Chart](../ui/horizon-ui.md) and +[UI Service and Ingress](../expose/ui-service-and-ingress.md). + +## Related + +- [The OAP Init Job](oap-init-job.md) — schema creation, and why OAP runs in `no-init` mode +- [Configure OAP](oap-configuration.md) — `oap.env`, `oap.config`, dynamic configuration +- [Satellite Gateway](satellite.md) — `satellite.replicas` and its own anti-affinity +- [skywalking Chart Values](../reference/skywalking-chart-values.md) — the full parameter table +- [Install and Startup Failures](../troubleshooting/install-and-startup.md) — pods `Pending` or + restarting +- [SkyWalking Configuration Vocabulary](https://skywalking.apache.org/docs/main/latest/en/setup/backend/configuration-vocabulary/) + — every `SW_*` variable, including the `cluster.kubernetes` block diff --git a/docs/operate/swck-adapter.md b/docs/operate/swck-adapter.md new file mode 100644 index 0000000..bcf6257 --- /dev/null +++ b/docs/operate/swck-adapter.md @@ -0,0 +1,142 @@ +# SWCK Metrics Adapter + +The `chart/adapter` chart deploys the [SWCK custom metrics adapter](https://github.com/apache/skywalking-swck/tree/master/adapter), +which exposes SkyWalking OAP metrics through the Kubernetes external metrics API so a +HorizontalPodAutoscaler can scale on them. This page covers what it deploys, its prerequisites, +and how to install and wire it to your OAP. + +## What it does + +The adapter is a Kubernetes aggregated API server. It queries OAP over GraphQL and serves the +answers as external metrics, so an HPA can target `service_cpm`, `service_percentile`, or any +other OAL metric instead of only CPU and memory. + +| Resource | Name (release `adapter`, no `fullnameOverride`) | Notes | +| --- | --- | --- | +| `Deployment` | `adapter-skywalking-helm-swck-adapter-adapter` | `replicas: 1` by default | +| `Service` | `…-custom-metrics-apiserver` | port `443` → container `service.port` (`6443`) | +| `APIService` | `v1beta1.external.metrics.k8s.io` | fixed name, `insecureSkipTLSVerify: true` | +| `ServiceAccount` | `skywalking-custom-metrics-apiserver` | in the release namespace | +| RBAC | `…-custom-metrics-resource-reader`, `…-custom-metrics-server-resources`, plus bindings | one `RoleBinding` lands in `kube-system` for `extension-apiserver-authentication-reader` | + +The container runs with these arguments, all derived from values: + +```text +--secure-port={{ .Values.service.port }} +--v=10 +--oap-addr=http://{{ .Values.oap.service.name }}.{{ .Values.oap.service.namespace }}:{{ .Values.oap.service.port }}/graphql +--cert-dir=/tmp +``` + +The `APIService` name is not templated, and `external.metrics.k8s.io` allows a single provider per +cluster — so install at most one adapter release, and expect a conflict if another external metrics +adapter is already registered. + +## Prerequisites + +- Kubernetes 1.24.0+ with the aggregation layer enabled, and Helm 3. See [Requirements](../evaluate/requirements.md). +- A running OAP whose REST/GraphQL port (`12800` in this chart) is reachable from the adapter's + namespace. Install SkyWalking first — [Quick Start](../install/quick-start.md). +- `image.tag` must resolve; the chart fails rendering with `.adapter.image.tag is required` if you + blank it. The default is `0.10.0`. +- cert-manager is **not** required for the adapter (only [`chart/operator`](swck-operator.md) needs it). + +The adapter chart is not published to the Docker Hub OCI registry — `make publish` only pushes +`chart/skywalking`. Install it from the source tree, or from the `master` snapshot at +`oci://ghcr.io/apache/skywalking-helm/skywalking-helm-swck-adapter`. See +[Where to Get the Chart](../install/chart-sources.md). + +## Install + +```shell +git clone https://github.com/apache/skywalking-helm +cd skywalking-helm +helm -n skywalking-custom-metrics-system install adapter chart/adapter --create-namespace +``` + +The chart has no `namespace` value: every resource except the `kube-system` RoleBinding lands in +the release namespace, so `-n` is what places it. + +## Point it at your OAP + +The defaults (`skywalking-system-oap` in `skywalking-system`) match an SWCK `OAPServer` custom +resource, **not** this repo's `skywalking` chart, whose OAP service is +`-skywalking-helm-oap` in the release namespace (`-oap` only if you installed it +with `--set fullnameOverride=`, as the e2e tests do). Override all three when they differ: + +```shell +helm -n skywalking-custom-metrics-system install adapter chart/adapter --create-namespace \ + --set fullnameOverride=skywalking-swck \ + --set oap.service.name=skywalking-skywalking-helm-oap \ + --set oap.service.namespace=default \ + --set oap.service.port=12800 +``` + +With `fullnameOverride=skywalking-swck` the deployment is `skywalking-swck-adapter` — the name the +e2e test in `test/e2e/swck/oap-agent-adapter-hpa.yaml` waits on. + +## Autoscale on a SkyWalking metric + +Use `type: External`, and prefix the metric name with the adapter's metric namespace, +`skywalking.apache.org`. This is the HPA the e2e test applies, which it `kubectl apply`s straight from +`skywalking-swck/v0.10.0/test/e2e/hpa-demo.yaml`. That upstream file still declares +`autoscaling/v2beta2` (the e2e cluster is Kubernetes 1.21); use `autoscaling/v2` as below, since the +beta API is gone on Kubernetes 1.26+: + +```yaml +apiVersion: autoscaling/v2 +kind: HorizontalPodAutoscaler +metadata: + name: hpa-demo + namespace: skywalking-system +spec: + scaleTargetRef: + apiVersion: apps/v1 + kind: Deployment + name: demo + minReplicas: 1 + maxReplicas: 3 + metrics: + - type: External + external: + metric: + name: skywalking.apache.org|service_cpm + selector: + matchLabels: + "service.str.0": "Your" + "service.byte.1": "5f" + "service.str.2": "ApplicationName" + target: + type: Value + value: 5 +``` + +The `service.str.` / `service.byte.` label pairs spell out the service name +(`5f` is the hex byte for `_`, so this selects `Your_ApplicationName`). The full label grammar, +including `instance`, `endpoint`, and `label` selectors, is in the upstream +[custom metrics adapter doc](https://github.com/apache/skywalking-swck/blob/master/docs/custom-metrics-adapter.md). + +## Verify + +```shell +kubectl -n skywalking-custom-metrics-system rollout status deployment/skywalking-swck-adapter +kubectl get apiservice v1beta1.external.metrics.k8s.io +kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1" +kubectl -n skywalking-system describe hpa hpa-demo +``` + +If the `APIService` reports `Available=False`, the adapter pod is not serving: check that +`--oap-addr` resolves by reading the deployment's args and the pod logs (the chart sets `--v=10`, +so the OAP queries are logged). + +## Uninstall + +```shell +helm -n skywalking-custom-metrics-system uninstall adapter +``` + +This removes the `APIService` too, so any HPA using `external.metrics.k8s.io` stops scaling. + +## Values + +Every parameter and default is listed in [adapter Chart Values](../reference/adapter-chart-values.md). diff --git a/docs/operate/swck-operator.md b/docs/operate/swck-operator.md new file mode 100644 index 0000000..3051972 --- /dev/null +++ b/docs/operate/swck-operator.md @@ -0,0 +1,173 @@ +# SWCK Operator + +`chart/operator` installs [SkyWalking Cloud on Kubernetes (SWCK)](https://github.com/apache/skywalking-swck/tree/master/operator) — +the controller that manages SkyWalking components through custom resources instead of a Helm +release. This page covers the cert-manager prerequisite, the install, and what the chart actually +creates. + +This is a **different chart from `chart/skywalking`**. The `skywalking` chart renders OAP, UI and +Satellite manifests directly; the operator watches `OAPServer`, `UI`, `BanyanDB` and other CRs and +reconciles them, and injects the Java agent into pods. You do not need the operator to run the +`skywalking` chart, and the two can coexist in one cluster. + +## Versioning + +`chart/operator` is versioned separately from the main chart — it tracks the SWCK release, not the +SkyWalking release. + +| | version | +| --- | --- | +| `chart/operator` chart version | `0.10.0` | +| `image.tag` (`docker.io/apache/skywalking-swck`) | `0.10.0` | +| `cert-manager` chart dependency | `1.9.1` (condition `cert-manager.enabled`, off by default) | +| `chart/skywalking` (for contrast) | `5.0.0` | +| Kubernetes | 1.24.0+ | + +It is **not** published to the Docker Hub OCI address or the JFrog repository. Get it from a clone, +or from the ghcr.io snapshot `oci://ghcr.io/apache/skywalking-helm/skywalking-helm-swck-operator`. +See [Where to Get the Chart](../install/chart-sources.md). + +## Prerequisite: cert-manager + +`webhook.enabled` is `true` by default, and the webhook path needs cert-manager: the chart renders a +`Certificate` and a self-signed `Issuer` as `post-install,post-upgrade` hooks, and annotates both +the Mutating and the Validating webhook configurations with `cert-manager.io/inject-ca-from`. +Without cert-manager in the cluster those objects never become real and the operator pod never gets +its serving cert. + +Install cert-manager first and wait for its pods to be Ready: + +```shell +kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.9.1/cert-manager.yaml +kubectl -n cert-manager wait --for=condition=Ready pod --all --timeout=300s +``` + +Two alternatives: + +- `--set cert-manager.enabled=true` pulls in the bundled cert-manager 1.9.1 subchart. It is a + quick-start convenience — do not use it in production, where cert-manager should be its own + release with its own lifecycle. +- `--set webhook.enabled=false` drops the webhook entirely (and with it the Java agent injector), + which removes the cert-manager requirement. + +## Install + +```shell +git clone https://github.com/apache/skywalking-helm +cd skywalking-helm + +helm dep up chart/operator +helm -n skywalking-swck-system install operator chart/operator \ + --create-namespace \ + --set fullnameOverride=skywalking-swck +``` + +`helm dep up` is required because of the `cert-manager` dependency, even though it is disabled. + +Without `fullnameOverride` the base name is `{release}-{chart name}`, so `helm install operator +chart/operator` gives you objects like `operator-skywalking-helm-swck-operator-operator`. The E2E +tests set `fullnameOverride=skywalking-swck`, giving the shorter names used below; do the same and +your runbooks stay readable. + +Verify: + +```shell +kubectl -n skywalking-swck-system rollout status deployment/skywalking-swck-operator +kubectl get crd | grep operator.skywalking.apache.org +``` + +## What the chart creates + +| Object | Name (with `fullnameOverride=skywalking-swck`) | Gated by | +| --- | --- | --- | +| CRDs (11 kinds) | `*.operator.skywalking.apache.org` | `crds.install` | +| Deployment | `skywalking-swck-operator` | always | +| ServiceAccount | `skywalking-swck-controller-manager` | always | +| RBAC | ClusterRoles `skywalking-swck-manager-role`, `-proxy-role`, `-metrics-reader` (+ bindings), namespaced Role `skywalking-swck-leader-election-role` | always | +| ConfigMap (manager config) | `skywalking-swck-manager-config` | always | +| Metrics Service | `skywalking-swck-controller-manager-metrics-service`, port `8443` | `metrics.enabled` | +| Webhook Service | `skywalking-swck-webhook-service`, `443` → `9443` | `webhook.enabled` | +| MutatingWebhookConfiguration | `skywalking-swck-mutating-webhook-configuration` | `webhook.enabled` | +| ValidatingWebhookConfiguration | `skywalking-swck-validating-webhook-configuration` | `webhook.enabled` | +| cert-manager `Certificate` + `Issuer` | `skywalking-swck-serving-cert`, `skywalking-swck-selfsigned-issuer`; the Certificate writes Secret `skywalking-swck-controller-manager-cert`, which the manager mounts at `/tmp/k8s-webhook-server/serving-certs` | `webhook.enabled` | +| Java agent ConfigMap | `skywalking-swck-java-agent-configmap` | `webhook.enabled` | + +The Deployment runs two containers: `manager` (`docker.io/apache/skywalking-swck:0.10.0`, health +probes on `8081`, leader election on) and `kube-rbac-proxy` +(`registry.k8s.io/kubebuilder/kube-rbac-proxy:v0.8.0`, fronting the metrics port). The +kube-rbac-proxy image is hardcoded in the template — mirror it explicitly for an air-gapped cluster. + +### Custom resources + +All in group `operator.skywalking.apache.org`, version `v1alpha1`: + +`BanyanDB`, `EventExporter`, `Fetcher`, `JavaAgent`, `OAPServer`, `OAPServerConfig`, +`OAPServerDynamicConfig`, `Satellite`, `Storage`, `SwAgent`, `UI`. + +See the [SWCK operator docs](https://github.com/apache/skywalking-swck/blob/master/docs/operator.md) +for the CR fields and examples; `test/e2e/swck/` in this repo has working manifests for each. + +## Java agent injection + +The pod mutating webhook (`mpod.kb.io`, path `/mutate-v1-pod`) is scoped by a `namespaceSelector`, +so it only touches namespaces you label: + +```shell +kubectl label namespace my-app swck-injection=enabled +``` + +`failurePolicy` is `Fail`. In a labeled namespace, pod creation is blocked while the operator is +unavailable — take the label off before taking the operator down for a long maintenance window. The +CR webhooks carry no `namespaceSelector`, so while the operator is down, writes to `OAPServer`, `UI` +and the other kinds are rejected cluster-wide too. + +The default agent config lives in `skywalking-swck-java-agent-configmap`: + +```properties +agent.service_name=${SW_AGENT_NAME:Your_ApplicationName} +collector.backend_service=${SW_AGENT_COLLECTOR_BACKEND_SERVICES:127.0.0.1:11800} +``` + +Point `SW_AGENT_COLLECTOR_BACKEND_SERVICES` at your OAP gRPC service. For a `skywalking` chart +release the Service is `{fullname}-oap:11800`, and `{fullname}` is `{release}-skywalking-helm` +unless you set `fullnameOverride` — see [OAP Endpoints](../expose/oap-endpoints.md). + +## Turning pieces off + +| value | default | effect when changed | +| --- | --- | --- | +| `crds.install` | `true` | `false` skips the CRDs — use when they are already installed and managed outside this release | +| `webhook.enabled` | `true` | `false` drops both webhook configurations, the webhook service, the Certificate/Issuer and the Java agent ConfigMap; cert-manager is then unnecessary | +| `metrics.enabled` | `true` | `false` drops the metrics Service and the proxy's `--secure-listen-address` | +| `cert-manager.enabled` | `false` | `true` installs the bundled cert-manager subchart (quick start only) | + +The full parameter list is in [operator Chart values](../reference/operator-chart-values.md). + +## Two values that do not do what their names suggest + +Both are real quirks of `chart/operator` at `0.10.0`, checkable with `helm template`: + +- `replicas` in `values.yaml` is never read. `templates/deployment.yaml` reads `.Values.replicaCount`, + which is undefined, so the rendered Deployment has an empty `replicas:` field and Kubernetes + defaults it to 1. To run more than one manager, set `--set replicaCount=2` (leader election is + already enabled). +- `serviceAccountName` in `values.yaml` is never read either. The ServiceAccount name is always + derived as `{fullname}-controller-manager`. + +## Uninstall + +```shell +helm -n skywalking-swck-system uninstall operator +``` + +The CRDs are ordinary templates (`templates/crds.yaml`), not files under a `crds/` directory, so +Helm deletes them on uninstall — and deleting a CRD cascades to every custom resource of that kind. +If you have live `OAPServer`, `UI` or `BanyanDB` resources you want to keep, back them up first, or +install with `crds.install=false` and manage the CRDs separately. + +## Next + +- [SWCK Metrics Adapter](swck-adapter.md) — the HPA custom-metrics side of SWCK. +- [operator Chart values](../reference/operator-chart-values.md) — the full values table. +- [apache/skywalking-swck](https://github.com/apache/skywalking-swck/tree/master/operator) — the + operator source and release notes. diff --git a/docs/reference/adapter-chart-values.md b/docs/reference/adapter-chart-values.md new file mode 100644 index 0000000..59d55d5 --- /dev/null +++ b/docs/reference/adapter-chart-values.md @@ -0,0 +1,40 @@ +# adapter Chart Values + +Every value the `adapter` chart accepts. See [SWCK Metrics Adapter](../operate/swck-adapter.md) for what it does and how to install it. + +The following table lists the configurable parameters of the adapter chart and their default values. + +| Parameter | Description | Default | +|--------------------------------------------------------------|--------------------------------------------------------------------------------------------------|--------------------------------------| +| `fullnameOverride` | Override fullname | `nil` | +| `.namespace` | Namespace of adapter deployment | `skywalking-custom-metrics-system` | +| `.replicas` | The replicas of adapter | `1` | +| `.serviceAccountName` | The service account name of adapter | `skywalking-custom-metrics-apiserver` | +| `.image.repository` | Adapter container image name | `docker.io/apache/skywalking-swck` | +| `.image.pullPolicy` | Adapter container image pull policy | `IfNotPresent` | +| `.image.tag` | Adapter container image tag | `0.10.0` | +| `.service.port` | The port for the adapter service | `6.1.0` | +| `.oap.service.name` | The service name of OAP | `skywalking-system-oap` | +| `.oap.service.namespace` | The service namespace of OAP | `skywalking-system` | +| `.oap.service.port` | The service port of OAP | `12800` | +| `.resources.limits.cpu` | The limits of cpu in the adapter | `100m` | +| `.resources.limits.memory` | The limits of memory in the adapter | `200Mi` | +| `.resources.requests.cpu` | The requests of cpu in the adapter | `100m` | +| `.resources.requests.memory` | The requests of memory in the adapter | `200Mi` | +| `.affinity` | The affinity policy of adapter | `{}` | + +Specify each parameter using the `--set key=value[,key=value]` argument to `helm install`. For example, + +**Note** You could refer to the [helm install](https://helm.sh/docs/helm/helm_install/) for more command information. + +```console +$ helm install myrelease adapter --set fullnameOverride=newadapter +``` + +Alternatively, a YAML file that specifies the values for the above parameters can be provided while installing the chart. For example, + +```console +$ helm install my-release adapter -f values.yaml +``` + +> **Tip**: You can use the default [values.yaml](../../chart/adapter/values.yaml) diff --git a/docs/reference/operator-chart-values.md b/docs/reference/operator-chart-values.md new file mode 100644 index 0000000..a5a50ed --- /dev/null +++ b/docs/reference/operator-chart-values.md @@ -0,0 +1,38 @@ +# operator Chart Values + +Every value the `operator` chart accepts. See [SWCK Operator](../operate/swck-operator.md) for what it does and how to install it. + +The following table lists the configurable parameters of the operator chart and their default values. + +| Parameter | Description | Default | +|------------------------------|------------------------------------------------------------------------------------------------------------------------------|--------------------------------------| +| `fullnameOverride` | Override fullname | `nil` | +| `.replicaCount` | The replicas of operator | `1` | +| `.serviceAccountName` | The service account name of operator | `skywalking-swck-controller-manager` | +| `.image.repository` | Operator container image name | `docker.io/apache/skywalking-swck` | +| `.image.pullPolicy` | Operator container image pull policy | `IfNotPresent` | +| `.image.tag` | Operator container image tag | `0.10.0` | +| `.metrics.service.port` | The port for the operator metrics service | `8443` | +| `.webhook.service.port` | The port for the operator web hook service | `9443` | +| `.resources.limits.cpu` | The limits of cpu in the operator | `200m` | +| `.resources.limits.memory` | The limits of memory in the operator | `300Mi` | +| `.resources.requests.cpu` | The requests of cpu in the operator | `200m` | +| `.resources.requests.memory` | The requests of memory in the operator | `300Mi` | +| `.affinity` | The affinity policy of operator | `{}` | +| `cert-manager.enabled` | Whether to install demo cert-manager. DO NOT use this in production, this is for quick start. | `false` | + +Specify each parameter using the `--set key=value[,key=value]` argument to `helm install`. For example, + +**Note** You could refer to the [helm install](https://helm.sh/docs/helm/helm_install/) for more command information. + +```console +$ helm install myrelease operator --set fullnameOverride=newoperator +``` + +Alternatively, a YAML file that specifies the values for the above parameters can be provided while installing the chart. For example, + +```console +$ helm install my-release operator -f values.yaml +``` + +> **Tip**: You can use the default [values.yaml](../../chart/operator/values.yaml) diff --git a/docs/reference/skywalking-chart-values.md b/docs/reference/skywalking-chart-values.md new file mode 100644 index 0000000..5d3704f --- /dev/null +++ b/docs/reference/skywalking-chart-values.md @@ -0,0 +1,244 @@ +# skywalking Chart Values + +Every value the `skywalking` chart accepts. Set them with `--set key=value` or a values file +(`-f my-values.yaml`). + +Three values have no default and must be set on every install: + +| name | description | example | +|---|---|---| +| `oap.image.tag` | OAP image tag | `11.0.0` | +| `oap.storageType` | storage backend | `elasticsearch`, `postgresql`, `banyandb` | +| `ui.image.tag` | Horizon UI image tag | `horizon-1.0.0` | + +Parameters for the bundled BanyanDB subchart are reachable by prefixing `banyandb.` — see +[BanyanDB](../storage/banyandb.md). + +The following table lists the configurable parameters of the Skywalking chart and their default values. + +| Parameter | Description | Default | +|----------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------| +| `nameOverride` | Override name | `nil` | +| `serviceAccounts.oap.create` | Create of the OAP service account | `true` | +| `serviceAccounts.oap.name` | Name of the OAP service account to use custom service account when `serviceAccounts.oap.create` is set to false | `` | +| `imagePullSecrets` | Image pull secrets | `[]` | +| `oap.name` | OAP deployment name | `oap` | +| `oap.dynamicConfig.enabled` | Enable oap dynamic configuration through k8s configmap | `false` | +| `oap.dynamicConfig.period` | Sync period in seconds | `60` | +| `oap.dynamicConfig.config` | Oap dynamic configuration [documentation](https://github.com/apache/skywalking/blob/master/docs/en/setup/backend/dynamic-config.md) | `{}` | +| `oap.image.repository` | OAP container image name | `skywalking.docker.scarf.sh/apache/skywalking-oap-server` | +| `oap.image.tag` | OAP container image tag | `6.1.0` | +| `oap.image.pullPolicy` | OAP container image pull policy | `IfNotPresent` | +| `oap.ports.grpc` | OAP grpc port for tracing or metric | `11800` | +| `oap.ports.rest` | OAP http port for the GraphQL query protocol (used by the UI and by `swctl`) | `12800` | +| `oap.ports.admin` | OAP admin REST port (admin-server, status, inspect, ui-management, dsl-debugging, runtime-rule). Introduced in OAP 11, which enables all of them by default and serves `/status/*` and `/debugging/*` here exclusively. Set to `null` on any OAP 10.x release, where 17128 is the AI-pipeline URI-recognition server instead | `17128` | +| `oap.ports.zipkin-receiver` | OAP http port for Zipkin receiver(not exposed by default) | `9411` | +| `oap.ports.zipkin-query` | OAP http port for querying Zipkin traces and UI(not exposed by default) | `9412` | +| `oap.replicas` | OAP k8s deployment replicas | `2` | +| `oap.service.type` | OAP svc type | `ClusterIP` | +| `oap.service.annotations` | OAP svc annotations | `{}` | +| `oap.javaOpts` | Parameters to be added to `JAVA_OPTS`environment variable for OAP | `-Xms2g -Xmx2g` | +| `oap.antiAffinity` | OAP anti-affinity policy | `soft` | +| `oap.nodeAffinity` | OAP node affinity policy | `{}` | +| `oap.nodeSelector` | OAP labels for master pod assignment | `{}` | +| `oap.tolerations` | OAP tolerations | `[]` | +| `oap.resources` | OAP node resources requests & limits | `{} - cpu limit must be an integer` | +| `oap.startupProbe` | Configuration fields for the [startupProbe](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/). The default budget (`failureThreshold` * `periodSeconds` = 300s) is large enough for OAP to wait in no-init mode while the OAP init Job creates the storage schema. | `tcpSocket.port: 12800`
`failureThreshold: 30`
`periodSeconds: 10` +| `oap.livenessProbe` | Configuration fields for the [livenessProbe](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/) | `tcpSocket.port: 12800`
`initialDelaySeconds: 5`
`periodSeconds: 10` +| `oap.readinessProbe` | Configuration fields for the [readinessProbe](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/) | `tcpSocket.port: 12800`
`initialDelaySeconds: 5`
`periodSeconds: 10` +| `oap.env` | OAP environment variables | `[]` | +| `oap.securityContext` | Allows you to set the [securityContext](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-pod) for the pod | `fsGroup: 1000`
`runAsUser: 1000` | +| `oap.sidecars` | Extra sidecar containers to run in the OAP pod (appended to the pod's `containers` list, rendered through `tpl`) | `[]` | +| `ui.enabled` | Deploy the Horizon UI. Set `false` to skip the UI Deployment, Service, Ingress, ConfigMap, and PVC entirely (useful when an external UI talks to OAP directly) | `true` | +| `ui.name` | Web UI deployment name | `ui` | +| `ui.replicas` | Web UI k8s deployment replicas. Keep at `1` unless your ingress provides sticky sessions — the Horizon BFF holds the session table in memory | `1` | +| `ui.image.repository` | Horizon UI container image. Release images: Docker Hub `apache/skywalking-ui` tagged `horizon-x.y.z`. Dev images: `ghcr.io/apache/skywalking-horizon-ui` | `skywalking.docker.scarf.sh/apache/skywalking-ui` | +| `ui.image.tag` | Horizon UI image tag (required), e.g. `horizon-1.0.0`. Horizon releases independently of OAP and 1.0.0 works against OAP 10.4.0 and 11.x alike (for 10.x also set `ui.config.templates.mode: readonly`). The legacy booster UI is not supported — `apache/skywalking-ui` publishes no 11.x tag | `null` | +| `ui.image.pullPolicy` | Web UI container image pull policy | `IfNotPresent` | +| `ui.nodeAffinity` | Web UI node affinity policy | `{}` | +| `ui.nodeSelector` | Web UI labels for pod assignment | `{}` | +| `ui.tolerations` | Web UI tolerations | `[]` | +| `ui.ingress.enabled` | Create Ingress for Web UI | `false` | +| `ui.ingress.annotations` | Associate annotations to the Ingress | `{}` | +| `ui.ingress.path` | Associate path with the Ingress | `/` | +| `ui.ingress.hosts` | Associate hosts with the Ingress | `[]` | +| `ui.ingress.tls` | Associate TLS with the Ingress | `[]` | +| `ui.service.type` | Web UI svc type | `ClusterIP` | +| `ui.service.externalPort` | external port for the service | `80` | +| `ui.service.internalPort` | internal port for the service (Horizon BFF binds 8081) | `8081` | +| `ui.service.externalIPs` | external IP addresses | `nil` | +| `ui.service.loadBalancerIP` | Load Balancer IP address | `nil` | +| `ui.service.annotations` | Kubernetes service annotations | `{}` | +| `ui.service.loadBalancerSourceRanges` | Limit load balancer source IPs to list of CIDRs (where available)) | `[]` | +| `ui.securityContext` | Pod [securityContext](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-pod). The image runs as the non-root `horizon` user; `fsGroup` makes mounted volumes group-writable for that user | `fsGroup: 101` | +| `ui.livenessProbe` | TCP liveness probe | `tcpSocket.port: 8081` | +| `ui.readinessProbe` | HTTP readiness probe; verifies the BFF is up and the auth backend is healthy. `/api/auth/health` is the only unauthenticated BFF health endpoint | `httpGet.path: /api/auth/health, port: 8081` | +| `ui.persistence.enabled` | Mount a PVC at `/data` for audit log / setup / alarm state / wire debug log. When `false`, state lands in the container's writable layer and is lost on pod restart | `false` | +| `ui.persistence.existingClaim` | Use a pre-created PVC; if empty, the chart creates one | `""` | +| `ui.persistence.storageClass` | Storage class for the chart-managed PVC | `""` | +| `ui.persistence.accessModes` | PVC access modes | `[ReadWriteOnce]` | +| `ui.persistence.size` | PVC size | `1Gi` | +| `ui.config` | `horizon.yaml` content (deep-merged onto chart defaults that point `oap.queryUrl`/`adminUrl`/`zipkinUrl` at the in-cluster OAP; `zipkinUrl` is derived from `oap.ports.zipkin-query` and only resolves to a usable URL when that port is set, and `server.publicUrl` is derived from `ui.ingress.hosts[0]` when an ingress is enabled). See the [horizon.yaml reference](https://github.com/apache/skywalking-horizon-ui/blob/main/docs/setup/horizon-yaml.md). Empty by default: the chart-computed values are written as `${HORIZON_*:}` tokens, so anything not set here stays overridable by env via `ui.extraEnv` / `ui.envFromSecret`. A field set here as a literal makes its `HORIZON_*` env var inert | `{}` | +| `ui.config.templates.mode` | `live` reads/writes dashboard templates through OAP 11's `/ui-management/templates*` admin REST API; `readonly` renders the templates bundled in the image and makes the config surface display-only. Set `readonly` against OAP 10.4.0 — it has no such REST surface, and `live` blocks every layer-driven page. Changing this needs a BFF restart | `live` | +| `ui.config.server.publicUrl` | Public base URL operators reach Horizon at; used for SSO callbacks and as the OAuth issuer. Derived from the first `ui.ingress.hosts` entry when an ingress is enabled; set explicitly to override | derived, else `""` | +| `ui.config.server.trustProxy` | Whether to believe `X-Forwarded-For` for the client address in the login audit. Use a hop count (`1` = one proxy in front) or the ingress address/CIDR; `true` is refused at boot | `false` | +| `ui.envFromSecret` | Reference a Secret whose keys are exposed as env vars in the BFF container, for use with `${VAR}` interpolation in `ui.config` (e.g. admin password hash) | `""` | +| `ui.extraEnv` | Extra env vars passed to the BFF container | `[]` | +| `ui.extraVolumes` | Extra volumes for the UI pod. Needed by the two Horizon settings that take a filesystem path: `auth.tokensFile` (API tokens Secret) and `sourceMaps.bootMountDir` (durable `.map` files at `/app/sourcemaps`) | `[]` | +| `ui.extraVolumeMounts` | Extra volume mounts for the BFF container, paired with `ui.extraVolumes` | `[]` | +| `oapInit.nodeAffinity` | OAP init job node affinity policy | `{}` | +| `oapInit.nodeSelector` | OAP init job labels for master pod assignment | `{}` | +| `oapInit.tolerations` | OAP init job tolerations | `[]` | +| `oapInit.extraPodLabels` | OAP init job metadata labels | `[]` | +| `oapInit.ttlSecondsAfterFinished` | Seconds after which the finished OAP init Job (and its Pod) is auto-deleted by the Kubernetes TTL-after-finished controller. Empty keeps the Job. Leave empty with GitOps tools (Argo CD/Flux), which would recreate it after deletion. | `""` | +| `satellite.name` | Satellite deployment name | `satellite` | +| `satellite.replicas` | Satellite k8s deployment replicas | `1` | +| `satellite.enabled` | Is enable Satellite | `false` | +| `satellite.image.repository` | Satellite container image name | `skywalking.docker.scarf.sh/apache/skywalking-satellite` | +| `satellite.image.tag` | Satellite container image tag | `v0.4.0` | +| `satellite.image.pullPolicy` | Satellite container image pull policy | `IfNotPresent` | +| `satellite.antiAffinity` | Satellite anti-affinity policy | `soft` | +| `satellite.nodeAffinity` | Satellite node affinity policy | `{}` | +| `satellite.nodeSelector` | Satellite labels for pod assignment | `{}` | +| `satellite.tolerations` | Satellite tolerations | `[]` | +| `satellite.service.type` | Satellite svc type | `ClusterIP` | +| `satellite.ports.grpc` | Satellite grpc port for tracing, metrics, logs, events | `11800` | +| `satellite.ports.prometheus` | Satellite http port for Prometheus monitoring | `1234` | +| `satellite.resources` | Satellite node resources requests & limits | `{} - cpu limit must be an integer` | +| `satellite.podAnnotations` | Configurable [annotations](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/) applied to all Satellite pods | `{}` | +| `satellite.env` | Satellite environment variables | `[]` | +| `satellite.securityContext` | Allows you to set the [securityContext](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-pod) for the pod | `fsGroup: 1000`
`runAsUser: 1000` | + +### Elasticsearch (ECK) + +Elasticsearch is deployed via [ECK (Elastic Cloud on Kubernetes)](https://github.com/elastic/cloud-on-k8s). +The chart includes the ECK operator and an `eck-elasticsearch` subchart, both controlled by `elasticsearch.enabled`. +Because Elasticsearch CRDs must exist before the ES custom resource can be created, the ECK operator CRDs need to be installed separately before deploying the chart. See the main [README](../../README.md) for installation steps. + +#### Top-level parameters + +| Parameter | Description | Default | +|---|---|---| +| `elasticsearch.enabled` | Deploy the ECK operator and an ECK-managed Elasticsearch cluster | `true` | +| `elasticsearch.version` | Elasticsearch version to deploy | `8.18.8` | +| `elasticsearch.fullnameOverride` | Override the Elasticsearch resource name. The ECK service will be `{name}-es-http` | `""` | +| `elasticsearch.labels` | Labels applied to the Elasticsearch resource | `{}` | +| `elasticsearch.annotations` | Annotations applied to the Elasticsearch resource | `{}` | +| `elasticsearch.http` | HTTP layer settings. TLS is disabled by default for OAP connectivity | `tls.selfSignedCertificate.disabled: true` | +| `elasticsearch.secureSettings` | [Secure settings](https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-es-secure-settings.html) to inject from Kubernetes secrets | `[]` | +| `elasticsearch.updateStrategy` | [Update strategy](https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-update-strategy.html) controlling simultaneous changes | `{}` | +| `elasticsearch.volumeClaimDeletePolicy` | Policy for PVC deletion on scale-down or cluster deletion | `""` | +| `elasticsearch.ingress.enabled` | Enable ingress to expose Elasticsearch externally | `false` | + +#### External Elasticsearch (when `elasticsearch.enabled` is `false`) + +| Parameter | Description | Default | +|---|---|---| +| `elasticsearch.config.host` | Elasticsearch host | `elasticsearch` | +| `elasticsearch.config.port.http` | Elasticsearch HTTP port | `9200` | +| `elasticsearch.config.user` | Elasticsearch user (optional) | `""` | +| `elasticsearch.config.password` | Elasticsearch password (optional) | `""` | + +#### Node sets (`elasticsearch.nodeSets[]`) + +ECK [node sets](https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-node-configuration.html) define the topology of the Elasticsearch cluster. Each entry in the list creates a group of Elasticsearch nodes. + +| Parameter | Description | Default | +|---|---|---| +| `nodeSets[].name` | Name of the node set | `default` | +| `nodeSets[].count` | Number of Elasticsearch nodes in this set | `3` | +| `nodeSets[].config` | Elasticsearch configuration (e.g. `node.store.allow_mmap`, `node.roles`) | `node.store.allow_mmap: false` | +| `nodeSets[].volumeClaimTemplates` | Persistent storage for Elasticsearch data | `[]` (ECK default: EmptyDir) | + +#### Pod template (`elasticsearch.nodeSets[].podTemplate`) + +The pod template follows standard Kubernetes Pod spec nested under `podTemplate.spec`. This controls scheduling, resources, init containers, etc. + +| Parameter | Description | Default | +|---|---|---| +| `podTemplate.metadata.annotations` | Pod annotations (e.g. `iam.amazonaws.com/role`) | `{}` | +| `podTemplate.metadata.labels` | Extra pod labels | `{}` | +| `podTemplate.spec.affinity` | Pod [affinity](https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-advanced-node-scheduling.html) rules | not set | +| `podTemplate.spec.nodeSelector` | Node selector for pod assignment | not set | +| `podTemplate.spec.tolerations` | Pod tolerations | not set | +| `podTemplate.spec.imagePullSecrets` | Image pull secrets | not set | +| `podTemplate.spec.priorityClassName` | Priority class name | not set | +| `podTemplate.spec.terminationGracePeriodSeconds` | Grace period for pod termination | not set | +| `podTemplate.spec.initContainers` | Init containers (e.g. sysctl `vm.max_map_count`) | not set | +| `podTemplate.spec.containers[].resources` | Container resource requests & limits | `requests: 100m cpu, 2Gi mem`
`limits: 2Gi mem` | +| `podTemplate.spec.containers[].env` | Environment variables (e.g. `ES_JAVA_OPTS`) | not set | +| `podTemplate.spec.containers[].securityContext` | Container-level security context | not set (ECK managed) | + +Specify each parameter using the `--set key=value[,key=value]` argument to `helm install`. For example, + +```console +$ helm install myrelease skywalking --set nameOverride=newSkywalking +``` + +Alternatively, a YAML file that specifies the values for the above parameters can be provided while installing the +chart. For example, + +```console +$ helm install my-release skywalking -f values.yaml +``` + +> **Tip**: You can use the default [values.yaml](../../chart/skywalking/values.yaml) + +### RBAC Configuration + +Roles and RoleBindings resources will be created automatically for `OAP` . + +> **Tip**: You can refer to the default `oap-role.yaml` file in [templates](../../chart/skywalking/templates/) to customize your own. + +### Ingress TLS + +If your cluster allows automatic create/retrieve of TLS certificates ( +e.g. [kube-lego](https://github.com/jetstack/kube-lego)), please refer to the documentation for that mechanism. + +To manually configure TLS, first create/retrieve a key & certificate pair for the address(skywalking ui) you wish to +protect. Then create a TLS secret in the namespace: + +```console +kubectl create secret tls skywalking-tls --cert=path/to/tls.cert --key=path/to/tls.key +``` + +Include the secret's name, along with the desired hostnames, in the skywalking-ui Ingress TLS section of your +custom `values.yaml` file: + +```yaml +ui: + ingress: + ## If true, Skywalking ui server Ingress will be created + ## + enabled: true + + ## Skywalking ui server Ingress hostnames + ## Must be provided if Ingress is enabled + ## + hosts: + - skywalking + + ## Skywalking ui server Ingress TLS configuration + ## Secrets must be manually created in the namespace + ## + tls: + - secretName: skywalking + hosts: + - skywalking +``` + +### Envoy ALS + +Envoy ALS(access log service) provides fully logs about RPC routed, including HTTP and TCP. + +If you want to open envoy ALS, you can do this by modifying values.yaml. default open. + +```yaml +serviceAccounts: + oap: + create: true +``` + +When envoy als ,will give ServiceAccount clusterrole permission. +More envoy als ,please refer +to https://github.com/apache/skywalking/blob/master/docs/en/setup/envoy/als_setting.md#observe-service-mesh-through-als diff --git a/docs/storage/banyandb.md b/docs/storage/banyandb.md new file mode 100644 index 0000000..34e5e92 --- /dev/null +++ b/docs/storage/banyandb.md @@ -0,0 +1,200 @@ +# BanyanDB + +How to run SkyWalking on BanyanDB with this chart: the install command, standalone versus cluster +mode, authentication, pointing OAP at an external cluster, and the OAP/BanyanDB version lock you +cannot ignore. + +BanyanDB is SkyWalking's own storage engine and the backend the project is moving to. The chart +deploys it as a subchart — [`skywalking-banyandb-helm`](https://github.com/apache/skywalking-banyandb-helm), +pinned to `0.7.0-rc1` in `chart/skywalking/Chart.yaml` under the alias `banyandb`, installed only +when `banyandb.enabled=true`. + +## Install + +```shell +helm install skywalking oci://registry-1.docker.io/apache/skywalking-helm \ + --version 5.0.0 -n skywalking --create-namespace \ + --set oap.image.tag=11.0.0 \ + --set oap.storageType=banyandb \ + --set ui.image.tag=horizon-1.0.0 \ + --set elasticsearch.enabled=false \ + --set banyandb.enabled=true \ + --set banyandb.image.tag=0.11.0 +``` + +`elasticsearch.enabled` defaults to `true`, so you must set it to `false` — otherwise the chart +also pulls in the ECK operator and an Elasticsearch cluster that nothing uses. + +`banyandb.image.tag` has no default in the subchart; the templates fail fast when it is empty, so +it must always be set. The subchart pulls from `docker.io/apache/skywalking-banyandb`, and the +`0.11.0` tag has not been pushed there yet — if you hit `ImagePullBackOff`, use the GHCR image +built from the v0.11.0 commit, which is what this repo's CI pins in `test/e2e/env`: + +```shell + --set banyandb.image.repository=ghcr.io/apache/skywalking-banyandb \ + --set banyandb.image.tag=3b83e18fb0481d02e44eaa5df137fcf7b000754b +``` + +## Version lock: OAP 11.0.0 accepts API 0.11 only + +OAP ships the list of BanyanDB server API versions it will talk to in +`SW_STORAGE_BANYANDB_COMPATIBLE_SERVER_API_VERSIONS`. **OAP 11.0.0 accepts API `0.11`**, which maps +to BanyanDB release `0.11.x`. Pairing OAP 11 with BanyanDB 0.10.x makes OAP refuse to start with: + +```text +Incompatible BanyanDB server API version: 0.10. But accepted versions: 0.11 +``` + +The list is an OAP setting, so `--set oap.env.SW_STORAGE_BANYANDB_COMPATIBLE_SERVER_API_VERSIONS=0.10` +would suppress the check — do not. It only turns a clear startup failure into schema and query +errors later. Upgrade or downgrade BanyanDB to match. The API-version to release mapping is +published at +[BanyanDB API versions](https://skywalking.apache.org/docs/skywalking-banyandb/latest/installation/versions/). + +This is why OAP 11.0.0, Horizon UI horizon-1.0.0 and BanyanDB 0.11.0 move as one set; see +[Version Compatibility](../evaluate/version-compatibility.md). + +## Standalone vs cluster + +The chart defaults to **standalone**: a single BanyanDB StatefulSet, fine for evaluation and small +installs. + +| | standalone (default) | cluster | +|---|---|---| +| `banyandb.standalone.enabled` | `true` | `false` | +| `banyandb.cluster.enabled` | `false` | `true` | +| pods | one `-banyandb` StatefulSet | `-banyandb-liaison` + `-banyandb-data-` StatefulSets, plus the FODC proxy | +| OAP-facing services | unchanged | unchanged | + +Cluster mode: + +```shell +helm install skywalking oci://registry-1.docker.io/apache/skywalking-helm \ + --version 5.0.0 -n skywalking --create-namespace \ + --set oap.image.tag=11.0.0 \ + --set oap.storageType=banyandb \ + --set ui.image.tag=horizon-1.0.0 \ + --set elasticsearch.enabled=false \ + --set banyandb.enabled=true \ + --set banyandb.image.tag=0.11.0 \ + --set banyandb.standalone.enabled=false \ + --set banyandb.cluster.enabled=true +``` + +Sizing knobs that actually take effect: + +| value | default as rendered by this chart | +|---|---| +| `banyandb.cluster.liaison.replicas` | `1` (this chart overrides the subchart's `2`) | +| `banyandb.cluster.data.nodeTemplate.replicas` | `2` (subchart default; per-role overrides live under `banyandb.cluster.data.roles..replicas`, merged over `nodeTemplate`) | +| `banyandb.cluster.fodc.agent.resources.{requests,limits}` | one entry, `memory` / `256Mi` | + +`banyandb.cluster.data.replicas` also appears in this chart's `values.yaml`, but no subchart +template reads it — it is inert. Use the `nodeTemplate` path above. + +The FODC agent's `resources.requests` and `resources.limits` are **lists of `{key, value}` pairs**, +not resource maps, so they are set by index: + +```shell + --set banyandb.cluster.fodc.agent.resources.requests[0].key=memory \ + --set banyandb.cluster.fodc.agent.resources.requests[0].value=256Mi \ + --set banyandb.cluster.fodc.agent.resources.limits[0].key=memory \ + --set banyandb.cluster.fodc.agent.resources.limits[0].value=256Mi +``` + +FODC (First Occurrence Data Collection) is enabled by default in cluster mode and adds a sidecar +agent to liaison and data pods plus an `-banyandb-fodc-proxy` deployment. + +## What the chart wires into OAP + +`_helpers.tpl` derives the addresses from the subchart's service names and ports — you do not set +them by hand when `banyandb.enabled=true`. + +| | value | +|---|---| +| gRPC service | `-banyandb-grpc` port `17912` (`banyandb.standalone.grpcSvc.port`, or `banyandb.cluster.liaison.grpcSvc.port`) | +| HTTP service | `-banyandb-http` port `17913` (`banyandb.standalone.httpSvc.port`, or `banyandb.cluster.liaison.httpSvc.port`) | +| OAP env | `SW_STORAGE=banyandb`, `SW_STORAGE_BANYANDB_TARGETS=-banyandb-grpc:17912` | +| startup gate | a `wait-for-banyandb` init container (`curlimages/curl`) that curls `/api/healthz` up to 60 times, 5s apart, then fails | + +The same init container and the same storage env are attached to both the OAP deployment and the +OAP init job that creates the storage schema — see [The OAP Init Job](../operate/oap-init-job.md). + +The `banyandb` name segment comes from the subchart shipping `nameOverride: banyandb`, so the +resources are `-banyandb-*` rather than `-skywalking-banyandb-helm-*`. Set +`banyandb.fullnameOverride` to rename them; the OAP wiring follows it. + +Note that the subchart defaults `standalone.httpSvc.type` and `cluster.liaison.httpSvc.type` to +`LoadBalancer`. On a cloud cluster that provisions a public load balancer for BanyanDB's HTTP API. +Set `banyandb.standalone.httpSvc.type=ClusterIP` (or the liaison equivalent) unless you want that. + +## Authentication + +BanyanDB auth is off by default. Turn it on with `banyandb.auth.enabled=true`: + +```shell + --set banyandb.auth.enabled=true \ + --set banyandb.auth.users[0].username=admin \ + --set banyandb.auth.users[0].password='' +``` + +The chart's own `values.yaml` already ships a placeholder user (`admin` / `banyandb`), so the +example above only needs the password change — but change it. + +Two things to know: + +- OAP takes its credentials from the **first entry** of `banyandb.auth.users`, rendered into + `SW_STORAGE_BANYANDB_USER` and `SW_STORAGE_BANYANDB_PASSWORD`. If you set `auth.enabled=true` + and clear `banyandb.auth.users`, the subchart generates a random `admin` password into its own + Secret and OAP is given no credentials at all — it will fail to authenticate. Always set the + users explicitly. +- Those two variables are rendered as **plain env values in the OAP pod spec**, not as a + `secretKeyRef`. Anyone who can read the deployment or the release values can read the password. + The chart has no secret-reference path for them today — `oap.env` also takes literal values + only — so treat the release values as sensitive and restrict RBAC on the namespace. + +## External BanyanDB + +To point OAP at a BanyanDB cluster the chart does not manage, leave the subchart off and give the +addresses directly: + +```shell +helm install skywalking oci://registry-1.docker.io/apache/skywalking-helm \ + --version 5.0.0 -n skywalking --create-namespace \ + --set oap.image.tag=11.0.0 \ + --set oap.storageType=banyandb \ + --set ui.image.tag=horizon-1.0.0 \ + --set elasticsearch.enabled=false \ + --set banyandb.enabled=false \ + --set banyandb.config.grpcAddress=banyandb-grpc.data:17912 \ + --set banyandb.config.httpAddress=banyandb-http.data:17913 +``` + +| value | default | used for | +|---|---|---| +| `banyandb.config.grpcAddress` | `banyandb-grpc:17912` | `SW_STORAGE_BANYANDB_TARGETS` | +| `banyandb.config.httpAddress` | `banyandb-http:17913` | the `wait-for-banyandb` health check | + +`banyandb.auth.*` still applies in this mode: with `banyandb.auth.enabled=true` the first user in +`banyandb.auth.users` is passed to OAP as the credentials for the external cluster. + +## Tuning anything else in BanyanDB + +Every parameter of the `skywalking-banyandb-helm` subchart is reachable from this chart by +**prefixing it with `banyandb.`**. `image.tag` becomes `banyandb.image.tag`, +`standalone.resources` becomes `banyandb.standalone.resources`, `storage.data` becomes +`banyandb.storage.data`, and so on — persistence, resources, node discovery, TLS, ingress, the +Canopy console, all of it. + +The full parameter list lives in the subchart's own docs: +[skywalking-banyandb-helm configuration](https://github.com/apache/skywalking-banyandb-helm?tab=readme-ov-file#configuration). + +For BanyanDB itself — groups, retention, cluster topology, backup — see the +[BanyanDB documentation](https://skywalking.apache.org/docs/skywalking-banyandb/latest/readme/). + +## See also + +- [Pick a Storage Backend](choose-a-backend.md) +- [Version Compatibility](../evaluate/version-compatibility.md) +- [The OAP Init Job](../operate/oap-init-job.md) +- [Install and Startup Failures](../troubleshooting/install-and-startup.md) diff --git a/docs/storage/choose-a-backend.md b/docs/storage/choose-a-backend.md new file mode 100644 index 0000000..492efb7 --- /dev/null +++ b/docs/storage/choose-a-backend.md @@ -0,0 +1,106 @@ +# Pick a Storage Backend + +`oap.storageType` is a required value, and this chart wires up three backends: `elasticsearch`, +`postgresql`, and `banyandb`. This page tells you which to pick, whether the chart can deploy the +backend for you, and exactly what each choice injects into the OAP pods. + +## Decision table + +| Backend | Pick it when | Chart can deploy it | Enable embedded with | Covered by e2e | +|---|---|---|---|---| +| `banyandb` | You want SkyWalking's purpose-built storage: standalone for small/medium clusters, cluster mode (liaison + data nodes) to scale out. | Yes, via the `skywalking-banyandb-helm` subchart (alias `banyandb`) | `banyandb.enabled=true` (+ `banyandb.image.tag`) | Yes — standalone and cluster | +| `elasticsearch` | You already run Elasticsearch, or you want the chart default and the widest set of OAP query features validated against it. | Yes, via ECK: the `eck-operator` + `eck-elasticsearch` subcharts | `elasticsearch.enabled=true` (**the default**) | Yes | +| `postgresql` | Demo, evaluation, or a tiny single-node install. | Yes, but the embedded deployment is **demo only** | `postgresql.enabled=true` | No | + +`postgresql.enabled` is commented in `values.yaml` as *"Whether to start a demo postgresql +deployment, don't use this for production."* — it ships with a hard-coded password (`123456`) and +`primary.persistence.enabled: false`, so the data is lost with the pod. For production PostgreSQL, +run your own server and set `postgresql.enabled=false` with `postgresql.config.host`. + +> **`elasticsearch.enabled` defaults to `true`.** If you pick `banyandb` or `postgresql`, also set +> `elasticsearch.enabled=false` — otherwise the chart still installs the ECK operator and a +> 3-node Elasticsearch cluster that nothing uses. + +## What each choice actually wires up + +Two helpers in `chart/skywalking/templates/_helpers.tpl` do all the work, and both the OAP +Deployment and the [OAP init Job](../operate/oap-init-job.md) include them: + +- `skywalking.oap.envs.storage` — the storage env vars on the OAP container. +- `skywalking.containers.wait-for-storage` — an init container that blocks the pod until the + backend answers. + +Every path sets `SW_STORAGE` to `oap.storageType` (rendered through `required`, so an empty value +fails the render). On top of that: + +| `oap.storageType` | Env vars added | Readiness gate (init container) | +|---|---|---| +| `elasticsearch` | `SW_STORAGE_ES_CLUSTER_NODES`, plus `SW_ES_USER` / `SW_ES_PASSWORD` (embedded: always; external: only when `elasticsearch.config.user` / `.password` are set) | `busybox` (`initContainer.image`/`tag`, default `busybox:1.30`) running `nc -z` against the ES HTTP port, 60 tries × 5s | +| `postgresql` | `SW_JDBC_URL`, `SW_DATA_SOURCE_USER`, `SW_DATA_SOURCE_PASSWORD` | `postgres:13` running `pg_isready` in a loop with a 3s sleep, no attempt cap | +| `banyandb` | `SW_STORAGE_BANYANDB_TARGETS`, plus `SW_STORAGE_BANYANDB_USER` / `SW_STORAGE_BANYANDB_PASSWORD` when `banyandb.auth.enabled` and `banyandb.auth.users` is non-empty | `curlimages/curl` polling `/api/healthz`, 60 tries × 5s | + +Anything else you pass to `oap.storageType` is forwarded to `SW_STORAGE` unchanged, but the chart +adds **no** connection env vars and **no** init container for it — you would have to supply the +whole connection through `oap.env.*` yourself, which emits literal `value:` entries only. + +### Where the connection details come from + +| | Embedded (`*.enabled=true`) | External (`*.enabled=false`) | +|---|---|---| +| Elasticsearch | `{release}-elasticsearch-es-http:9200` (name from `elasticsearch.fullnameOverride` when set; the port is hard-coded `9200`, `elasticsearch.config.port.http` applies to external clusters only); user is `elastic`, password read from the ECK-generated secret `{release}-elasticsearch-es-elastic-user`, key `elastic` | `elasticsearch.config.host` + `elasticsearch.config.port.http`; `elasticsearch.config.user` / `.password` are rendered as **plaintext env values**, and `oap.env` is no better — the OAP templates have no `envFrom` / `secretKeyRef` path, so OAP credentials always land in the Deployment as literals | +| PostgreSQL | host `{release}-postgresql`, port `postgresql.containerPorts.postgresql` (`5432`), database `postgresql.auth.database` (`skywalking`) | `postgresql.config.host`; the port, database, username and password still come from `postgresql.containerPorts.postgresql` and `postgresql.auth.*` — there is no `postgresql.config.port` | +| BanyanDB | `{release}-banyandb-grpc:` / `-http:`, ports taken from `banyandb.standalone.*Svc.port` or `banyandb.cluster.liaison.*Svc.port` (defaults `17912` gRPC, `17913` HTTP) | `banyandb.config.grpcAddress` (default `banyandb-grpc:17912`) and `banyandb.config.httpAddress` (default `banyandb-http:17913`) | + +## Install examples + +BanyanDB (standalone is the chart's default BanyanDB mode): + +```shell +helm install skywalking oci://registry-1.docker.io/apache/skywalking-helm \ + --version 5.0.0 \ + --set oap.image.tag=11.0.0 \ + --set ui.image.tag=horizon-1.0.0 \ + --set oap.storageType=banyandb \ + --set elasticsearch.enabled=false \ + --set banyandb.enabled=true \ + --set banyandb.image.tag=0.11.0 +``` + +`banyandb.image.tag` has no default — the subchart fails fast when it is empty. And OAP and +BanyanDB versions are locked: OAP 11.0.0 accepts BanyanDB server API `0.11`, i.e. BanyanDB 0.11.x. +See [Version Compatibility](../evaluate/version-compatibility.md). + +Elasticsearch (the default): + +```shell +helm install skywalking oci://registry-1.docker.io/apache/skywalking-helm \ + --version 5.0.0 \ + --set oap.image.tag=11.0.0 \ + --set ui.image.tag=horizon-1.0.0 \ + --set oap.storageType=elasticsearch \ + --set eck-operator.installCRDs=false +``` + +`eck-operator.installCRDs=false` assumes the ECK CRDs are already in the cluster — install the +`eck-operator-crds` chart first, or drop the flag and let the bundled `eck-operator` install them +(its own default is `installCRDs: true`). See [Elasticsearch](./elasticsearch.md). + +PostgreSQL, demo only: + +```shell +helm install skywalking oci://registry-1.docker.io/apache/skywalking-helm \ + --version 5.0.0 \ + --set oap.image.tag=11.0.0 \ + --set ui.image.tag=horizon-1.0.0 \ + --set oap.storageType=postgresql \ + --set elasticsearch.enabled=false \ + --set postgresql.enabled=true +``` + +## Next + +- [Elasticsearch](./elasticsearch.md) — ECK operator, CRDs, node sets, external clusters +- [BanyanDB](./banyandb.md) — standalone vs cluster, auth, version locking +- [PostgreSQL](./postgresql.md) — the demo deployment and pointing at your own server +- [The OAP Init Job](../operate/oap-init-job.md) — how the schema gets created in whichever + backend you picked diff --git a/docs/storage/elasticsearch.md b/docs/storage/elasticsearch.md new file mode 100644 index 0000000..a52711a --- /dev/null +++ b/docs/storage/elasticsearch.md @@ -0,0 +1,234 @@ +# Elasticsearch + +How to run SkyWalking OAP on Elasticsearch with this chart — either the ECK-managed cluster the chart +ships (`elasticsearch.enabled: true`, the default) or an Elasticsearch cluster you already run. + +| | Embedded (default) | External | +|---|---|---| +| `elasticsearch.enabled` | `true` | `false` | +| Deploys | ECK operator + an `Elasticsearch` custom resource | nothing | +| CRD pre-install | **required** | not needed | +| Connection settings | computed from the release name | `elasticsearch.config.*` | +| Credentials | auto-generated `elastic` user secret | `elasticsearch.config.user` / `.password` | + +Both modes need the chart's three required values, with `oap.storageType` set to `elasticsearch`: + +```shell +--set oap.image.tag=11.0.0 --set oap.storageType=elasticsearch --set ui.image.tag=horizon-1.0.0 +``` + +## Embedded ECK cluster + +Elasticsearch is deployed through [ECK (Elastic Cloud on Kubernetes)](https://github.com/elastic/cloud-on-k8s). +Two subchart dependencies in `chart/skywalking/Chart.yaml` are gated on `elasticsearch.enabled`: + +| Subchart | Version | Role | +|---|---|---| +| `eck-operator` | 3.3.1 | the ECK controller that turns the custom resource into pods | +| `eck-elasticsearch` (alias `elasticsearch`) | 0.18.1 | renders the `Elasticsearch` custom resource | + +The default Elasticsearch version is `8.18.8` (`elasticsearch.version`). + +### Install the CRDs first + +The chart creates an `elasticsearch.k8s.elastic.co/v1` `Elasticsearch` object, so the ECK CRDs must +already exist when `helm install` runs. Install them once per cluster as their own release, at the +version pinned for `eck-operator` in `chart/skywalking/Chart.yaml`: + +```shell +helm install eck-crds eck-operator-crds \ + --repo https://helm.elastic.co --version 3.3.1 \ + -n "${SKYWALKING_RELEASE_NAMESPACE}" --create-namespace +``` + +Then install SkyWalking with `--set eck-operator.installCRDs=false` so the bundled operator does not +try to create the same CRDs a second time: + +```shell +helm install skywalking oci://registry-1.docker.io/apache/skywalking-helm \ + --version 5.0.0 \ + -n "${SKYWALKING_RELEASE_NAMESPACE}" \ + --set oap.image.tag=11.0.0 \ + --set oap.storageType=elasticsearch \ + --set ui.image.tag=horizon-1.0.0 \ + --set eck-operator.installCRDs=false +``` + +CI installs the same CRD release from a checkout instead, unpacking `eck-operator-crds` out of the +vendored `chart/skywalking/charts/eck-operator-3.3.1.tgz` after `helm dep up` — see +`test/e2e/e2e-elasticsearch.yaml`. + +### What the chart wires up + +The Elasticsearch resource is named `{release}-elasticsearch` (override with +`elasticsearch.fullnameOverride`). ECK derives the rest of the names from it: + +| Object | Name (release `skywalking`) | +|---|---| +| `Elasticsearch` custom resource | `skywalking-elasticsearch` | +| HTTP service | `skywalking-elasticsearch-es-http`, port `9200` | +| Auto-generated auth secret | `skywalking-elasticsearch-es-elastic-user`, key `elastic` | + +Both the OAP Deployment and the OAP init Job get the same storage env block, with the password read +straight from the ECK secret — you never set an Elasticsearch password yourself in this mode: + +```yaml +- name: SW_STORAGE + value: elasticsearch +- name: SW_STORAGE_ES_CLUSTER_NODES + value: "skywalking-elasticsearch-es-http:9200" +- name: SW_ES_USER + value: "elastic" +- name: SW_ES_PASSWORD + valueFrom: + secretKeyRef: + name: skywalking-elasticsearch-es-elastic-user + key: elastic +``` + +To read that password yourself (for `curl`, Kibana, or a support dump): + +```shell +kubectl -n "${SKYWALKING_RELEASE_NAMESPACE}" get secret skywalking-elasticsearch-es-elastic-user \ + -o go-template='{{.data.elastic | base64decode}}' +``` + +The OAP Deployment and the init Job also both get a `wait-for-elasticsearch` init container +(`busybox:1.30`, from `initContainer.image` / `initContainer.tag`) that TCP-probes +`{release}-elasticsearch-es-http:9200` up to 60 times, 5 seconds apart, before OAP starts. If ES +never becomes reachable the init container exits 1 — the Deployment pod sits in +`Init:CrashLoopBackOff` and retries, while the init Job's pod fails and the Job creates another. + +### HTTP TLS is disabled by default + +`values.yaml` ships: + +```yaml +elasticsearch: + http: + tls: + selfSignedCertificate: + disabled: true +``` + +so OAP reaches Elasticsearch over plain HTTP inside the cluster and does not need to trust the ECK +self-signed CA. To re-enable TLS, remove the `tls` section (or set `disabled: false`) **and** +configure OAP to trust the certificate — the chart does not do that part for you. Anything extra OAP +needs can be passed with `--set oap.env.=`, for example +`--set oap.env.SW_STORAGE_ES_HTTP_PROTOCOL=https`. + +`elasticsearch.http` is passed through verbatim to the custom resource, so it also carries service +settings such as `http.service.spec.type: LoadBalancer`. See +[Accessing Elastic services](https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-accessing-elastic-services.html). + +### Cluster topology (`nodeSets`) + +`elasticsearch.nodeSets[]` is passed straight through to the ECK +[node sets](https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-node-configuration.html). +Each entry creates a group of Elasticsearch nodes. + +| Parameter | Default | Notes | +|---|---|---| +| `nodeSets[].name` | `default` | node set name | +| `nodeSets[].count` | `3` | nodes in this set | +| `nodeSets[].config` | `node.store.allow_mmap: false` | Elasticsearch settings (`node.roles`, watermarks, …) | +| `nodeSets[].volumeClaimTemplates` | not set | commented example in `values.yaml`; ECK applies its own default claim when omitted | +| `nodeSets[].podTemplate` | resources `100m` CPU / `2Gi` memory (request), `2Gi` memory (limit) | a normal Pod template under `podTemplate.spec` | + +`nodeSets` is a list, so `--set` replaces the whole thing. Use a values file: + +```yaml +elasticsearch: + nodeSets: + - name: default + count: 3 + config: + node.store.allow_mmap: false + volumeClaimTemplates: + - metadata: + name: elasticsearch-data + spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 30Gi + storageClassName: standard + podTemplate: + spec: + containers: + - name: elasticsearch + resources: + requests: + cpu: 1 + memory: 4Gi + limits: + memory: 4Gi + env: + - name: ES_JAVA_OPTS + value: "-Xms2g -Xmx2g" +``` + +For production, `values.yaml` recommends raising the kernel setting `vm.max_map_count` to `262144` +with a privileged init container and leaving `node.store.allow_mmap` unset instead of `false` — see +[Virtual memory](https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-virtual-memory.html). +`test/e2e/values.yaml` shows the opposite end of the scale: a single node with relaxed disk +watermarks, sized for kind. + +### Other passthrough parameters + +| Parameter | Default | Description | +|---|---|---| +| `elasticsearch.version` | `8.18.8` | Elasticsearch version | +| `elasticsearch.fullnameOverride` | `""` | rename the resource; the service becomes `{name}-es-http` | +| `elasticsearch.labels` / `.annotations` | `{}` | applied to the `Elasticsearch` resource | +| `elasticsearch.secureSettings` | `[]` | [secure settings](https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-es-secure-settings.html) from Kubernetes secrets | +| `elasticsearch.updateStrategy` | `{}` | [change budget](https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-update-strategy.html) for rolling changes | +| `elasticsearch.volumeClaimDeletePolicy` | `""` | `DeleteOnScaledownOnly` or `DeleteOnScaledownAndClusterDeletion` | +| `elasticsearch.ingress.enabled` | `false` | expose Elasticsearch itself through an Ingress | + +## Use an existing Elasticsearch + +Set `elasticsearch.enabled: false` and point `elasticsearch.config` at your cluster. This skips both +the ECK operator and the custom resource, so **no CRD pre-install is needed**. + +```yaml +elasticsearch: + enabled: false + config: + host: elasticsearch-es-http + port: + http: 9200 + user: "xxx" # [optional] + password: "xxx" # [optional] +``` + +| Parameter | Default | Description | +|---|---|---| +| `elasticsearch.config.host` | `elasticsearch` | hostname reachable from the OAP pods | +| `elasticsearch.config.port.http` | `9200` | HTTP port | +| `elasticsearch.config.user` | `""` | optional; sets `SW_ES_USER` | +| `elasticsearch.config.password` | `""` | optional; sets `SW_ES_PASSWORD` | + +`host` and `port.http` feed both `SW_STORAGE_ES_CLUSTER_NODES` and the `wait-for-elasticsearch` init +container probe. `user` and `password` are only emitted when non-empty, and they are rendered as +literal env values in the OAP pod spec — treat the values file as a secret. For a TLS-fronted +cluster add `--set oap.env.SW_STORAGE_ES_HTTP_PROTOCOL=https` (OAP defaults to `http`). + +`chart/skywalking/values-my-es.yaml` is the ready-made example. It already carries the three required +values, so nothing else has to be passed: + +```shell +helm dep up chart/skywalking # charts/ is gitignored — vendor the dependencies first +helm install "${SKYWALKING_RELEASE_NAME}" chart/skywalking \ + -n "${SKYWALKING_RELEASE_NAMESPACE}" \ + -f chart/skywalking/values-my-es.yaml +``` + +## Related + +- [Pick a Storage Backend](choose-a-backend.md) — how Elasticsearch compares with BanyanDB and PostgreSQL +- [The OAP Init Job](../operate/oap-init-job.md) — what creates the Elasticsearch indices +- [Install and Startup Failures](../troubleshooting/install-and-startup.md) — CRD and `wait-for-elasticsearch` errors +- [skywalking Chart Values](../reference/skywalking-chart-values.md) — full parameter list +- [OAP Configuration Vocabulary](https://skywalking.apache.org/docs/main/latest/en/setup/backend/configuration-vocabulary/) — every `SW_STORAGE_ES_*` variable diff --git a/docs/storage/postgresql.md b/docs/storage/postgresql.md new file mode 100644 index 0000000..75c96ff --- /dev/null +++ b/docs/storage/postgresql.md @@ -0,0 +1,151 @@ +# PostgreSQL + +How to run OAP on PostgreSQL with this chart — either the bundled demo database (`postgresql.enabled=true`) or a PostgreSQL server you already operate — and exactly which connection environment variables the chart derives for the OAP pods. + +## The bundled PostgreSQL is a demo, not a production database + +`chart/skywalking/values.yaml` says so in the value itself: + +```yaml +postgresql: + enabled: false # Whether to start a demo postgresql deployment, don't use this for production. +``` + +What that means concretely: + +| What the chart does | Consequence | +| --- | --- | +| Ships `primary.persistence.enabled: false` and `readReplicas.persistence.enabled: false` | The Bitnami StatefulSet gets no `volumeClaimTemplates` — its `data` volume is `emptyDir: {}` and is not even mounted, so PGDATA lives in the pod's writable layer and every trace and metric is gone when the pod restarts. | +| Ships literal credentials (`auth.password: "123456"`) | They are rendered as plain `env` values in the OAP pod spec, not a `secretKeyRef` (see below). | +| Pins the Bitnami `postgresql` subchart to 12.1.2 from the `archive-full-index` repo (`chart/skywalking/Chart.yaml`) | The image is `docker.io/bitnami/postgresql:15.1.0-debian-11-r0`; the chart and image are frozen, so a chart bump brings you no PostgreSQL patches. | +| Runs no PostgreSQL e2e job — `.github/workflows/e2e.ci.yaml` covers Elasticsearch and BanyanDB only | This path is not exercised on every commit. | + +For anything real, either set `postgresql.enabled: false` and point at a PostgreSQL you (or your cloud provider) run, or pick [BanyanDB](banyandb.md) / [Elasticsearch](elasticsearch.md) — see [Pick a Storage Backend](choose-a-backend.md). + +## Values + +| Value | Default | Effect | +| --- | --- | --- | +| `postgresql.enabled` | `false` | Deploy the bundled Bitnami PostgreSQL subchart (`condition: postgresql.enabled`). | +| `postgresql.config.host` | `postgresql-service.your-awesome-company.com` | Hostname of your own PostgreSQL. Used **only** when `postgresql.enabled: false`. | +| `postgresql.auth.postgresPassword` | `"123456"` | Password of the `postgres` superuser in the bundled database. | +| `postgresql.auth.username` | `postgres` | Goes into `SW_DATA_SOURCE_USER` and into the `pg_isready -U` probe. | +| `postgresql.auth.password` | `"123456"` | Goes into `SW_DATA_SOURCE_PASSWORD`. | +| `postgresql.auth.database` | `skywalking` | Database name in `SW_JDBC_URL`. | +| `postgresql.containerPorts.postgresql` | `5432` | Port in `SW_JDBC_URL` and in the wait probe — in **both** embedded and external mode. Both point at a Service, so in embedded mode change `postgresql.primary.service.ports.postgresql` (Bitnami's Service port, also `5432`) to match, or the URL points at a port nothing serves. | +| `postgresql.primary.persistence.enabled` | `false` | Bundled primary uses `emptyDir`. | +| `postgresql.readReplicas.persistence.enabled` | `false` | Same for read replicas (the subchart's default `architecture` is `standalone`, so none are deployed unless you change it). | + +Any other key under `postgresql.` is passed straight through to the Bitnami subchart. + +## Install with the bundled demo database + +```shell +helm install skywalking \ + oci://registry-1.docker.io/apache/skywalking-helm \ + --version 5.0.0 \ + -n default \ + --set oap.image.tag=11.0.0 \ + --set oap.storageType=postgresql \ + --set ui.image.tag=horizon-1.0.0 \ + --set elasticsearch.enabled=false \ + --set postgresql.enabled=true +``` + +`elasticsearch.enabled=false` matters: the ES subchart and the ECK operator are enabled by default, so leaving it out deploys an Elasticsearch nobody reads from. + +The chart addresses the bundled database as `-postgresql` on `postgresql.containerPorts.postgresql`, which is the Service the Bitnami subchart creates for a release named `skywalking`. + +## Point at your own PostgreSQL + +Leave `postgresql.enabled` at `false` (the default) and set the host — the rest of the `auth` block still supplies the user, password, database and port: + +```yaml +# my-postgres.yaml +oap: + storageType: postgresql +elasticsearch: + enabled: false +postgresql: + enabled: false + config: + host: pg.example.com + auth: + username: skywalking + password: "change-me" + database: skywalking + containerPorts: + postgresql: 5432 +``` + +```shell +helm install skywalking \ + oci://registry-1.docker.io/apache/skywalking-helm \ + --version 5.0.0 \ + -n default \ + --set oap.image.tag=11.0.0 \ + --set ui.image.tag=horizon-1.0.0 \ + -f my-postgres.yaml +``` + +The database itself must exist before install; the chart only creates the SkyWalking schema inside it (via the [OAP init Job](../operate/oap-init-job.md)). + +## What the chart derives + +`skywalking.oap.envs.storage` in `templates/_helpers.tpl` builds these for **both** the OAP Deployment and the OAP init Job — rendered from the `my-postgres.yaml` above: + +```yaml +- name: SW_STORAGE + value: postgresql +- name: SW_JDBC_URL + value: "jdbc:postgresql://pg.example.com:5432/skywalking" +- name: SW_DATA_SOURCE_USER + value: "skywalking" +- name: SW_DATA_SOURCE_PASSWORD + value: "change-me" +``` + +| Env var | Built from | +| --- | --- | +| `SW_STORAGE` | `oap.storageType` | +| `SW_JDBC_URL` | `jdbc:postgresql://:/`, where `` is `-postgresql` when `postgresql.enabled: true`, otherwise `postgresql.config.host` | +| `SW_DATA_SOURCE_USER` | `postgresql.auth.username` | +| `SW_DATA_SOURCE_PASSWORD` | `postgresql.auth.password` | + +These are the same knobs as the upstream `storage.postgresql` block: https://skywalking.apache.org/docs/main/latest/en/setup/backend/storages/postgresql/ — anything else on that page (HikariCP pool sizing, `SW_STORAGE_MAX_SIZE_OF_BATCH_SQL`, …) is not templated, so set it through `oap.env`: + +```yaml +oap: + env: + SW_STORAGE_MAX_SIZE_OF_BATCH_SQL: "2000" +``` + +Both the Deployment and the init Job also get a `wait-for-postgresql` init container (image `postgres:13`, not overridable by `initContainer.image`) that loops on `pg_isready -h -p -U ` every 3 seconds: + +```shell +until pg_isready -h 'skywalking-postgresql' -p '5432' -U 'postgres'; do + echo "Waiting for postgresql..." + sleep 3 +done +``` + +Verify the whole thing without installing anything: + +```shell +helm template skywalking chart/skywalking \ + --set oap.image.tag=11.0.0 \ + --set oap.storageType=postgresql \ + --set ui.image.tag=horizon-1.0.0 \ + --set elasticsearch.enabled=false \ + --set postgresql.enabled=true | grep -A1 'SW_JDBC_URL' +``` + +## Gotchas + +**`auth.password` is ignored by the bundled database when the user is `postgres`.** With the default `auth.username: postgres`, the Bitnami StatefulSet takes `POSTGRES_PASSWORD` from the Secret key `postgres-password` (i.e. `auth.postgresPassword`) and never uses the `password` key it also writes, while OAP is handed `auth.password` as `SW_DATA_SOURCE_PASSWORD`. The defaults are both `"123456"` so it works out of the box, but changing only one of them gives OAP a password the database never had. Change both, or set a non-`postgres` `auth.username` — then Bitnami creates that user with `auth.password` and the two agree. + +**Do not name the release `*postgresql*` with the bundled database.** The chart computes the host as `-postgresql`, but Bitnami collapses its own name when the release already contains the chart name. Release `my-postgresql` yields the Service `my-postgresql` and `SW_JDBC_URL` of `jdbc:postgresql://my-postgresql-postgresql:5432/skywalking`, which never connects. The same mismatch appears if you set `postgresql.fullnameOverride` / `postgresql.nameOverride`. Workaround: run the database separately and use `postgresql.config.host`. + +**The wait loop is unbounded.** Unlike the Elasticsearch and BanyanDB init containers, which give up after 60 attempts, `wait-for-postgresql` retries forever. An OAP pod stuck in `Init:0/1` with `Waiting for postgresql...` in the init container log means a wrong host, port, user, or a firewalled database — it will not fail on its own. See [Install and Startup Failures](../troubleshooting/install-and-startup.md). + +**Credentials are visible in the pod spec.** `SW_DATA_SOURCE_PASSWORD` is a literal `value:`, and `oap.env` renders string values only (no `valueFrom`), so there is no supported way to source it from a Secret today. Anyone who can read the OAP Pod, Deployment or Job can read the password. diff --git a/docs/troubleshooting/install-and-startup.md b/docs/troubleshooting/install-and-startup.md new file mode 100644 index 0000000..f06a04d --- /dev/null +++ b/docs/troubleshooting/install-and-startup.md @@ -0,0 +1,325 @@ +# Install and Startup Failures + +Symptom → cause → fix for everything between `helm install` and the first Ready OAP pod: template +errors, hanging `--wait`, init containers that never finish, and OAP crash loops. + +Examples below assume release `skywalking` in namespace `skywalking`. With that release name the +chart's resources are `skywalking-skywalking-helm-oap`, `-ui`, and a one-shot Job named +`skywalking-skywalking-helm-oap-init-` (add `--set fullnameOverride=skywalking` to get the +short `skywalking-oap` names instead). + +## Triage first + +```shell +helm status skywalking -n skywalking +kubectl get pods,jobs -n skywalking +kubectl describe pod -n skywalking -l component=oap +kubectl logs -n skywalking -l release=skywalking --all-containers --tail=200 +``` + +`kubectl describe` is where init-container state (`Init:0/1`, `Init:Error`) and probe/OOM kill +reasons appear; `kubectl logs -l release=skywalking` covers the OAP pods, the UI pod and the init +Job pod in one shot. + +--- + +## Helm fails before anything is created + +### `execution error at (...): ui.image.tag is required` + +```text +Error: execution error at (skywalking-helm/templates/ui-deployment.yaml:77:52): ui.image.tag is required +Error: execution error at (skywalking-helm/templates/oap-init.job.yaml:83:53): oap.image.tag is required +Error: execution error at (skywalking-helm/templates/oap-init.job.yaml:92:12): oap.storageType is required +``` + +**Cause.** Three values have no default and are wrapped in Helm's `required` function. Rendering +stops at the first one missing, so with none of them set you hit them one at a time, in the order +above. + +**Fix.** Set all three (and `satellite.image.tag` too, if `satellite.enabled=true`): + +| value | example | notes | +|---|---|---| +| `oap.image.tag` | `11.0.0` | | +| `oap.storageType` | `elasticsearch` | also `banyandb`, `postgresql` | +| `ui.image.tag` | `horizon-1.0.0` | must be a `horizon-*` tag | +| `satellite.image.tag` | `v0.4.0` | only when `satellite.enabled=true` | + +### `no matches for kind "Elasticsearch" ... ensure CRDs are installed first` + +```text +Error: INSTALLATION FAILED: unable to build kubernetes objects from release manifest: +resource mapping not found for name: "skywalking-elasticsearch" ... no matches for kind +"Elasticsearch" in version "elasticsearch.k8s.elastic.co/v1": ensure CRDs are installed first +``` + +**Cause.** `elasticsearch.enabled` is `true` (the default) so the chart renders an `Elasticsearch` +custom resource. The `eck-operator` dependency ships its CRDs as ordinary templates in the *same* +release, and Helm has to map every object against the API server before it applies any of them — +so the CR cannot be validated in the install that would also create its CRD. + +**Fix.** Install the CRDs as their own release first, then tell the chart not to install them again: + +```shell +helm install eck-crds eck-operator-crds \ + --repo https://helm.elastic.co --version 3.3.1 \ + -n skywalking --create-namespace + +helm install skywalking oci://registry-1.docker.io/apache/skywalking-helm --version 5.0.0 \ + -n skywalking \ + --set oap.image.tag=11.0.0 \ + --set oap.storageType=elasticsearch \ + --set ui.image.tag=horizon-1.0.0 \ + --set eck-operator.installCRDs=false +``` + +Only `elasticsearch.enabled=false` switches this off: both subcharts are gated on that value alone, +so pointing `oap.storageType` at `banyandb` or `postgresql` still renders the ECK CRDs and the +`Elasticsearch` CR. Add `--set elasticsearch.enabled=false` and the step does not apply. + +### `exists and cannot be imported into the current release` + +```text +Error: INSTALLATION FAILED: Unable to continue with install: CustomResourceDefinition +"elasticsearches.elasticsearch.k8s.elastic.co" in namespace "" exists and cannot be imported +into the current release: invalid ownership metadata +``` + +**Cause.** The ECK CRDs are already owned by another release (your `eck-crds` release, or an ECK +operator installed cluster-wide earlier) and `eck-operator.installCRDs` is still `true`, so this +release tries to claim them too. CRDs are cluster-scoped — only one release can own them. + +**Fix.** `--set eck-operator.installCRDs=false`. Note that the `eck-operator` and `elasticsearch` +subcharts share one condition (`elasticsearch.enabled`) — you cannot install the CR without the +operator through this chart. If a full ECK operator already runs in the cluster, set +`elasticsearch.enabled=false` and point OAP at the existing cluster with `elasticsearch.config.*` +(see [Elasticsearch](../storage/elasticsearch.md)). + +--- + +## `helm install` hangs, then times out + +### `Error: ... timed out waiting for the condition` with `--wait` + +**Cause.** With `--wait`, Helm blocks until every resource is Ready. The OAP Deployment runs with +`JAVA_OPTS="-Dmode=no-init ..."`: it deliberately keeps port `12800` closed until the storage +schema exists, and the schema is created by the one-shot `*-oap-init-*` Job. So a hang means the +init Job has not succeeded — it is failing, or still waiting on storage. + +The init Job is a **normal release resource**, not a Helm hook, precisely so this resolves: Job and +Deployment run in the same install phase. (A `post-install` hook would deadlock forever — Helm +waits for the Deployment to be Ready before running post-hooks, and the Deployment is waiting for +the hook.) + +**Fix.** Add `--wait-for-jobs` so Helm surfaces the Job's failure directly instead of only +reporting that OAP never became Ready, and read the Job's logs: + +```shell +kubectl get jobs -n skywalking +kubectl logs -n skywalking -l component=skywalking-skywalking-helm-job --tail=200 +``` + +Then work down the list below: the Job's own `wait-for-*` init container is the usual culprit. +See [The OAP Init Job](../operate/oap-init-job.md) for the full mechanism. + +### OAP pod is `Running` but never Ready + +**Cause.** Same handshake seen from the pod side: `no-init` OAP is alive but has not opened `12800` +because the schema is missing. Either the init Job never ran, or it failed. The pod stays at `0/1` +and its `RESTARTS` count climbs every ~5 minutes, as the default startup-probe budget expires. + +**Fix.** Confirm the Job completed, and if it is missing, re-run it: + +```shell +kubectl get jobs -n skywalking # want COMPLETIONS 1/1 +kubectl delete job -n skywalking -l release=skywalking +helm upgrade skywalking -n skywalking --reuse-values +``` + +Helm recreates the deleted Job and init runs again. The Job's name carries a hash of the chart +values, so any `helm upgrade` that changes a value re-runs init on its own — an upgrade that +changes nothing leaves the identical Job in place and does **not** re-run it. + +--- + +## Pods never get past init + +### Pod stuck in `Init:0/1` + +**Cause.** The `wait-for-storage` init container (attached to both the OAP Deployment and the init +Job) is still polling the storage backend. What it polls depends on `oap.storageType`: + +| `oap.storageType` | init container | probe | budget | +|---|---|---|---| +| `elasticsearch` | `wait-for-elasticsearch` (`busybox:1.30`) | `nc -z -w3 9200` | 60 tries, 5s apart, then `exit 1` | +| `banyandb` | `wait-for-banyandb` (`curlimages/curl`) | `curl -k /api/healthz` | 60 tries, 5s apart, then `exit 1` | +| `postgresql` | `wait-for-postgresql` (`postgres:13`) | `pg_isready -h -p 5432 -U ` | **unbounded** — retries every 3s forever | + +The PostgreSQL loop never gives up, so a wrong host there shows as a pod that sits in `Init:0/1` +indefinitely rather than erroring. Check what it is actually waiting on: + +```shell +kubectl logs -n skywalking -l component=oap -c wait-for-postgresql +kubectl get pods,svc -n skywalking # is the backend even scheduled? +``` + +**Fix.** Usually the backend itself is not up yet (see the Pending entry below) — wait. If the +address is wrong, correct it: `elasticsearch.config.host` / `.port.http`, +`banyandb.config.httpAddress`, or `postgresql.config.host` for external backends; for embedded +ones the addresses are derived by the chart and a mismatch means the subchart is disabled or +renamed (`*.fullnameOverride`). + +### Pod goes `Init:Error` / `Init:CrashLoopBackOff` after about five minutes + +**Cause.** The Elasticsearch or BanyanDB wait container exhausted its 60 × 5s budget and exited +`1`. The pod restarts and tries again, so this repeats until storage answers. + +**Fix.** Same as above — but five minutes of silence normally means the backend is broken or +unreachable, not slow. For the embedded ECK path, check the Elasticsearch resource and its pods: + +```shell +kubectl get elasticsearch,pods -n skywalking +kubectl logs -n skywalking -l elasticsearch.k8s.elastic.co/cluster-name=skywalking-elasticsearch +``` + +### Storage pods stay `Pending` + +**Cause.** The default Elasticsearch topology is **3 nodes** (`elasticsearch.nodeSets[0].count: 3`) +each requesting `2Gi` of memory, and each ES node also takes a PersistentVolume from the default +StorageClass (ECK's own default claim — the chart ships `nodeSets[0].volumeClaimTemplates` +commented out). BanyanDB claims volumes for its data nodes in cluster mode +(`banyandb.storage.data.enabled: true`); BanyanDB standalone +(`banyandb.storage.standalone.enabled: false`) and the demo PostgreSQL +(`postgresql.primary.persistence.enabled: false`) claim none. A laptop-sized cluster usually has +neither the memory nor a default StorageClass. + +**Fix.** `kubectl describe pod` names the reason (`Insufficient memory`, `no persistent volumes +available`). Shrink the cluster the way the e2e suite does — `test/e2e/values.yaml` sets +`count: 1` — or switch to BanyanDB standalone. See [Requirements](../evaluate/requirements.md). + +### `ImagePullBackOff` on the UI image + +```text +Failed to pull image "skywalking.docker.scarf.sh/apache/skywalking-ui:11.0.0": manifest unknown +``` + +**Cause.** `ui.image.tag` was set to the OAP version. Horizon UI releases independently and its +images are tagged `horizon-`; there is no `11.0.0` tag on `apache/skywalking-ui`. + +**Fix.** `--set ui.image.tag=horizon-1.0.0`. Dev builds live at +`ghcr.io/apache/skywalking-horizon-ui` and need `ui.image.repository` changed too. The legacy +booster UI is not supported by this chart — see [Horizon UI in This Chart](../ui/horizon-ui.md). + +--- + +## OAP starts and then dies + +### OAP restarts every few minutes during a cold start + +**Cause.** The startup probe budget ran out. The default `startupProbe` is a TCP check on `12800` +with `failureThreshold: 30` and `periodSeconds: 10` — a 300-second budget covering storage startup +*plus* schema creation by the init Job. On a slow cluster (cold image pulls, a large Elasticsearch +coming up, a big BanyanDB schema) 300s can be too short, and the kubelet restarts a pod that was +legitimately waiting. + +**Fix.** Raise the budget — `oap.startupProbe` is empty by default and, when set, replaces the +whole block: + +```yaml +oap: + startupProbe: + tcpSocket: + port: 12800 + failureThreshold: 90 # 90 * 10s = 15 minutes + periodSeconds: 10 +``` + +Do not "fix" this by loosening `livenessProbe` instead: liveness only starts counting once the +startup probe has passed. + +### `Incompatible BanyanDB server API version` + +**Cause.** OAP and BanyanDB are version-locked. OAP ships the list of server API versions it +accepts in `SW_STORAGE_BANYANDB_COMPATIBLE_SERVER_API_VERSIONS`; **OAP 11.0.0 accepts API `0.11`**, +which maps to BanyanDB release `0.11.x`. Pairing OAP 11 with BanyanDB 0.10.x makes OAP refuse to +start. + +**Fix.** Match the versions — `--set banyandb.image.tag=0.11.0` for OAP 11.0.0. The API-version to +release mapping is at +[BanyanDB API versions](https://skywalking.apache.org/docs/skywalking-banyandb/latest/installation/versions/); +the chart-side details are in [BanyanDB](../storage/banyandb.md) and +[Version Compatibility](../evaluate/version-compatibility.md). + +### OAP `OOMKilled` + +**Cause.** `oap.javaOpts` defaults to `-Xmx2g -Xms2g` while `oap.resources` is `{}`. Adding a memory +limit below roughly 2.5Gi without changing the heap gets the JVM killed as soon as it touches its +heap — the heap alone is 2Gi before metaspace, direct buffers and thread stacks. + +**Fix.** Change both together, keeping the limit comfortably above `-Xmx`: + +```shell + --set oap.javaOpts="-Xmx4g -Xms4g" \ + --set oap.resources.limits.memory=6Gi \ + --set oap.resources.requests.memory=6Gi +``` + +### `pods is forbidden: User "system:serviceaccount:..." cannot list resource "pods"` + +**Cause.** OAP always runs with `SW_CLUSTER=kubernetes` in this chart and needs read access to +pods, endpoints, services, nodes, namespaces and configmaps to find its peers. The chart creates +that Role/ClusterRole and binding only when `serviceAccounts.oap.create` is `true`; setting it to +`false` and pointing `serviceAccounts.oap.name` at an unprivileged ServiceAccount leaves OAP unable +to form a cluster. + +**Fix.** Either leave `serviceAccounts.oap.create=true`, or grant your own ServiceAccount the same +rules as `chart/skywalking/templates/oap-clusterrole.yaml` and `oap-role.yaml`. + +### `FORBIDDEN/12/index read-only / allow delete (api)` in OAP logs + +**Cause.** Elasticsearch hit its flood-stage disk watermark (95% by default) and put indices into +read-only mode. Common on kind / small dev clusters where the node disk is shared. + +**Fix.** Free disk, or relax the watermarks the way the e2e overlay does in `test/e2e/values.yaml`: + +```yaml +elasticsearch: + nodeSets: + - name: default + # nodeSets is a list: your entry replaces the shipped one whole, so `count` + # (required by the CRD) and `node.store.allow_mmap` have to be repeated here. + count: 3 + config: + node.store.allow_mmap: false + cluster.routing.allocation.disk.watermark.low: 90% + cluster.routing.allocation.disk.watermark.high: 99% + cluster.routing.allocation.disk.watermark.flood_stage: 99% +``` + +Existing indices stay read-only until the block is cleared on the Elasticsearch side. + +--- + +## GitOps: init runs on every reconcile + +**Cause.** `oapInit.ttlSecondsAfterFinished` was set. The Kubernetes TTL-after-finished controller +deletes the completed Job, and Argo CD / Flux then see a missing resource and recreate it — so init +re-runs on every reconcile loop. + +**Fix.** Leave `oapInit.ttlSecondsAfterFinished` empty (the default) under GitOps. The Job name is +value-hashed, so upgrades already replace it correctly without a TTL; the setting exists only to +tidy finished Jobs in non-GitOps installs. + +--- + +## Still stuck? + +- [The OAP Init Job](../operate/oap-init-job.md) — how the `init` / `no-init` handshake works. +- [UI and Login Problems](ui-and-login.md) — the UI is Ready but nobody can log in. +- [BanyanDB](../storage/banyandb.md) · [Elasticsearch](../storage/elasticsearch.md) · + [PostgreSQL](../storage/postgresql.md) — backend-specific settings. +- [Requirements](../evaluate/requirements.md) — cluster version, RBAC and volume prerequisites. +- Ask on the mailing list `dev@skywalking.apache.org`, or open an + [issue](https://github.com/apache/skywalking/issues) with `helm get manifest` output and the + failing pod's `describe` + logs. diff --git a/docs/troubleshooting/ui-and-login.md b/docs/troubleshooting/ui-and-login.md new file mode 100644 index 0000000..d91f5b0 --- /dev/null +++ b/docs/troubleshooting/ui-and-login.md @@ -0,0 +1,277 @@ +# UI and Login Problems + +Symptom-keyed fixes for the Horizon UI deployed by this chart: a Ready pod nobody can sign into, +empty layer pages, `HORIZON_*` variables that do nothing, broken SSO redirects, sessions that drop, +and `swctl` calls that used to work against the UI. For install-time crashes and `CrashLoopBackOff`, +see [Install and Startup Failures](install-and-startup.md). + +Two names are used throughout: the UI Service is `${SKYWALKING_RELEASE_NAME}-skywalking-helm-ui`, +the OAP Service is `${SKYWALKING_RELEASE_NAME}-skywalking-helm-oap`. + +| Symptom | Cause | Fix | +|---|---|---| +| Pod Ready, every login rejected | No users configured | [Seed `auth.local.users`](#the-pod-is-ready-but-nobody-can-log-in) | +| Layer pages blank, "template store unreachable" | OAP 10.x + `templates.mode: live` | [`templates.mode: readonly`](#layer-pages-are-empty-behind-a-dashboard-template-store-unreachable-banner) | +| A `HORIZON_*` env var is ignored | A literal in `ui.config` shadows it | [Write a `${...}` token instead](#a-horizon_-env-var-has-no-effect) | +| SSO callback hits the internal address | `server.publicUrl` unset or wrong | [Set `publicUrl`](#sso-callback-or-oauth-issuer-points-at-the-internal-address) | +| Logged out on every other request | `ui.replicas > 1`, no sticky routing | [Back to one replica](#logins-drop-on-every-other-request) | +| `swctl` against the UI fails | No `/graphql` passthrough | [Target OAP `12800`](#swctl-against-the-ui-returns-404-or-html) | + +## The pod is Ready but nobody can log in + +Horizon has no `admin/admin` fallback and the chart configures no users. The BFF **does not fail +closed**: with an empty user list it boots, logs an error, serves the login page, and answers its +readiness probe with `200`. Kubernetes therefore reports a perfectly healthy Deployment while every +credential is rejected. + +`ui.readinessProbe` hits `/api/auth/health`, which is a public route by design — it is what the login +page reads to render its setup banner — so its `200` says nothing about whether auth works. The +response body does: + +```shell +kubectl port-forward -n "${SKYWALKING_RELEASE_NAMESPACE}" \ + svc/${SKYWALKING_RELEASE_NAME}-skywalking-helm-ui 8080:80 +curl -s http://127.0.0.1:8080/api/auth/health +``` + +```json +{"backend":"local","configured":false,"setupHint":"No users configured. Add at least one entry to auth.local.users in horizon.yaml (use `pnpm --filter bff cli:hash` for the password hash) or switch to LDAP.","ldap":null,"breakGlass":{"armed":false}} +``` + +`configured: false` is the whole diagnosis. The same state appears once in the container log at boot: + +```shell +kubectl logs -n "${SKYWALKING_RELEASE_NAMESPACE}" \ + deploy/${SKYWALKING_RELEASE_NAME}-skywalking-helm-ui | grep 'auth.local.users is empty' +``` + +Fix it by seeding users — see [Set Up Logins](../ui/logins.md) for the demo snippet and the +production Secret pattern. Re-check `/api/auth/health` after the rollout; `configured: true` is the +confirmation. + +Two things also produce `configured: false` with an LDAP backend: `auth.backend: ldap` with no +`auth.ldap` block, and an `auth.ldap` whose `groupMappings` list is empty. `setupHint` names which. + +## Layer pages are empty behind a "Dashboard template store unreachable" banner + +Full-width banner reading **Dashboard template store unreachable**, with "Layer dashboards, overviews +and topology are blocked until OAP's UI-template store is reachable." Traces, logs and alarms may +still render. + +Horizon's default `templates.mode` is `live`, which reads dashboard definitions from OAP 11's +`/ui-management/templates*` admin REST API and treats the OAP-stored row as the only source — an +unreachable store blocks the page rather than quietly substituting the bundled defaults. Two things +trigger it: + +- **You are running OAP 10.x.** OAP 10 has only a legacy GraphQL template API, which Horizon does + not consume, and Horizon does not fall back on your behalf. Set `readonly`: + + ```shell + --set ui.config.templates.mode=readonly + ``` + + In `readonly` mode Horizon renders the templates bundled in its own image and never calls a + template-management API. Dashboards, traces, logs, topology, alarms and profiling all work; the + template configuration surface becomes display-only. + +- **You are on OAP 11 and the admin port is not reachable.** The chart points `oap.adminUrl` at + `oap.ports.admin` (`17128`). If that port was removed from `oap.ports`, or a NetworkPolicy blocks + it, `live` mode has nothing to read. Confirm from inside the UI pod: + + ```shell + kubectl exec -n "${SKYWALKING_RELEASE_NAMESPACE}" \ + deploy/${SKYWALKING_RELEASE_NAME}-skywalking-helm-ui -- \ + wget -qO- http://${SKYWALKING_RELEASE_NAME}-skywalking-helm-oap:17128/ui-management/templates + ``` + + That is the exact path Horizon probes at boot. A connection refused or a 404 is the problem. + +Note that `oap.ports.admin` is required for OAP 11 regardless of template mode — `/status/*` and +`/debugging/*` are served only there. See [Horizon UI in This Chart](../ui/horizon-ui.md). + +## A `HORIZON_*` env var has no effect + +You added a variable through `ui.extraEnv` or `ui.envFromSecret`, the pod restarted, and the setting +did not change. + +The chart mounts its ConfigMap **over** the image's `/app/horizon.yaml`. Horizon expands `${...}` +over the raw **text** of whatever file sits at that path, before parsing it as YAML. So a +`HORIZON_*` variable is read in exactly two situations: + +1. A matching `${HORIZON_*:default}` token appears in the mounted file's text, or +2. The field is absent from the file **and** its schema default itself reads `process.env` — true for + only a handful of fields, `templates.mode` and `server.publicUrl` among them. + +That is why the chart writes its computed values as tokens rather than literals: + +```yaml +oap: + adminUrl: ${HORIZON_OAP_ADMIN_URL:http://sw-skywalking-helm-oap:17128} + queryUrl: ${HORIZON_OAP_QUERY_URL:http://sw-skywalking-helm-oap:12800} +server: + port: 8081 +``` + +The in-cluster address is the default and the env var still wins. Anything **you** write into +`ui.config` is emitted as a plain literal, and a literal makes the matching variable inert — which is +why `ui.config` is empty by default. + +Check what actually got mounted before anything else: + +```shell +kubectl get cm -n "${SKYWALKING_RELEASE_NAMESPACE}" \ + ${SKYWALKING_RELEASE_NAME}-skywalking-helm-ui -o jsonpath='{.data.horizon\.yaml}' +``` + +| What the file shows | What the env var does | +|---|---| +| `queryUrl: ${HORIZON_OAP_QUERY_URL:...}` | Overrides it | +| `queryUrl: http://my-oap:12800` (your literal) | Ignored | +| Field absent, schema default reads env (e.g. `templates.mode`) | Overrides it | +| Field absent, schema default is a constant (e.g. `auth.local.users`) | Ignored | + +The fix is to write the token yourself in `ui.config` rather than the value: + +```yaml +ui: + envFromSecret: horizon-secrets + config: + oap: + queryUrl: "${HORIZON_OAP_QUERY_URL:http://sw-skywalking-helm-oap:12800}" +``` + +One more failure mode in the same family: changing only the **Secret** does not restart anything. The +Deployment carries a `checksum/config` annotation over the rendered ConfigMap, so `ui.config` edits +roll the pod, but env vars are read once at container start. After editing a Secret: + +```shell +kubectl rollout restart -n "${SKYWALKING_RELEASE_NAMESPACE}" \ + deploy/${SKYWALKING_RELEASE_NAME}-skywalking-helm-ui +``` + +Details and the full field table are in [Configure Horizon](../ui/configure.md). + +## SSO callback or OAuth issuer points at the internal address + +The provider rejects the redirect URI, or the browser is bounced to something like +`http://sw-skywalking-helm-ui:8081/api/auth/oidc/callback` after signing in. + +Horizon builds the OIDC callback URI from `server.publicUrl`; when that is blank it falls back to the +request's own origin (`req.protocol` + the `Host` header). Behind an ingress that rewrites `Host`, or +anywhere TLS terminates upstream, that fallback produces the internal address. `oauth.issuer` also +defaults to `server.publicUrl`, so the same blank value gives you a wrong issuer. + +The chart derives `publicUrl` automatically, but **only when both** `ui.ingress.enabled` is true +**and** `ui.ingress.hosts` is non-empty — it takes the first host, with `https` when +`ui.ingress.tls` is set and `http` otherwise: + +```yaml +server: + port: 8081 + publicUrl: ${HORIZON_PUBLIC_URL:http://skywalking.example.com} +``` + +So it is absent when you expose the UI by LoadBalancer, NodePort, or a Gateway API / service-mesh +route instead of the chart's Ingress. Set it yourself in that case, either as env: + +```yaml +ui: + extraEnv: + - name: HORIZON_PUBLIC_URL + value: https://skywalking.example.com +``` + +or pinned in `ui.config.server.publicUrl`. Two rules: it must be **byte-identical** to the redirect +URI registered with the provider, and if a gateway serves Horizon under a path prefix +(`https://example.com/horizon/`), `publicUrl` must carry that prefix — it is also what the BFF uses +to build root-relative redirects back to `/login`. + +While you are here, two neighbouring settings — and both belong in `ui.config`, not in `ui.extraEnv`. +Unlike `server.publicUrl`, neither field's schema default reads the environment, so with the chart's +ConfigMap mounted over `/app/horizon.yaml` neither of the two situations above applies, and a +`HORIZON_SESSION_COOKIE_SECURE` or `HORIZON_TRUST_PROXY` variable is inert: + +```yaml +ui: + config: + session: + cookieSecure: true # serving over HTTPS + server: + trustProxy: 1 # hop count — 1 = one proxy in front; an address/CIDR list also works +``` + +`trustProxy` is what makes the login audit record the real client address rather than the ingress. +`trustProxy: true` is refused at boot. See [TLS](../expose/tls.md) and +[UI Service and Ingress](../expose/ui-service-and-ingress.md). + +## Logins drop on every other request + +You sign in, click once, and land back on the login page — then it works again, then it does not. + +The Horizon BFF keeps its session table **in memory**, per pod. Two pods hold disjoint session state, +so requests round-robined between them are authenticated only half the time. This is why +`ui.replicas` defaults to `1` and why the UI Deployment uses `strategy: Recreate` rather than +`RollingUpdate` — a rolling update would open the same window during every upgrade. + +```shell +kubectl get deploy -n "${SKYWALKING_RELEASE_NAMESPACE}" \ + ${SKYWALKING_RELEASE_NAME}-skywalking-helm-ui -o jsonpath='{.spec.replicas}' +``` + +If that is greater than `1`, either go back to one replica: + +```shell +--set ui.replicas=1 +``` + +or configure session affinity on your ingress controller. The chart exposes no `sessionAffinity` +field on the UI Service, so this has to come from `ui.ingress.annotations` — for ingress-nginx: + +```yaml +ui: + replicas: 2 + ingress: + enabled: true + annotations: + nginx.ingress.kubernetes.io/affinity: "cookie" + nginx.ingress.kubernetes.io/session-cookie-name: "horizon-route" + hosts: + - skywalking.example.com +``` + +Two related non-bugs: a `helm upgrade` that changes `ui.config` rolls the pod and drops every session +by design, and `ui.persistence` with the default `ReadWriteOnce` access mode cannot be shared by two +pods at all. + +## `swctl` against the UI returns 404 or HTML + +A command that used to work — `swctl --base-url=http:///graphql ...` — now fails. + +Horizon's BFF **does not proxy `/graphql`** to OAP. It talks to OAP itself on the query port +(`12800`) and the admin port (`17128`) and exposes its own `/api/*` surface; there is no passthrough +for external GraphQL clients, and the UI container listens on `8081` rather than the `8080` the +legacy booster UI used. Point every API caller at the OAP Service directly: + +```shell +kubectl port-forward -n "${SKYWALKING_RELEASE_NAMESPACE}" \ + svc/${SKYWALKING_RELEASE_NAME}-skywalking-helm-oap 12800:12800 + +swctl --display yaml --base-url=http://127.0.0.1:12800/graphql service ls +``` + +In-cluster, that is +`http://${SKYWALKING_RELEASE_NAME}-skywalking-helm-oap:12800/graphql`. The chart's own e2e suites do +exactly this — every `swctl` invocation under `test/e2e/` targets OAP port `12800`. + +For a caller that genuinely needs Horizon's own API rather than OAP's — a script, CI job, or MCP +client — use Horizon API tokens instead. `auth.tokensFile` takes a filesystem **path**, so it needs a +mounted Secret via `ui.extraVolumes` / `ui.extraVolumeMounts`; see +[Configure Horizon](../ui/configure.md). + +## Related pages + +- [Set Up Logins](../ui/logins.md) — seeding users, hashes, roles +- [Configure Horizon](../ui/configure.md) — the full `ui.config` / env surface +- [Horizon UI in This Chart](../ui/horizon-ui.md) — ports, versions, migration from booster UI +- [UI Service and Ingress](../expose/ui-service-and-ingress.md) — exposing the UI +- [`horizon.yaml` reference](https://github.com/apache/skywalking-horizon-ui/blob/main/docs/setup/horizon-yaml.md) — the upstream schema diff --git a/docs/ui/configure.md b/docs/ui/configure.md new file mode 100644 index 0000000..d8ad9ef --- /dev/null +++ b/docs/ui/configure.md @@ -0,0 +1,219 @@ +# Configure Horizon + +How the chart configures the Horizon UI container: what it writes into `horizon.yaml`, why almost everything else should be an environment variable instead, and the two settings — `templates.mode` and `/data` persistence — that most deployments need to touch. + +## The image is configured by environment variable + +The Horizon image ships `/app/horizon.yaml` in which **every field is a `${HORIZON_*:default}` token**. Horizon expands `${...}` over the raw *text* of that file before parsing it as YAML, so the container is meant to run with the shipped file and only the env vars you care about. + +Precedence is: **env var → the file's `:default` → the built-in schema default.** + +Two consequences fall out of that, and both are the reason this chart writes as little as it does: + +- A field written as a **plain literal** in the config file makes its `HORIZON_*` env var **silently inert** — the token it would have replaced is no longer there to expand. +- A field the file **omits** falls back to a built-in default, and only a handful of those consult the environment. + +So a config file full of literals does not merely restate defaults — it disables most of the image's configuration surface. **Prefer [`ui.extraEnv` and `ui.envFromSecret`](#prefer-env-vars) for everything the chart does not compute.** + +## What the chart writes + +`ui.enabled` creates a ConfigMap named `{release}-skywalking-helm-ui` (key `horizon.yaml`), mounted over `/app/horizon.yaml` as a read-only `subPath`. The chart writes **only the values the image cannot know**, and writes them as tokens so env still wins: + +| `horizon.yaml` field | Source in `values.yaml` | Written when | +|---|---|---| +| `oap.queryUrl` | in-cluster OAP service + `oap.ports.rest` (`12800`) | always | +| `oap.adminUrl` | in-cluster OAP service + `oap.ports.admin` (`17128`) | always | +| `oap.zipkinUrl` | in-cluster OAP service + `oap.ports.zipkin-query` + `/zipkin` | only when `oap.ports.zipkin-query` is set | +| `server.publicUrl` | first entry of `ui.ingress.hosts`, `https` if `ui.ingress.tls` is non-empty else `http` | only when `ui.ingress.enabled` **and** `ui.ingress.hosts` are set | +| `server.port` | `ui.service.internalPort` (`8081`) | always | + +With chart defaults, the rendered file is just: + +```yaml +oap: + adminUrl: ${HORIZON_OAP_ADMIN_URL:http://sw-skywalking-helm-oap:17128} + queryUrl: ${HORIZON_OAP_QUERY_URL:http://sw-skywalking-helm-oap:12800} +server: + port: 8081 +``` + +Add a Zipkin query port and an ingress and two more lines appear: + +```yaml +oap: + adminUrl: ${HORIZON_OAP_ADMIN_URL:http://sw-skywalking-helm-oap:17128} + queryUrl: ${HORIZON_OAP_QUERY_URL:http://sw-skywalking-helm-oap:12800} + zipkinUrl: ${HORIZON_OAP_ZIPKIN_URL:http://sw-skywalking-helm-oap:9412/zipkin} +server: + port: 8081 + publicUrl: ${HORIZON_PUBLIC_URL:http://skywalking.example.com} +``` + +Render it yourself before installing: + +```shell +helm template sw chart/skywalking \ + --set oap.image.tag=11.0.0 \ + --set oap.storageType=banyandb \ + --set ui.image.tag=horizon-1.0.0 \ + -s templates/ui-configmap.yaml +``` + +Notes on the table: + +- `oap.zipkinUrl` is omitted rather than blanked when no Zipkin port is exposed: Horizon's schema requires a valid URL there, and an empty value fails at boot. +- `server.port` is the one **literal**, not a token — the container port comes from `ui.service.internalPort`, so the BFF has to bind that same port. `HORIZON_SERVER_PORT` is therefore inert; change `ui.service.internalPort` instead. The probes are *not* derived from it: `ui.livenessProbe` and `ui.readinessProbe` hard-code port `8081` in `values.yaml`, so override them too. +- `server.host` is not written at all — the image's own `ENV` already sets it to `0.0.0.0`. +- Anything equal to Horizon's own default is deliberately left out, so upstream owns it. + +## Prefer env vars + +`ui.extraEnv` adds env vars to the BFF container; `ui.envFromSecret` names a pre-created Secret whose keys all become env vars: + +```yaml +ui: + envFromSecret: horizon-secrets + extraEnv: + - name: HORIZON_SESSION_COOKIE_SECURE + value: "true" + - name: HORIZON_TRUST_PROXY + value: "1" + - name: HORIZON_OAP_AUTH + valueFrom: + secretKeyRef: + name: horizon-secrets + key: oapAuth +``` + +Common settings and the env var that does the same job: + +| `horizon.yaml` field | Env var | Typical use | +|---|---|---| +| `templates.mode` | `HORIZON_TEMPLATES_MODE` | `readonly` against OAP 10.x | +| `session.cookieSecure` | `HORIZON_SESSION_COOKIE_SECURE` | `true` when serving HTTPS | +| `server.trustProxy` | `HORIZON_TRUST_PROXY` | hop count or ingress CIDR (`true` is refused at boot) | +| `auth.local.users` | `HORIZON_AUTH_LOCAL_USERS` | JSON array — see [Set Up Logins](logins.md) | +| `oap.auth` | `HORIZON_OAP_AUTH` | JSON, basic auth for outbound OAP calls | + +The full field list is owned upstream: [horizon.yaml reference](https://github.com/apache/skywalking-horizon-ui/blob/main/docs/setup/horizon-yaml.md). + +## When to use `ui.config` anyway + +`ui.config` is deep-merged **over** the chart-computed values, so it can pin any field regardless of env — which is exactly what you want for a value that must not be overridable, and exactly what you do not want everywhere else. It is `{}` by default. + +Two rules: + +- **A literal you write here kills that field's `HORIZON_*` var.** Writing `templates.mode: readonly` into `ui.config` means `HORIZON_TEMPLATES_MODE` no longer does anything for this deployment. That is fine when pinning is the intent. +- **`${VAR}` tokens you write here still expand.** This is how secrets stay out of the ConfigMap — write the shape in `ui.config`, keep the secret in a Secret referenced by `ui.envFromSecret` / `ui.extraEnv`: + +```yaml +ui: + envFromSecret: horizon-admin + config: + auth: + local: + users: + - username: admin + passwordHash: "${HORIZON_ADMIN_HASH}" + roles: [admin] +``` + +Overriding a chart-computed URL — for example to point Horizon at an OAP outside the release — works the same way, but the env var is the lighter option: + +```yaml +ui: + extraEnv: + - name: HORIZON_OAP_QUERY_URL + value: http://oap.observability.svc:12800 +``` + +## `templates.mode` + +`live` (Horizon's default) reads and writes dashboard templates through OAP 11's `/ui-management/templates*` admin REST API and persists them in OAP storage. In that mode OAP is the only source: if the template store cannot be read, layer pages are blocked rather than falling back to the bundled templates. + +**Against OAP 10.x you must set `readonly`.** OAP 10 does not serve that REST surface, so `live` blocks every layer-driven page. `readonly` renders the templates bundled in the image and makes the configuration surface display-only; dashboards, traces, logs, topology, alarms and profiling all work. + +```yaml +ui: + extraEnv: + - name: HORIZON_TEMPLATES_MODE + value: readonly +``` + +or, pinned in the file: + +```yaml +ui: + config: + templates: + mode: readonly +``` + +Changing the mode requires a BFF restart, not just a config reload. + +## Persistence (`/data`) + +The image declares `/data` as its state volume and routes the BFF's OAP wire debug log there (`HORIZON_WIRE_LOG_FILE=/data/horizon-wire.jsonl`, written only when `debugLog.enabled`); anything else you point at a path under `/data` lands there too. The chart always mounts a volume at `/data` — an `emptyDir` by default, so **that state is gone whenever the pod is replaced** (upgrade, reschedule, delete). Turn on a PVC for anything you intend to keep: + +```yaml +ui: + persistence: + enabled: true + size: 1Gi + # storageClass: standard + # existingClaim: my-horizon-data +``` + +| Value | Default | Notes | +|---|---|---| +| `ui.persistence.enabled` | `false` | `false` → `emptyDir`; `true` → PVC mounted at `/data` | +| `ui.persistence.existingClaim` | unset | use a pre-created PVC; otherwise the chart creates `{release}-skywalking-helm-ui-data` | +| `ui.persistence.storageClass` | unset | `-` renders an empty `storageClassName` | +| `ui.persistence.accessModes` | `[ReadWriteOnce]` | matches `ui.replicas: 1` | +| `ui.persistence.size` | `1Gi` | | +| `ui.persistence.annotations` | `{}` | applied to the chart-managed PVC | + +The image runs as the non-root `horizon` user, so any volume mounted into the container must be group-writable by it. `ui.securityContext.fsGroup` defaults to `101` for exactly this reason — keep it (or set an equivalent) when you override `ui.securityContext`, and apply the same thought to anything you add through `ui.extraVolumeMounts`. + +Keep `ui.replicas: 1`. The BFF holds its session table in memory, the Deployment uses the `Recreate` strategy for that reason, and a `ReadWriteOnce` PVC cannot be mounted by pods on two different nodes anyway. + +## Settings that take a path, not a value + +Two Horizon 1.0.0 settings are configured by filesystem path, so they need `ui.extraVolumes` / `ui.extraVolumeMounts`: + +- `auth.tokensFile` — API tokens for callers with no browser (scripts, CI, MCP clients). Mount a Secret. +- `sourceMaps.bootMountDir` — durable `.map` files for the Browser Errors tab. The image sets this to `/app/sourcemaps`; without a volume there, runtime uploads live in BFF memory only and are lost on pod restart. + +```yaml +ui: + extraVolumes: + - name: horizon-tokens + secret: + secretName: horizon-tokens + extraVolumeMounts: + - name: horizon-tokens + mountPath: /app/tokens + readOnly: true + config: + auth: + tokensFile: /app/tokens/tokens.json +``` + +## Applying a change + +The UI Deployment carries a `checksum/config` annotation over the rendered ConfigMap, because a `subPath` ConfigMap mount does **not** update inside a running container. Any change to `ui.config` — or to a value that feeds a computed field — therefore rolls the pod on the next `helm upgrade`. Changing `ui.extraEnv` rolls it too, and env is read once at process start — but editing the *contents* of the Secret behind `ui.envFromSecret` changes no pod field, so nothing rolls: `kubectl rollout restart` the Deployment yourself. + +Verify what actually landed in the container: + +```shell +kubectl exec -n "${SKYWALKING_RELEASE_NAMESPACE}" \ + deploy/${SKYWALKING_RELEASE_NAME}-skywalking-helm-ui -- cat /app/horizon.yaml +``` + +## Next + +- [Horizon UI in This Chart](horizon-ui.md) — what the image is, and why booster UI is gone +- [Set Up Logins](logins.md) — no login is configured by default +- [UI Service and Ingress](../expose/ui-service-and-ingress.md) — where `server.publicUrl` comes from +- [UI and Login Problems](../troubleshooting/ui-and-login.md) +- [skywalking Chart values](../reference/skywalking-chart-values.md) — every `ui.*` value diff --git a/docs/ui/horizon-ui.md b/docs/ui/horizon-ui.md new file mode 100644 index 0000000..1b45c81 --- /dev/null +++ b/docs/ui/horizon-ui.md @@ -0,0 +1,118 @@ +# Horizon UI in This Chart + +This page explains what web UI the chart deploys, how it talks to OAP, and why the legacy booster UI is no longer an option. + +[Apache SkyWalking Horizon UI](https://github.com/apache/skywalking-horizon-ui) is the official SkyWalking web console and the only UI this chart deploys. It is a Vue SPA served by a Node **BFF** (backend-for-frontend) inside the same container — not a thin reverse proxy like the UI it replaces. + +## What the chart ships + +| | value | set by | +|---|---|---| +| image | `skywalking.docker.scarf.sh/apache/skywalking-ui` | `ui.image.repository` | +| tag | required, no default; a `horizon-*` tag on the default repository (e.g. `horizon-1.0.0`) | `ui.image.tag` | +| container port | `8081` | `ui.service.internalPort` | +| service port | `80` → `8081` | `ui.service.externalPort` | +| replicas | `1` | `ui.replicas` | +| config file | ConfigMap mounted at `/app/horizon.yaml` | `ui.config` (empty by default) | + +Set `ui.enabled: false` to skip the UI entirely — no Deployment, Service, Ingress, ConfigMap or PVC is created, and OAP stays reachable on its own query port. + +## The BFF talks to OAP on two ports + +Booster UI needed only the GraphQL query port. Horizon's BFF needs two, and the chart writes both into `horizon.yaml`: + +| field | OAP port | chart value | what it carries | +|---|---|---|---| +| `oap.queryUrl` | `12800` | `oap.ports.rest` | the GraphQL query API | +| `oap.adminUrl` | `17128` | `oap.ports.admin` | `/status/*`, `/debugging/*`, inspect, DSL debugging, runtime rules, dashboard templates | +| `oap.zipkinUrl` | commented out by default | `oap.ports.zipkin-query` | written as `http://:/zipkin`, only when that port is set | + +OAP 11 serves the admin surfaces **only** on the admin port, so `oap.ports.admin` is required. It defaults to `17128`. Set it to `null` on any OAP 10.x release — the admin server arrived in OAP 11, and on 10.x that port belongs to the AI-pipeline URI-recognition server. + +A rendered `horizon.yaml` for release `skywalking` looks like this: + +```yaml +oap: + adminUrl: ${HORIZON_OAP_ADMIN_URL:http://skywalking-skywalking-helm-oap:17128} + queryUrl: ${HORIZON_OAP_QUERY_URL:http://skywalking-skywalking-helm-oap:12800} +server: + port: 8081 +``` + +Every chart-computed field is written as a `${VAR:default}` **token**, so the in-cluster address is the default and the matching `HORIZON_*` environment variable still overrides it. `server.port` is the one exception — it is derived from `ui.service.internalPort` so the BFF binds the port the container actually exposes. + +When `ui.ingress.enabled` is true and `ui.ingress.hosts` is non-empty the chart writes one more field, `server.publicUrl` — `${HORIZON_PUBLIC_URL:://}`, `https` when `ui.ingress.tls` is non-empty, `http` otherwise. Horizon otherwise derives its public base URL per request, which is wrong behind an ingress that rewrites `Host`. See [UI service and ingress](../expose/ui-service-and-ingress.md). + +## Port 8081, and no `/graphql` passthrough + +Two breaking differences from booster UI: + +- The container listens on **8081** (booster listened on 8080). The Service still fronts it on port 80. +- The BFF **does not proxy `/graphql`** to OAP. Anything that used to query the UI's GraphQL endpoint must now address the OAP service directly. + +```shell +# before: swctl --base-url=http:///graphql ... +swctl --display yaml \ + --base-url=http://skywalking-skywalking-helm-oap:12800/graphql \ + service ls +``` + +The chart's own e2e suites already do this — every `swctl` invocation in `test/e2e/` points at OAP port `12800`. + +Both probes in `values.yaml` hardcode `8081` (`livenessProbe.tcpSocket.port`, and `readinessProbe.httpGet` on `/api/auth/health`, the only unauthenticated BFF health endpoint). If you change `ui.service.internalPort`, change both probes to match. + +The Deployment uses `strategy: Recreate` and defaults to one replica: the BFF keeps its session table in memory, so two pods serving at once means logins break on alternating requests unless your ingress does sticky routing. + +## Image tags + +Horizon releases **independently** of OAP — there is no 1:1 version mapping, and you pin the two tags separately. + +| channel | image | tags | +|---|---|---| +| release | `apache/skywalking-ui` (Docker Hub) | `horizon-x.y.z`, e.g. `horizon-1.0.0` | +| dev / pre-release | `ghcr.io/apache/skywalking-horizon-ui` | full commit SHA, `x.y.z`, `main` | + +Note the asymmetry: the same version is `horizon-1.0.0` on Docker Hub and `1.0.0` on ghcr.io. + +Horizon 1.0.0 works against OAP 10.4.0 as well as OAP 11.x, so pin it whichever OAP release you run. Against a 10.x OAP also set `ui.config.templates.mode: readonly` (or `HORIZON_TEMPLATES_MODE=readonly`) — Horizon reads dashboard templates from OAP 11's `/ui-management` admin REST API, which OAP 10 does not serve, so `readonly` renders the templates bundled in the image instead. + +## Booster UI is not supported + +The legacy `skywalking-booster-ui` (and `skywalking-rocketbot-ui` before it) is not supported by this chart, and that is not a chart policy — SkyWalking 11.0.0 removed the UI from the distribution: + +- `apm-webapp/` — the Armeria reverse proxy behind the `skywalking/ui` image — was deleted, along with the `skywalking-ui` git submodule tracking `apache/skywalking-booster-ui`, the `docker.ui` Maven target, and the image build. +- The last booster image published to `apache/skywalking-ui` is therefore `10.4.0`. There is no `11.x` tag and there will not be one; the repository now carries only `horizon-x.y.z` tags. +- The OAP-side surfaces booster depended on are gone too: the `ui-initialized-templates` seed files, the sidebar menu storage, the `UIConfigurationManagement` GraphQL mutations and queries, and the `SW_ENABLE_UPDATE_UI_TEMPLATE` flag. Horizon ships its own dashboard library and menu, and manages templates over the admin REST port. + +### Upgrading from a chart release that ran booster UI + +1. Replace `--set ui.image.tag=` with `--set ui.image.tag=horizon-1.0.0`. +2. Make sure `oap.ports.admin` is set (it is, by default). +3. Repoint any `swctl` / API caller from `http:///graphql` to `http://:12800/graphql`. +4. Configure login users — see [Set up logins](logins.md). + +## Authentication is not optional + +Horizon has **no built-in `admin/admin` fallback** and no login configured by default. The BFF does not fail closed: with no users it still boots, logs an error, serves the login page, and answers the readiness probe with 200 — so the pod reports **Ready and nobody can log in**. Configure users before you rely on the deployment — see [Set up logins](logins.md). + +Prefer environment variables (`ui.extraEnv`, `ui.envFromSecret`) over `ui.config`: the image's `/app/horizon.yaml` is fully env-tokenized, and any field you write as a plain literal makes its `HORIZON_*` variable inert. `${VAR}` tokens written in `ui.config` do expand, which is how the Secret pattern works: + +```yaml +ui: + envFromSecret: horizon-admin # provides HORIZON_ADMIN_HASH + config: + auth: + local: + users: + - username: admin + passwordHash: "${HORIZON_ADMIN_HASH}" + roles: [admin] +``` + +## See also + +- [Set up logins](logins.md) — the demo credentials snippet and the production Secret pattern +- [Configure Horizon](configure.md) — `ui.config` vs `ui.extraEnv`, persistence, extra volumes +- [Quick start](../install/quick-start.md) — a full install command with the required values +- [`horizon.yaml` reference](https://github.com/apache/skywalking-horizon-ui/blob/main/docs/setup/horizon-yaml.md) — the full upstream schema (server, templates, oap, auth, rbac, session, audit, ai, mcp, oauth, debugLog) +- [Apache SkyWalking Horizon UI](https://github.com/apache/skywalking-horizon-ui) — upstream repository diff --git a/docs/ui/logins.md b/docs/ui/logins.md new file mode 100644 index 0000000..0fc04e5 --- /dev/null +++ b/docs/ui/logins.md @@ -0,0 +1,220 @@ +# Set Up Logins + +Horizon UI ships with **no accounts at all**, and a chart install that skips this page produces a +Deployment that reports healthy while nobody can sign in. This page shows how to confirm that state, +how to seed a throwaway demo login, and how to configure real users from a Kubernetes Secret. + +## There is no default login, and the pod still goes Ready + +Horizon has no built-in `admin/admin` fallback, and the chart configures no users of its own. The BFF +does **not** fail closed when it finds none: it boots, logs an error, serves the login page, and +answers its readiness probe with `200`. The result is a green deployment nobody can use. + +| what you see | what is actually happening | +|---|---| +| Pod `1/1 Running`, Ready | `ui.readinessProbe` hits `/api/auth/health`, which is public and always answers `200` | +| Login page renders, with a setup banner | The page reads `configured: false` from that same endpoint | +| Every username/password is rejected | `auth.local.users` is empty, so no credential can match | + +Confirm it from outside the pod — `configured` and `setupHint` are the two fields that matter: + +```shell +kubectl port-forward -n "${SKYWALKING_RELEASE_NAMESPACE}" \ + svc/${SKYWALKING_RELEASE_NAME}-skywalking-helm-ui 8080:80 +curl -s http://127.0.0.1:8080/api/auth/health +``` + +```json +{"backend":"local","configured":false,"setupHint":"No users configured. Add at least one entry to auth.local.users in horizon.yaml ...","ldap":null,"breakGlass":{"armed":false}} +``` + +The same state appears once in the UI container log at startup: + +```shell +kubectl logs -n "${SKYWALKING_RELEASE_NAMESPACE}" \ + deploy/${SKYWALKING_RELEASE_NAME}-skywalking-helm-ui | grep 'auth.local.users is empty' +``` + +## Demo logins (publicly-known credentials) + +For a first run on a trusted network, paste this into a values file. It seeds `admin/admin` and +`skywalking/skywalking` using `argon2id` hashes of those exact plaintexts — the same pair the chart's +own e2e tests use. + +> **These hashes are published in this repository.** Anyone can read them and derive the passwords. +> Use them only on a network you control, and replace them before the UI is reachable by anyone else. + +```yaml +# demo-values.yaml +ui: + config: + auth: + backend: local # the default; shown for clarity + local: + users: + - username: admin # password: admin + passwordHash: "$argon2id$v=19$m=65536,t=3,p=4$eemqy1r72oSXR58y8VpRqw$Bn/dULrmJTHEi3263KfgWDEwQmUsqNLi3xwyv/DekHM" + roles: [admin] + - username: skywalking # password: skywalking + passwordHash: "$argon2id$v=19$m=65536,t=3,p=4$Zqj8HhQDqm8d5c2MipHYZw$BsaCnu4bdd4uadIldx3wwYLsdo47Thxb7Lv1MXpWG2Q" + roles: [viewer, maintainer] +``` + +```shell +helm install "${SKYWALKING_RELEASE_NAME}" \ + oci://registry-1.docker.io/apache/skywalking-helm \ + --version 5.0.0 \ + -n "${SKYWALKING_RELEASE_NAMESPACE}" \ + --set oap.image.tag=11.0.0 \ + --set oap.storageType=banyandb \ + --set ui.image.tag=horizon-1.0.0 \ + -f demo-values.yaml +``` + +Then port-forward and log in as `admin/admin`: + +```shell +kubectl port-forward -n "${SKYWALKING_RELEASE_NAMESPACE}" \ + svc/${SKYWALKING_RELEASE_NAME}-skywalking-helm-ui 8080:80 +open http://127.0.0.1:8080 +``` + +Pass the hashes through a values **file**, not `--set`: a hash is full of `,` and `=`, which `--set` +reads as its own separators (and of `$`, which the shell would expand first). + +## Production: hashes from a Secret + +Generate your own hash first. The CLI lives in the Horizon UI repository and reads the password from +`argv` or stdin: + +```shell +git clone https://github.com/apache/skywalking-horizon-ui.git +cd skywalking-horizon-ui && pnpm install +HASH=$(pnpm --filter bff cli:hash 'your-strong-password' | tail -1) +``` + +Passwords longer than 64 characters are refused — the login route rejects them too, so a hash of one +could never be signed in with. + +From there, pick one of two shapes. Both put the hash in a Secret and reference it with +`ui.envFromSecret`, which the chart turns into an `envFrom.secretRef` on the UI container. + +### Why a token in `ui.config` is required either way + +The chart mounts its ConfigMap **over** the image's `/app/horizon.yaml`, and that rendered file +contains only `oap.*` and `server.*` — no `auth:` block. Horizon expands `${VAR}` over the raw text of +whatever file is at that path, so a `HORIZON_*` variable is read only if a matching token is present +in the text. `auth.local.users` has a plain `[]` schema default and is **not** env-backed. + +**Setting `HORIZON_AUTH_LOCAL_USERS` through `ui.envFromSecret` alone therefore does nothing** — the +token it would fill is not in the file the chart mounted. You must write the token into `ui.config`. + +### Option A — one JSON array for all users + +Best when users are managed as a unit and you would rather not restate them in the values file. + +```shell +kubectl create secret generic horizon-users \ + -n "${SKYWALKING_RELEASE_NAMESPACE}" \ + --from-literal=HORIZON_AUTH_LOCAL_USERS='[{"username":"admin","passwordHash":"'"$HASH"'","roles":["admin"]}]' +``` + +```yaml +# my-values.yaml +ui: + envFromSecret: horizon-users + config: + auth: + local: + users: "${HORIZON_AUTH_LOCAL_USERS:[]}" +``` + +The JSON must be a **single line** — it is substituted into YAML text, where a newline would end the +value. If the Secret key is missing or empty the token falls back to `[]`, which is the silent +lockout again, so check `/api/auth/health` after rolling out. + +### Option B — a `${VAR}` per hash + +Best when the user list is stable and belongs in version control, with only the secrets held out. + +```shell +kubectl create secret generic horizon-admin \ + -n "${SKYWALKING_RELEASE_NAMESPACE}" \ + --from-literal=HORIZON_ADMIN_HASH="$HASH" +``` + +```yaml +# my-values.yaml +ui: + envFromSecret: horizon-admin + config: + auth: + local: + users: + - username: admin + passwordHash: "${HORIZON_ADMIN_HASH}" + roles: [admin] +``` + +Use `ui.extraEnv` instead of `ui.envFromSecret` when you want to pick individual keys out of an +existing Secret: + +```yaml +ui: + extraEnv: + - name: HORIZON_ADMIN_HASH + valueFrom: + secretKeyRef: + name: horizon-admin + key: passwordHash +``` + +### Install with it + +```shell +helm install "${SKYWALKING_RELEASE_NAME}" \ + oci://registry-1.docker.io/apache/skywalking-helm \ + --version 5.0.0 \ + -n "${SKYWALKING_RELEASE_NAMESPACE}" \ + --set oap.image.tag=11.0.0 \ + --set oap.storageType=banyandb \ + --set ui.image.tag=horizon-1.0.0 \ + -f my-values.yaml +``` + +A `helm upgrade` that changes `ui.config` rolls the UI pod on its own: the Deployment carries a +`checksum/config` annotation over the rendered ConfigMap. Changing only the **Secret** does not — env +vars are read once at container start, so restart the Deployment yourself: + +```shell +kubectl rollout restart -n "${SKYWALKING_RELEASE_NAMESPACE}" \ + deploy/${SKYWALKING_RELEASE_NAME}-skywalking-helm-ui +``` + +## Roles + +`roles` on a user is a list of role names from `rbac.roles`. Horizon ships four: + +| role | grants | +|---|---| +| `viewer` | Read the data: metrics, traces, logs, alarms, events, topology, profiling, browser errors, overviews, inspect | +| `maintainer` | Viewer, plus platform reads — cluster health, TTL, OAP configuration | +| `operator` | Maintainer, plus writes — dashboard and overview templates, DSL rules, live debugging, profiling tasks, source maps. Alarm rules stay read-only for every role | +| `admin` | `*` | + +A user with an empty `roles` list can sign in and see nothing. Define your own names by setting +`ui.config.rbac.roles`; see the [horizon.yaml reference](https://github.com/apache/skywalking-horizon-ui/blob/main/docs/setup/horizon-yaml.md). + +## Beyond local users + +LDAP, SSO (OIDC/OAuth2), break-glass accounts and API tokens are all configured under `auth` in the +same `ui.config` block, and follow the same rule: write the field there, keep the secret in a Secret +and reference it with a `${VAR}` token. + +`auth.tokensFile` — API tokens for callers with no browser (scripts, CI, MCP clients) — takes a +**path**, not a value, so it also needs `ui.extraVolumes` / `ui.extraVolumeMounts`. See +[Configure Horizon](configure.md). + +- [Horizon UI in This Chart](horizon-ui.md) — what the BFF is and how it talks to OAP +- [UI and Login Problems](../troubleshooting/ui-and-login.md) — symptoms and fixes +- [Access control (upstream)](https://github.com/apache/skywalking-horizon-ui/blob/main/docs/access-control/local-backend.md) diff --git a/docs/upgrade/upgrading.md b/docs/upgrade/upgrading.md new file mode 100644 index 0000000..4ead36a --- /dev/null +++ b/docs/upgrade/upgrading.md @@ -0,0 +1,212 @@ +# Upgrade + +How to run `helm upgrade` on this chart, and the breaking changes chart 5.0.0 brings — the booster +UI removal, the OAP 11 dashboard-template changes, and the new `ui.config` default. + +## Run the upgrade + +```shell +helm upgrade skywalking oci://registry-1.docker.io/apache/skywalking-helm \ + --version 5.0.0 -n skywalking \ + --set oap.image.tag=11.0.0 \ + --set oap.storageType=banyandb \ + --set ui.image.tag=horizon-1.0.0 \ + --set elasticsearch.enabled=false \ + --set banyandb.enabled=true \ + --set banyandb.image.tag=0.11.0 \ + --wait --wait-for-jobs +``` + +Three values have no defaults and must be present on **every** upgrade, not just the first install: + +| value | example | +|---|---| +| `oap.image.tag` | `11.0.0` | +| `oap.storageType` | `elasticsearch`, `postgresql`, `banyandb` | +| `ui.image.tag` | `horizon-1.0.0` (only when `ui.enabled` is true) | + +`helm upgrade` does not remember `--set` flags from the previous release, so either repeat them, +pass a values file with `-f`, or add `--reuse-values`. `--wait-for-jobs` alongside `--wait` makes +Helm report an init-Job failure directly instead of leaving you to guess why OAP never went Ready. + +## Chart 5.0.0 is a version set + +OAP and BanyanDB are hard-coupled and must move together — that pair is the most common upgrade +failure. Horizon releases independently of OAP (1.0.0 covers both OAP 10.4.0 and 11.x), so its tag +is pinned separately, but it must be a `horizon-*` tag: the chart no longer supports booster UI. + +| component | value | version | +|---|---|---| +| chart | `--version` | `5.0.0` | +| OAP | `oap.image.tag` | `11.0.0` | +| Horizon UI | `ui.image.tag` | `horizon-1.0.0` | +| BanyanDB | `banyandb.image.tag` | `0.11.0` | + +OAP ships the BanyanDB server API versions it accepts in +`SW_STORAGE_BANYANDB_COMPATIBLE_SERVER_API_VERSIONS`. OAP 11.0.0 accepts API `0.11`, which maps to +BanyanDB release `0.11.x`. The check is an exact string match with no lenient fallback, so OAP 11 +against BanyanDB 0.10.x refuses to start with `Incompatible BanyanDB server API version`. The +mapping is published at +[BanyanDB API versions](https://skywalking.apache.org/docs/skywalking-banyandb/latest/installation/versions/); +see also [BanyanDB](../storage/banyandb.md). + +The bump from `4.9.0` to `5.0.0` is a major one because the UI image line, several `values.yaml` +keys and the init Job's identity all changed incompatibly. The rest of this page is that list. + +## Breaking: booster UI is gone + +OAP 11 deleted `apm-webapp` — the Armeria reverse proxy behind the `skywalking/ui` image — along +with the `skywalking-ui` git submodule and the `docker.ui` Maven target. The last booster image +published to `apache/skywalking-ui` is `10.4.0`; there is no `11.x` tag and there will not be one. +Only `horizon-*` tags are published going forward. + +What to change when coming from a release that ran booster UI: + +| | before (4.9.0) | after (5.0.0) | +|---|---|---| +| image tag | `ui.image.tag=` | `ui.image.tag=horizon-1.0.0` | +| container port | `8080` | `8081` (`ui.service.internalPort`; Service still fronts `80`) | +| OAP admin port | no `oap.ports.admin` key at all | `oap.ports.admin: 17128`, required | +| login | none — booster was unauthenticated | must be configured — no fallback | +| `/graphql` on the UI | proxied to OAP | **not proxied**; call OAP directly | +| UI env | `ui.env` (map); chart set `SW_OAP_ADDRESS` / `SW_ZIPKIN_ADDRESS` | `ui.extraEnv` (list) and `ui.envFromSecret`; OAP URLs come from the `horizon.yaml` ConfigMap | +| Zipkin port keys | `oap.ports.zipkinreceiver` / `zipkinquery` | `oap.ports.zipkin-receiver` / `zipkin-query` | + +`ui.env` no longer exists — a values file that still sets it renders nothing and the BFF gets no +extra environment. The Zipkin keys were renamed too: `oap-deployment.yaml` reads them as +`zipkin-receiver` / `zipkin-query`, so a carried-over `zipkinquery` still opens a container port but +no longer sets `SW_QUERY_ZIPKIN` / `SW_QUERY_ZIPKIN_REST_PORT`, and the UI's `oap.zipkinUrl` is not +derived. + +**Configure logins before you cut over.** Horizon has no built-in `admin/admin`, and the BFF does +**not** fail closed: with no users it boots, logs an error, serves the login page, and answers +`/api/auth/health` with 200 — which is the chart's own readiness probe. The pod goes Ready and +nobody can sign in. See [Set Up Logins](../ui/logins.md). + +**Expose the OAP admin port.** OAP 11 serves `/status/*` and `/debugging/*` on the admin REST port +only, and Horizon's BFF reads inspect, DSL debugging, runtime rules and (in the default +`templates.mode: live`) the dashboard template store from it. `oap.ports.admin` defaults to `17128`; +set it to `null` on any OAP 10.x release — the admin server is an OAP 11 addition. + +**Retarget `swctl` and any other GraphQL caller.** The BFF does not pass `/graphql` through to OAP, +so anything that used to query the UI must address the OAP Service: + +```shell +# before +swctl --base-url=http://skywalking-skywalking-helm-ui/graphql service ls + +# after — OAP directly, on oap.ports.rest +swctl --display yaml \ + --base-url=http://skywalking-skywalking-helm-oap:12800/graphql \ + service ls +``` + +Every `swctl` invocation under `test/e2e/` already points at OAP port `12800`; copy the form from +there if you need the in-cluster hostname pattern. + +### Upgrading the UI first + +Horizon 1.0.0 runs against OAP 10.4.0 as well as OAP 11.x, so you can split the move in two: swap +the UI while still on OAP 10, then upgrade OAP. On OAP 10 add one setting, because OAP 10 does not +serve the `/ui-management` REST API Horizon reads templates from: + +```shell +--set ui.config.templates.mode=readonly +``` + +That renders the templates bundled in the image and makes the configuration surface display-only. +Dashboards, traces, logs, topology, alarms and profiling all work. Remove it once OAP is on 11. The +mode is read at boot, so it needs a BFF restart — the UI Deployment's `checksum/config` annotation +rolls the pod for you whenever the ConfigMap changes. + +## Breaking: OAP 11 removed the UI template seeds + +These OAP-side surfaces went with the bundled UI. If your values still set them, they are read by +nothing — delete them: + +| removed | where it used to live | +|---|---| +| `ui-initialized-templates` | a key under `oap.config`, seeding on-disk dashboard JSON | +| `SW_ENABLE_UPDATE_UI_TEMPLATE` | an env var under `oap.env` | +| sidebar menu storage | OAP storage | +| `UIConfigurationManagement` GraphQL queries and mutations | OAP query API | + +The on-disk dashboard seed files were deleted along with `UITemplateInitializer`. Horizon ships its +own dashboard library and manages templates over the admin REST port instead — that is what +`oap.ports.admin` and `ui.config.templates.mode` are for. + +## Breaking: `ui.config` is empty by default + +The interim Horizon work on `main` mounted a ConfigMap of literal values over `/app/horizon.yaml`. +As of 5.0.0 `ui.config` defaults to `{}` and the ConfigMap carries only the fields the chart has to +compute: + +| field | derived from | +|---|---| +| `oap.queryUrl` | the in-cluster OAP Service and `oap.ports.rest` | +| `oap.adminUrl` | the in-cluster OAP Service and `oap.ports.admin` | +| `oap.zipkinUrl` | the OAP Service and `oap.ports.zipkin-query`, only when that port is set | +| `server.publicUrl` | the first `ui.ingress.hosts` entry, only when `ui.ingress.enabled` and `ui.ingress.hosts` are both set (`https` when `ui.ingress.tls` is non-empty) | +| `server.port` | `ui.service.internalPort`, so the BFF binds the port the container exposes | + +The four URLs are written as `${VAR:default}` **tokens**, not literals, so the in-cluster value is +only the default and the matching `HORIZON_*` variable still wins. `server.port` is the deliberate +exception — the chart writes it as a plain number, because the container port and both probes are +derived from `ui.service.internalPort` and the BFF has to bind that same port. + +This matters because Horizon expands `${...}` over the raw *text* of the config file before parsing +it. A field written as a literal makes its `HORIZON_*` environment variable silently inert, and a +field the file omits falls back to a schema default that consults the environment for only a +fraction of the image's variables. A config file full of literals therefore does not merely +duplicate defaults — it disables most of the image's configuration surface. + +**If you are carrying a full `ui.config` block forward** — from the pre-release `main` values, not +from 4.x, which had no `ui.config` at all — drop every field that only restates what the image +already sets (`server.host`, `oap.timeoutMs`, `auth.backend`, `rbac.enabled`, `session.*` and +`templates.mode` all did), and move the rest to `ui.extraEnv` or `ui.envFromSecret`: + +| config field | environment variable | +|---|---| +| `templates.mode` | `HORIZON_TEMPLATES_MODE` | +| `session.cookieSecure` | `HORIZON_SESSION_COOKIE_SECURE` | +| `server.trustProxy` | `HORIZON_TRUST_PROXY` | +| `auth.local.users` | `HORIZON_AUTH_LOCAL_USERS` (a JSON array) | +| `oap.auth` | `HORIZON_OAP_AUTH` (JSON) | + +`ui.config` is still there for pinning a field regardless of the environment, and `${VAR}` tokens +you write in it do expand — that is how the Secret pattern works. See +[Configure Horizon](../ui/configure.md). + +## The init Job re-runs on every upgrade that changes a value + +The storage schema is created by a one-shot `*-oap-init-*` Job that runs OAP with `-Dmode=init`; the +main Deployment runs `-Dmode=no-init` and blocks until the schema exists. The Job name carries a +`sha256` of the chart values, so any `helm upgrade` that changes a value produces a new Job name and +re-runs init automatically, with Helm pruning the previous one. An upgrade that changes nothing +reuses the existing Job. + +Before 5.0.0 this Job was a `post-install,post-upgrade,post-rollback` Helm hook with a fixed name — +the OAP Service name plus `-init`, e.g. `skywalking-skywalking-helm-oap-init`. Helm does not own +hook resources, so an upgrade from 4.9.0 leaves that completed Job behind in the namespace. It is +inert; delete it by hand. + +To force a rerun without changing a value, delete the Job and upgrade again: + +```shell +kubectl delete job -n skywalking -l release=skywalking +helm upgrade skywalking oci://registry-1.docker.io/apache/skywalking-helm \ + --version 5.0.0 -n skywalking --reuse-values +``` + +More detail in [The OAP Init Job](../operate/oap-init-job.md). + +## Rolling back + +`helm rollback` restores the previous release's manifests, so image tags and configuration revert. +It does **not** revert anything the init Job wrote to storage — the chart has no downgrade path for +the storage schema. Before an OAP major upgrade, make sure the storage backend is one you can +restore independently. + +If the UI comes back Ready but unusable after an upgrade, start at +[UI and Login Problems](../troubleshooting/ui-and-login.md); if OAP or the init Job fails, see +[Install and Startup Failures](../troubleshooting/install-and-startup.md). diff --git a/test/e2e/e2e-adapter-chart.yaml b/test/e2e/e2e-adapter-chart.yaml new file mode 100644 index 0000000..dbf2556 --- /dev/null +++ b/test/e2e/e2e-adapter-chart.yaml @@ -0,0 +1,54 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Tests this repository's Helm packaging of the SWCK custom-metrics adapter. +# The adapter's own behaviour (autoscaling on SkyWalking metrics) is covered by +# apache/skywalking-swck's oap-agent-adapter-hpa case; what is only testable +# here is whether the chart registers a working APIService. + +setup: + env: kind + file: kind28.yaml + init-system-environment: env + steps: + - name: Install yq + command: bash test/e2e/setup-e2e-shell/install.sh yq + - name: Install kubectl + command: bash test/e2e/setup-e2e-shell/install.sh kubectl + - name: Install helm + command: bash test/e2e/setup-e2e-shell/install.sh helm + - name: Install the adapter chart + command: | + helm install adapter chart/adapter --namespace=skywalking-custom-metrics-system \ + --create-namespace \ + --set image.repository=$SWCK_OPERATOR_REPO \ + --set image.tag=$SWCK_OPERATOR_TAG + wait: + - namespace: skywalking-custom-metrics-system + resource: deployment + for: condition=Available + timeout: 10m + +verify: + retry: + count: 20 + interval: 5s + cases: + # The APIService must be accepted by the aggregation layer. If the chart's + # Service name, namespace or CA wiring is wrong it stays False forever while + # the Deployment still reports Available. + - query: | + kubectl get apiservice -o json | yq -p json -o yaml '{"available": ([.items[] | select(.spec.service.namespace == "skywalking-custom-metrics-system") | .status.conditions[] | select(.type == "Available") | .status] | .[0])}' + expected: expected/swck-adapter-apiservice.yml diff --git a/test/e2e/e2e-banyandb-cluster.yaml b/test/e2e/e2e-banyandb-cluster.yaml deleted file mode 100644 index d21c050..0000000 --- a/test/e2e/e2e-banyandb-cluster.yaml +++ /dev/null @@ -1,242 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -# This file is used to show how to write configuration files and can be used to test. - -setup: - env: kind - file: kind28.yaml - init-system-environment: env - kind: - expose-ports: - - namespace: istio-system - resource: service/skywalking-oap - port: 12800 - steps: - - name: install yq - command: bash test/e2e/setup-e2e-shell/install.sh yq - - name: install swctl - command: bash test/e2e/setup-e2e-shell/install.sh swctl - - name: install kubectl - command: bash test/e2e/setup-e2e-shell/install.sh kubectl - - name: install istio - command: | - bash test/e2e/setup-e2e-shell/install.sh istioctl - istioctl install -y --set profile=demo \ - --set meshConfig.defaultConfig.envoyAccessLogService.address=skywalking-satellite.istio-system:11800 \ - --set meshConfig.enableEnvoyAccessLogService=true - kubectl label namespace default istio-injection=enabled - - name: Install helm - command: bash test/e2e/setup-e2e-shell/install.sh helm - - name: Install SkyWalking - command: | - helm dep up chart/skywalking - helm -n istio-system install --timeout 10m skywalking chart/skywalking \ - --set fullnameOverride=skywalking \ - --set oap.env.SW_ENVOY_METRIC_ALS_HTTP_ANALYSIS=k8s-mesh \ - --set oap.env.SW_ENVOY_METRIC_ALS_TCP_ANALYSIS=k8s-mesh \ - --set oap.env.K8S_SERVICE_NAME_RULE='e2e::${service.metadata.name}' \ - --set oap.replicas=1 \ - --set ui.image.repository=$UI_REPO \ - --set ui.image.tag=$UI_TAG \ - --set oap.image.repository=$OAP_REPO \ - --set oap.image.tag=$OAP_TAG \ - --set oap.storageType=banyandb \ - --set elasticsearch.enabled=false \ - --set banyandb.image.repository=$BANYANDB_REPO \ - --set banyandb.image.tag=$BANYANDB_TAG \ - --set banyandb.enabled=true \ - --set banyandb.standalone.enabled=false \ - --set banyandb.cluster.enabled=true \ - --set banyandb.cluster.fodc.agent.resources.requests[0].key=memory \ - --set banyandb.cluster.fodc.agent.resources.requests[0].value=256Mi \ - --set banyandb.cluster.fodc.agent.resources.limits[0].key=memory \ - --set banyandb.cluster.fodc.agent.resources.limits[0].value=256Mi \ - --set satellite.enabled=true \ - --set satellite.image.repository=$SATELLITE_REPO \ - --set satellite.image.tag=$SATELLITE_TAG \ - -f test/e2e/values.yaml - wait: - - namespace: istio-system - resource: deployments/skywalking-oap - for: condition=available - - namespace: istio-system - resource: deployments/skywalking-satellite - for: condition=available - - namespace: istio-system - resource: deployments/skywalking-ui - for: condition=available - - name: Deploy demo services - command: | - kubectl apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/platform/kube/bookinfo.yaml - kubectl apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/networking/bookinfo-gateway.yaml - # Enable TCP services - kubectl apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/platform/kube/bookinfo-ratings-v2.yaml - kubectl apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/platform/kube/bookinfo-db.yaml - kubectl apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/networking/destination-rule-all.yaml - kubectl apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/networking/virtual-service-ratings-db.yaml - wait: - - namespace: default - resource: pod - for: condition=Ready - - name: Generate traffic - path: traffic-gen.yaml - wait: - - namespace: default - resource: pod - for: condition=Ready - timeout: 25m - -verify: - retry: - count: 20 - interval: 10s - cases: - # service list - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql service ls - expected: expected/service.yml - # service instance list - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql instance list --service-name=e2e::productpage - expected: expected/service-instance.yml - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql instance list --service-name=e2e::reviews - expected: expected/service-instance.yml - # service endpoint - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql endpoint list --service-name=e2e::productpage - expected: expected/service-endpoint-productpage.yml - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql endpoint list --service-name=e2e::reviews - expected: expected/service-endpoint-reviews.yml - - # service metrics: e2e::productpage - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_sla --service-name=e2e::productpage - expected: expected/metrics-has-value.yml - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_cpm --service-name=e2e::productpage - expected: expected/metrics-has-value.yml - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_resp_time --service-name=e2e::productpage - expected: expected/metrics-has-value.yml - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_apdex --service-name=e2e::productpage - expected: expected/metrics-has-value.yml - # service metrics: e2e::reviews - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_sla --service-name=e2e::reviews - expected: expected/metrics-has-value.yml - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_cpm --service-name=e2e::reviews - expected: expected/metrics-has-value.yml - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_resp_time --service-name=e2e::reviews - expected: expected/metrics-has-value.yml - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_apdex --service-name=e2e::reviews - expected: expected/metrics-has-value.yml - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_sidecar_internal_req_latency_nanos --service-name=e2e::reviews - expected: expected/metrics-has-value.yml - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_sidecar_internal_resp_latency_nanos --service-name=e2e::reviews - expected: expected/metrics-has-value.yml - # service instance metrics: e2e::productpage - - query: | - swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_instance_resp_time --service-name=e2e::productpage --instance-name=$( \ - swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql instance list --service-name=e2e::productpage | yq e '.[0].name' - - ) - expected: expected/metrics-has-value.yml - - query: | - swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_instance_cpm --service-name=e2e::productpage --instance-name=$( \ - swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql instance list --service-name=e2e::productpage | yq e '.[0].name' - - ) - expected: expected/metrics-has-value.yml - - query: | - swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_instance_sla --service-name=e2e::productpage --instance-name=$( \ - swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql instance list --service-name=e2e::productpage | yq e '.[0].name' - - ) - expected: expected/metrics-has-value.yml - # service instance metrics: e2e::reviews - - query: | - swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_instance_resp_time --service-name=e2e::reviews --instance-name=$( \ - swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql instance list --service-name=e2e::reviews | yq e '.[0].name' - - ) - expected: expected/metrics-has-value.yml - - query: | - swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_instance_cpm --service-name=e2e::reviews --instance-name=$( \ - swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql instance list --service-name=e2e::reviews | yq e '.[0].name' - - ) - expected: expected/metrics-has-value.yml - - query: | - swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_instance_sla --service-name=e2e::reviews --instance-name=$( \ - swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql instance list --service-name=e2e::reviews | yq e '.[0].name' - - ) - expected: expected/metrics-has-value.yml - - query: | - swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_instance_sidecar_internal_req_latency_nanos --service-name=e2e::reviews --instance-name=$( \ - swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql instance list --service-name=e2e::reviews | yq e '.[0].name' - - ) - expected: expected/metrics-has-value.yml - - query: | - swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_instance_sidecar_internal_resp_latency_nanos --service-name=e2e::reviews --instance-name=$( \ - swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql instance list --service-name=e2e::reviews | yq e '.[0].name' - - ) - expected: expected/metrics-has-value.yml - - # service endpoint metrics: e2e::productpage GET:/productpage - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=endpoint_cpm --endpoint-name=GET:/productpage --service-name=e2e::productpage - expected: expected/metrics-has-value.yml - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=endpoint_resp_time --endpoint-name=GET:/productpage --service-name=e2e::productpage - expected: expected/metrics-has-value.yml - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=endpoint_sla --endpoint-name=GET:/productpage --service-name=e2e::productpage - expected: expected/metrics-has-value.yml - # service endpoint metrics: e2e::reviews GET:/reviews/0 - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=endpoint_cpm --endpoint-name=GET:/reviews/0 --service-name=e2e::reviews - expected: expected/metrics-has-value.yml - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=endpoint_resp_time --endpoint-name=GET:/reviews/0 --service-name=e2e::reviews - expected: expected/metrics-has-value.yml - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=endpoint_sla --endpoint-name=GET:/reviews/0 --service-name=e2e::reviews - expected: expected/metrics-has-value.yml - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=endpoint_sidecar_internal_req_latency_nanos --endpoint-name=GET:/reviews/0 --service-name=e2e::reviews - expected: expected/metrics-has-value.yml - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=endpoint_sidecar_internal_resp_latency_nanos --endpoint-name=GET:/reviews/0 --service-name=e2e::reviews - expected: expected/metrics-has-value.yml - - # dependency service - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql dependency service --service-name=e2e::productpage - expected: expected/dependency-services-productpage.yml - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql dependency service --service-name=e2e::reviews - expected: expected/dependency-services-reviews.yml - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql dependency instance --service-name=e2e::productpage --dest-service-name=e2e::reviews - expected: expected/dependency-services-instance-productpage.yml - - # service relation metrics - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_relation_client_cpm --service-name=e2e::productpage --dest-service-name=e2e::reviews - expected: expected/metrics-has-value.yml - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_relation_server_cpm --service-name=e2e::productpage --dest-service-name=e2e::reviews - expected: expected/metrics-has-value.yml - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_client_sidecar_internal_req_latency_nanos --service-name=e2e::productpage --dest-service-name=e2e::reviews - expected: expected/metrics-has-value.yml - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_server_sidecar_internal_req_latency_nanos --service-name=e2e::productpage --dest-service-name=e2e::reviews - expected: expected/metrics-has-value.yml - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_client_sidecar_internal_resp_latency_nanos --service-name=e2e::productpage --dest-service-name=e2e::reviews - expected: expected/metrics-has-value.yml - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_server_sidecar_internal_resp_latency_nanos --service-name=e2e::productpage --dest-service-name=e2e::reviews - expected: expected/metrics-has-value.yml - # service instance relation metrics, e2e::productpage -> e2e::reviews - - query: | - swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_instance_relation_client_cpm \ - --service-name=e2e::productpage --instance-name=$( \ - swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql instance list --service-name=e2e::productpage | yq e '.[0].name' - ) \ - --dest-service-name=e2e::reviews --dest-instance-name=$( \ - swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql instance list --service-name=e2e::reviews | yq e '.[0].name' - ) \ - - expected: expected/metrics-has-value.yml - - query: | - swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_instance_relation_server_cpm \ - --service-name=e2e::productpage --instance-name=$( \ - swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql instance list --service-name=e2e::productpage | yq e '.[0].name' - ) \ - --dest-service-name=e2e::reviews --dest-instance-name=$( \ - swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql instance list --service-name=e2e::reviews | yq e '.[0].name' - ) \ - - expected: expected/metrics-has-value.yml diff --git a/test/e2e/e2e-banyandb-standalone.yaml b/test/e2e/e2e-banyandb-standalone.yaml deleted file mode 100644 index adf2e03..0000000 --- a/test/e2e/e2e-banyandb-standalone.yaml +++ /dev/null @@ -1,239 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -# This file is used to show how to write configuration files and can be used to test. - -setup: - env: kind - file: kind28.yaml - init-system-environment: env - kind: - expose-ports: - - namespace: istio-system - resource: service/skywalking-oap - port: 12800 - steps: - - name: install yq - command: bash test/e2e/setup-e2e-shell/install.sh yq - - name: install swctl - command: bash test/e2e/setup-e2e-shell/install.sh swctl - - name: install kubectl - command: bash test/e2e/setup-e2e-shell/install.sh kubectl - - name: install istio - command: | - bash test/e2e/setup-e2e-shell/install.sh istioctl - istioctl install -y --set profile=demo \ - --set meshConfig.defaultConfig.envoyAccessLogService.address=skywalking-satellite.istio-system:11800 \ - --set meshConfig.enableEnvoyAccessLogService=true - kubectl label namespace default istio-injection=enabled - - name: Install helm - command: bash test/e2e/setup-e2e-shell/install.sh helm - - name: Install SkyWalking - command: | - helm dep up chart/skywalking - helm -n istio-system install skywalking chart/skywalking \ - --set fullnameOverride=skywalking \ - --set oap.env.SW_ENVOY_METRIC_ALS_HTTP_ANALYSIS=k8s-mesh \ - --set oap.env.SW_ENVOY_METRIC_ALS_TCP_ANALYSIS=k8s-mesh \ - --set oap.env.K8S_SERVICE_NAME_RULE='e2e::${service.metadata.name}' \ - --set oap.replicas=1 \ - --set ui.image.repository=$UI_REPO \ - --set ui.image.tag=$UI_TAG \ - --set oap.image.repository=$OAP_REPO \ - --set oap.image.tag=$OAP_TAG \ - --set oap.storageType=banyandb \ - --set elasticsearch.enabled=false \ - --set banyandb.image.repository=$BANYANDB_REPO \ - --set banyandb.image.tag=$BANYANDB_TAG \ - --set banyandb.enabled=true \ - --set banyandb.standalone.enabled=true \ - --set banyandb.cluster.enabled=false \ - --set banyandb.auth.enabled=true \ - --set satellite.enabled=true \ - --set satellite.image.repository=$SATELLITE_REPO \ - --set satellite.image.tag=$SATELLITE_TAG \ - -f test/e2e/values.yaml - wait: - - namespace: istio-system - resource: deployments/skywalking-oap - for: condition=available - - namespace: istio-system - resource: deployments/skywalking-satellite - for: condition=available - - namespace: istio-system - resource: deployments/skywalking-ui - for: condition=available - - name: Deploy demo services - command: | - kubectl apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/platform/kube/bookinfo.yaml - kubectl apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/networking/bookinfo-gateway.yaml - # Enable TCP services - kubectl apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/platform/kube/bookinfo-ratings-v2.yaml - kubectl apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/platform/kube/bookinfo-db.yaml - kubectl apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/networking/destination-rule-all.yaml - kubectl apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/networking/virtual-service-ratings-db.yaml - wait: - - namespace: default - resource: pod - for: condition=Ready - - name: Generate traffic - path: traffic-gen.yaml - wait: - - namespace: default - resource: pod - for: condition=Ready - timeout: 25m - -verify: - retry: - count: 20 - interval: 10s - cases: - # service list - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql service ls - expected: expected/service.yml - # service instance list - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql instance list --service-name=e2e::productpage - expected: expected/service-instance.yml - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql instance list --service-name=e2e::reviews - expected: expected/service-instance.yml - # service endpoint - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql endpoint list --service-name=e2e::productpage - expected: expected/service-endpoint-productpage.yml - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql endpoint list --service-name=e2e::reviews - expected: expected/service-endpoint-reviews.yml - - # service metrics: e2e::productpage - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_sla --service-name=e2e::productpage - expected: expected/metrics-has-value.yml - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_cpm --service-name=e2e::productpage - expected: expected/metrics-has-value.yml - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_resp_time --service-name=e2e::productpage - expected: expected/metrics-has-value.yml - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_apdex --service-name=e2e::productpage - expected: expected/metrics-has-value.yml - # service metrics: e2e::reviews - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_sla --service-name=e2e::reviews - expected: expected/metrics-has-value.yml - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_cpm --service-name=e2e::reviews - expected: expected/metrics-has-value.yml - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_resp_time --service-name=e2e::reviews - expected: expected/metrics-has-value.yml - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_apdex --service-name=e2e::reviews - expected: expected/metrics-has-value.yml - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_sidecar_internal_req_latency_nanos --service-name=e2e::reviews - expected: expected/metrics-has-value.yml - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_sidecar_internal_resp_latency_nanos --service-name=e2e::reviews - expected: expected/metrics-has-value.yml - # service instance metrics: e2e::productpage - - query: | - swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_instance_resp_time --service-name=e2e::productpage --instance-name=$( \ - swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql instance list --service-name=e2e::productpage | yq e '.[0].name' - - ) - expected: expected/metrics-has-value.yml - - query: | - swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_instance_cpm --service-name=e2e::productpage --instance-name=$( \ - swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql instance list --service-name=e2e::productpage | yq e '.[0].name' - - ) - expected: expected/metrics-has-value.yml - - query: | - swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_instance_sla --service-name=e2e::productpage --instance-name=$( \ - swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql instance list --service-name=e2e::productpage | yq e '.[0].name' - - ) - expected: expected/metrics-has-value.yml - # service instance metrics: e2e::reviews - - query: | - swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_instance_resp_time --service-name=e2e::reviews --instance-name=$( \ - swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql instance list --service-name=e2e::reviews | yq e '.[0].name' - - ) - expected: expected/metrics-has-value.yml - - query: | - swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_instance_cpm --service-name=e2e::reviews --instance-name=$( \ - swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql instance list --service-name=e2e::reviews | yq e '.[0].name' - - ) - expected: expected/metrics-has-value.yml - - query: | - swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_instance_sla --service-name=e2e::reviews --instance-name=$( \ - swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql instance list --service-name=e2e::reviews | yq e '.[0].name' - - ) - expected: expected/metrics-has-value.yml - - query: | - swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_instance_sidecar_internal_req_latency_nanos --service-name=e2e::reviews --instance-name=$( \ - swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql instance list --service-name=e2e::reviews | yq e '.[0].name' - - ) - expected: expected/metrics-has-value.yml - - query: | - swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_instance_sidecar_internal_resp_latency_nanos --service-name=e2e::reviews --instance-name=$( \ - swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql instance list --service-name=e2e::reviews | yq e '.[0].name' - - ) - expected: expected/metrics-has-value.yml - - # service endpoint metrics: e2e::productpage GET:/productpage - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=endpoint_cpm --endpoint-name=GET:/productpage --service-name=e2e::productpage - expected: expected/metrics-has-value.yml - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=endpoint_resp_time --endpoint-name=GET:/productpage --service-name=e2e::productpage - expected: expected/metrics-has-value.yml - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=endpoint_sla --endpoint-name=GET:/productpage --service-name=e2e::productpage - expected: expected/metrics-has-value.yml - # service endpoint metrics: e2e::reviews GET:/reviews/0 - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=endpoint_cpm --endpoint-name=GET:/reviews/0 --service-name=e2e::reviews - expected: expected/metrics-has-value.yml - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=endpoint_resp_time --endpoint-name=GET:/reviews/0 --service-name=e2e::reviews - expected: expected/metrics-has-value.yml - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=endpoint_sla --endpoint-name=GET:/reviews/0 --service-name=e2e::reviews - expected: expected/metrics-has-value.yml - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=endpoint_sidecar_internal_req_latency_nanos --endpoint-name=GET:/reviews/0 --service-name=e2e::reviews - expected: expected/metrics-has-value.yml - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=endpoint_sidecar_internal_resp_latency_nanos --endpoint-name=GET:/reviews/0 --service-name=e2e::reviews - expected: expected/metrics-has-value.yml - - # dependency service - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql dependency service --service-name=e2e::productpage - expected: expected/dependency-services-productpage.yml - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql dependency service --service-name=e2e::reviews - expected: expected/dependency-services-reviews.yml - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql dependency instance --service-name=e2e::productpage --dest-service-name=e2e::reviews - expected: expected/dependency-services-instance-productpage.yml - - # service relation metrics - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_relation_client_cpm --service-name=e2e::productpage --dest-service-name=e2e::reviews - expected: expected/metrics-has-value.yml - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_relation_server_cpm --service-name=e2e::productpage --dest-service-name=e2e::reviews - expected: expected/metrics-has-value.yml - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_client_sidecar_internal_req_latency_nanos --service-name=e2e::productpage --dest-service-name=e2e::reviews - expected: expected/metrics-has-value.yml - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_server_sidecar_internal_req_latency_nanos --service-name=e2e::productpage --dest-service-name=e2e::reviews - expected: expected/metrics-has-value.yml - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_client_sidecar_internal_resp_latency_nanos --service-name=e2e::productpage --dest-service-name=e2e::reviews - expected: expected/metrics-has-value.yml - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_server_sidecar_internal_resp_latency_nanos --service-name=e2e::productpage --dest-service-name=e2e::reviews - expected: expected/metrics-has-value.yml - # service instance relation metrics, e2e::productpage -> e2e::reviews - - query: | - swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_instance_relation_client_cpm \ - --service-name=e2e::productpage --instance-name=$( \ - swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql instance list --service-name=e2e::productpage | yq e '.[0].name' - ) \ - --dest-service-name=e2e::reviews --dest-instance-name=$( \ - swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql instance list --service-name=e2e::reviews | yq e '.[0].name' - ) \ - - expected: expected/metrics-has-value.yml - - query: | - swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_instance_relation_server_cpm \ - --service-name=e2e::productpage --instance-name=$( \ - swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql instance list --service-name=e2e::productpage | yq e '.[0].name' - ) \ - --dest-service-name=e2e::reviews --dest-instance-name=$( \ - swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql instance list --service-name=e2e::reviews | yq e '.[0].name' - ) \ - - expected: expected/metrics-has-value.yml diff --git a/test/e2e/e2e-elasticsearch.yaml b/test/e2e/e2e-elasticsearch.yaml deleted file mode 100644 index 54133e7..0000000 --- a/test/e2e/e2e-elasticsearch.yaml +++ /dev/null @@ -1,238 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -# This file is used to show how to write configuration files and can be used to test. - -setup: - env: kind - file: kind28.yaml - init-system-environment: env - kind: - expose-ports: - - namespace: istio-system - resource: service/skywalking-oap - port: 12800 - steps: - - name: install yq - command: bash test/e2e/setup-e2e-shell/install.sh yq - - name: install swctl - command: bash test/e2e/setup-e2e-shell/install.sh swctl - - name: install kubectl - command: bash test/e2e/setup-e2e-shell/install.sh kubectl - - name: install istio - command: | - bash test/e2e/setup-e2e-shell/install.sh istioctl - istioctl install -y --set profile=demo \ - --set meshConfig.defaultConfig.envoyAccessLogService.address=skywalking-satellite.istio-system:11800 \ - --set meshConfig.enableEnvoyAccessLogService=true - kubectl label namespace default istio-injection=enabled - - name: Install helm - command: bash test/e2e/setup-e2e-shell/install.sh helm - - name: Install ECK operator - command: | - helm dep up chart/skywalking - tar xzf chart/skywalking/charts/eck-operator-3.3.1.tgz -C /tmp eck-operator/charts/eck-operator-crds - helm -n istio-system install eck-crds /tmp/eck-operator/charts/eck-operator-crds \ - --create-namespace - - name: Install SkyWalking - command: | - helm -n istio-system install skywalking chart/skywalking \ - --set fullnameOverride=skywalking \ - --set eck-operator.installCRDs=false \ - --set oap.env.SW_ENVOY_METRIC_ALS_HTTP_ANALYSIS=k8s-mesh \ - --set oap.env.SW_ENVOY_METRIC_ALS_TCP_ANALYSIS=k8s-mesh \ - --set oap.env.K8S_SERVICE_NAME_RULE='e2e::${service.metadata.name}' \ - --set oap.replicas=1 \ - --set ui.image.repository=$UI_REPO \ - --set ui.image.tag=$UI_TAG \ - --set oap.image.repository=$OAP_REPO \ - --set oap.image.tag=$OAP_TAG \ - --set oap.storageType=elasticsearch \ - --set satellite.enabled=true \ - --set satellite.image.repository=$SATELLITE_REPO \ - --set satellite.image.tag=$SATELLITE_TAG \ - -f test/e2e/values.yaml - wait: - - namespace: istio-system - resource: deployments/skywalking-oap - for: condition=available - - namespace: istio-system - resource: deployments/skywalking-satellite - for: condition=available - - namespace: istio-system - resource: deployments/skywalking-ui - for: condition=available - - name: Deploy demo services - command: | - kubectl apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/platform/kube/bookinfo.yaml - kubectl apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/networking/bookinfo-gateway.yaml - # Enable TCP services - kubectl apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/platform/kube/bookinfo-ratings-v2.yaml - kubectl apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/platform/kube/bookinfo-db.yaml - kubectl apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/networking/destination-rule-all.yaml - kubectl apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/networking/virtual-service-ratings-db.yaml - wait: - - namespace: default - resource: pod - for: condition=Ready - - name: Generate traffic - path: traffic-gen.yaml - wait: - - namespace: default - resource: pod - for: condition=Ready - timeout: 25m - -verify: - retry: - count: 20 - interval: 10s - cases: - # service list - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql service ls - expected: expected/service.yml - # service instance list - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql instance list --service-name=e2e::productpage - expected: expected/service-instance.yml - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql instance list --service-name=e2e::reviews - expected: expected/service-instance.yml - # service endpoint - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql endpoint list --service-name=e2e::productpage - expected: expected/service-endpoint-productpage.yml - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql endpoint list --service-name=e2e::reviews - expected: expected/service-endpoint-reviews.yml - - # service metrics: e2e::productpage - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_sla --service-name=e2e::productpage - expected: expected/metrics-has-value.yml - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_cpm --service-name=e2e::productpage - expected: expected/metrics-has-value.yml - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_resp_time --service-name=e2e::productpage - expected: expected/metrics-has-value.yml - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_apdex --service-name=e2e::productpage - expected: expected/metrics-has-value.yml - # service metrics: e2e::reviews - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_sla --service-name=e2e::reviews - expected: expected/metrics-has-value.yml - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_cpm --service-name=e2e::reviews - expected: expected/metrics-has-value.yml - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_resp_time --service-name=e2e::reviews - expected: expected/metrics-has-value.yml - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_apdex --service-name=e2e::reviews - expected: expected/metrics-has-value.yml - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_sidecar_internal_req_latency_nanos --service-name=e2e::reviews - expected: expected/metrics-has-value.yml - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_sidecar_internal_resp_latency_nanos --service-name=e2e::reviews - expected: expected/metrics-has-value.yml - # service instance metrics: e2e::productpage - - query: | - swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_instance_resp_time --service-name=e2e::productpage --instance-name=$( \ - swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql instance list --service-name=e2e::productpage | yq e '.[0].name' - - ) - expected: expected/metrics-has-value.yml - - query: | - swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_instance_cpm --service-name=e2e::productpage --instance-name=$( \ - swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql instance list --service-name=e2e::productpage | yq e '.[0].name' - - ) - expected: expected/metrics-has-value.yml - - query: | - swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_instance_sla --service-name=e2e::productpage --instance-name=$( \ - swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql instance list --service-name=e2e::productpage | yq e '.[0].name' - - ) - expected: expected/metrics-has-value.yml - # service instance metrics: e2e::reviews - - query: | - swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_instance_resp_time --service-name=e2e::reviews --instance-name=$( \ - swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql instance list --service-name=e2e::reviews | yq e '.[0].name' - - ) - expected: expected/metrics-has-value.yml - - query: | - swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_instance_cpm --service-name=e2e::reviews --instance-name=$( \ - swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql instance list --service-name=e2e::reviews | yq e '.[0].name' - - ) - expected: expected/metrics-has-value.yml - - query: | - swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_instance_sla --service-name=e2e::reviews --instance-name=$( \ - swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql instance list --service-name=e2e::reviews | yq e '.[0].name' - - ) - expected: expected/metrics-has-value.yml - - query: | - swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_instance_sidecar_internal_req_latency_nanos --service-name=e2e::reviews --instance-name=$( \ - swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql instance list --service-name=e2e::reviews | yq e '.[0].name' - - ) - expected: expected/metrics-has-value.yml - - query: | - swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_instance_sidecar_internal_resp_latency_nanos --service-name=e2e::reviews --instance-name=$( \ - swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql instance list --service-name=e2e::reviews | yq e '.[0].name' - - ) - expected: expected/metrics-has-value.yml - - # service endpoint metrics: e2e::productpage GET:/productpage - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=endpoint_cpm --endpoint-name=GET:/productpage --service-name=e2e::productpage - expected: expected/metrics-has-value.yml - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=endpoint_resp_time --endpoint-name=GET:/productpage --service-name=e2e::productpage - expected: expected/metrics-has-value.yml - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=endpoint_sla --endpoint-name=GET:/productpage --service-name=e2e::productpage - expected: expected/metrics-has-value.yml - # service endpoint metrics: e2e::reviews GET:/reviews/0 - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=endpoint_cpm --endpoint-name=GET:/reviews/0 --service-name=e2e::reviews - expected: expected/metrics-has-value.yml - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=endpoint_resp_time --endpoint-name=GET:/reviews/0 --service-name=e2e::reviews - expected: expected/metrics-has-value.yml - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=endpoint_sla --endpoint-name=GET:/reviews/0 --service-name=e2e::reviews - expected: expected/metrics-has-value.yml - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=endpoint_sidecar_internal_req_latency_nanos --endpoint-name=GET:/reviews/0 --service-name=e2e::reviews - expected: expected/metrics-has-value.yml - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=endpoint_sidecar_internal_resp_latency_nanos --endpoint-name=GET:/reviews/0 --service-name=e2e::reviews - expected: expected/metrics-has-value.yml - - # dependency service - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql dependency service --service-name=e2e::productpage - expected: expected/dependency-services-productpage.yml - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql dependency service --service-name=e2e::reviews - expected: expected/dependency-services-reviews.yml - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql dependency instance --service-name=e2e::productpage --dest-service-name=e2e::reviews - expected: expected/dependency-services-instance-productpage.yml - - # service relation metrics - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_relation_client_cpm --service-name=e2e::productpage --dest-service-name=e2e::reviews - expected: expected/metrics-has-value.yml - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_relation_server_cpm --service-name=e2e::productpage --dest-service-name=e2e::reviews - expected: expected/metrics-has-value.yml - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_client_sidecar_internal_req_latency_nanos --service-name=e2e::productpage --dest-service-name=e2e::reviews - expected: expected/metrics-has-value.yml - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_server_sidecar_internal_req_latency_nanos --service-name=e2e::productpage --dest-service-name=e2e::reviews - expected: expected/metrics-has-value.yml - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_client_sidecar_internal_resp_latency_nanos --service-name=e2e::productpage --dest-service-name=e2e::reviews - expected: expected/metrics-has-value.yml - - query: swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_server_sidecar_internal_resp_latency_nanos --service-name=e2e::productpage --dest-service-name=e2e::reviews - expected: expected/metrics-has-value.yml - # service instance relation metrics, e2e::productpage -> e2e::reviews - - query: | - swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_instance_relation_client_cpm \ - --service-name=e2e::productpage --instance-name=$( \ - swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql instance list --service-name=e2e::productpage | yq e '.[0].name' - ) \ - --dest-service-name=e2e::reviews --dest-instance-name=$( \ - swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql instance list --service-name=e2e::reviews | yq e '.[0].name' - ) \ - - expected: expected/metrics-has-value.yml - - query: | - swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql metrics exec --expression=service_instance_relation_server_cpm \ - --service-name=e2e::productpage --instance-name=$( \ - swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql instance list --service-name=e2e::productpage | yq e '.[0].name' - ) \ - --dest-service-name=e2e::reviews --dest-instance-name=$( \ - swctl --display yaml --base-url=http://${service_skywalking_oap_host}:${service_skywalking_oap_12800}/graphql instance list --service-name=e2e::reviews | yq e '.[0].name' - ) \ - - expected: expected/metrics-has-value.yml diff --git a/test/e2e/e2e-oap10-banyandb.yaml b/test/e2e/e2e-oap10-banyandb.yaml new file mode 100644 index 0000000..abda5a7 --- /dev/null +++ b/test/e2e/e2e-oap10-banyandb.yaml @@ -0,0 +1,143 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# This file is used to show how to write configuration files and can be used to test. + +setup: + env: kind + file: kind28.yaml + init-system-environment: env + kind: + expose-ports: + - namespace: istio-system + resource: service/skywalking-ui + port: 80 + steps: + - name: install yq + command: bash test/e2e/setup-e2e-shell/install.sh yq + - name: install kubectl + command: bash test/e2e/setup-e2e-shell/install.sh kubectl + - name: install istio + command: | + bash test/e2e/setup-e2e-shell/install.sh istioctl + istioctl install -y --set profile=demo \ + --set meshConfig.defaultConfig.envoyAccessLogService.address=skywalking-satellite.istio-system:11800 \ + --set meshConfig.enableEnvoyAccessLogService=true + kubectl label namespace default istio-injection=enabled + - name: Install helm + command: bash test/e2e/setup-e2e-shell/install.sh helm + - name: Update chart dependencies + command: helm dep up chart/skywalking + - name: Install SkyWalking + command: | + helm -n istio-system install skywalking chart/skywalking \ + --set fullnameOverride=skywalking \ + --set oap.env.SW_ENVOY_METRIC_ALS_HTTP_ANALYSIS=k8s-mesh \ + --set oap.env.SW_ENVOY_METRIC_ALS_TCP_ANALYSIS=k8s-mesh \ + --set oap.env.K8S_SERVICE_NAME_RULE='e2e::${service.metadata.name}' \ + --set oap.replicas=1 \ + --set ui.image.repository=$UI_REPO \ + --set ui.image.tag=$UI_TAG \ + --set oap.image.repository=$OAP_10_REPO \ + --set oap.image.tag=$OAP_10_TAG \ + --set oap.storageType=banyandb \ + --set elasticsearch.enabled=false \ + --set banyandb.enabled=true \ + --set banyandb.image.repository=$BANYANDB_0_10_REPO \ + --set banyandb.image.tag=$BANYANDB_0_10_TAG \ + --set banyandb.standalone.enabled=true \ + --set banyandb.cluster.enabled=false \ + --set banyandb.auth.enabled=true \ + --set oap.ports.admin=null \ + --set ui.config.templates.mode=readonly \ + --set satellite.enabled=true \ + --set satellite.image.repository=$SATELLITE_REPO \ + --set satellite.image.tag=$SATELLITE_TAG \ + -f test/e2e/values.yaml + wait: + - namespace: istio-system + resource: deployments/skywalking-oap + for: condition=available + - namespace: istio-system + resource: deployments/skywalking-satellite + for: condition=available + - namespace: istio-system + resource: deployments/skywalking-ui + for: condition=available + - name: Deploy demo services + command: | + kubectl apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/platform/kube/bookinfo.yaml + kubectl apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/networking/bookinfo-gateway.yaml + # Enable TCP services + kubectl apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/platform/kube/bookinfo-ratings-v2.yaml + kubectl apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/platform/kube/bookinfo-db.yaml + kubectl apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/networking/destination-rule-all.yaml + kubectl apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/networking/virtual-service-ratings-db.yaml + wait: + - namespace: default + resource: pod + for: condition=Ready + - name: Generate traffic + path: traffic-gen.yaml + wait: + - namespace: default + resource: pod + for: condition=Ready + timeout: 25m + +verify: + retry: + count: 20 + interval: 10s + cases: + # Every assertion below goes through Horizon's BFF: horizon -> oap -> storage. + # Nothing talks to OAP's GraphQL directly -- that would test OAP, not the chart. + + # The operator can log in, and lands with the role the values file granted. + - query: | + bash test/e2e/script/horizon.sh http://${service_skywalking_ui_host}:${service_skywalking_ui_80} get /api/auth/me | yq -p json -o yaml '{"username": .username, "roles": .roles}' + expected: expected/horizon-me.yml + + # Cluster Status -> Query pane. Proves oap.queryUrl reaches the in-cluster OAP. + - query: | + bash test/e2e/script/horizon.sh http://${service_skywalking_ui_host}:${service_skywalking_ui_80} get /api/oap/info | yq -p json -o yaml '{"reachable": .reachable, "version": .version}' + expected: expected/horizon-oap-info.yml + + # Inverted: the admin host arrived in OAP 11, so 10.4 has none and Horizon must + # be running readonly. These endpoints answer HTTP 200 either way and report the + # failure in the body, so the assertion flips rather than disappearing. + - query: | + bash test/e2e/script/horizon.sh http://${service_skywalking_ui_host}:${service_skywalking_ui_80} get '/api/preflight?refresh=1' | yq -p json -o yaml '{"adminReachable": .adminReachable, "templatesMode": .templatesMode}' + expected: expected/horizon-admin-readonly.yml + + # readonly still serves templates -- from the bundle in the image, without ever + # contacting OAP. + - query: | + bash test/e2e/script/horizon.sh http://${service_skywalking_ui_host}:${service_skywalking_ui_80} get '/api/admin/templates/sync-status?force=true' | yq -p json -o yaml '{"mode": .mode, "unreachable": .unreachable, "hasRows": ((.rows | length) > 0)}' + expected: expected/horizon-templates-readonly.yml + + # The mesh services the demo app produces, listed through Horizon. + # NOTE: the layer is MESH, not GENERAL -- the fixture feeds OAP via Envoy ALS + # with k8s-mesh analysis, so nothing lands in GENERAL. Asserting .reachable here + # would be worthless: the handler returns reachable:true with an empty list even + # when OAP is down. + - query: | + bash test/e2e/script/horizon.sh http://${service_skywalking_ui_host}:${service_skywalking_ui_80} get /api/layer/MESH/services | yq -p json -o yaml '{"layer": .layer, "services": ([.services[].name] | sort)}' + expected: expected/horizon-mesh-services.yml + + # service_cpm for one of them -- the full read path through to storage. + - query: | + bash test/e2e/script/horizon.sh http://${service_skywalking_ui_host}:${service_skywalking_ui_80} post /api/layer/MESH/dashboard '{"service":"e2e::productpage","scope":"service","widgets":[{"id":"cpm","title":"Service CPM","type":"card","expressions":["service_cpm"]}]}' | yq -p json -o yaml '{"reachable": .reachable, "id": .widgets[0].id, "value": .widgets[0].value}' + expected: expected/horizon-service-cpm.yml diff --git a/test/e2e/e2e-oap10-elasticsearch.yaml b/test/e2e/e2e-oap10-elasticsearch.yaml new file mode 100644 index 0000000..6b0a1d9 --- /dev/null +++ b/test/e2e/e2e-oap10-elasticsearch.yaml @@ -0,0 +1,141 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# This file is used to show how to write configuration files and can be used to test. + +setup: + env: kind + file: kind28.yaml + init-system-environment: env + kind: + expose-ports: + - namespace: istio-system + resource: service/skywalking-ui + port: 80 + steps: + - name: install yq + command: bash test/e2e/setup-e2e-shell/install.sh yq + - name: install kubectl + command: bash test/e2e/setup-e2e-shell/install.sh kubectl + - name: install istio + command: | + bash test/e2e/setup-e2e-shell/install.sh istioctl + istioctl install -y --set profile=demo \ + --set meshConfig.defaultConfig.envoyAccessLogService.address=skywalking-satellite.istio-system:11800 \ + --set meshConfig.enableEnvoyAccessLogService=true + kubectl label namespace default istio-injection=enabled + - name: Install helm + command: bash test/e2e/setup-e2e-shell/install.sh helm + - name: Install ECK operator + command: | + helm dep up chart/skywalking + tar xzf chart/skywalking/charts/eck-operator-3.3.1.tgz -C /tmp eck-operator/charts/eck-operator-crds + helm -n istio-system install eck-crds /tmp/eck-operator/charts/eck-operator-crds \ + --create-namespace + - name: Install SkyWalking + command: | + helm -n istio-system install skywalking chart/skywalking \ + --set fullnameOverride=skywalking \ + --set eck-operator.installCRDs=false \ + --set oap.env.SW_ENVOY_METRIC_ALS_HTTP_ANALYSIS=k8s-mesh \ + --set oap.env.SW_ENVOY_METRIC_ALS_TCP_ANALYSIS=k8s-mesh \ + --set oap.env.K8S_SERVICE_NAME_RULE='e2e::${service.metadata.name}' \ + --set oap.replicas=1 \ + --set ui.image.repository=$UI_REPO \ + --set ui.image.tag=$UI_TAG \ + --set oap.image.repository=$OAP_10_REPO \ + --set oap.image.tag=$OAP_10_TAG \ + --set oap.storageType=elasticsearch \ + --set oap.ports.admin=null \ + --set ui.config.templates.mode=readonly \ + --set satellite.enabled=true \ + --set satellite.image.repository=$SATELLITE_REPO \ + --set satellite.image.tag=$SATELLITE_TAG \ + -f test/e2e/values.yaml + wait: + - namespace: istio-system + resource: deployments/skywalking-oap + for: condition=available + - namespace: istio-system + resource: deployments/skywalking-satellite + for: condition=available + - namespace: istio-system + resource: deployments/skywalking-ui + for: condition=available + - name: Deploy demo services + command: | + kubectl apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/platform/kube/bookinfo.yaml + kubectl apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/networking/bookinfo-gateway.yaml + # Enable TCP services + kubectl apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/platform/kube/bookinfo-ratings-v2.yaml + kubectl apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/platform/kube/bookinfo-db.yaml + kubectl apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/networking/destination-rule-all.yaml + kubectl apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/networking/virtual-service-ratings-db.yaml + wait: + - namespace: default + resource: pod + for: condition=Ready + - name: Generate traffic + path: traffic-gen.yaml + wait: + - namespace: default + resource: pod + for: condition=Ready + timeout: 25m + +verify: + retry: + count: 20 + interval: 10s + cases: + # Every assertion below goes through Horizon's BFF: horizon -> oap -> storage. + # Nothing talks to OAP's GraphQL directly -- that would test OAP, not the chart. + + # The operator can log in, and lands with the role the values file granted. + - query: | + bash test/e2e/script/horizon.sh http://${service_skywalking_ui_host}:${service_skywalking_ui_80} get /api/auth/me | yq -p json -o yaml '{"username": .username, "roles": .roles}' + expected: expected/horizon-me.yml + + # Cluster Status -> Query pane. Proves oap.queryUrl reaches the in-cluster OAP. + - query: | + bash test/e2e/script/horizon.sh http://${service_skywalking_ui_host}:${service_skywalking_ui_80} get /api/oap/info | yq -p json -o yaml '{"reachable": .reachable, "version": .version}' + expected: expected/horizon-oap-info.yml + + # Inverted: the admin host arrived in OAP 11, so 10.4 has none and Horizon must + # be running readonly. These endpoints answer HTTP 200 either way and report the + # failure in the body, so the assertion flips rather than disappearing. + - query: | + bash test/e2e/script/horizon.sh http://${service_skywalking_ui_host}:${service_skywalking_ui_80} get '/api/preflight?refresh=1' | yq -p json -o yaml '{"adminReachable": .adminReachable, "templatesMode": .templatesMode}' + expected: expected/horizon-admin-readonly.yml + + # readonly still serves templates -- from the bundle in the image, without ever + # contacting OAP. + - query: | + bash test/e2e/script/horizon.sh http://${service_skywalking_ui_host}:${service_skywalking_ui_80} get '/api/admin/templates/sync-status?force=true' | yq -p json -o yaml '{"mode": .mode, "unreachable": .unreachable, "hasRows": ((.rows | length) > 0)}' + expected: expected/horizon-templates-readonly.yml + + # The mesh services the demo app produces, listed through Horizon. + # NOTE: the layer is MESH, not GENERAL -- the fixture feeds OAP via Envoy ALS + # with k8s-mesh analysis, so nothing lands in GENERAL. Asserting .reachable here + # would be worthless: the handler returns reachable:true with an empty list even + # when OAP is down. + - query: | + bash test/e2e/script/horizon.sh http://${service_skywalking_ui_host}:${service_skywalking_ui_80} get /api/layer/MESH/services | yq -p json -o yaml '{"layer": .layer, "services": ([.services[].name] | sort)}' + expected: expected/horizon-mesh-services.yml + + # service_cpm for one of them -- the full read path through to storage. + - query: | + bash test/e2e/script/horizon.sh http://${service_skywalking_ui_host}:${service_skywalking_ui_80} post /api/layer/MESH/dashboard '{"service":"e2e::productpage","scope":"service","widgets":[{"id":"cpm","title":"Service CPM","type":"card","expressions":["service_cpm"]}]}' | yq -p json -o yaml '{"reachable": .reachable, "id": .widgets[0].id, "value": .widgets[0].value}' + expected: expected/horizon-service-cpm.yml diff --git a/test/e2e/e2e-oap11-banyandb-cluster.yaml b/test/e2e/e2e-oap11-banyandb-cluster.yaml new file mode 100644 index 0000000..fa3c93f --- /dev/null +++ b/test/e2e/e2e-oap11-banyandb-cluster.yaml @@ -0,0 +1,148 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# This file is used to show how to write configuration files and can be used to test. + +setup: + env: kind + file: kind28.yaml + init-system-environment: env + kind: + expose-ports: + - namespace: istio-system + resource: service/skywalking-ui + port: 80 + steps: + - name: install yq + command: bash test/e2e/setup-e2e-shell/install.sh yq + - name: install kubectl + command: bash test/e2e/setup-e2e-shell/install.sh kubectl + - name: install istio + command: | + bash test/e2e/setup-e2e-shell/install.sh istioctl + istioctl install -y --set profile=demo \ + --set meshConfig.defaultConfig.envoyAccessLogService.address=skywalking-satellite.istio-system:11800 \ + --set meshConfig.enableEnvoyAccessLogService=true + kubectl label namespace default istio-injection=enabled + - name: Install helm + command: bash test/e2e/setup-e2e-shell/install.sh helm + - name: Update chart dependencies + command: helm dep up chart/skywalking + - name: Install SkyWalking + command: | + helm -n istio-system install skywalking chart/skywalking \ + --set fullnameOverride=skywalking \ + --set oap.env.SW_ENVOY_METRIC_ALS_HTTP_ANALYSIS=k8s-mesh \ + --set oap.env.SW_ENVOY_METRIC_ALS_TCP_ANALYSIS=k8s-mesh \ + --set oap.env.K8S_SERVICE_NAME_RULE='e2e::${service.metadata.name}' \ + --set oap.replicas=1 \ + --set ui.image.repository=$UI_REPO \ + --set ui.image.tag=$UI_TAG \ + --set oap.image.repository=$OAP_REPO \ + --set oap.image.tag=$OAP_TAG \ + --set oap.storageType=banyandb \ + --set elasticsearch.enabled=false \ + --set banyandb.enabled=true \ + --set banyandb.image.repository=$BANYANDB_REPO \ + --set banyandb.image.tag=$BANYANDB_TAG \ + --set banyandb.standalone.enabled=false \ + --set banyandb.cluster.enabled=true \ + --set banyandb.auth.enabled=true \ + --set oap.ports.zipkin-query=9412 \ + --set oap.env.SW_RECEIVER_ZIPKIN=default \ + --set oap.env.SW_QUERY_ZIPKIN=default \ + --set satellite.enabled=true \ + --set satellite.image.repository=$SATELLITE_REPO \ + --set satellite.image.tag=$SATELLITE_TAG \ + -f test/e2e/values.yaml + wait: + - namespace: istio-system + resource: deployments/skywalking-oap + for: condition=available + - namespace: istio-system + resource: deployments/skywalking-satellite + for: condition=available + - namespace: istio-system + resource: deployments/skywalking-ui + for: condition=available + - name: Deploy demo services + command: | + kubectl apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/platform/kube/bookinfo.yaml + kubectl apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/networking/bookinfo-gateway.yaml + # Enable TCP services + kubectl apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/platform/kube/bookinfo-ratings-v2.yaml + kubectl apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/platform/kube/bookinfo-db.yaml + kubectl apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/networking/destination-rule-all.yaml + kubectl apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/networking/virtual-service-ratings-db.yaml + wait: + - namespace: default + resource: pod + for: condition=Ready + - name: Generate traffic + path: traffic-gen.yaml + wait: + - namespace: default + resource: pod + for: condition=Ready + timeout: 25m + +verify: + retry: + count: 20 + interval: 10s + cases: + # Every assertion below goes through Horizon's BFF: horizon -> oap -> storage. + # Nothing talks to OAP's GraphQL directly -- that would test OAP, not the chart. + + # The operator can log in, and lands with the role the values file granted. + - query: | + bash test/e2e/script/horizon.sh http://${service_skywalking_ui_host}:${service_skywalking_ui_80} get /api/auth/me | yq -p json -o yaml '{"username": .username, "roles": .roles}' + expected: expected/horizon-me.yml + + # Cluster Status -> Query pane. Proves oap.queryUrl reaches the in-cluster OAP. + - query: | + bash test/e2e/script/horizon.sh http://${service_skywalking_ui_host}:${service_skywalking_ui_80} get /api/oap/info | yq -p json -o yaml '{"reachable": .reachable, "version": .version}' + expected: expected/horizon-oap-info.yml + + # Cluster Status -> Zipkin pane. Proves oap.zipkinUrl, which the chart only + # emits when oap.ports.zipkin-query is set. + - query: | + bash test/e2e/script/horizon.sh http://${service_skywalking_ui_host}:${service_skywalking_ui_80} get /api/oap/info | yq -p json -o yaml '{"zipkinReachable": .zipkinReachable}' + expected: expected/horizon-zipkin.yml + + # Cluster Status -> Admin pane. Proves oap.adminUrl reaches the OAP 11 admin + # host and that ui-management is mounted on it. + - query: | + bash test/e2e/script/horizon.sh http://${service_skywalking_ui_host}:${service_skywalking_ui_80} get '/api/preflight?refresh=1' | yq -p json -o yaml '{"adminReachable": .adminReachable, "templatesMode": .templatesMode, "uiManagement": ([.modules[] | select(.name == "ui-management") | .reachable] | .[0])}' + expected: expected/horizon-admin-live.yml + + # The dashboard template store, read through /ui-management on the admin port. + - query: | + bash test/e2e/script/horizon.sh http://${service_skywalking_ui_host}:${service_skywalking_ui_80} get '/api/admin/templates/sync-status?force=true' | yq -p json -o yaml '{"mode": .mode, "unreachable": .unreachable, "hasRows": ((.rows | length) > 0)}' + expected: expected/horizon-templates-live.yml + + # The mesh services the demo app produces, listed through Horizon. + # NOTE: the layer is MESH, not GENERAL -- the fixture feeds OAP via Envoy ALS + # with k8s-mesh analysis, so nothing lands in GENERAL. Asserting .reachable here + # would be worthless: the handler returns reachable:true with an empty list even + # when OAP is down. + - query: | + bash test/e2e/script/horizon.sh http://${service_skywalking_ui_host}:${service_skywalking_ui_80} get /api/layer/MESH/services | yq -p json -o yaml '{"layer": .layer, "services": ([.services[].name] | sort)}' + expected: expected/horizon-mesh-services.yml + + # service_cpm for one of them -- the full read path through to storage. + - query: | + bash test/e2e/script/horizon.sh http://${service_skywalking_ui_host}:${service_skywalking_ui_80} post /api/layer/MESH/dashboard '{"service":"e2e::productpage","scope":"service","widgets":[{"id":"cpm","title":"Service CPM","type":"card","expressions":["service_cpm"]}]}' | yq -p json -o yaml '{"reachable": .reachable, "id": .widgets[0].id, "value": .widgets[0].value}' + expected: expected/horizon-service-cpm.yml diff --git a/test/e2e/e2e-oap11-banyandb-standalone.yaml b/test/e2e/e2e-oap11-banyandb-standalone.yaml new file mode 100644 index 0000000..f5244cd --- /dev/null +++ b/test/e2e/e2e-oap11-banyandb-standalone.yaml @@ -0,0 +1,148 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# This file is used to show how to write configuration files and can be used to test. + +setup: + env: kind + file: kind28.yaml + init-system-environment: env + kind: + expose-ports: + - namespace: istio-system + resource: service/skywalking-ui + port: 80 + steps: + - name: install yq + command: bash test/e2e/setup-e2e-shell/install.sh yq + - name: install kubectl + command: bash test/e2e/setup-e2e-shell/install.sh kubectl + - name: install istio + command: | + bash test/e2e/setup-e2e-shell/install.sh istioctl + istioctl install -y --set profile=demo \ + --set meshConfig.defaultConfig.envoyAccessLogService.address=skywalking-satellite.istio-system:11800 \ + --set meshConfig.enableEnvoyAccessLogService=true + kubectl label namespace default istio-injection=enabled + - name: Install helm + command: bash test/e2e/setup-e2e-shell/install.sh helm + - name: Update chart dependencies + command: helm dep up chart/skywalking + - name: Install SkyWalking + command: | + helm -n istio-system install skywalking chart/skywalking \ + --set fullnameOverride=skywalking \ + --set oap.env.SW_ENVOY_METRIC_ALS_HTTP_ANALYSIS=k8s-mesh \ + --set oap.env.SW_ENVOY_METRIC_ALS_TCP_ANALYSIS=k8s-mesh \ + --set oap.env.K8S_SERVICE_NAME_RULE='e2e::${service.metadata.name}' \ + --set oap.replicas=1 \ + --set ui.image.repository=$UI_REPO \ + --set ui.image.tag=$UI_TAG \ + --set oap.image.repository=$OAP_REPO \ + --set oap.image.tag=$OAP_TAG \ + --set oap.storageType=banyandb \ + --set elasticsearch.enabled=false \ + --set banyandb.enabled=true \ + --set banyandb.image.repository=$BANYANDB_REPO \ + --set banyandb.image.tag=$BANYANDB_TAG \ + --set banyandb.standalone.enabled=true \ + --set banyandb.cluster.enabled=false \ + --set banyandb.auth.enabled=true \ + --set oap.ports.zipkin-query=9412 \ + --set oap.env.SW_RECEIVER_ZIPKIN=default \ + --set oap.env.SW_QUERY_ZIPKIN=default \ + --set satellite.enabled=true \ + --set satellite.image.repository=$SATELLITE_REPO \ + --set satellite.image.tag=$SATELLITE_TAG \ + -f test/e2e/values.yaml + wait: + - namespace: istio-system + resource: deployments/skywalking-oap + for: condition=available + - namespace: istio-system + resource: deployments/skywalking-satellite + for: condition=available + - namespace: istio-system + resource: deployments/skywalking-ui + for: condition=available + - name: Deploy demo services + command: | + kubectl apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/platform/kube/bookinfo.yaml + kubectl apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/networking/bookinfo-gateway.yaml + # Enable TCP services + kubectl apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/platform/kube/bookinfo-ratings-v2.yaml + kubectl apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/platform/kube/bookinfo-db.yaml + kubectl apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/networking/destination-rule-all.yaml + kubectl apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/networking/virtual-service-ratings-db.yaml + wait: + - namespace: default + resource: pod + for: condition=Ready + - name: Generate traffic + path: traffic-gen.yaml + wait: + - namespace: default + resource: pod + for: condition=Ready + timeout: 25m + +verify: + retry: + count: 20 + interval: 10s + cases: + # Every assertion below goes through Horizon's BFF: horizon -> oap -> storage. + # Nothing talks to OAP's GraphQL directly -- that would test OAP, not the chart. + + # The operator can log in, and lands with the role the values file granted. + - query: | + bash test/e2e/script/horizon.sh http://${service_skywalking_ui_host}:${service_skywalking_ui_80} get /api/auth/me | yq -p json -o yaml '{"username": .username, "roles": .roles}' + expected: expected/horizon-me.yml + + # Cluster Status -> Query pane. Proves oap.queryUrl reaches the in-cluster OAP. + - query: | + bash test/e2e/script/horizon.sh http://${service_skywalking_ui_host}:${service_skywalking_ui_80} get /api/oap/info | yq -p json -o yaml '{"reachable": .reachable, "version": .version}' + expected: expected/horizon-oap-info.yml + + # Cluster Status -> Zipkin pane. Proves oap.zipkinUrl, which the chart only + # emits when oap.ports.zipkin-query is set. + - query: | + bash test/e2e/script/horizon.sh http://${service_skywalking_ui_host}:${service_skywalking_ui_80} get /api/oap/info | yq -p json -o yaml '{"zipkinReachable": .zipkinReachable}' + expected: expected/horizon-zipkin.yml + + # Cluster Status -> Admin pane. Proves oap.adminUrl reaches the OAP 11 admin + # host and that ui-management is mounted on it. + - query: | + bash test/e2e/script/horizon.sh http://${service_skywalking_ui_host}:${service_skywalking_ui_80} get '/api/preflight?refresh=1' | yq -p json -o yaml '{"adminReachable": .adminReachable, "templatesMode": .templatesMode, "uiManagement": ([.modules[] | select(.name == "ui-management") | .reachable] | .[0])}' + expected: expected/horizon-admin-live.yml + + # The dashboard template store, read through /ui-management on the admin port. + - query: | + bash test/e2e/script/horizon.sh http://${service_skywalking_ui_host}:${service_skywalking_ui_80} get '/api/admin/templates/sync-status?force=true' | yq -p json -o yaml '{"mode": .mode, "unreachable": .unreachable, "hasRows": ((.rows | length) > 0)}' + expected: expected/horizon-templates-live.yml + + # The mesh services the demo app produces, listed through Horizon. + # NOTE: the layer is MESH, not GENERAL -- the fixture feeds OAP via Envoy ALS + # with k8s-mesh analysis, so nothing lands in GENERAL. Asserting .reachable here + # would be worthless: the handler returns reachable:true with an empty list even + # when OAP is down. + - query: | + bash test/e2e/script/horizon.sh http://${service_skywalking_ui_host}:${service_skywalking_ui_80} get /api/layer/MESH/services | yq -p json -o yaml '{"layer": .layer, "services": ([.services[].name] | sort)}' + expected: expected/horizon-mesh-services.yml + + # service_cpm for one of them -- the full read path through to storage. + - query: | + bash test/e2e/script/horizon.sh http://${service_skywalking_ui_host}:${service_skywalking_ui_80} post /api/layer/MESH/dashboard '{"service":"e2e::productpage","scope":"service","widgets":[{"id":"cpm","title":"Service CPM","type":"card","expressions":["service_cpm"]}]}' | yq -p json -o yaml '{"reachable": .reachable, "id": .widgets[0].id, "value": .widgets[0].value}' + expected: expected/horizon-service-cpm.yml diff --git a/test/e2e/e2e-oap11-elasticsearch.yaml b/test/e2e/e2e-oap11-elasticsearch.yaml new file mode 100644 index 0000000..4329d8c --- /dev/null +++ b/test/e2e/e2e-oap11-elasticsearch.yaml @@ -0,0 +1,146 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# This file is used to show how to write configuration files and can be used to test. + +setup: + env: kind + file: kind28.yaml + init-system-environment: env + kind: + expose-ports: + - namespace: istio-system + resource: service/skywalking-ui + port: 80 + steps: + - name: install yq + command: bash test/e2e/setup-e2e-shell/install.sh yq + - name: install kubectl + command: bash test/e2e/setup-e2e-shell/install.sh kubectl + - name: install istio + command: | + bash test/e2e/setup-e2e-shell/install.sh istioctl + istioctl install -y --set profile=demo \ + --set meshConfig.defaultConfig.envoyAccessLogService.address=skywalking-satellite.istio-system:11800 \ + --set meshConfig.enableEnvoyAccessLogService=true + kubectl label namespace default istio-injection=enabled + - name: Install helm + command: bash test/e2e/setup-e2e-shell/install.sh helm + - name: Install ECK operator + command: | + helm dep up chart/skywalking + tar xzf chart/skywalking/charts/eck-operator-3.3.1.tgz -C /tmp eck-operator/charts/eck-operator-crds + helm -n istio-system install eck-crds /tmp/eck-operator/charts/eck-operator-crds \ + --create-namespace + - name: Install SkyWalking + command: | + helm -n istio-system install skywalking chart/skywalking \ + --set fullnameOverride=skywalking \ + --set eck-operator.installCRDs=false \ + --set oap.env.SW_ENVOY_METRIC_ALS_HTTP_ANALYSIS=k8s-mesh \ + --set oap.env.SW_ENVOY_METRIC_ALS_TCP_ANALYSIS=k8s-mesh \ + --set oap.env.K8S_SERVICE_NAME_RULE='e2e::${service.metadata.name}' \ + --set oap.replicas=1 \ + --set ui.image.repository=$UI_REPO \ + --set ui.image.tag=$UI_TAG \ + --set oap.image.repository=$OAP_REPO \ + --set oap.image.tag=$OAP_TAG \ + --set oap.storageType=elasticsearch \ + --set oap.ports.zipkin-query=9412 \ + --set oap.env.SW_RECEIVER_ZIPKIN=default \ + --set oap.env.SW_QUERY_ZIPKIN=default \ + --set satellite.enabled=true \ + --set satellite.image.repository=$SATELLITE_REPO \ + --set satellite.image.tag=$SATELLITE_TAG \ + -f test/e2e/values.yaml + wait: + - namespace: istio-system + resource: deployments/skywalking-oap + for: condition=available + - namespace: istio-system + resource: deployments/skywalking-satellite + for: condition=available + - namespace: istio-system + resource: deployments/skywalking-ui + for: condition=available + - name: Deploy demo services + command: | + kubectl apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/platform/kube/bookinfo.yaml + kubectl apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/networking/bookinfo-gateway.yaml + # Enable TCP services + kubectl apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/platform/kube/bookinfo-ratings-v2.yaml + kubectl apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/platform/kube/bookinfo-db.yaml + kubectl apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/networking/destination-rule-all.yaml + kubectl apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/networking/virtual-service-ratings-db.yaml + wait: + - namespace: default + resource: pod + for: condition=Ready + - name: Generate traffic + path: traffic-gen.yaml + wait: + - namespace: default + resource: pod + for: condition=Ready + timeout: 25m + +verify: + retry: + count: 20 + interval: 10s + cases: + # Every assertion below goes through Horizon's BFF: horizon -> oap -> storage. + # Nothing talks to OAP's GraphQL directly -- that would test OAP, not the chart. + + # The operator can log in, and lands with the role the values file granted. + - query: | + bash test/e2e/script/horizon.sh http://${service_skywalking_ui_host}:${service_skywalking_ui_80} get /api/auth/me | yq -p json -o yaml '{"username": .username, "roles": .roles}' + expected: expected/horizon-me.yml + + # Cluster Status -> Query pane. Proves oap.queryUrl reaches the in-cluster OAP. + - query: | + bash test/e2e/script/horizon.sh http://${service_skywalking_ui_host}:${service_skywalking_ui_80} get /api/oap/info | yq -p json -o yaml '{"reachable": .reachable, "version": .version}' + expected: expected/horizon-oap-info.yml + + # Cluster Status -> Zipkin pane. Proves oap.zipkinUrl, which the chart only + # emits when oap.ports.zipkin-query is set. + - query: | + bash test/e2e/script/horizon.sh http://${service_skywalking_ui_host}:${service_skywalking_ui_80} get /api/oap/info | yq -p json -o yaml '{"zipkinReachable": .zipkinReachable}' + expected: expected/horizon-zipkin.yml + + # Cluster Status -> Admin pane. Proves oap.adminUrl reaches the OAP 11 admin + # host and that ui-management is mounted on it. + - query: | + bash test/e2e/script/horizon.sh http://${service_skywalking_ui_host}:${service_skywalking_ui_80} get '/api/preflight?refresh=1' | yq -p json -o yaml '{"adminReachable": .adminReachable, "templatesMode": .templatesMode, "uiManagement": ([.modules[] | select(.name == "ui-management") | .reachable] | .[0])}' + expected: expected/horizon-admin-live.yml + + # The dashboard template store, read through /ui-management on the admin port. + - query: | + bash test/e2e/script/horizon.sh http://${service_skywalking_ui_host}:${service_skywalking_ui_80} get '/api/admin/templates/sync-status?force=true' | yq -p json -o yaml '{"mode": .mode, "unreachable": .unreachable, "hasRows": ((.rows | length) > 0)}' + expected: expected/horizon-templates-live.yml + + # The mesh services the demo app produces, listed through Horizon. + # NOTE: the layer is MESH, not GENERAL -- the fixture feeds OAP via Envoy ALS + # with k8s-mesh analysis, so nothing lands in GENERAL. Asserting .reachable here + # would be worthless: the handler returns reachable:true with an empty list even + # when OAP is down. + - query: | + bash test/e2e/script/horizon.sh http://${service_skywalking_ui_host}:${service_skywalking_ui_80} get /api/layer/MESH/services | yq -p json -o yaml '{"layer": .layer, "services": ([.services[].name] | sort)}' + expected: expected/horizon-mesh-services.yml + + # service_cpm for one of them -- the full read path through to storage. + - query: | + bash test/e2e/script/horizon.sh http://${service_skywalking_ui_host}:${service_skywalking_ui_80} post /api/layer/MESH/dashboard '{"service":"e2e::productpage","scope":"service","widgets":[{"id":"cpm","title":"Service CPM","type":"card","expressions":["service_cpm"]}]}' | yq -p json -o yaml '{"reachable": .reachable, "id": .widgets[0].id, "value": .widgets[0].value}' + expected: expected/horizon-service-cpm.yml diff --git a/test/e2e/e2e-operator-chart.yaml b/test/e2e/e2e-operator-chart.yaml new file mode 100644 index 0000000..0cc8822 --- /dev/null +++ b/test/e2e/e2e-operator-chart.yaml @@ -0,0 +1,94 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Tests THIS repository's Helm packaging of the SWCK operator -- not SWCK's +# behaviour, which apache/skywalking-swck covers in its own e2e suite. +# +# What is unique here: chart/operator ships hand-written cert.yaml (a +# cert-manager Certificate + self-signed Issuer), webhook.yaml, and a +# java-agent ConfigMap, all named through Helm helpers that truncate to 63 +# characters. SWCK deploys itself with kustomize and never exercises any of it. +# +# Agent injection is the probe because it is the cheapest assertion that +# requires the whole chain to work: cert issued -> CA injected into the webhook +# config -> API server can reach the webhook Service -> RBAC permits the +# mutation. The webhook's failurePolicy is Fail, so a broken chain rejects the +# pod outright rather than silently skipping injection. + +setup: + env: kind + file: kind28.yaml + init-system-environment: env + steps: + - name: Install yq + command: bash test/e2e/setup-e2e-shell/install.sh yq + - name: Install kubectl + command: bash test/e2e/setup-e2e-shell/install.sh kubectl + - name: Install helm + command: bash test/e2e/setup-e2e-shell/install.sh helm + - name: Install cert-manager + command: | + kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.9.1/cert-manager.yaml + wait: + - namespace: cert-manager + resource: pod + for: condition=Ready + - name: Install the operator chart + command: | + helm dep up chart/operator + helm install operator chart/operator --namespace=skywalking-swck-system \ + --create-namespace \ + --set fullnameOverride=skywalking-swck \ + --set image.repository=$SWCK_OPERATOR_REPO \ + --set image.tag=$SWCK_OPERATOR_TAG + wait: + - namespace: skywalking-swck-system + resource: deployment/skywalking-swck-operator + for: condition=Available + - name: Create a pod for the injector to mutate + command: | + kubectl create namespace inject-test + kubectl label namespace inject-test swck-injection=enabled + kubectl apply -f test/e2e/inject-probe.yaml + wait: + - namespace: inject-test + resource: pod/inject-probe + for: condition=Ready + timeout: 15m + +verify: + retry: + count: 10 + interval: 5s + cases: + # The chart's CRDs are a hand-copied snapshot of SWCK's generated ones, with + # no sync check. If they drift from the operator image pinned beside them, + # this is what notices. + - query: | + kubectl get crd -o json | yq -p json -o yaml '{"kinds": ([.items[].spec.names.kind] | sort | map(select(. == "OAPServer" or . == "UI" or . == "Satellite" or . == "JavaAgent")))}' + expected: expected/swck-crds.yml + + # The mutation happened at all -- proves cert, CA injection, webhook routing + # and RBAC are all correct in the chart's templates. + - query: | + kubectl -n inject-test get pod inject-probe -o json | yq -p json -o yaml '{"injected": ((.spec.initContainers // []) | length > 0)}' + expected: expected/swck-injected.yml + + # The injected pod mounts the ConfigMap THIS chart renders. The name goes + # through the 63-char truncation helper, so a mismatch here is a chart bug + # the operator's own tests can never see. + - query: | + kubectl -n inject-test get pod inject-probe -o json | yq -p json -o yaml '{"configMaps": ([.spec.volumes[] | select(.configMap != null) | .configMap.name] | sort)}' + expected: expected/swck-agent-configmap.yml diff --git a/test/e2e/env b/test/e2e/env index 9e4cf0e..fa22d9d 100644 --- a/test/e2e/env +++ b/test/e2e/env @@ -13,14 +13,17 @@ # See the License for the specific language governing permissions and # limitations under the License. -OAP_TAG=2aa0e8502a928c7066a01f864ee32122f12dfcae -OAP_REPO=ghcr.io/apache/skywalking/oap -# Horizon UI dev images are published to GHCR by the apache/skywalking-horizon-ui -# repo (tags: full commit SHA, vX.Y.Z, main). Release images live on Docker Hub -# at apache/skywalking-ui:horizon-x.y.z. CI pins to a SHA so re-runs are -# reproducible — bump this when a new horizon-ui feature/fix needs CI coverage. -UI_TAG=58a4c1fb61478936f317136400d6dbd0cdda5ee1 -UI_REPO=ghcr.io/apache/skywalking-horizon-ui +# CI runs the released artifacts so it validates exactly what the README tells +# users to install. OAP 11.0.0 accepts BanyanDB server API 0.11 only, so this +# trio (OAP 11.0.0 / Horizon 1.0.0 / BanyanDB 0.11.0) has to move together. +OAP_TAG=11.0.0 +OAP_REPO=docker.io/apache/skywalking-oap-server +# Horizon UI release images live on Docker Hub at apache/skywalking-ui:horizon-x.y.z. +# Dev images are published to GHCR by the apache/skywalking-horizon-ui repo +# (tags: full commit SHA, x.y.z, main) — switch to a GHCR SHA pin when CI needs +# an unreleased horizon-ui fix. +UI_TAG=horizon-1.0.0 +UI_REPO=docker.io/apache/skywalking-ui # SWCK operator release image — apache/skywalking-swck:0.10.0 includes # Horizon UI support (PR #199). Match the chart default in # chart/operator/values.yaml so CI exercises what users get. @@ -28,6 +31,25 @@ SWCK_OPERATOR_REPO=docker.io/apache/skywalking-swck SWCK_OPERATOR_TAG=0.10.0 SATELLITE_TAG=v35bfaff6352b4dc351a706772796a1f79b651c14 SATELLITE_REPO=ghcr.io/apache/skywalking-satellite/skywalking-satellite -BANYANDB_TAG=36921708b3bd3e0cd0ca37db9761fadf685c99a1 +# BanyanDB 0.11.0 is released (source on dist.apache.org, listed on the website) +# but its release image has not been pushed to Docker Hub yet: the +# `publish-docker` workflow's Docker Hub path arrived in apache/skywalking-banyandb#1315, +# which landed on main AFTER the v0.11.0 tag, so it has never been dispatched for +# that tag. Until `docker.io/apache/skywalking-banyandb:0.11.0` exists, pin the +# GHCR dev image built from the exact v0.11.0 commit (3b83e18 "feat: advertise +# BanyanDB API version 0.11") so CI still exercises a real 0.11 server. The +# previous pin (3692170, 2026-08-03) predates 0.11 and cannot satisfy OAP 11. +# TODO: switch to BANYANDB_REPO=docker.io/apache/skywalking-banyandb + BANYANDB_TAG=0.11.0. +BANYANDB_TAG=3b83e18fb0481d02e44eaa5df137fcf7b000754b BANYANDB_REPO=ghcr.io/apache/skywalking-banyandb -SW_CTL_COMMIT=9a1beab08413ce415a00a8547a238a14691c5655 + +# The OAP 10 compatibility line. FROZEN: 10.4.0 is the last v10 release, and it +# pins compatibleServerApiVersions to 0.10, so this pair never moves again. +# Horizon 1.0.0 supports this combination with `oap.ports.admin=null` and +# `ui.config.templates.mode=readonly` -- the admin host and its /ui-management +# template store are OAP 11 additions. +# Unlike the 0.11 line above, both of these are real published release tags. +OAP_10_TAG=10.4.0 +OAP_10_REPO=docker.io/apache/skywalking-oap-server +BANYANDB_0_10_TAG=0.10.3 +BANYANDB_0_10_REPO=docker.io/apache/skywalking-banyandb diff --git a/test/e2e/expected/dependency-services-instance-productpage.yml b/test/e2e/expected/dependency-services-instance-productpage.yml deleted file mode 100644 index ed31a89..0000000 --- a/test/e2e/expected/dependency-services-instance-productpage.yml +++ /dev/null @@ -1,42 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -debuggingtrace: null -nodes: -{{- contains .nodes }} -- id: {{ notEmpty .id }} - name: {{ notEmpty .name }} - serviceid: {{ b64enc "e2e::reviews" }}.1 - servicename: e2e::reviews - type: "" - isreal: true -- id: {{ notEmpty .id }} - name: {{ notEmpty .name }} - serviceid: {{ b64enc "e2e::productpage" }}.1 - servicename: e2e::productpage - type: "" - isreal: true -{{- end }} -calls: -{{- contains .calls }} -- source: {{ notEmpty .source }} - sourcecomponents: [] - target: {{ notEmpty .target }} - targetcomponents: [] - id: {{ notEmpty .source }}-{{ notEmpty .target }} - detectpoints: - - CLIENT - - SERVER -{{- end }} diff --git a/test/e2e/expected/dependency-services-productpage.yml b/test/e2e/expected/dependency-services-productpage.yml deleted file mode 100644 index 07b003e..0000000 --- a/test/e2e/expected/dependency-services-productpage.yml +++ /dev/null @@ -1,85 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -debuggingtrace: null -nodes: -{{- contains .nodes }} -- id: {{ b64enc "e2e::reviews"}}.1 - name: e2e::reviews - type: http - isreal: true - layers: - - MESH -- id: {{ b64enc "e2e::productpage"}}.1 - name: e2e::productpage - type: http - isreal: true - layers: - - MESH -- id: {{ b64enc "e2e::details" }}.1 - name: e2e::details - type: http - isreal: true - layers: - - MESH -- id: {{ b64enc "e2e::istio-ingressgateway" }}.1 - name: e2e::istio-ingressgateway - type: http - isreal: true - layers: - - MESH -{{- end }} -calls: -{{- contains .calls }} -- source: {{ b64enc "e2e::istio-ingressgateway"}}.1 - sourcecomponents: - - http - target: {{ b64enc "e2e::productpage"}}.1 - targetcomponents: - {{- contains .targetcomponents }} - - http - - mtls - {{- end }} - id: {{ b64enc "e2e::istio-ingressgateway"}}.1-{{ b64enc "e2e::productpage"}}.1 - detectpoints: - - CLIENT - - SERVER -- source: {{ b64enc "e2e::productpage"}}.1 - sourcecomponents: - - http - target: {{ b64enc "e2e::details"}}.1 - targetcomponents: - {{- contains .targetcomponents }} - - http - - mtls - {{- end }} - id: {{ b64enc "e2e::productpage"}}.1-{{ b64enc "e2e::details"}}.1 - detectpoints: - - CLIENT - - SERVER -- source: {{ b64enc "e2e::productpage" }}.1 - sourcecomponents: - - http - target: {{ b64enc "e2e::reviews"}}.1 - targetcomponents: - {{- contains .targetcomponents }} - - http - - mtls - {{- end }} - id: {{ b64enc "e2e::productpage" }}.1-{{ b64enc "e2e::reviews"}}.1 - detectpoints: - - CLIENT - - SERVER -{{- end }} diff --git a/test/e2e/expected/dependency-services-reviews.yml b/test/e2e/expected/dependency-services-reviews.yml deleted file mode 100644 index 7dd6601..0000000 --- a/test/e2e/expected/dependency-services-reviews.yml +++ /dev/null @@ -1,66 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -debuggingtrace: null -nodes: -{{- contains .nodes }} -- id: {{ b64enc "e2e::reviews"}}.1 - name: e2e::reviews - type: http - isreal: true - layers: - - MESH -- id: {{ b64enc "e2e::ratings"}}.1 - name: e2e::ratings - type: http - isreal: true - layers: - - MESH -- id: {{ b64enc "e2e::productpage" }}.1 - name: e2e::productpage - type: http - isreal: true - layers: - - MESH -{{- end }} -calls: -{{- contains .calls }} -- source: {{ b64enc "e2e::productpage"}}.1 - sourcecomponents: - - http - target: {{ b64enc "e2e::reviews"}}.1 - targetcomponents: - {{- contains .targetcomponents }} - - http - - mtls - {{- end }} - id: {{ b64enc "e2e::productpage"}}.1-{{ b64enc "e2e::reviews"}}.1 - detectpoints: - - CLIENT - - SERVER -- source: {{ b64enc "e2e::reviews" }}.1 - sourcecomponents: - - http - target: {{ b64enc "e2e::ratings"}}.1 - targetcomponents: - {{- contains .targetcomponents }} - - http - - mtls - {{- end }} - id: {{ b64enc "e2e::reviews" }}.1-{{ b64enc "e2e::ratings"}}.1 - detectpoints: - - CLIENT - - SERVER -{{- end }} diff --git a/test/e2e/expected/endpoint.yaml b/test/e2e/expected/horizon-admin-live.yml similarity index 86% rename from test/e2e/expected/endpoint.yaml rename to test/e2e/expected/horizon-admin-live.yml index 2f8ef70..bd35b69 100644 --- a/test/e2e/expected/endpoint.yaml +++ b/test/e2e/expected/horizon-admin-live.yml @@ -13,7 +13,6 @@ # See the License for the specific language governing permissions and # limitations under the License. -{{- contains .}} -- id: {{ b64enc "Your_ApplicationName" }}.1_{{ b64enc "GET:/hello" }} - name: 'GET:/hello' -{{- end}} +adminReachable: true +templatesMode: live +uiManagement: true diff --git a/test/e2e/expected/replicas.yaml b/test/e2e/expected/horizon-admin-readonly.yml similarity index 93% rename from test/e2e/expected/replicas.yaml rename to test/e2e/expected/horizon-admin-readonly.yml index 7c2dadd..1e129e2 100644 --- a/test/e2e/expected/replicas.yaml +++ b/test/e2e/expected/horizon-admin-readonly.yml @@ -13,7 +13,5 @@ # See the License for the specific language governing permissions and # limitations under the License. -{{- contains . }} -- key: "replicas" - value: 3 -{{- end }} \ No newline at end of file +adminReachable: false +templatesMode: readonly diff --git a/test/e2e/expected/service-endpoint-reviews.yml b/test/e2e/expected/horizon-me.yml similarity index 86% rename from test/e2e/expected/service-endpoint-reviews.yml rename to test/e2e/expected/horizon-me.yml index f26305d..6f51e0c 100644 --- a/test/e2e/expected/service-endpoint-reviews.yml +++ b/test/e2e/expected/horizon-me.yml @@ -13,7 +13,6 @@ # See the License for the specific language governing permissions and # limitations under the License. -{{- contains . }} -- id: {{ b64enc "e2e::reviews" }}.1_{{ b64enc "GET:/reviews/0" }} - name: GET:/reviews/0 -{{- end }} +username: admin +roles: + - admin diff --git a/test/e2e/expected/service-endpoint-productpage.yml b/test/e2e/expected/horizon-mesh-services.yml similarity index 86% rename from test/e2e/expected/service-endpoint-productpage.yml rename to test/e2e/expected/horizon-mesh-services.yml index d26fc8f..9a75372 100644 --- a/test/e2e/expected/service-endpoint-productpage.yml +++ b/test/e2e/expected/horizon-mesh-services.yml @@ -13,7 +13,9 @@ # See the License for the specific language governing permissions and # limitations under the License. -{{- contains . }} -- id: {{ b64enc "e2e::productpage" }}.1_{{ b64enc "GET:/productpage" }} - name: GET:/productpage -{{- end }} +layer: MESH +services: + {{- contains .services }} + - e2e::productpage + - e2e::reviews + {{- end }} diff --git a/test/e2e/expected/horizon-oap-info.yml b/test/e2e/expected/horizon-oap-info.yml new file mode 100644 index 0000000..3bf93d1 --- /dev/null +++ b/test/e2e/expected/horizon-oap-info.yml @@ -0,0 +1,17 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +reachable: true +version: {{ notEmpty .version }} diff --git a/test/e2e/expected/horizon-service-cpm.yml b/test/e2e/expected/horizon-service-cpm.yml new file mode 100644 index 0000000..81d08b1 --- /dev/null +++ b/test/e2e/expected/horizon-service-cpm.yml @@ -0,0 +1,18 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +reachable: true +id: cpm +value: {{ notEmpty .value }} diff --git a/test/e2e/expected/horizon-templates-live.yml b/test/e2e/expected/horizon-templates-live.yml new file mode 100644 index 0000000..7d5b4f4 --- /dev/null +++ b/test/e2e/expected/horizon-templates-live.yml @@ -0,0 +1,18 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +mode: live +unreachable: false +hasRows: true diff --git a/test/e2e/expected/horizon-templates-readonly.yml b/test/e2e/expected/horizon-templates-readonly.yml new file mode 100644 index 0000000..abdece2 --- /dev/null +++ b/test/e2e/expected/horizon-templates-readonly.yml @@ -0,0 +1,18 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +mode: readonly +unreachable: false +hasRows: true diff --git a/test/e2e/expected/horizon-zipkin.yml b/test/e2e/expected/horizon-zipkin.yml new file mode 100644 index 0000000..081edca --- /dev/null +++ b/test/e2e/expected/horizon-zipkin.yml @@ -0,0 +1,16 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +zipkinReachable: true diff --git a/test/e2e/expected/metrics-has-value.yml b/test/e2e/expected/metrics-has-value.yml deleted file mode 100644 index cae6688..0000000 --- a/test/e2e/expected/metrics-has-value.yml +++ /dev/null @@ -1,34 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -debuggingtrace: null -type: TIME_SERIES_VALUES -results: - {{- contains .results }} - - metric: - labels: [] - values: - {{- contains .values }} - - id: {{ notEmpty .id }} - value: {{ notEmpty .value }} - owner: null - traceid: null - - id: {{ notEmpty .id }} - value: null - owner: null - traceid: null - {{- end}} - {{- end}} -error: null diff --git a/test/e2e/expected/metrics.yaml b/test/e2e/expected/metrics.yaml deleted file mode 100644 index e26df45..0000000 --- a/test/e2e/expected/metrics.yaml +++ /dev/null @@ -1,25 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -{{- contains . }} -- key: {{ notEmpty .key }} - value: - value: {{ gt .value.value 0 }} - isemptyvalue: false -- key: {{ notEmpty .key }} - value: - value: 0 - isemptyvalue: true -{{- end }} diff --git a/test/e2e/expected/service-apdex.yaml b/test/e2e/expected/service-apdex.yaml deleted file mode 100644 index 76fee01..0000000 --- a/test/e2e/expected/service-apdex.yaml +++ /dev/null @@ -1,25 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -{{- contains . }} -- key: {{ notEmpty .key }} - value: - value: 0 - isemptyvalue: true -- key: {{ notEmpty .key }} - value: - value: {{ gt .value.value 0 }} - isemptyvalue: false -{{- end }} diff --git a/test/e2e/expected/service-instance.yml b/test/e2e/expected/service-instance.yml deleted file mode 100644 index 8210388..0000000 --- a/test/e2e/expected/service-instance.yml +++ /dev/null @@ -1,40 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -{{- contains . }} -- id: {{ notEmpty .id }} - name: {{ notEmpty .name }} - attributes: - {{- contains .attributes }} - - name: pod - value: {{ notEmpty .value }} - - name: namespace - value: default - - name: app - value: {{ notEmpty .value }} - - name: pod-template-hash - value: {{ notEmpty .value }} - - name: security.istio.io/tlsMode - value: istio - - name: service.istio.io/canonical-name - value: {{ notEmpty .value }} - - name: service.istio.io/canonical-revision - value: {{ notEmpty .value }} - - name: version - value: {{ notEmpty .value }} - {{- end }} - language: UNKNOWN - instanceuuid: {{ notEmpty .instanceuuid }} -{{- end }} diff --git a/test/e2e/expected/service.yml b/test/e2e/expected/service.yml deleted file mode 100644 index e29cc7f..0000000 --- a/test/e2e/expected/service.yml +++ /dev/null @@ -1,59 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -{{- contains . }} -- id: {{ b64enc "e2e::istio-ingressgateway" }}.1 - name: e2e::istio-ingressgateway - group: "e2e" - shortname: "istio-ingressgateway" - layers: - - MESH - normal: true -- id: {{ b64enc "e2e::reviews" }}.1 - name: e2e::reviews - group: "e2e" - shortname: "reviews" - layers: - - MESH - normal: true -- id: {{ b64enc "e2e::ratings" }}.1 - name: e2e::ratings - group: "e2e" - shortname: "ratings" - layers: - - MESH - normal: true -- id: {{ b64enc "e2e::productpage" }}.1 - name: e2e::productpage - group: "e2e" - shortname: "productpage" - layers: - - MESH - normal: true -- id: {{ b64enc "e2e::details" }}.1 - name: e2e::details - group: "e2e" - shortname: "details" - layers: - - MESH - normal: true -- id: {{ b64enc "e2e::mongodb" }}.1 - name: e2e::mongodb - group: "e2e" - shortname: "mongodb" - layers: - - MESH - normal: true -{{- end }} diff --git a/test/e2e/expected/swck-adapter-apiservice.yml b/test/e2e/expected/swck-adapter-apiservice.yml new file mode 100644 index 0000000..cf65300 --- /dev/null +++ b/test/e2e/expected/swck-adapter-apiservice.yml @@ -0,0 +1,16 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +available: "True" diff --git a/test/e2e/expected/swck-agent-configmap.yml b/test/e2e/expected/swck-agent-configmap.yml new file mode 100644 index 0000000..9a72b1a --- /dev/null +++ b/test/e2e/expected/swck-agent-configmap.yml @@ -0,0 +1,19 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +configMaps: + {{- contains .configMaps }} + - skywalking-swck-java-agent-configmap + {{- end }} diff --git a/test/e2e/expected/swck-crds.yml b/test/e2e/expected/swck-crds.yml new file mode 100644 index 0000000..3d9768b --- /dev/null +++ b/test/e2e/expected/swck-crds.yml @@ -0,0 +1,20 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +kinds: + - JavaAgent + - OAPServer + - Satellite + - UI diff --git a/test/e2e/expected/swck-demo-service.yaml b/test/e2e/expected/swck-demo-service.yaml deleted file mode 100644 index 798857b..0000000 --- a/test/e2e/expected/swck-demo-service.yaml +++ /dev/null @@ -1,26 +0,0 @@ -# Licensed to Apache Software Foundation (ASF) under one or more contributor -# license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright -# ownership. Apache Software Foundation (ASF) licenses this file to you under -# the Apache License, Version 2.0 (the "License"); you may -# not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. - -{{- contains . }} -- id: {{ b64enc "Your_ApplicationName" }}.1 - name: Your_ApplicationName - group: "" - shortname: "Your_ApplicationName" - layers: - - GENERAL - normal: true -{{- end }} diff --git a/test/e2e/expected/swck-injected.yml b/test/e2e/expected/swck-injected.yml new file mode 100644 index 0000000..4e4df43 --- /dev/null +++ b/test/e2e/expected/swck-injected.yml @@ -0,0 +1,16 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +injected: true diff --git a/test/e2e/expected/hpa-metrics.yaml b/test/e2e/inject-probe.yaml similarity index 57% rename from test/e2e/expected/hpa-metrics.yaml rename to test/e2e/inject-probe.yaml index 264ecf9..611a6f3 100644 --- a/test/e2e/expected/hpa-metrics.yaml +++ b/test/e2e/inject-probe.yaml @@ -13,13 +13,20 @@ # See the License for the specific language governing permissions and # limitations under the License. -{{- contains . }} -- key: {{ notEmpty .key }} - value: - value: 0 - isemptyvalue: true -- key: {{ notEmpty .key }} - value: - value: {{ gt .value.value 15 }} - isemptyvalue: false -{{- end }} \ No newline at end of file +# A pod that exists only to be mutated. The SWCK injector fires on any pod +# CREATE in a namespace labelled `swck-injection: enabled` when the pod carries +# `swck-java-agent-injected: "true"`, so this needs no Java runtime and no OAP -- +# what is under test is the operator chart's webhook wiring, not the agent. +apiVersion: v1 +kind: Pod +metadata: + name: inject-probe + namespace: inject-test + labels: + swck-java-agent-injected: "true" +spec: + restartPolicy: Never + containers: + - name: app + image: busybox:1.30 + command: ["sh", "-c", "sleep 3600"] diff --git a/test/e2e/kind.yaml b/test/e2e/kind.yaml deleted file mode 100644 index 5234f9d..0000000 --- a/test/e2e/kind.yaml +++ /dev/null @@ -1,30 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -# this config file contains all config fields with comments -# NOTE: this is not a particularly useful config file -kind: Cluster -apiVersion: kind.x-k8s.io/v1alpha4 -nodes: - # the control plane node config - - role: control-plane - image: kindest/node:v1.21.14 - # the three workers - - role: worker - image: kindest/node:v1.21.14 - - role: worker - image: kindest/node:v1.21.14 - - role: worker - image: kindest/node:v1.21.14 diff --git a/test/e2e/script/horizon.sh b/test/e2e/script/horizon.sh new file mode 100755 index 0000000..a599cf1 --- /dev/null +++ b/test/e2e/script/horizon.sh @@ -0,0 +1,53 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Query Horizon UI's BFF the way an operator does: log in, then call an API with +# the session cookie. Every request in the e2e goes through here, so the whole +# suite exercises horizon -> oap -> storage rather than talking to OAP directly. +# +# horizon.sh get +# horizon.sh post +# +# Credentials come from HORIZON_USERNAME / HORIZON_PASSWORD and default to the +# admin/admin pair that test/e2e/values.yaml seeds. There is no built-in login: +# Horizon ships `auth.local.users` empty, so without that values file every call +# here fails at the login step -- which is the intended signal, not a flake. + +set -eo pipefail + +BASE_URL=$1 +VERB=$2 +API_PATH=$3 +BODY=${4:-} + +USERNAME=${HORIZON_USERNAME:-admin} +PASSWORD=${HORIZON_PASSWORD:-admin} + +JAR=$(mktemp) +trap 'rm -f "$JAR"' EXIT + +curl -sS --fail-with-body -c "$JAR" -X POST "${BASE_URL}/api/auth/login" \ + -H 'Content-Type: application/json' \ + -d "{\"username\":\"${USERNAME}\",\"password\":\"${PASSWORD}\"}" >/dev/null + +grep -q 'horizon_sid' "$JAR" || { echo "login did not set a session cookie" >&2; exit 1; } + +if [ "$VERB" = "post" ]; then + curl -sS --fail-with-body -b "$JAR" -X POST "${BASE_URL}${API_PATH}" \ + -H 'Content-Type: application/json' -d "$BODY" +else + curl -sS --fail-with-body -b "$JAR" "${BASE_URL}${API_PATH}" +fi diff --git a/test/e2e/setup-e2e-shell/install-swctl.sh b/test/e2e/setup-e2e-shell/install-swctl.sh deleted file mode 100644 index f9bcf34..0000000 --- a/test/e2e/setup-e2e-shell/install-swctl.sh +++ /dev/null @@ -1,39 +0,0 @@ -#!/usr/bin/env bash - -# ---------------------------------------------------------------------------- -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. -# ---------------------------------------------------------------------------- - -BASE_DIR=$1 -BIN_DIR=$2 - -install_swctl() { - mkdir -p $BASE_DIR/swctl && cd $BASE_DIR/swctl - curl -kLo skywalking-cli.tar.gz https://github.com/apache/skywalking-cli/archive/${SW_CTL_COMMIT}.tar.gz - tar -zxf skywalking-cli.tar.gz --strip=1 - VERSION=${SW_CTL_COMMIT} make install DESTDIR=$BIN_DIR -} - -if ! command -v swctl &> /dev/null; then - echo "swctl is not installed" - install_swctl -elif ! swctl --version | grep -q "${SW_CTL_COMMIT}"; then - # Check if the installed version is correct - echo "swctl is already installed, but version is not ${SW_CTL_COMMIT}, will re-install it" - install_swctl -fi diff --git a/test/e2e/swck/deploy-elasticsearch.yaml b/test/e2e/swck/deploy-elasticsearch.yaml deleted file mode 100644 index bc3180c..0000000 --- a/test/e2e/swck/deploy-elasticsearch.yaml +++ /dev/null @@ -1,53 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -apiVersion: apps/v1 -kind: StatefulSet -metadata: - name: es -spec: - serviceName: es - replicas: 1 - selector: - matchLabels: - app: es - template: - metadata: - labels: - app: es - spec: - containers: - - name: es - image: docker.elastic.co/elasticsearch/elasticsearch:8.18.8 - env: - - name: discovery.type - value: single-node - - name: xpack.security.enabled - value: "false" - ports: - - containerPort: 9200 ---- -apiVersion: v1 -kind: Service -metadata: - name: es-out -spec: - type: ClusterIP - selector: - app: es - ports: - - port: 9200 - targetPort: 9200 - protocol: TCP diff --git a/test/e2e/swck/oap-agent-adapter-hpa.yaml b/test/e2e/swck/oap-agent-adapter-hpa.yaml deleted file mode 100644 index 4028cee..0000000 --- a/test/e2e/swck/oap-agent-adapter-hpa.yaml +++ /dev/null @@ -1,127 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -setup: - env: kind - file: ../kind.yaml - init-system-environment: ../env - kind: - expose-ports: - - namespace: skywalking-system - resource: service/demo - port: 8085 - - namespace: skywalking-system - resource: service/skywalking-system-oap - port: 12800 - steps: - - name: Install yq - command: bash test/e2e/setup-e2e-shell/install.sh yq - - name: Install swctl - command: bash test/e2e/setup-e2e-shell/install.sh swctl - - name: Install kubectl - command: bash test/e2e/setup-e2e-shell/install.sh kubectl - - name: Install helm - command: bash test/e2e/setup-e2e-shell/install.sh helm - - name: Install cert-manager - command: | - kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.9.1/cert-manager.yaml - wait: - - namespace: cert-manager - resource: pod - for: condition=Ready - - name: Install adapter - command: | - helm install adapter chart/adapter --namespace=skywalking-custom-metrics-system \ - --create-namespace \ - --set fullnameOverride=skywalking-swck \ - --set oap.service.name=skywalking-system-oap \ - --set oap.service.namespace=skywalking-system \ - --set oap.service.port=12800 - wait: - - namespace: skywalking-custom-metrics-system - resource: deployment/skywalking-swck-adapter - for: condition=Available - - name: Install operator - command: | - helm dep up chart/operator - helm install operator chart/operator --namespace=skywalking-swck-system \ - --create-namespace \ - --set fullnameOverride=skywalking-swck \ - --set image.repository=$SWCK_OPERATOR_REPO \ - --set image.tag=$SWCK_OPERATOR_TAG - wait: - - namespace: skywalking-swck-system - resource: deployment/skywalking-swck-operator - for: condition=Available - - name: Setup BanyanDB - command: | - export BANYANDB_VERSION=$(yq e '.dependencies[] | select(.name == "skywalking-banyandb-helm") | .version' chart/skywalking/Chart.yaml) - helm install banyandb oci://registry-1.docker.io/apache/skywalking-banyandb-helm \ - --version ${BANYANDB_VERSION} \ - --set fullnameOverride=banyandb \ - --set image.repository=$BANYANDB_REPO \ - --set image.tag=$BANYANDB_TAG \ - --set standalone.enabled=true \ - --set cluster.enabled=false - wait: - - namespace: default - resource: pod - for: condition=Ready - - name: Setup oapserver - command: | - kubectl create namespace skywalking-system - sed "s#OAP_IMAGE_PLACEHOLDER#${OAP_REPO}:${OAP_TAG}#g;s#UI_IMAGE_PLACEHOLDER#${UI_REPO}:${UI_TAG}#g;s#PLACEHOLDER#latest#g" \ - test/e2e/swck/skywalking-components.yaml | grep -B30 "\-\-\-" | kubectl apply -f - - wait: - - namespace: skywalking-system - resource: OAPServer/skywalking-system - for: condition=Available - - name: Setup java agent demo - command: | - kubectl label namespace skywalking-system swck-injection=enabled - curl https://raw.githubusercontent.com/apache/skywalking-swck/v0.10.0/test/e2e/demo.yaml | sed 's/oap-service/skywalking-system-oap.skywalking-system/' | kubectl apply -f - - wait: - - namespace: skywalking-system - resource: deployment/demo - for: condition=Available - - name: Setup the hpa of java agent demo - command: | - curl https://raw.githubusercontent.com/apache/skywalking-swck/v0.10.0/test/e2e/hpa-demo.yaml | kubectl apply -f - - timeout: 20m - -cleanup: - # always never success failure - on: always - -trigger: - action: http - interval: 30s - times: 30 - url: http://${service_demo_host}:${service_demo_8085}/hello - method: GET - -verify: - # verify with retry strategy - retry: - # max retry count - count: 30 - # the interval between two attempts, e.g. 10s, 1m. - interval: 10s - cases: - - query: swctl --display yaml --base-url=http://${service_skywalking_system_oap_host}:${service_skywalking_system_oap_12800}/graphql metrics linear --name service_cpm --service-name Your_ApplicationName | yq e 'to_entries' - - expected: ../expected/hpa-metrics.yaml - # the trigger will do 30 times, as the hpa's value is set as 5, so the deployment's replicas will be 3(maxReplicas). - - query: 'kubectl get deployment demo -n skywalking-system -o jsonpath=''{.status.replicas}'' | yq e ''{"replicas": .}'' - | yq e ''to_entries'' -' - expected: ../expected/replicas.yaml diff --git a/test/e2e/swck/oap-agent-banyandb.yaml b/test/e2e/swck/oap-agent-banyandb.yaml deleted file mode 100644 index 7d42c67..0000000 --- a/test/e2e/swck/oap-agent-banyandb.yaml +++ /dev/null @@ -1,118 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -setup: - env: kind - file: ../kind.yaml - init-system-environment: ../env - kind: - expose-ports: - - namespace: skywalking-system - resource: service/demo - port: 8085 - - namespace: skywalking-system - resource: service/skywalking-system-oap - port: 12800 - steps: - - name: Install yq - command: bash test/e2e/setup-e2e-shell/install.sh yq - - name: Install swctl - command: bash test/e2e/setup-e2e-shell/install.sh swctl - - name: Install kubectl - command: bash test/e2e/setup-e2e-shell/install.sh kubectl - - name: Install helm - command: bash test/e2e/setup-e2e-shell/install.sh helm - - name: Install cert-manager - command: | - kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.9.1/cert-manager.yaml - wait: - - namespace: cert-manager - resource: pod - for: condition=Ready - - name: Install operator - command: | - helm dep up chart/operator - helm install operator chart/operator --namespace=skywalking-swck-system \ - --create-namespace \ - --set fullnameOverride=skywalking-swck \ - --set image.repository=$SWCK_OPERATOR_REPO \ - --set image.tag=$SWCK_OPERATOR_TAG - wait: - - namespace: skywalking-swck-system - resource: deployment/skywalking-swck-operator - for: condition=Available - - name: Setup BanyanDB - command: | - export BANYANDB_VERSION=$(yq e '.dependencies[] | select(.name == "skywalking-banyandb-helm") | .version' chart/skywalking/Chart.yaml) - helm install banyandb oci://registry-1.docker.io/apache/skywalking-banyandb-helm \ - --version ${BANYANDB_VERSION} \ - --set fullnameOverride=banyandb \ - --set image.repository=$BANYANDB_REPO \ - --set image.tag=$BANYANDB_TAG \ - --set standalone.enabled=true \ - --set cluster.enabled=false - wait: - - namespace: default - resource: pod - for: condition=Ready - - name: setup oapserver and ui - command: | - kubectl create namespace skywalking-system - sed "s#OAP_IMAGE_PLACEHOLDER#${OAP_REPO}:${OAP_TAG}#g;s#UI_IMAGE_PLACEHOLDER#${UI_REPO}:${UI_TAG}#g;s#PLACEHOLDER#latest#g" \ - test/e2e/swck/skywalking-components.yaml | kubectl apply -f - - wait: - - namespace: skywalking-system - resource: OAPServer/skywalking-system - for: condition=Available - - namespace: skywalking-system - resource: UI/skywalking-system - for: condition=Available - - name: Setup java agent demo - command: | - kubectl label namespace skywalking-system swck-injection=enabled - curl https://raw.githubusercontent.com/apache/skywalking-swck/v0.10.0/test/e2e/demo.yaml | sed 's/oap-service/skywalking-system-oap.skywalking-system/' | kubectl apply -f - - wait: - - namespace: skywalking-system - resource: deployment/demo - for: condition=Available - timeout: 20m - -cleanup: - # always never success failure - on: always - -trigger: - action: http - interval: 10s - times: 0 - url: http://${service_demo_host}:${service_demo_8085}/hello - method: GET - -verify: - # verify with retry strategy - retry: - # max retry count - count: 30 - # the interval between two attempts, e.g. 10s, 1m. - interval: 10s - cases: - - query: swctl --display yaml --base-url=http://${service_skywalking_system_oap_host}:${service_skywalking_system_oap_12800}/graphql service ls - expected: ../expected/swck-demo-service.yaml - - query: swctl --display yaml --base-url=http://${service_skywalking_system_oap_host}:${service_skywalking_system_oap_12800}/graphql metrics linear --name service_cpm --service-name Your_ApplicationName | yq e 'to_entries' - - expected: ../expected/metrics.yaml - - query: swctl --display yaml --base-url=http://${service_skywalking_system_oap_host}:${service_skywalking_system_oap_12800}/graphql endpoint list --keyword=hello --service-name Your_ApplicationName - expected: ../expected/endpoint.yaml - - query: swctl --display yaml --base-url=http://${service_skywalking_system_oap_host}:${service_skywalking_system_oap_12800}/graphql metrics linear --name endpoint_cpm --endpoint-name GET:/hello --service-name Your_ApplicationName | yq e 'to_entries' - - expected: ../expected/metrics.yaml diff --git a/test/e2e/swck/oap-ui-agent-elasticsearch.yaml b/test/e2e/swck/oap-ui-agent-elasticsearch.yaml deleted file mode 100644 index 6c5e5fb..0000000 --- a/test/e2e/swck/oap-ui-agent-elasticsearch.yaml +++ /dev/null @@ -1,108 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -setup: - env: kind - file: ../kind.yaml - init-system-environment: ../env - kind: - expose-ports: - - namespace: skywalking-system - resource: service/demo - port: 8085 - - namespace: skywalking-system - resource: service/skywalking-system-oap - port: 12800 - steps: - - name: Install yq - command: bash test/e2e/setup-e2e-shell/install.sh yq - - name: Install swctl - command: bash test/e2e/setup-e2e-shell/install.sh swctl - - name: Install kubectl - command: bash test/e2e/setup-e2e-shell/install.sh kubectl - - name: Install helm - command: bash test/e2e/setup-e2e-shell/install.sh helm - - name: Install cert-manager - command: | - kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.9.1/cert-manager.yaml - wait: - - namespace: cert-manager - resource: pod - for: condition=Ready - - name: Install operator - command: | - helm dep up chart/operator - helm install operator chart/operator --namespace=skywalking-swck-system \ - --create-namespace \ - --set fullnameOverride=skywalking-swck \ - --set image.repository=$SWCK_OPERATOR_REPO \ - --set image.tag=$SWCK_OPERATOR_TAG - wait: - - namespace: skywalking-swck-system - resource: deployment/skywalking-swck-operator - for: condition=Available - - name: setup elasticsearch - command: | - kubectl apply -f test/e2e/swck/deploy-elasticsearch.yaml - kubectl rollout status --watch --timeout=120s statefulset/es - - name: setup oapserver and ui - command: | - kubectl create namespace skywalking-system - sed "s#OAP_IMAGE_PLACEHOLDER#${OAP_REPO}:${OAP_TAG}#g;s#UI_IMAGE_PLACEHOLDER#${UI_REPO}:${UI_TAG}#g;s#PLACEHOLDER#latest#g" \ - test/e2e/swck/skywalking-components-elasticsearch.yaml | kubectl apply -f - - wait: - - namespace: skywalking-system - resource: OAPServer/skywalking-system - for: condition=Available - - namespace: skywalking-system - resource: UI/skywalking-system - for: condition=Available - - name: Setup java agent demo - command: | - kubectl label namespace skywalking-system swck-injection=enabled - curl https://raw.githubusercontent.com/apache/skywalking-swck/v0.10.0/test/e2e/demo.yaml | sed 's/oap-service/skywalking-system-oap.skywalking-system/' | kubectl apply -f - - wait: - - namespace: skywalking-system - resource: deployment/demo - for: condition=Available - timeout: 20m - -cleanup: - # always never success failure - on: always - -trigger: - action: http - interval: 10s - times: 0 - url: http://${service_demo_host}:${service_demo_8085}/hello - method: GET - -verify: - # verify with retry strategy - retry: - # max retry count - count: 30 - # the interval between two attempts, e.g. 10s, 1m. - interval: 10s - cases: - - query: swctl --display yaml --base-url=http://${service_skywalking_system_oap_host}:${service_skywalking_system_oap_12800}/graphql service ls - expected: ../expected/swck-demo-service.yaml - - query: swctl --display yaml --base-url=http://${service_skywalking_system_oap_host}:${service_skywalking_system_oap_12800}/graphql metrics linear --name service_cpm --service-name Your_ApplicationName | yq e 'to_entries' - - expected: ../expected/metrics.yaml - - query: swctl --display yaml --base-url=http://${service_skywalking_system_oap_host}:${service_skywalking_system_oap_12800}/graphql endpoint list --keyword=hello --service-name Your_ApplicationName - expected: ../expected/endpoint.yaml - - query: swctl --display yaml --base-url=http://${service_skywalking_system_oap_host}:${service_skywalking_system_oap_12800}/graphql metrics linear --name endpoint_cpm --endpoint-name GET:/hello --service-name Your_ApplicationName | yq e 'to_entries' - - expected: ../expected/metrics.yaml diff --git a/test/e2e/swck/oap-ui-agent-oapserverconfig.yaml b/test/e2e/swck/oap-ui-agent-oapserverconfig.yaml deleted file mode 100644 index 0e1f4c1..0000000 --- a/test/e2e/swck/oap-ui-agent-oapserverconfig.yaml +++ /dev/null @@ -1,148 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -setup: - env: kind - file: ../kind.yaml - init-system-environment: ../env - kind: - expose-ports: - - namespace: skywalking-system - resource: service/demo - port: 8085 - - namespace: skywalking-system - resource: service/skywalking-system-oap - port: 12800 - - namespace: skywalking-system - resource: service/songs - port: 80 - steps: - - name: Install yq - command: bash test/e2e/setup-e2e-shell/install.sh yq - - name: Install swctl - command: bash test/e2e/setup-e2e-shell/install.sh swctl - - name: Install kubectl - command: bash test/e2e/setup-e2e-shell/install.sh kubectl - - name: Install helm - command: bash test/e2e/setup-e2e-shell/install.sh helm - - name: Install cert-manager - command: | - kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.9.1/cert-manager.yaml - wait: - - namespace: cert-manager - resource: pod - for: condition=Ready - - name: Install operator - command: | - helm dep up chart/operator - helm install operator chart/operator --namespace=skywalking-swck-system \ - --create-namespace \ - --set fullnameOverride=skywalking-swck \ - --set image.repository=$SWCK_OPERATOR_REPO \ - --set image.tag=$SWCK_OPERATOR_TAG - wait: - - namespace: skywalking-swck-system - resource: deployment/skywalking-swck-operator - for: condition=Available - - name: Setup BanyanDB - command: | - export BANYANDB_VERSION=$(yq e '.dependencies[] | select(.name == "skywalking-banyandb-helm") | .version' chart/skywalking/Chart.yaml) - helm install banyandb oci://registry-1.docker.io/apache/skywalking-banyandb-helm \ - --version ${BANYANDB_VERSION} \ - --set fullnameOverride=banyandb \ - --set image.repository=$BANYANDB_REPO \ - --set image.tag=$BANYANDB_TAG \ - --set standalone.enabled=true \ - --set cluster.enabled=false - wait: - - namespace: default - resource: pod - for: condition=Ready - - name: setup oapserver and ui - command: | - kubectl create namespace skywalking-system - sed "s#OAP_IMAGE_PLACEHOLDER#${OAP_REPO}:${OAP_TAG}#g;s#UI_IMAGE_PLACEHOLDER#${UI_REPO}:${UI_TAG}#g;s#PLACEHOLDER#latest#g" \ - test/e2e/swck/skywalking-components.yaml | kubectl apply -f - - wait: - - namespace: skywalking-system - resource: OAPServer/skywalking-system - for: condition=Available - - namespace: skywalking-system - resource: UI/skywalking-system - for: condition=Available - - name: setup java agent demo(test for dynamic configuration) - command: | - kubectl label namespace skywalking-system swck-injection=enabled - curl https://raw.githubusercontent.com/apache/skywalking-swck/v0.10.0/test/e2e/demo.yaml | sed 's/oap-service/skywalking-system-oap.skywalking-system/' | kubectl create -f - - wait: - - namespace: skywalking-system - resource: deployment/demo - for: condition=Available - - name: setup oapserverconfig(static configuration) and wait for pod restart - command: | - oap_podname=$(kubectl get pod -lapp=oap -n skywalking-system -o jsonpath='{.items[*].metadata.name}') - curl https://raw.githubusercontent.com/apache/skywalking-swck/v0.10.0/test/e2e/oapserverconfig-demo.yaml | \ - sed 's/version: 9.5.0/version: latest/' | \ - sed '/env:/a\ - name: SW_STORAGE\n value: banyandb\n - name: SW_STORAGE_BANYANDB_TARGETS\n value: banyandb-grpc.default:17912' | \ - kubectl apply -f - - while kubectl get pod $oap_podname -n skywalking-system > /dev/null 2>&1; \ - do \ - sleep 0.1; \ - done - - name: setup java agent demo(test for static configuration) - command: | - curl https://raw.githubusercontent.com/apache/skywalking-swck/v0.10.0/test/e2e/resource.yaml | sed 's/oap-service/skywalking-system-oap.skywalking-system/' | kubectl create -f - - wait: - - namespace: skywalking-system - resource: deployment/songs-deployment - for: condition=Available - - name: setup oapserverdynamicconfig(dynamic configuration) - command: | - curl https://raw.githubusercontent.com/apache/skywalking-swck/v0.10.0/test/e2e/oapserverdynamicconfig-demo.yaml | sed 's/default: 0.1/default: 500/' | kubectl apply -f - - timeout: 20m - -cleanup: - # always never success failure - on: always - -trigger: - action: http - interval: 5s - times: 0 - url: http://${service_demo_host}:${service_demo_8085}/hello - method: GET - -verify: - # verify with retry strategy - retry: - # max retry count - count: 30 - # the interval between two attempts, e.g. 10s, 1m. - interval: 10s - cases: - - query: swctl --display yaml --base-url=http://${service_skywalking_system_oap_host}:${service_skywalking_system_oap_12800}/graphql service ls - expected: ../expected/swck-demo-service.yaml - - query: swctl --display yaml --base-url=http://${service_skywalking_system_oap_host}:${service_skywalking_system_oap_12800}/graphql metrics linear --name service_cpm --service-name Your_ApplicationName | yq e 'to_entries' - - expected: ../expected/metrics.yaml - - query: swctl --display yaml --base-url=http://${service_skywalking_system_oap_host}:${service_skywalking_system_oap_12800}/graphql endpoint list --keyword=hello --service-name Your_ApplicationName - expected: ../expected/endpoint.yaml - - query: swctl --display yaml --base-url=http://${service_skywalking_system_oap_host}:${service_skywalking_system_oap_12800}/graphql metrics linear --name endpoint_cpm --endpoint-name GET:/hello --service-name Your_ApplicationName | yq e 'to_entries' - - expected: ../expected/metrics.yaml - # test oapsever's static configuration - - query: swctl --display yaml --base-url=http://${service_skywalking_system_oap_host}:${service_skywalking_system_oap_12800}/graphql metrics linear --name=log_count_info --instance-name=songs --service-name=agent::songs | yq e 'to_entries' - - expected: ../expected/metrics.yaml - # test oapsever's dynamic configuration - - query: swctl --display yaml --base-url=http://${service_skywalking_system_oap_host}:${service_skywalking_system_oap_12800}/graphql metrics linear --name=service_apdex --service-name=Your_ApplicationName | yq e 'to_entries' - - expected: ../expected/service-apdex.yaml diff --git a/test/e2e/swck/oap-ui-agent-satellite.yaml b/test/e2e/swck/oap-ui-agent-satellite.yaml deleted file mode 100644 index 88b5a53..0000000 --- a/test/e2e/swck/oap-ui-agent-satellite.yaml +++ /dev/null @@ -1,121 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -setup: - env: kind - file: ../kind.yaml - init-system-environment: ../env - kind: - expose-ports: - - namespace: skywalking-system - resource: service/demo - port: 8085 - - namespace: skywalking-system - resource: service/skywalking-system-oap - port: 12800 - steps: - - name: Install yq - command: bash test/e2e/setup-e2e-shell/install.sh yq - - name: Install swctl - command: bash test/e2e/setup-e2e-shell/install.sh swctl - - name: Install kubectl - command: bash test/e2e/setup-e2e-shell/install.sh kubectl - - name: Install helm - command: bash test/e2e/setup-e2e-shell/install.sh helm - - name: Install cert-manager - command: | - kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.9.1/cert-manager.yaml - wait: - - namespace: cert-manager - resource: pod - for: condition=Ready - - name: Install operator - command: | - helm dep up chart/operator - helm install operator chart/operator --namespace=skywalking-swck-system \ - --create-namespace \ - --set fullnameOverride=skywalking-swck \ - --set image.repository=$SWCK_OPERATOR_REPO \ - --set image.tag=$SWCK_OPERATOR_TAG - wait: - - namespace: skywalking-swck-system - resource: deployment/skywalking-swck-operator - for: condition=Available - - name: Setup BanyanDB - command: | - export BANYANDB_VERSION=$(yq e '.dependencies[] | select(.name == "skywalking-banyandb-helm") | .version' chart/skywalking/Chart.yaml) - helm install banyandb oci://registry-1.docker.io/apache/skywalking-banyandb-helm \ - --version ${BANYANDB_VERSION} \ - --set fullnameOverride=banyandb \ - --set image.repository=$BANYANDB_REPO \ - --set image.tag=$BANYANDB_TAG \ - --set standalone.enabled=true \ - --set cluster.enabled=false - wait: - - namespace: default - resource: pod - for: condition=Ready - - name: setup oapserver and ui - command: | - kubectl create namespace skywalking-system - sed "s#OAP_IMAGE_PLACEHOLDER#${OAP_REPO}:${OAP_TAG}#g;s#UI_IMAGE_PLACEHOLDER#${UI_REPO}:${UI_TAG}#g;s#SATELLITE_IMAGE_PLACEHOLDER#${SATELLITE_REPO}:${SATELLITE_TAG}#g;s#PLACEHOLDER#latest#g" \ - test/e2e/swck/skywalking-components-satellite.yaml | kubectl apply -f - - wait: - - namespace: skywalking-system - resource: OAPServer/skywalking-system - for: condition=Available - - namespace: skywalking-system - resource: UI/skywalking-system - for: condition=Available - - namespace: skywalking-system - resource: Satellite/skywalking-system - for: condition=Available - - name: Setup java agent demo - command: | - kubectl label namespace skywalking-system swck-injection=enabled - curl https://raw.githubusercontent.com/apache/skywalking-swck/v0.10.0/test/e2e/demo.yaml | sed 's/oap-service/skywalking-system-satellite.skywalking-system/' | kubectl apply -f - - wait: - - namespace: skywalking-system - resource: deployment/demo - for: condition=Available - timeout: 20m - -cleanup: - # always never success failure - on: always - -trigger: - action: http - interval: 30s - times: 30 - url: http://${service_demo_host}:${service_demo_8085}/hello - method: GET - -verify: - # verify with retry strategy - retry: - # max retry count - count: 30 - # the interval between two attempts, e.g. 10s, 1m. - interval: 10s - cases: - - query: swctl --display yaml --base-url=http://${service_skywalking_system_oap_host}:${service_skywalking_system_oap_12800}/graphql service ls - expected: ../expected/swck-demo-service.yaml - - query: swctl --display yaml --base-url=http://${service_skywalking_system_oap_host}:${service_skywalking_system_oap_12800}/graphql metrics linear --name service_cpm --service-name Your_ApplicationName | yq e 'to_entries' - - expected: ../expected/metrics.yaml - - query: swctl --display yaml --base-url=http://${service_skywalking_system_oap_host}:${service_skywalking_system_oap_12800}/graphql endpoint list --keyword=hello --service-name Your_ApplicationName - expected: ../expected/endpoint.yaml - - query: swctl --display yaml --base-url=http://${service_skywalking_system_oap_host}:${service_skywalking_system_oap_12800}/graphql metrics linear --name endpoint_cpm --endpoint-name GET:/hello --service-name Your_ApplicationName | yq e 'to_entries' - - expected: ../expected/metrics.yaml diff --git a/test/e2e/swck/skywalking-components-elasticsearch.yaml b/test/e2e/swck/skywalking-components-elasticsearch.yaml deleted file mode 100644 index 9df8ef3..0000000 --- a/test/e2e/swck/skywalking-components-elasticsearch.yaml +++ /dev/null @@ -1,49 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -# OAPServer with Elasticsearch storage -apiVersion: operator.skywalking.apache.org/v1alpha1 -kind: OAPServer -metadata: - name: skywalking-system - namespace: skywalking-system -spec: - version: PLACEHOLDER - instances: 1 - image: OAP_IMAGE_PLACEHOLDER - service: - template: - type: ClusterIP - config: - - name: SW_STORAGE - value: elasticsearch - - name: SW_STORAGE_ES_CLUSTER_NODES - value: es-out.default:9200 ---- -apiVersion: operator.skywalking.apache.org/v1alpha1 -kind: UI -metadata: - name: skywalking-system - namespace: skywalking-system -spec: - version: PLACEHOLDER - instances: 1 - image: UI_IMAGE_PLACEHOLDER - OAPServerAddress: http://skywalking-system-oap.skywalking-system:12800 - service: - template: - type: ClusterIP - ingress: - host: demo.ui.skywalking diff --git a/test/e2e/swck/skywalking-components-satellite.yaml b/test/e2e/swck/skywalking-components-satellite.yaml deleted file mode 100644 index 7b8f4c6..0000000 --- a/test/e2e/swck/skywalking-components-satellite.yaml +++ /dev/null @@ -1,67 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -# OAPServer with BanyanDB storage and satellite meter config -apiVersion: operator.skywalking.apache.org/v1alpha1 -kind: OAPServer -metadata: - name: skywalking-system - namespace: skywalking-system -spec: - version: PLACEHOLDER - instances: 1 - image: OAP_IMAGE_PLACEHOLDER - config: - - name: SW_STORAGE - value: banyandb - - name: SW_STORAGE_BANYANDB_TARGETS - value: banyandb-grpc.default:17912 - - name: SW_METER_ANALYZER_ACTIVE_FILES - value: satellite - service: - template: - type: ClusterIP ---- -apiVersion: operator.skywalking.apache.org/v1alpha1 -kind: Satellite -metadata: - name: skywalking-system - namespace: skywalking-system -spec: - version: PLACEHOLDER - instances: 1 - image: SATELLITE_IMAGE_PLACEHOLDER - config: - - name: SATELLITE_TELEMETRY_EXPORT_TYPE - value: metrics_service - service: - template: - type: ClusterIP ---- -apiVersion: operator.skywalking.apache.org/v1alpha1 -kind: UI -metadata: - name: skywalking-system - namespace: skywalking-system -spec: - version: PLACEHOLDER - instances: 1 - image: UI_IMAGE_PLACEHOLDER - OAPServerAddress: http://skywalking-system-oap.skywalking-system:12800 - service: - template: - type: ClusterIP - ingress: - host: demo.ui.skywalking diff --git a/test/e2e/swck/skywalking-components.yaml b/test/e2e/swck/skywalking-components.yaml deleted file mode 100644 index 63ad62f..0000000 --- a/test/e2e/swck/skywalking-components.yaml +++ /dev/null @@ -1,49 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -# OAPServer with BanyanDB storage -apiVersion: operator.skywalking.apache.org/v1alpha1 -kind: OAPServer -metadata: - name: skywalking-system - namespace: skywalking-system -spec: - version: PLACEHOLDER - instances: 1 - image: OAP_IMAGE_PLACEHOLDER - service: - template: - type: ClusterIP - config: - - name: SW_STORAGE - value: banyandb - - name: SW_STORAGE_BANYANDB_TARGETS - value: banyandb-grpc.default:17912 ---- -apiVersion: operator.skywalking.apache.org/v1alpha1 -kind: UI -metadata: - name: skywalking-system - namespace: skywalking-system -spec: - version: PLACEHOLDER - instances: 1 - image: UI_IMAGE_PLACEHOLDER - OAPServerAddress: http://skywalking-system-oap.skywalking-system:12800 - service: - template: - type: ClusterIP - ingress: - host: demo.ui.skywalking From aa16968ff6c8ede192fc69aa7c8e1c8cc4c49ff8 Mon Sep 17 00:00:00 2001 From: Wu Sheng Date: Sat, 29 Aug 2026 23:16:23 +0800 Subject: [PATCH 2/4] Assert what the injector actually does, and let every cell report CI proved the important half: injection happened, so the operator chart's cert-manager Certificate, CA injection, webhook Service routing and RBAC are all correct. The third assertion was wrong by design -- it expected the pod to mount the chart's java-agent ConfigMap, and the actual output was 'configMaps: []'. The injector adds an emptyDir, not a ConfigMap: an initContainer copies the agent into a shared volume that the app container mounts, and the agent config is overlaid by the operator through annotations before admission. The chart's java-agent ConfigMap is read by the operator, never mounted into the workload. So the volume assertion now checks the emptyDir and the app container's mount, and the ConfigMap is checked where it actually lives -- by name, in the operator namespace, which is what exercises the 63-char truncation helper. Also set fail-fast: false. The operator cell failing cancelled the other six before any of them reported, which is the opposite of what a matrix is for on a change this size. --- .github/workflows/e2e.ci.yaml | 1 + test/e2e/e2e-operator-chart.yaml | 17 +++++++++++++---- test/e2e/expected/swck-agent-configmap.yml | 6 ++---- test/e2e/expected/swck-injected-volume.yml | 17 +++++++++++++++++ 4 files changed, 33 insertions(+), 8 deletions(-) create mode 100644 test/e2e/expected/swck-injected-volume.yml diff --git a/.github/workflows/e2e.ci.yaml b/.github/workflows/e2e.ci.yaml index a2dbb23..58d1f31 100644 --- a/.github/workflows/e2e.ci.yaml +++ b/.github/workflows/e2e.ci.yaml @@ -33,6 +33,7 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 strategy: + fail-fast: false matrix: test: - name: Horizon + OAP 11 + Elasticsearch diff --git a/test/e2e/e2e-operator-chart.yaml b/test/e2e/e2e-operator-chart.yaml index 0cc8822..7d4641d 100644 --- a/test/e2e/e2e-operator-chart.yaml +++ b/test/e2e/e2e-operator-chart.yaml @@ -86,9 +86,18 @@ verify: kubectl -n inject-test get pod inject-probe -o json | yq -p json -o yaml '{"injected": ((.spec.initContainers // []) | length > 0)}' expected: expected/swck-injected.yml - # The injected pod mounts the ConfigMap THIS chart renders. The name goes - # through the 63-char truncation helper, so a mismatch here is a chart bug - # the operator's own tests can never see. + # The injector's shared volume is present. It copies the agent into an + # emptyDir that the app container then mounts -- no ConfigMap is mounted + # into the workload, the agent config is overlaid by the operator through + # annotations before admission. - query: | - kubectl -n inject-test get pod inject-probe -o json | yq -p json -o yaml '{"configMaps": ([.spec.volumes[] | select(.configMap != null) | .configMap.name] | sort)}' + kubectl -n inject-test get pod inject-probe -o json | yq -p json -o yaml '{"sharedVolume": ([.spec.volumes[] | select(.emptyDir != null)] | length > 0), "appMounted": ([.spec.containers[] | select(.name == "app") | (.volumeMounts // []) | length > 0] | .[0])}' + expected: expected/swck-injected-volume.yml + + # The java-agent ConfigMap THIS chart renders, checked where it actually + # lives -- the operator reads it, it is not mounted into injected pods. Its + # name goes through the 63-char truncation helper, so a mismatch is a chart + # bug the operator's own kustomize-based tests can never see. + - query: | + kubectl -n skywalking-swck-system get configmap skywalking-swck-java-agent-configmap -o json | yq -p json -o yaml '{"name": .metadata.name, "hasAgentConfig": (.data | has("agent.config"))}' expected: expected/swck-agent-configmap.yml diff --git a/test/e2e/expected/swck-agent-configmap.yml b/test/e2e/expected/swck-agent-configmap.yml index 9a72b1a..c7e363b 100644 --- a/test/e2e/expected/swck-agent-configmap.yml +++ b/test/e2e/expected/swck-agent-configmap.yml @@ -13,7 +13,5 @@ # See the License for the specific language governing permissions and # limitations under the License. -configMaps: - {{- contains .configMaps }} - - skywalking-swck-java-agent-configmap - {{- end }} +name: skywalking-swck-java-agent-configmap +hasAgentConfig: true diff --git a/test/e2e/expected/swck-injected-volume.yml b/test/e2e/expected/swck-injected-volume.yml new file mode 100644 index 0000000..5b8b9c3 --- /dev/null +++ b/test/e2e/expected/swck-injected-volume.yml @@ -0,0 +1,17 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +sharedVolume: true +appMounted: true From 912c02cb44051e25bae795a8153a879ac7b64fd5 Mon Sep 17 00:00:00 2001 From: Wu Sheng Date: Sat, 29 Aug 2026 23:25:26 +0800 Subject: [PATCH 3/4] Drop a tautological template assertion, widen the retry budget Two findings from an adversarial pass over the new suite. hasRows was asserting nothing. Both live and readonly build their row set from the ~370-file bundle on disk (logic/templates/sync.ts), so rows|length > 0 is true in every cell whether or not OAP was ever contacted. The load-bearing field is unreachable, which in live mode is false only when the /ui-management call against OAP actually succeeded. Keep that, drop the rest. Retry goes 20 -> 30 at 10s. GET /api/layer/:key/services sits behind a 60s catalog TTL, so 200s of budget is only a few useful samples once the demo traffic has to accumulate first. --- test/e2e/e2e-oap10-banyandb.yaml | 4 ++-- test/e2e/e2e-oap10-elasticsearch.yaml | 4 ++-- test/e2e/e2e-oap11-banyandb-cluster.yaml | 4 ++-- test/e2e/e2e-oap11-banyandb-standalone.yaml | 4 ++-- test/e2e/e2e-oap11-elasticsearch.yaml | 4 ++-- test/e2e/expected/horizon-templates-live.yml | 1 - test/e2e/expected/horizon-templates-readonly.yml | 1 - 7 files changed, 10 insertions(+), 12 deletions(-) diff --git a/test/e2e/e2e-oap10-banyandb.yaml b/test/e2e/e2e-oap10-banyandb.yaml index abda5a7..2d8e4b2 100644 --- a/test/e2e/e2e-oap10-banyandb.yaml +++ b/test/e2e/e2e-oap10-banyandb.yaml @@ -99,7 +99,7 @@ setup: verify: retry: - count: 20 + count: 30 interval: 10s cases: # Every assertion below goes through Horizon's BFF: horizon -> oap -> storage. @@ -125,7 +125,7 @@ verify: # readonly still serves templates -- from the bundle in the image, without ever # contacting OAP. - query: | - bash test/e2e/script/horizon.sh http://${service_skywalking_ui_host}:${service_skywalking_ui_80} get '/api/admin/templates/sync-status?force=true' | yq -p json -o yaml '{"mode": .mode, "unreachable": .unreachable, "hasRows": ((.rows | length) > 0)}' + bash test/e2e/script/horizon.sh http://${service_skywalking_ui_host}:${service_skywalking_ui_80} get '/api/admin/templates/sync-status?force=true' | yq -p json -o yaml '{"mode": .mode, "unreachable": .unreachable}' expected: expected/horizon-templates-readonly.yml # The mesh services the demo app produces, listed through Horizon. diff --git a/test/e2e/e2e-oap10-elasticsearch.yaml b/test/e2e/e2e-oap10-elasticsearch.yaml index 6b0a1d9..41873bf 100644 --- a/test/e2e/e2e-oap10-elasticsearch.yaml +++ b/test/e2e/e2e-oap10-elasticsearch.yaml @@ -97,7 +97,7 @@ setup: verify: retry: - count: 20 + count: 30 interval: 10s cases: # Every assertion below goes through Horizon's BFF: horizon -> oap -> storage. @@ -123,7 +123,7 @@ verify: # readonly still serves templates -- from the bundle in the image, without ever # contacting OAP. - query: | - bash test/e2e/script/horizon.sh http://${service_skywalking_ui_host}:${service_skywalking_ui_80} get '/api/admin/templates/sync-status?force=true' | yq -p json -o yaml '{"mode": .mode, "unreachable": .unreachable, "hasRows": ((.rows | length) > 0)}' + bash test/e2e/script/horizon.sh http://${service_skywalking_ui_host}:${service_skywalking_ui_80} get '/api/admin/templates/sync-status?force=true' | yq -p json -o yaml '{"mode": .mode, "unreachable": .unreachable}' expected: expected/horizon-templates-readonly.yml # The mesh services the demo app produces, listed through Horizon. diff --git a/test/e2e/e2e-oap11-banyandb-cluster.yaml b/test/e2e/e2e-oap11-banyandb-cluster.yaml index fa3c93f..3907b4f 100644 --- a/test/e2e/e2e-oap11-banyandb-cluster.yaml +++ b/test/e2e/e2e-oap11-banyandb-cluster.yaml @@ -100,7 +100,7 @@ setup: verify: retry: - count: 20 + count: 30 interval: 10s cases: # Every assertion below goes through Horizon's BFF: horizon -> oap -> storage. @@ -130,7 +130,7 @@ verify: # The dashboard template store, read through /ui-management on the admin port. - query: | - bash test/e2e/script/horizon.sh http://${service_skywalking_ui_host}:${service_skywalking_ui_80} get '/api/admin/templates/sync-status?force=true' | yq -p json -o yaml '{"mode": .mode, "unreachable": .unreachable, "hasRows": ((.rows | length) > 0)}' + bash test/e2e/script/horizon.sh http://${service_skywalking_ui_host}:${service_skywalking_ui_80} get '/api/admin/templates/sync-status?force=true' | yq -p json -o yaml '{"mode": .mode, "unreachable": .unreachable}' expected: expected/horizon-templates-live.yml # The mesh services the demo app produces, listed through Horizon. diff --git a/test/e2e/e2e-oap11-banyandb-standalone.yaml b/test/e2e/e2e-oap11-banyandb-standalone.yaml index f5244cd..9eb2322 100644 --- a/test/e2e/e2e-oap11-banyandb-standalone.yaml +++ b/test/e2e/e2e-oap11-banyandb-standalone.yaml @@ -100,7 +100,7 @@ setup: verify: retry: - count: 20 + count: 30 interval: 10s cases: # Every assertion below goes through Horizon's BFF: horizon -> oap -> storage. @@ -130,7 +130,7 @@ verify: # The dashboard template store, read through /ui-management on the admin port. - query: | - bash test/e2e/script/horizon.sh http://${service_skywalking_ui_host}:${service_skywalking_ui_80} get '/api/admin/templates/sync-status?force=true' | yq -p json -o yaml '{"mode": .mode, "unreachable": .unreachable, "hasRows": ((.rows | length) > 0)}' + bash test/e2e/script/horizon.sh http://${service_skywalking_ui_host}:${service_skywalking_ui_80} get '/api/admin/templates/sync-status?force=true' | yq -p json -o yaml '{"mode": .mode, "unreachable": .unreachable}' expected: expected/horizon-templates-live.yml # The mesh services the demo app produces, listed through Horizon. diff --git a/test/e2e/e2e-oap11-elasticsearch.yaml b/test/e2e/e2e-oap11-elasticsearch.yaml index 4329d8c..5378a91 100644 --- a/test/e2e/e2e-oap11-elasticsearch.yaml +++ b/test/e2e/e2e-oap11-elasticsearch.yaml @@ -98,7 +98,7 @@ setup: verify: retry: - count: 20 + count: 30 interval: 10s cases: # Every assertion below goes through Horizon's BFF: horizon -> oap -> storage. @@ -128,7 +128,7 @@ verify: # The dashboard template store, read through /ui-management on the admin port. - query: | - bash test/e2e/script/horizon.sh http://${service_skywalking_ui_host}:${service_skywalking_ui_80} get '/api/admin/templates/sync-status?force=true' | yq -p json -o yaml '{"mode": .mode, "unreachable": .unreachable, "hasRows": ((.rows | length) > 0)}' + bash test/e2e/script/horizon.sh http://${service_skywalking_ui_host}:${service_skywalking_ui_80} get '/api/admin/templates/sync-status?force=true' | yq -p json -o yaml '{"mode": .mode, "unreachable": .unreachable}' expected: expected/horizon-templates-live.yml # The mesh services the demo app produces, listed through Horizon. diff --git a/test/e2e/expected/horizon-templates-live.yml b/test/e2e/expected/horizon-templates-live.yml index 7d5b4f4..0c880d1 100644 --- a/test/e2e/expected/horizon-templates-live.yml +++ b/test/e2e/expected/horizon-templates-live.yml @@ -15,4 +15,3 @@ mode: live unreachable: false -hasRows: true diff --git a/test/e2e/expected/horizon-templates-readonly.yml b/test/e2e/expected/horizon-templates-readonly.yml index abdece2..deebfc6 100644 --- a/test/e2e/expected/horizon-templates-readonly.yml +++ b/test/e2e/expected/horizon-templates-readonly.yml @@ -15,4 +15,3 @@ mode: readonly unreachable: false -hasRows: true From ddb35c71248d22a16d62d0d38e64bbb5ad01eab2 Mon Sep 17 00:00:00 2001 From: Wu Sheng Date: Sat, 29 Aug 2026 23:42:05 +0800 Subject: [PATCH 4/4] Assert service_cpm is positive, not notEmpty The metric came back as 2719.86 through Horizon -> OAP -> Elasticsearch on OAP 10.4, so the read path works; the case failed only because infra-e2e's notEmpty matcher accepts nil or string and the value is a float64. Asserting the number is greater than zero is both valid and a stronger claim than presence. --- test/e2e/e2e-oap10-banyandb.yaml | 2 +- test/e2e/e2e-oap10-elasticsearch.yaml | 2 +- test/e2e/e2e-oap11-banyandb-cluster.yaml | 2 +- test/e2e/e2e-oap11-banyandb-standalone.yaml | 2 +- test/e2e/e2e-oap11-elasticsearch.yaml | 2 +- test/e2e/expected/horizon-service-cpm.yml | 2 +- 6 files changed, 6 insertions(+), 6 deletions(-) diff --git a/test/e2e/e2e-oap10-banyandb.yaml b/test/e2e/e2e-oap10-banyandb.yaml index 2d8e4b2..0508ddf 100644 --- a/test/e2e/e2e-oap10-banyandb.yaml +++ b/test/e2e/e2e-oap10-banyandb.yaml @@ -139,5 +139,5 @@ verify: # service_cpm for one of them -- the full read path through to storage. - query: | - bash test/e2e/script/horizon.sh http://${service_skywalking_ui_host}:${service_skywalking_ui_80} post /api/layer/MESH/dashboard '{"service":"e2e::productpage","scope":"service","widgets":[{"id":"cpm","title":"Service CPM","type":"card","expressions":["service_cpm"]}]}' | yq -p json -o yaml '{"reachable": .reachable, "id": .widgets[0].id, "value": .widgets[0].value}' + bash test/e2e/script/horizon.sh http://${service_skywalking_ui_host}:${service_skywalking_ui_80} post /api/layer/MESH/dashboard '{"service":"e2e::productpage","scope":"service","widgets":[{"id":"cpm","title":"Service CPM","type":"card","expressions":["service_cpm"]}]}' | yq -p json -o yaml '{"reachable": .reachable, "id": .widgets[0].id, "positive": (.widgets[0].value != null and .widgets[0].value > 0)}' expected: expected/horizon-service-cpm.yml diff --git a/test/e2e/e2e-oap10-elasticsearch.yaml b/test/e2e/e2e-oap10-elasticsearch.yaml index 41873bf..c201182 100644 --- a/test/e2e/e2e-oap10-elasticsearch.yaml +++ b/test/e2e/e2e-oap10-elasticsearch.yaml @@ -137,5 +137,5 @@ verify: # service_cpm for one of them -- the full read path through to storage. - query: | - bash test/e2e/script/horizon.sh http://${service_skywalking_ui_host}:${service_skywalking_ui_80} post /api/layer/MESH/dashboard '{"service":"e2e::productpage","scope":"service","widgets":[{"id":"cpm","title":"Service CPM","type":"card","expressions":["service_cpm"]}]}' | yq -p json -o yaml '{"reachable": .reachable, "id": .widgets[0].id, "value": .widgets[0].value}' + bash test/e2e/script/horizon.sh http://${service_skywalking_ui_host}:${service_skywalking_ui_80} post /api/layer/MESH/dashboard '{"service":"e2e::productpage","scope":"service","widgets":[{"id":"cpm","title":"Service CPM","type":"card","expressions":["service_cpm"]}]}' | yq -p json -o yaml '{"reachable": .reachable, "id": .widgets[0].id, "positive": (.widgets[0].value != null and .widgets[0].value > 0)}' expected: expected/horizon-service-cpm.yml diff --git a/test/e2e/e2e-oap11-banyandb-cluster.yaml b/test/e2e/e2e-oap11-banyandb-cluster.yaml index 3907b4f..cf3a74d 100644 --- a/test/e2e/e2e-oap11-banyandb-cluster.yaml +++ b/test/e2e/e2e-oap11-banyandb-cluster.yaml @@ -144,5 +144,5 @@ verify: # service_cpm for one of them -- the full read path through to storage. - query: | - bash test/e2e/script/horizon.sh http://${service_skywalking_ui_host}:${service_skywalking_ui_80} post /api/layer/MESH/dashboard '{"service":"e2e::productpage","scope":"service","widgets":[{"id":"cpm","title":"Service CPM","type":"card","expressions":["service_cpm"]}]}' | yq -p json -o yaml '{"reachable": .reachable, "id": .widgets[0].id, "value": .widgets[0].value}' + bash test/e2e/script/horizon.sh http://${service_skywalking_ui_host}:${service_skywalking_ui_80} post /api/layer/MESH/dashboard '{"service":"e2e::productpage","scope":"service","widgets":[{"id":"cpm","title":"Service CPM","type":"card","expressions":["service_cpm"]}]}' | yq -p json -o yaml '{"reachable": .reachable, "id": .widgets[0].id, "positive": (.widgets[0].value != null and .widgets[0].value > 0)}' expected: expected/horizon-service-cpm.yml diff --git a/test/e2e/e2e-oap11-banyandb-standalone.yaml b/test/e2e/e2e-oap11-banyandb-standalone.yaml index 9eb2322..40c73c9 100644 --- a/test/e2e/e2e-oap11-banyandb-standalone.yaml +++ b/test/e2e/e2e-oap11-banyandb-standalone.yaml @@ -144,5 +144,5 @@ verify: # service_cpm for one of them -- the full read path through to storage. - query: | - bash test/e2e/script/horizon.sh http://${service_skywalking_ui_host}:${service_skywalking_ui_80} post /api/layer/MESH/dashboard '{"service":"e2e::productpage","scope":"service","widgets":[{"id":"cpm","title":"Service CPM","type":"card","expressions":["service_cpm"]}]}' | yq -p json -o yaml '{"reachable": .reachable, "id": .widgets[0].id, "value": .widgets[0].value}' + bash test/e2e/script/horizon.sh http://${service_skywalking_ui_host}:${service_skywalking_ui_80} post /api/layer/MESH/dashboard '{"service":"e2e::productpage","scope":"service","widgets":[{"id":"cpm","title":"Service CPM","type":"card","expressions":["service_cpm"]}]}' | yq -p json -o yaml '{"reachable": .reachable, "id": .widgets[0].id, "positive": (.widgets[0].value != null and .widgets[0].value > 0)}' expected: expected/horizon-service-cpm.yml diff --git a/test/e2e/e2e-oap11-elasticsearch.yaml b/test/e2e/e2e-oap11-elasticsearch.yaml index 5378a91..1d5606a 100644 --- a/test/e2e/e2e-oap11-elasticsearch.yaml +++ b/test/e2e/e2e-oap11-elasticsearch.yaml @@ -142,5 +142,5 @@ verify: # service_cpm for one of them -- the full read path through to storage. - query: | - bash test/e2e/script/horizon.sh http://${service_skywalking_ui_host}:${service_skywalking_ui_80} post /api/layer/MESH/dashboard '{"service":"e2e::productpage","scope":"service","widgets":[{"id":"cpm","title":"Service CPM","type":"card","expressions":["service_cpm"]}]}' | yq -p json -o yaml '{"reachable": .reachable, "id": .widgets[0].id, "value": .widgets[0].value}' + bash test/e2e/script/horizon.sh http://${service_skywalking_ui_host}:${service_skywalking_ui_80} post /api/layer/MESH/dashboard '{"service":"e2e::productpage","scope":"service","widgets":[{"id":"cpm","title":"Service CPM","type":"card","expressions":["service_cpm"]}]}' | yq -p json -o yaml '{"reachable": .reachable, "id": .widgets[0].id, "positive": (.widgets[0].value != null and .widgets[0].value > 0)}' expected: expected/horizon-service-cpm.yml diff --git a/test/e2e/expected/horizon-service-cpm.yml b/test/e2e/expected/horizon-service-cpm.yml index 81d08b1..0a3b0f7 100644 --- a/test/e2e/expected/horizon-service-cpm.yml +++ b/test/e2e/expected/horizon-service-cpm.yml @@ -15,4 +15,4 @@ reachable: true id: cpm -value: {{ notEmpty .value }} +positive: true