Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions .github/actions/preview-delivery/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,7 @@ runs:
run: bash "$GITHUB_ACTION_PATH/../../../scripts/automation/preview_delivery.sh" resolve-apps

- name: Push and release preview peer transports
id: release
shell: bash
env:
APP_NAME: ${{ steps.heroku.outputs.app_name }}
Expand All @@ -91,6 +92,7 @@ runs:
run: bash "$GITHUB_ACTION_PATH/../../../scripts/automation/preview_delivery.sh" release

- name: Converge preview Peer advertisement
id: advertisement
shell: bash
env:
DATABASE_URL: ${{ steps.database.outputs.runtime_database_url }}
Expand All @@ -100,6 +102,7 @@ runs:
run: bash "$GITHUB_ACTION_PATH/../../../scripts/automation/preview_delivery.sh" advertise

- name: Probe deployed preview
id: probe
shell: bash
env:
APP_NAME: ${{ steps.heroku.outputs.app_name }}
Expand All @@ -109,3 +112,20 @@ runs:
POSTGREST_URL: ${{ steps.heroku.outputs.postgrest_url }}
WEB_URL: ${{ steps.heroku.outputs.web_url }}
run: bash "$GITHUB_ACTION_PATH/../../../scripts/automation/preview_delivery.sh" probe

- name: Report runtime stages even after failure
if: ${{ always() }}
shell: bash
env:
HEAD_SHA: ${{ inputs.head_sha }}
PR_NUMBER: ${{ inputs.pr_number }}
HEROKU_API_KEY: ${{ inputs.heroku_api_key }}
EXTENSION_REGISTRY_URL: ${{ inputs.extension_registry_url }}
BRANCH_ID: ${{ steps.neon.outputs.branch_id }}
DATABASE_RESULT: ${{ steps.database.outcome }}
DATABASE_BEFORE: ${{ steps.database.outputs.database_before }}
DATABASE_AFTER: ${{ steps.database.outputs.database_after }}
RELEASE_RESULT: ${{ steps.release.outcome }}
ADVERTISEMENT_RESULT: ${{ steps.advertisement.outcome }}
PROBE_RESULT: ${{ steps.probe.outcome }}
run: bash "$GITHUB_ACTION_PATH/../../../scripts/automation/preview_delivery.sh" summarize
18 changes: 16 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -75,8 +75,22 @@ jobs:
- name: Run repository contract
run: pdm run check

- name: Verify native documentation publication
run: pdm run python -m pytest -q scripts/tests/test_extension_documentation.py
- name: Initialize disposable business-test database
env:
INKCRE_ENV_FILE: ""
MIGRATION_DATABASE_URL: postgresql+psycopg://postgres:postgres@127.0.0.1:5432/inkcre
CORE_DATABASE_PASSWORD: ci-core-test-password-at-least-32-bytes
POSTGREST_DATABASE_PASSWORD: ci-postgrest-test-password-at-least-32-bytes
run: pdm run db:init --profile development

- name: Run required PostgreSQL business scenarios
env:
INKCRE_TEST_DATABASE_URL: postgresql+psycopg://inkcre_core:ci-core-test-password-at-least-32-bytes@127.0.0.1:5432/inkcre
MIGRATION_DATABASE_URL: postgresql+psycopg://postgres:postgres@127.0.0.1:5432/inkcre
run: pdm run test:database

- name: Verify documentation publication and partial Preview reporting
run: pdm run python -m pytest -q scripts/tests/test_extension_documentation.py scripts/tests/test_preview_delivery.py

- name: Build and verify all first-party wheels
env:
Expand Down
12 changes: 12 additions & 0 deletions .github/workflows/preview-deploy.yml
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ jobs:
environment: preview
steps:
- name: Checkout trusted delivery implementation
id: controller
uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09 # v5
with:
ref: ${{ github.workflow_sha }}
Expand Down Expand Up @@ -80,6 +81,7 @@ jobs:
pr_number: ${{ github.event.pull_request.number }}

- name: Deploy exact-head sibling Registry to isolated Pages branch
id: registry
uses: cloudflare/wrangler-action@ebbaa1584979971c8614a24965b4405ff95890e0 # v4.0.0
with:
apiToken: ${{ secrets.CLOUDFLARE_API_TOKEN }}
Expand All @@ -102,6 +104,7 @@ jobs:
pr_number: ${{ github.event.pull_request.number }}

- name: Deliver preview
id: delivery
uses: ./.delivery/.github/actions/preview-delivery
with:
core_database_password: ${{ secrets.CORE_DATABASE_PASSWORD }}
Expand All @@ -115,6 +118,15 @@ jobs:
jwt_secret: ${{ secrets.JWT_SECRET }}
pr_number: ${{ github.event.pull_request.number }}

- name: Report complete or partial candidate delivery
if: ${{ always() && steps.controller.outcome == 'success' }}
env:
HEAD_SHA: ${{ github.event.pull_request.head.sha }}
REGISTRY_RESULT: ${{ steps.registry.outcome }}
REGISTRY_SNAPSHOT: ${{ steps.registry.outputs.deployment-url }}
DELIVERY_RESULT: ${{ steps.delivery.outcome }}
run: bash .delivery/scripts/automation/preview_workflow.sh summarize

cleanup:
name: Retire preview resources
if: >-
Expand Down
8 changes: 8 additions & 0 deletions docs/40-deployment/development-environment.md
Original file line number Diff line number Diff line change
Expand Up @@ -153,6 +153,14 @@ Credentialed semantic-quality acceptance is explicit and separate from the ordin
selected migrated `INKCRE_TEST_DATABASE_URL`; see the local Semantic Retrieval Unit TDD for the command and pass/fail
authority.

`pdm run test:database` 是无需真实 AI/IMAP 服务的 PostgreSQL 业务回归入口,具体场景清单由
`scripts/automation/repository_check.sh` 拥有。它要求显式 `INKCRE_TEST_DATABASE_URL`;未设置时直接失败,
不会将整组 skip 当作通过。数据库必须已经迁移且可丢弃:场景包含清空数据、事务回滚和迁移往返验证,
不能连接共享开发库、Preview 或生产。Memos 迁移场景还要求该独占 PostgreSQL 的 `MIGRATION_DATABASE_URL`
具有创建数据库权限;它建立随机命名的历史数据库,只往返附件所属 revision,并在结束后删除该数据库。
这保留实际 migration 编排和应用下载读回,不会回退当前业务测试库或声称验证了整条历史 downgrade 链。
CI 在自身临时 PostgreSQL service 上先初始化再执行这一组;外部服务和真实凭据验收继续由各自专项入口负责。

## Database Branch Workflow In CI

The repository currently uses Neon branch automation for pull requests.
Expand Down
12 changes: 12 additions & 0 deletions docs/40-deployment/heroku.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,18 @@ reads and consumer acceptance are deliberately independent and do not block Core
Repository-qualified app and branch identities keep an impl-py PR and an impl-ts PR with
the same number from addressing the same review resources.

交付摘要在失败时也报告目标 SHA、Registry snapshot、数据库初始化、进程 release、Peer 广告和探测的
各自结果。数据库输出保留初始化前后的实际 migration head 与候选期望 head/contract;没有取得的观察
明确留空,不推断旧服务已经停止。Core 镜像写入目标 SHA label。Heroku 当前 release 身份另外只读查询,
该身份本身不证明本轮源码已生效。进程 Registry fallback 也不能覆盖 Peer/deployment 的优先配置,
跨仓业务验收仍需核对实际 Distribution 和 Registry 来源。

同一 PR 的 Neon 分支保留状态。若候选移除了该分支已经执行的未发布迁移,重复交付不会自动 reset、
downgrade 或补回迁移。先保留原分支作诊断:需要验证保留状态升级时,修复候选,使它能从当前 head
前进;需要验证全新候选时,将同一明确源码放到新的 PR,使现有分支 owner 从 `preview-base` 创建新的
隔离分支。原 PR 的关闭清理只在不再需要其数据与证据时执行。这里不增加隐式“自愈”或复用其他 PR
数据库的行为;失败时 Registry 可能已经更新,不表示整个组合已经就绪。

## Canonical Production

Production is one logical environment containing two peer transports:
Expand Down
1 change: 1 addition & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,7 @@ format = "ruff format ."
"format:check" = "ruff format --check ."
typecheck = "pyrefly check --min-severity=warn --output-format=min-text --progress-bar=no"
test = "python -m pytest -q"
"test:database" = "bash scripts/automation/repository_check.sh database-tests"
"test:acceptance" = "python -m pytest -q -m acceptance tests/semantic_retrieval/acceptance"
"test:organization-acceptance" = "python -m pytest -q -s tests/organization/acceptance/test_black_box.py"
"check:lock" = "python scripts/check_lock.py"
Expand Down
57 changes: 54 additions & 3 deletions scripts/automation/preview_delivery.sh
Original file line number Diff line number Diff line change
Expand Up @@ -110,11 +110,27 @@ case "${1:-}" in
converge-database)
for name in CORE_DATABASE_PASSWORD HEAD_SHA MIGRATION_DATABASE_URL \
POSTGREST_DATABASE_PASSWORD SOURCE_DATABASE_URL; do require_env "$name"; done
readiness_file="$(mktemp)"
trap 'rm -f "$readiness_file"' EXIT
before_status=0
docker run --rm --env MIGRATION_DATABASE_URL "inkcre-preview-web:$HEAD_SHA" \
python scripts/container.py db ready --profile runtime --json > "$readiness_file" || before_status=$?
emit_output database_before "$(jq -c --argjson probe_exit "$before_status" \
'{probe_exit:$probe_exit,status,contract,migration,database}' "$readiness_file")"
initialization_status=0
docker run --rm --env CORE_DATABASE_PASSWORD --env MIGRATION_DATABASE_URL \
--env POSTGREST_DATABASE_PASSWORD "inkcre-preview-web:$HEAD_SHA" \
python scripts/container.py db init --profile runtime --environment preview
python scripts/container.py db init --profile runtime --environment preview || initialization_status=$?
readiness_status=0
docker run --rm --env MIGRATION_DATABASE_URL "inkcre-preview-web:$HEAD_SHA" \
python scripts/container.py db ready --profile runtime --json
python scripts/container.py db ready --profile runtime --json > "$readiness_file" || readiness_status=$?
emit_output database_after "$(jq -c --argjson probe_exit "$readiness_status" \
'{probe_exit:$probe_exit,status,contract,migration,database}' "$readiness_file")"
if [ "$initialization_status" != 0 ]; then
echo "Preview database initialization failed; preserve this branch for inspection. Choose a forward-compatible candidate or a new PR/isolated branch from preview-base; do not reset automatically." >&2
exit "$initialization_status"
fi
test "$readiness_status" = 0
runtime_database_url="$(docker run --rm --env SOURCE_DATABASE_URL \
--env TARGET_DATABASE_PASSWORD="$CORE_DATABASE_PASSWORD" \
--env TARGET_DATABASE_ROLE=inkcre_core --entrypoint python \
Expand Down Expand Up @@ -233,8 +249,43 @@ EOF
heroku logs --num 200 --app "$POSTGREST_APP_NAME"
exit 1
;;
summarize)
require_env HEAD_SHA
require_env PR_NUMBER
[[ "$PR_NUMBER" =~ ^[0-9]+$ ]]
append_summary <<EOF
### Preview 运行时分阶段结果

- 目标源码:\`$HEAD_SHA\`
- Neon 分支:\`${BRANCH_ID:-未取得}\`
- 配置的进程 Registry fallback:${EXTENSION_REGISTRY_URL:-未取得};Peer/deployment override 仍可能优先,业务验收需核对实际选择。
- 数据库初始化:\`${DATABASE_RESULT:-未执行}\`
- 进程 release:\`${RELEASE_RESULT:-未执行}\`
- Peer 广告:\`${ADVERTISEMENT_RESULT:-未执行}\`
- 基础探测:\`${PROBE_RESULT:-未执行}\`

以下 contract.revision 是候选期望合同,migration.current 是实际读取的数据库 head;空值表示没有取得观察。

初始化前:
\`\`\`json
${DATABASE_BEFORE:-null}
\`\`\`
初始化后:
\`\`\`json
${DATABASE_AFTER:-null}
\`\`\`
EOF
for app_name in "inkcre-core-py-pr-$PR_NUMBER" "inkcre-postgrest-pr-$PR_NUMBER"; do
if releases="$(heroku releases --app "$app_name" --json 2>/dev/null)"; then
observed="$(jq -c '.[0] | {version,status,description}' <<<"$releases")"
printf -- '- 当前 Heroku release `%s`:`%s`(不单独证明源码或探测通过)\n' "$app_name" "$observed" | append_summary
else
printf -- '- 当前 Heroku release `%s`:未取得观察,不能推断服务不存在。\n' "$app_name" | append_summary
fi
done
;;
*)
echo "usage: $0 resolve-neon|normalize-roles|converge-database|resolve-apps|release|advertise|probe" >&2
echo "usage: $0 resolve-neon|normalize-roles|converge-database|resolve-apps|release|advertise|probe|summarize" >&2
exit 2
;;
esac
19 changes: 18 additions & 1 deletion scripts/automation/preview_workflow.sh
Original file line number Diff line number Diff line change
Expand Up @@ -23,14 +23,31 @@ case "${1:-}" in
require_env HEAD_SHA
require_env SOURCE_DIRECTORY
docker build --platform linux/amd64 --provenance=false --target heroku-web \
--build-arg "SOURCE_REVISION=$HEAD_SHA" \
--tag "inkcre-preview-web:$HEAD_SHA" "$SOURCE_DIRECTORY"
docker build --platform linux/amd64 --provenance=false --target heroku-release \
--build-arg "SOURCE_REVISION=$HEAD_SHA" \
--tag "inkcre-preview-release:$HEAD_SHA" "$SOURCE_DIRECTORY"
docker build --platform linux/amd64 --provenance=false \
--file "$SOURCE_DIRECTORY/Dockerfile.postgrest" \
--build-arg "SOURCE_REVISION=$HEAD_SHA" \
--tag "inkcre-preview-postgrest:$HEAD_SHA" "$SOURCE_DIRECTORY"
;;
summarize)
require_env HEAD_SHA
append_summary <<EOF
### Preview 候选交付结果

| 层 | 本次结果 | 候选身份 |
| --- | --- | --- |
| 源码 | 精确 PR head | \`$HEAD_SHA\` |
| wheel Registry | \`${REGISTRY_RESULT:-未执行}\` | ${REGISTRY_SNAPSHOT:-未取得快照 URL} |
| Core/数据库/PostgREST 交付 | \`${DELIVERY_RESULT:-未执行}\` | 见下方分阶段结果或对应失败步骤 |

Registry 与运行时不是原子交付。Registry 成功而运行时失败时,不能将新的 Registry 当作整个候选已就绪;
旧 Core 或数据库可能仍存在,本流程没有自动回滚。基础探测也不代表 Web 已连接、扩展已启用或业务旅程通过。
EOF
;;
validate-identity)
validate_preview_identity
;;
Expand All @@ -44,5 +61,5 @@ case "${1:-}" in
fi
done
;;
*) echo "usage: $0 build-extensions|build-images|validate-identity|cleanup-apps" >&2; exit 2 ;;
*) echo "usage: $0 build-extensions|build-images|summarize|validate-identity|cleanup-apps" >&2; exit 2 ;;
esac
12 changes: 11 additions & 1 deletion scripts/automation/repository_check.sh
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,16 @@
source "$(dirname "$0")/_common.sh"

case "${1:-}" in
database-tests)
require_env INKCRE_TEST_DATABASE_URL
# This admitted suite needs migrated, disposable PostgreSQL, but no live AI or IMAP service.
pdm run python -m pytest -q -rs \
tests/agent/integration tests/ai/integration tests/deployment_config/integration \
tests/info_base/integration tests/lexical_retrieval/integration \
tests/organization/integration tests/peer/integration tests/semantic_retrieval/integration \
tests/extensions/memos/integration tests/extensions/rss/integration/test_feed_collection.py \
tests/migrations/test_memos_attachment_v2_migration.py
;;
release-intent)
arguments=()
if [[ "${BASE_REVISION:-}" =~ ^[0-9a-f]{40}$ ]] &&
Expand All @@ -25,5 +35,5 @@ case "${1:-}" in
--project "extensions/$extension" --wheel "$wheel"
done
;;
*) echo "usage: $0 release-intent|extension-wheels" >&2; exit 2 ;;
*) echo "usage: $0 database-tests|release-intent|extension-wheels" >&2; exit 2 ;;
esac
98 changes: 98 additions & 0 deletions scripts/tests/test_preview_delivery.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
"""Exercise partial delivery reporting without provider credentials or shared state."""

import json
import os
from pathlib import Path
import subprocess


ROOT = Path(__file__).resolve().parents[2]


def _run(script: str, command: str, environment: dict[str, str]):
return subprocess.run( # noqa: S603 - runs a fixed repository script with isolated provider shims.
["bash", str(ROOT / "scripts/automation" / script), command],
env=environment,
capture_output=True,
text=True,
check=False,
)


def test_failed_migration_keeps_observed_heads_and_reports_partial_delivery(tmp_path):
binaries = tmp_path / "bin"
binaries.mkdir()
observed = {
"status": "error",
"contract": {"revision": "candidate-contract"},
"migration": {"current": ["removed-revision"], "expected": ["candidate-head"]},
"database": {"status": "error"},
}
docker = binaries / "docker"
docker.write_text(
"#!/bin/sh\n"
'case "$*" in\n'
f" *' db ready '*) printf '%s\\n' '{json.dumps(observed)}' ;;\n"
" *' db init '*) echo 'missing revision' >&2; exit 17 ;;\n"
" *) echo 'unexpected provider operation' >&2; exit 88 ;;\n"
"esac\n"
)
docker.chmod(0o700)
heroku = binaries / "heroku"
releases = json.dumps(
[{"version": 7, "status": "succeeded", "description": "Previous web"}]
)
heroku.write_text(f"#!/bin/sh\nprintf '%s\\n' '{releases}'\n")
heroku.chmod(0o700)
output = tmp_path / "outputs"
environment = {
**os.environ,
"PATH": f"{binaries}:{os.environ['PATH']}",
"GITHUB_OUTPUT": str(output),
"GITHUB_STEP_SUMMARY": str(tmp_path / "summary"),
"HEAD_SHA": "a" * 40,
"PR_NUMBER": "123",
"CORE_DATABASE_PASSWORD": "secret-not-for-report",
"POSTGREST_DATABASE_PASSWORD": "secret-not-for-report",
"MIGRATION_DATABASE_URL": "secret-not-for-report",
"SOURCE_DATABASE_URL": "secret-not-for-report",
}
failed = _run("preview_delivery.sh", "converge-database", environment)
assert failed.returncode == 17
values = dict(line.split("=", 1) for line in output.read_text().splitlines())
assert json.loads(values["database_before"])["migration"] == observed["migration"]
assert json.loads(values["database_after"])["migration"] == observed["migration"]
assert "runtime_database_url" not in values

runtime_report = _run(
"preview_delivery.sh",
"summarize",
{
**environment,
"DATABASE_BEFORE": values["database_before"],
"DATABASE_AFTER": values["database_after"],
"DATABASE_RESULT": "failure",
"RELEASE_RESULT": "skipped",
"ADVERTISEMENT_RESULT": "skipped",
"PROBE_RESULT": "skipped",
},
)
assert runtime_report.returncode == 0, runtime_report.stderr
candidate_report = _run(
"preview_workflow.sh",
"summarize",
{
**environment,
"REGISTRY_RESULT": "success",
"REGISTRY_SNAPSHOT": "https://snapshot.pages.dev",
"DELIVERY_RESULT": "failure",
},
)
assert candidate_report.returncode == 0, candidate_report.stderr
summary = (tmp_path / "summary").read_text()
assert "数据库初始化:`failure`" in summary
assert "进程 release:`skipped`" in summary
assert "removed-revision" in summary
assert '"version":7' in summary
assert "https://snapshot.pages.dev" in summary
assert "secret-not-for-report" not in summary + failed.stdout + failed.stderr
Loading
Loading