Feature/590 introduction ak integrations test harness

[SandunYL]

Description

Introduces a true end-to-end test harness for Agent Kernel's messaging integrations against
real platform accounts. A real user account sends a message on the real platform → the
platform delivers it by webhook to a long-lived AWS ECS deployment running one OpenAI agent
(gpt-4.1-mini) with all integrations enabled → the test reads the agent's real reply back from
the platform.

This closes the gap left by the existing nightly smoke tests, which are health-check-only —
nothing today drives a message through a platform and verifies a reply arrives. Only a deployed
instance receiving real webhooks exercises the transport layer (API Gateway routing, Slack
signature verification, Telegram secret token, webhook payload shape).

The harness ships as a self-contained e2e/ tree (deployable app + Terraform + pytest suite),
plus a weekly CI workflow that drives it, and the design/spec/plan docs under docs/specs/.

Coverage — every messaging platform Agent Kernel can construct today:

Platform	Verification	CI
Slack	Reads bot's threaded reply back from the platform (full round-trip)	Automated
Telegram	Reads bot's reply back via MTProto/Telethon (bots can't message bots)	Automated
Gmail	Polling-based; reads bot's reply back (two accounts — send-to-self loops)	Automated
WhatsApp	No read-back API — verified via CloudWatch send-success logs; needs a production sender number	Opt-in (E2E_WHATSAPP_AUTOMATED=1), else manual
Messenger	No API to DM a Page as a user — manual DM only; log-based check	Opt-in (E2E_MESSENGER_AUTOMATED=1), else manual
Instagram	No API to DM an account as a user — manual DM only; log-based check	Opt-in (E2E_INSTAGRAM_AUTOMATED=1), else manual

Teams is out of scope: core/config.py has no _TeamsConfig/teams: field, so its handler
can't be constructed without a core change first.

Assertion depth (first cut): the bot replied with something, and the reply is not one of the
handlers' known error-fallback strings (which would mean transport OK but the agent run failed).
Reply content is deliberately not asserted.

Type of Change

• New feature (non-breaking change which adds functionality)
• Test update
• CI/CD update
• Documentation update

Related Issues

Closes #590

Changes Made

• e2e/app/ — deployable agent app: one app.py registers a single OpenAI agent and serves
  all enabled integrations via RESTAPI.run([...]). Optional integrations activate only when
  their env vars are present, so the app degrades to Slack + Telegram when the rest are empty.
• e2e/app/deploy/ — Terraform using the yaalalabs/ak-containerized/aws module (ECS +
  HTTPS API Gateway via VPC link) with gateway_endpoints for each platform's webhook path;
  webhook URLs are outputs. Includes Dockerfile, build.sh, and deploy.sh (loads a gitignored
  .env and exports TF_VAR_*). Remote state via backend.tf. Deployment is long-lived /
  one-time — deploy once, run tests on demand.
• e2e/tests/ — plain pytest harness driving the deployed instance. Credentials come from env
  vars; tests skip when creds are missing, so platforms can be run one at a time. Each test
  sends a uniquely-tagged message and polls up to 3 minutes for the reply.
• e2e/tests/scripts/ — one-time interactive helpers: Telegram MTProto login (session
  string), Telegram webhook registration, and Gmail OAuth token generation.
• .github/workflows/integration-test-weekly.yaml — adds two jobs: e2e-messaging-deploy
  (opt-in via the provision_e2e_messaging dispatch input; applies Terraform in place, no destroy)
  and e2e-messaging-test (probes the deployment, re-registers the Telegram webhook, runs the
  suite). Runs on the weekly schedule + manual dispatch. Must run on Linux (the image vendors deps
  at build time; macOS wheels crash the linux/amd64 container).
• docs/specs/590-messaging-integration-e2e-tests/ — design/spec/plan documents.
• e2e/README.md — full one-time setup, per-platform account provisioning, deploy, webhook
  registration, run instructions, an env-var reference table, and troubleshooting.

Testing

• Manual testing completed — Slack, Telegram, and Gmail verified end-to-end against the live
  ECS deployment; WhatsApp, Messenger, and Instagram verified manually via CloudWatch logs (their
  inbound legs can't be automated).
• New tests added for changes
• Unit tests pass locally
• Integration tests pass locally

Checklist

• My code follows the project's style guidelines
• I have performed a self-review of my code
• I have made corresponding changes to the documentation
• I have added tests that prove my fix is effective or that my feature works

Additional Notes

• Not wired into per-PR or nightly runs — weekly schedule + manual dispatch only, to keep the
  real-account traffic and deployment cost bounded.
• Several platforms have hard automation ceilings imposed by the vendors (no API to send to a
  Page/account as a user; two WhatsApp Cloud API test numbers can't message each other). Those
  are documented per-platform in e2e/README.md, and their automated tests skip by default behind
  explicit E2E_*_AUTOMATED=1 flags.
• Meta test-number and Gmail "Testing"-status tokens are short-lived; the README calls out using
  system-user tokens / "In production" consent status for a long-lived deployment.

[amithad]

Overall: a solid, unusually well-documented harness. The factual claims in the README/docstrings were verified against the integration source and they all check out (Slack fallback string "Error handling your request." at slack_chat.py:183, Telegram fallback strings, ak.api.<platform> logger names, "Message sent successfully"/"Error handling message" log lines, AK_GMAIL__* env reads in gmail_chat.py, _TeamsConfig absence, gateway_endpoints/agent_invoke_url module surface, remote-state bucket convention).

Findings: 2 questions, 2 suggestions, 2 nits.

• [question] Spec documents are missing: the PR description's Changes Made lists docs/specs/590-messaging-integration-e2e-tests/ (design/spec/plan), and issue #590 links to docs/specs/590-messaging-integration-e2e-tests/design.md on develop — but no docs/specs/590-* path exists in this PR's diff or on develop (the issue link 404s). Were the spec docs approved somewhere else and dropped from the branch? Per the staged spec process they should land in the repo; as-is the implementation can't be checked against its design.
• [suggestion] AGENTS.md's repo map doesn't mention the new top-level e2e/ tree — please add a line (like the examples/ / use-cases/ entries) so the map stays accurate for agents and humans navigating the repo.
• CI note: the failing checks look infra-flaky rather than caused by this PR (script-tests log ends with 26 passed, 0 failed, lint was cancelled, setup's log blob is gone; the PR touches no ak-py code) — worth a rerun to confirm.
• Scope note: the PR intentionally exceeds issue #590's first cut (Slack+Telegram, no CI wiring) by adding Gmail/WhatsApp/Messenger/Instagram and the weekly jobs; this is documented in the description, so no action needed — just confirming it was noticed, not missed.

[Copilot]

Pull request overview

This PR introduces a new e2e/ harness that deploys a long-lived ECS-backed Agent Kernel instance and runs true end-to-end messaging integration tests against real platform accounts, plus a weekly GitHub Actions workflow to drive deploy/test runs.

Changes:

• Added deployable E2E app (e2e/app/) with Terraform-based ECS + API Gateway deployment (e2e/app/deploy/).
• Added pytest-based E2E test suite (e2e/tests/) for Slack, Telegram, Gmail, plus opt-in/manual log-based verification for WhatsApp/Messenger/Instagram.
• Extended the weekly integration workflow to optionally provision the deployment and run the E2E suite.

Reviewed changes

Copilot reviewed 28 out of 31 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
e2e/tests/uv.lock	Pinned dependency lockfile for the E2E test harness environment.
e2e/tests/pyproject.toml	Defines the E2E test harness dependencies and formatting config.
e2e/tests/conftest.py	Shared test constants and require_env() helper to skip when creds are missing.
e2e/tests/test_slack.py	Slack round-trip test: user-token send + threaded bot reply read-back.
e2e/tests/test_telegram.py	Telegram round-trip test via Telethon user session + bot reply read-back.
e2e/tests/test_gmail.py	Gmail round-trip test via Gmail API thread polling for bot reply.
e2e/tests/test_whatsapp.py	WhatsApp opt-in test: sends template message and verifies via CloudWatch logs.
e2e/tests/test_messenger.py	Messenger opt-in/manual verification via CloudWatch logs.
e2e/tests/test_instagram.py	Instagram opt-in/manual verification via CloudWatch logs.
e2e/tests/scripts/telegram_login.py	Helper to generate Telethon StringSession for the tester user.
e2e/tests/scripts/set_telegram_webhook.py	Helper to register the deployed Telegram webhook URL/secret.
e2e/tests/scripts/gmail_login.py	Helper to generate base64 OAuth token payloads for Gmail accounts.
e2e/tests/.gitignore	Ignores local venv/cache/session artifacts for the E2E tests.
e2e/README.md	End-user documentation for provisioning accounts, deploying, and running E2E tests.
e2e/app/pyproject.toml	Defines the deployable app dependencies (agentkernel extras + aiohttp).
e2e/app/app.py	E2E app entrypoint: registers one OpenAI agent and wires integrations/handlers.
e2e/app/config.yaml	Runtime config for integrations (agent routing, Gmail polling config, logging).
e2e/app/build.sh	Local helper to build the app environment (including local dist install option).
e2e/app/.env.example	Example environment file for deployment credentials/config.
e2e/app/.gitignore	Ignores local env/dist/venv artifacts for the deployable app.
e2e/app/deploy/main.tf	Terraform module wiring for ECS + API Gateway endpoints + env var injection.
e2e/app/deploy/variables.tf	Terraform variables for platform credentials and optional integrations.
e2e/app/deploy/outputs.tf	Exposes webhook URLs for each integration endpoint via terraform outputs.
e2e/app/deploy/terraform.tfvars	Default aliases/region/module_name values for the deployment.
e2e/app/deploy/backend.tf	Configures remote S3 state for the long-lived E2E deployment.
e2e/app/deploy/Dockerfile	Minimal image entrypoint for the packaged app bundle.
e2e/app/deploy/deploy.sh	Local deploy helper: builds dist bundle, applies terraform, waits for ECS stable.
e2e/app/deploy/.terraform.lock.hcl	Provider lockfile for terraform init reproducibility.
e2e/app/deploy/.gitignore	Ignores terraform state artifacts locally.
.github/workflows/integration-test-weekly.yaml	Adds E2E deploy + test jobs (manual dispatch + weekly schedule).

Files not reviewed (1)

• e2e/app/deploy/.terraform.lock.hcl: Generated file

Suppressed comments (2)

e2e/app/app.py:67

• Messenger is treated as “optional” but this guard only checks AK_MESSENGER__ACCESS_TOKEN. The Messenger handler requires both access_token and verify_token at construction time, so a partial config will crash app startup instead of skipping the integration.

    # Messenger is optional: the handler refuses to construct without credentials.
    if os.environ.get("AK_MESSENGER__ACCESS_TOKEN"):
        from agentkernel.messenger import AgentMessengerRequestHandler

e2e/app/app.py:74

• Instagram is treated as “optional” but this guard only checks AK_INSTAGRAM__ACCESS_TOKEN. The Instagram handler requires both access_token and verify_token at construction time, so a partial config will crash app startup instead of skipping the integration.

    # Instagram is optional: the handler refuses to construct without credentials.
    if os.environ.get("AK_INSTAGRAM__ACCESS_TOKEN"):
        from agentkernel.instagram import AgentInstagramRequestHandler

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[amithad]

Publish pipeline should be updated to change the agent kernel dependency version once a new version is published. Just like how we do it for examples.

Then the test.yaml should be updated to upgrade uv.lock of the harness. The reason for the decoupling is, GitHub actions can't immediately know when a new version is upgraded