Skip to content

UNOMI-1000: Run background tasks for every tenant - #911

Merged
sergehuber merged 7 commits into
masterfrom
UNOMI-1000-background-tasks-tenants
Oct 9, 2026
Merged

sergehuber merged 7 commits into
masterfrom
UNOMI-1000-background-tasks-tenants

Conversation

@sergehuber

@sergehuber sergehuber commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Plain-language summary

Several background jobs in 4.0 only saw the internal "system" tenant. Purge, daily segment/scoring updates, rule statistics refresh and router startup after a restart skipped real tenants (including "default", where migrated data lives). Other jobs such as rule refresh already looped over every tenant. This PR applies the same pattern, and fixes the problems that running per tenant exposed.

What changed

Background tasks

  • Profile purge, segment date recalculation and rule statistics refresh keep the outer executeAsSystem for permissions, then run under each tenant (system included) through a shared executeForEachTenant helper in AbstractMultiTypeCachingService. One tenant's failure does not stop the others, and the context is restored afterward.
  • The purge and segment tasks now fail when any tenant failed or was skipped by an interruption, naming those tenants, instead of reporting success. The scheduler retries a failed task (3 times, a minute apart, by default) for every tenant; both tasks are safe to repeat.
  • The date-relative recalculation includes the segments and scoring plans a tenant inherits from the system tenant, since those apply to the tenant's profiles too.
  • Rule statistics: syncRuleStatistics() no longer copies the system tenant's statistics into each tenant's in-memory map. Once the sync ran per tenant, that copy made a tenant save system statistics under its own tenant id and the system counts were lost. Reads already fall back to the system tenant. The sync skips tenants with nothing to sync.
  • The system persistence filter is not widened.

Router

  • Startup starts the recurrent import/export routes of every tenant, so recurring configs survive a restart without a manual save. Each route is added on its own once the Camel context runs: a configuration whose route cannot be built is handed to the route refresh (retried, then marked ROUTE_CREATION_FAILED) and does not keep the routes of other configurations or tenants from starting.
  • A tenant whose configurations cannot be loaded at startup, or a tenant listing that fails, no longer leaves routes missing until a restart: the refresh timer retries with a delay that doubles up to one minute, and builds the routes through the normal refresh once the tenant can be loaded.
  • Camel route ids of recurrent configurations are now <tenantId>:<configId> (new RouteIds in router-api). Configuration ids are only unique within a tenant, so two tenants using the same id previously shared one route id, and updating or deleting one tenant's configuration removed the other tenant's route.
  • API change: IRouterCamelContext.killExistingRoute, updateProfileImportReaderRoute and updateProfileExportReaderRoute take the tenant id as a new first parameter. RouterCamelContext is the only implementation and caller in this repository; a plugin calling these methods has to pass the tenant.
  • The route completion processors recover the configuration id from the route id. The import one now logs an error and returns when the configuration cannot be found, as the export one did, instead of throwing a NullPointerException.
  • A tenant that cannot be switched to during a route refresh no longer keeps the other tenants' refreshes of that tick from being processed; its own queued refreshes are retried on the next ticks and given up on after five attempts.
  • Removing a route also removes a definition left behind by a route that failed to start. RouteIds.of refuses a tenant id containing a colon.

Documentation

  • migrate-3.0-to-4.0.adoc: new section on background tasks and recurrent routes running for every tenant (first purge after upgrade, task failure reporting, route id format, IRouterCamelContext change).

Integration tests

  • New BackgroundTenantTasksIT (registered in AllITs and CorePersistenceITs): statistics of a tenant rule are persisted for that tenant; a system date-relative segment is applied to a tenant profile by the recalculation; two tenants using the same import configuration id each get their own route, and deleting one keeps the other.

  • The scheduled purge is switched off in the IT container. Now that it reaches every tenant, it deleted the years-old snapshot data of the migration ITs seconds after startup (the Migrate16xToCurrentVersionIT failure on the first CI run of this PR). No IT relies on the scheduled purge; the purge ITs call the purge methods directly.

  • The Camel route helpers in BaseIT take the tenant id and the configuration id.

Tests

  • BackgroundTenantTasksTest: purge (profiles, sessions, events), segment recalculation (including a system-owned segment) and rule statistics are checked on data in two tenants through the real scheduled executors; one tenant failing is reported and leaves the others processed; an interrupted run is not reported as done; system statistics stay in the system tenant over several ticks.
  • AbstractMultiTypeCachingServiceTest: executeForEachTenant directly (every tenant plus system, a failing tenant, an interruption, both callback outcomes).
  • RouterCamelContextTenantLoadTest: the real route setup builds the routes of two tenants and the system tenant sharing the same configuration ids; removing one tenant's route keeps the other's; a configuration whose route cannot be built does not block the others and ends up marked failed; startup failures are retried with back-off; a queued removal wins over the startup retry; refreshes of an unswitchable tenant are retried then given up on.
  • RouteIdsTest, RouteCompletionProcessorTest: route id round trip (including colons in configuration ids) and completion through a tenant-qualified route id.

Known limits

  • The route completion processors still load and save the configuration in the tenant context left by an earlier step of the route; they do not switch tenant themselves.
  • A failing tenant fails the purge or segment task as a whole, so the scheduler's retries re-run the work for every tenant.

Test plan

  • mvn -pl services-common,services,extensions/router/router-api,extensions/router/router-core clean install (unit tests: services 820, services-common 51, router-core 99)
  • Integration tests green on Elasticsearch
  • Integration tests green on OpenSearch

🤖 Generated with Claude Code

sergehuber and others added 7 commits October 7, 2026 13:36
Profile purge, segment date recalculation, rule statistics refresh and
router startup only saw the system tenant. Walk each tenant with
executeAsTenant (restoring context afterward) instead of widening the
system persistence filter.
Disable the scheduled profile purge in ITs so years-old snapshot data is
not deleted once purge walks every tenant. Also harden tenant task
execution and use tenant-scoped router route IDs.
A tenant that cannot be switched to during a route refresh no longer
drops the refreshes already taken off the queue for the other tenants,
nor skips the export pass of that tick.

Also quiet the startup retry (stack trace once per tenant), stop the
purge and segment tasks from logging "executed" after a tenant failed,
log a missing export configuration as an error like the import one, and
tighten two tests that could pass for the wrong reason.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…o end

Router startup adds each recurrent route on its own once the Camel
context runs, so a configuration whose route cannot be built is handed
to the route refresh instead of stopping the routes of every tenant.
Removing a route also removes a definition left by a failed start, and
the refresh retries, then gives up on, a tenant it cannot switch to
rather than dropping its queued updates.

The date-relative recalculation now includes the segments and scoring
plans inherited from the system tenant, which apply to every tenant's
profiles. The rule statistics sync skips tenants with nothing to sync.

Adds BackgroundTenantTasksIT (tenant rule statistics, system segment
applied to tenant profiles, same import configuration id in two
tenants), unit tests for the per-tenant helper, session and event purge
and the router failure paths, and an upgrade note in the 3.0 to 4.0
migration guide.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Removing what a failed attempt left in the Camel context no longer keeps
the configuration from being marked ROUTE_CREATION_FAILED when the
refresh gives up. A tenant that cannot be switched to logs its stack
trace once instead of once per queued configuration, and the related
error messages name what actually failed.

Covers the statistics sync skipping tenants with nothing to sync, makes
the rule statistics IT wait until nothing is left to flush, keeps the
routes IT cleanup from hiding the test's own failure, and notes in the
migration guide that the migration tenant identifier must follow the
tenant identifier rules.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A rule only applies to events of its own scope, so the event sent by
ruleStatisticsOfATenantRuleArePersisted never fired the rule and the
test failed before reaching the statistics refresh it is about.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sergehuber
sergehuber merged commit 5a263b8 into master Oct 9, 2026
7 checks passed
@sergehuber
sergehuber deleted the UNOMI-1000-background-tasks-tenants branch October 9, 2026 13:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant