Skip to content

UNOMI-998: Cover a 3.0 to 4.0 migration in the integration tests - #910

Merged
sergehuber merged 7 commits into
masterfrom
UNOMI-998-migrate-from-30
Oct 8, 2026
Merged

sergehuber merged 7 commits into
masterfrom
UNOMI-998-migrate-from-30

Conversation

@sergehuber

@sergehuber sergehuber commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Plain-language summary

The existing migration test starts from a 1.6 snapshot with the full feature set already installed. That setup did not catch the 3.0 upgrade problems fixed in UNOMI-996 and UNOMI-997. This PR adds an integration test that seeds a small synthetic 3.0 data set over HTTP, runs unomi:migrate 3.0.0 before Unomi starts, then checks the post-migration shape.

What changed

  • New Migrate30ToCurrentVersionIT, registered first in AllITs / CorePersistenceITs so it runs before the 1.6 snapshot restore.
  • Seeds systemitems (shipped definition + user rule/segment), profile, geoname, cluster node and Salesforce configuration indices, then migrates.
  • Asserts: default tenant exists; user content stays out of system; geonames are system; cluster nodes and unknown indices use the configured tenant; every document has a tenant; the 4.0 booleanCondition wins over the seeded 3.0 marker; migration-mappings are present in the shell-commands bundle; the API key file names the real headers.
  • Fix initialize_tenant_and_audit_fields.painless: during the migration reindex ctx._index is the -cloned source index, so the -systemitems / -geonameentry checks never matched. System items whose id lacked the _itemtype suffix did not get it, and geonames were not skipped for audit fields. The suffix is now stripped before the checks.
  • The raw checks on what the migration wrote (shipped definition under system, default tenant document) run before Unomi starts, since startup redeploys definitions and BaseIT creates the test tenant.
  • Both migration ITs clear data/migration/history.json first, so Migrate16xToCurrentVersionIT no longer skips the 4.0.0 steps after this test has run.
  • OpenSearchITBackend mirrors its connection settings into the org.apache.unomi.elasticsearch.* properties, which org.apache.unomi.migration.cfg reads.
  • On Elasticsearch, where Migrate16xToCurrentVersionIT restores its snapshot next, this test leaves Unomi stopped and only checks the migrated indices; the two checks that need a running Unomi (tenant via tenantService, 4.0 booleanCondition winning) run on OpenSearch only.
  • Correct the API key file text in migrate-4.0.0-10-tenantInitialization.groovy (X-Unomi-Api-Key / Basic auth) so it matches what Unomi actually accepts.

Test plan

  • ./build.sh --skip-unit-tests --no-javadoc --no-clean --integration-tests --single-test org.apache.unomi.itests.migration.Migrate30ToCurrentVersionIT
  • Full Elasticsearch IT suite locally
  • Full OpenSearch IT suite locally
  • CI green on Elasticsearch and OpenSearch jobs

Add a synthetic 3.0 seed that runs unomi:migrate before Unomi starts, and
correct the tenant API key file header names written by the migration.
Clear open context-* indices before the 1.6 snapshot restore, and mirror
OpenSearch connection settings into the elasticsearch.* keys that
migration.cfg uses so unomi:migrate does not hit localhost:9200.
@sergehuber

Copy link
Copy Markdown
Contributor Author

CI diagnosis and follow-up push (fbaff8fde):

Elasticsearch job: Migrate30ToCurrentVersionIT passed, then Migrate16xToCurrentVersionIT failed restoring the 1.6 snapshot because open context-* indices from Migrate30 were still present. That aborted the 1.6 setup and cascaded into many matchAllCondition type not found failures.

OpenSearch job: unomi:migrate connected to localhost:9200 (migration.cfg default). OpenSearch ITs only set org.apache.unomi.opensearch.addresses, so migrate never reached the real cluster after seeding.

Fix: delete existing context-* indices before the 1.6 snapshot restore; mirror OpenSearch connection settings into the elasticsearch.* keys that org.apache.unomi.migration.cfg resolves.

sergehuber and others added 5 commits October 7, 2026 17:10
Clear the migration history between migration ITs so the 1.6 run no
longer skips the 4.0.0 steps, and check what the migration itself wrote
before Unomi starts and redeploys definitions and the test tenant.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…clone

The reindex script sees the "-cloned" source index in ctx._index, so the
systemitems and geonameentry checks never matched during a real
migration. Strip the suffix before testing the index name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Started by the 3.0 migration test, Unomi kept running on indices that
the 1.6 test then deletes and restores, and never redeployed its
definitions and schemas. Where the snapshot migration runs next, the
3.0 test now only checks the migrated indices.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
GitHub's runner image now ships Maven 3.10.0, on which extension 2.5.0
fails with an internal error before the build starts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sergehuber
sergehuber merged commit 670319c into master Oct 8, 2026
8 of 9 checks passed
@sergehuber
sergehuber deleted the UNOMI-998-migrate-from-30 branch October 8, 2026 10:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant