insert --watch does not recover from stale IRSA credentials after IAM role recreation

[nmiakushina]

Summary

A long-running ice insert --watch process on EKS can get stuck retrying
S3 operations with 403 Forbidden after its IAM/IRSA environment is
reconciled or the IAM role is recreated. The effective IAM policy is correct,
but the process does not recover until its Kubernetes pod is recreated.

After the pod restart, the same SQS messages and S3 objects are processed
successfully and Iceberg snapshots are committed. This suggests that the
long-running process keeps stale AWS credentials or an S3 client/session that
is not refreshed after the underlying IRSA identity changes.

Environment

• ice: 0.16.0
• Image: cr-enam.altinity.io/altinity/ice:0.16.0
• AWS authentication: IRSA / web identity ServiceAccount annotation
• Java: 21
• Source: S3 ObjectCreated events delivered through SQS
• Destination: Iceberg REST Catalog
• Partitioning: hour(timestamp)

Command shape

The watcher is run as a long-lived process equivalent to:

ice insert \
  -p \
  --force-no-copy \
  --skip-duplicates \
  --partition='[{"column":"timestamp","transform":"hour"}]' \
  's3://<bucket>/cold/metrics-v5/**/*.parquet' \
  --watch='<SQS queue URL>'

Observed behavior

After an infrastructure reconciliation, the watcher continues receiving SQS
messages but fails while reading S3 object metadata:

ERROR c.a.i.c.i.cmd.InsertWatch > Failed to process batch of messages (retry in PT20S)
java.io.IOException: Error processing file(s)
...
Caused by: software.amazon.awssdk.services.s3.model.S3Exception: Forbidden
(Service: S3, Status Code: 403)
...
at software.amazon.awssdk.services.s3.DefaultS3Client.headObject(...)
at org.apache.iceberg.aws.s3.BaseS3File.getObjectMetadata(...)
at org.apache.iceberg.aws.s3.S3InputFile.getLength(...)
at com.altinity.ice.cli.internal.iceberg.parquet.Metadata.readFooter(...)
at com.altinity.ice.cli.internal.cmd.Insert.processFile(...)

The watcher retries every 20 seconds but does not recover by itself.

The live inline role policy was checked while the failure was happening and
contained the required permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:ListBucket"],
      "Resource": [
        "arn:aws:s3:::<bucket>",
        "arn:aws:s3:::<bucket>/cold/metrics-v5/*"
      ]
    }
  ]
}

The failing object was under the allowed cold/metrics-v5/ prefix.

Recovery and evidence

Recreating only the watcher pod fixes the problem without changing the IAM
policy, SQS queue, S3 object, Iceberg table, or watcher arguments:

kubectl rollout restart deployment/vector-metrics-ice-watch-v5

Immediately after restart, the new process successfully processes the queued
objects:

INFO o.a.i.SnapshotProducer > Committed snapshot 6236223900153511381 (MergeAppend)
INFO c.a.i.c.i.cmd.InsertWatch > Received S3 event: ObjectCreated:Put -> s3://<bucket>/cold/metrics-v5/dt=2026-07-27/hour=18/<uuid>.parquet
...
INFO o.a.i.SnapshotProducer > Committed snapshot 197678787631752742 (MergeAppend)

It then drains the accumulated SQS backlog. No Forbidden or
Failed to process messages appear after the restart. --skip-duplicates
prevents already registered files from being added twice.

Timeline from this incident:

• 2026-07-28 13:40:46 UTC: repeated S3 403 Forbidden from headObject
• IAM inline policy verified and found correct
• watcher Deployment restarted; no policy change
• 2026-07-28 14:08:53 UTC: successful MergeAppend snapshot commit
• 2026-07-28 14:09:15 UTC: another successful snapshot commit while draining
  old events

Expected behavior

A long-running watcher using IRSA should recover when temporary web-identity
credentials expire or when the role behind the same ServiceAccount is
reconciled/recreated. It should not require an external pod restart.

At minimum, after an S3 authentication failure the watcher should
invalidate/rebuild the AWS credentials/client state and retry the message with
freshly resolved web-identity credentials.

Suggested investigation

• Verify whether the S3 client/FileIO is constructed with a refreshable
  web-identity credentials provider or with credentials resolved once at
  startup.
• Check whether the S3 client is cached for the entire lifetime of
  InsertWatch without credential invalidation.
• On 401/403, consider rebuilding the S3 client/FileIO and retrying once with
  newly resolved credentials before returning to the normal 20-second retry
  loop.
• Add an integration test that rotates/recreates the IRSA role or otherwise
  invalidates temporary credentials while insert --watch remains running.

Current workaround

Restart the watcher Deployment to force a new web-identity session:

kubectl rollout restart deployment/<ice-watch-deployment>