Skip to content

Only close AWS discovery clients that were actually built - #954

Open
pjfanning wants to merge 1 commit into
apache:mainfrom
pjfanning:aws-discovery-shutdown-fix
Open

pjfanning wants to merge 1 commit into
apache:mainfrom
pjfanning:aws-discovery-shutdown-fix

Conversation

@pjfanning

Copy link
Copy Markdown
Member

Motivation

Follow-up to #907.

The ec2ClientUsed / ecsClientUsed flags added in #907 are set at the top of the lazy client initialiser, before the client exists:

private lazy val ec2Client: AmazonEC2 = {
  ec2ClientUsed = true          // <-- before the client is built
  ...
  builder.build()
}

A Scala lazy val whose initialiser throws stays uninitialised and is re-evaluated on the next access. So if builder.build() fails, or the custom client-config FQCN cannot be instantiated (throw new Exception(s"Could not create instance of '$fqcn'", ex)), we end up with ec2ClientUsed == true and no client. The PhaseServiceUnbind task then calls ec2Client.shutdown(), re-enters the initialiser and throws again — which is precisely the shutdown-time client creation the flag was added to avoid (see @He-Pin's review comment on #907). Setting the flag before builder.build() returns also leaves a window where shutdown observes true while another thread is still constructing the client.

Separately, the two collection changes in #907 are not the optimisations the description claimed:

  • taskArns.grouped(100).toSeqIterableOnceOps.toSeq is immutable.Seq.from(this), i.e. strict, and List is the default immutable.Seq. This materialises exactly like the .toList it replaced.
  • EC2 getInstancesjava.util.List.asScala yields a strict mutable.Buffer, not a lazy view, so flatMap then map build two intermediate buffers before .toList. No fewer allocations than the original.

Both paginated loops also do accumulator ++ page per page (List in EC2, immutable.Seq in ECS), which is O(pages²) copying.

Modification

  • Replace both @volatile var flags with an AtomicReference that is populated after builder.build() succeeds. The shutdown task closes whatever client it finds there and returns Done when it is null, so it never touches the lazy val.
  • EC2 getInstances: use .view so flatMap/map are genuinely lazy, and accumulate into a Vector.
  • ECS describeTasks: flatMap straight off the grouped iterator instead of materialising the groups first.
  • ECS listTaskArns: accumulate into a Vector so appends are amortised constant time.
  • New Ec2ClientShutdownSpec and EcsResolveTasksSpec. EcsServiceDiscovery.resolveTasks is relaxed from private to private[ecs] so the latter can drive it — a new method in bytecode, so MiMa-clean.

Result

Shutdown never creates, or re-attempts to create, an AWS client — including after a failed creation. Pagination is linear in the number of pages rather than quadratic, and the EC2 chain allocates one list instead of three.

Tests

  • sbt "discovery-aws-api/test" — 8 succeeded, 0 failed.
  • Directional check: sbt "discovery-aws-api/testOnly *Ec2ClientShutdownSpec" with Ec2TagBasedServiceDiscovery.scala reverted to main"should not re-attempt EC2 client creation during shutdown when creation previously failed" FAILED (2 configuration attempts instead of 1). Passes with the fix.
  • sbt "+discovery-aws-api/mimaReportBinaryIssues" — success.
  • sbt "discovery-aws-api/scalafmt" "discovery-aws-api/Test/scalafmt" and sbt headerCreateAll — clean. sortImports is not a task in this build (only javafmtSortImports); no Java sources changed.
  • Not exercised against live AWS — no account available. The specs stub AbstractAmazonECS and ClientConfiguration so no credentials, region or network access are needed.

References

Refs #907

Motivation:
The `ec2ClientUsed` / `ecsClientUsed` flags added in apache#907 are set at the top of
the lazy client initialiser, before the client exists. A `lazy val` whose
initialiser throws stays uninitialised and is re-evaluated on the next access,
so a failed client build (for example an unusable `client-config` FQCN, or an
unresolvable region/credential chain) leaves the flag `true` with no client. The
`PhaseServiceUnbind` task then re-enters the initialiser and throws again -
exactly the shutdown-time client creation the flags were meant to prevent. The
flag is also set before `builder.build()` returns, so shutdown can observe
`true` while the client is still being constructed.

The two collection changes in apache#907 were not the optimisations they were
described as. `Iterator.toSeq` is `immutable.Seq.from`, so `.grouped(100).toSeq`
materialises exactly like the `.toList` it replaced, and `java.util.List.asScala`
yields a strict `mutable.Buffer`, so the EC2 `flatMap`/`map` chain still builds
two intermediate buffers. Both paginated loops also append with `accumulator ++
page`, which is quadratic in the number of pages.

Modification:
Replace both flags with an `AtomicReference` that is set only after
`builder.build()` succeeds; the shutdown task closes the client it finds there
and does nothing when it is null, never touching the lazy val. Use a lazy `view`
in the EC2 `getInstances` chain, stream `describeTasks` straight off the
`grouped` iterator, and accumulate both paginated loops into a `Vector` so
appends are amortised constant time.

Add `Ec2ClientShutdownSpec` covering both shutdown paths, and
`EcsResolveTasksSpec` covering `listTasks` pagination and `describeTasks`
batching. `resolveTasks` is relaxed from `private` to `private[ecs]` so the
latter can drive it.

Result:
Shutdown never creates, or re-attempts to create, an AWS client. Pagination is
linear rather than quadratic, and the EC2 chain allocates one list instead of
three.

Tests:
- sbt "discovery-aws-api/test" - 8 succeeded, 0 failed
- sbt "discovery-aws-api/testOnly *Ec2ClientShutdownSpec" against the pre-fix
  Ec2TagBasedServiceDiscovery - "should not re-attempt EC2 client creation
  during shutdown when creation previously failed" FAILED (2 attempts, not 1)
- sbt "+discovery-aws-api/mimaReportBinaryIssues" - success
- sbt "discovery-aws-api/scalafmt" "discovery-aws-api/Test/scalafmt",
  sbt headerCreateAll - clean. `sortImports` is not a task in this build.
- Not run against live AWS - no account available; the specs stub AmazonECS and
  ClientConfiguration instead.

References:
Refs apache#907
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant