Conversation
Motivation: The `ec2ClientUsed` / `ecsClientUsed` flags added in apache#907 are set at the top of the lazy client initialiser, before the client exists. A `lazy val` whose initialiser throws stays uninitialised and is re-evaluated on the next access, so a failed client build (for example an unusable `client-config` FQCN, or an unresolvable region/credential chain) leaves the flag `true` with no client. The `PhaseServiceUnbind` task then re-enters the initialiser and throws again - exactly the shutdown-time client creation the flags were meant to prevent. The flag is also set before `builder.build()` returns, so shutdown can observe `true` while the client is still being constructed. The two collection changes in apache#907 were not the optimisations they were described as. `Iterator.toSeq` is `immutable.Seq.from`, so `.grouped(100).toSeq` materialises exactly like the `.toList` it replaced, and `java.util.List.asScala` yields a strict `mutable.Buffer`, so the EC2 `flatMap`/`map` chain still builds two intermediate buffers. Both paginated loops also append with `accumulator ++ page`, which is quadratic in the number of pages. Modification: Replace both flags with an `AtomicReference` that is set only after `builder.build()` succeeds; the shutdown task closes the client it finds there and does nothing when it is null, never touching the lazy val. Use a lazy `view` in the EC2 `getInstances` chain, stream `describeTasks` straight off the `grouped` iterator, and accumulate both paginated loops into a `Vector` so appends are amortised constant time. Add `Ec2ClientShutdownSpec` covering both shutdown paths, and `EcsResolveTasksSpec` covering `listTasks` pagination and `describeTasks` batching. `resolveTasks` is relaxed from `private` to `private[ecs]` so the latter can drive it. Result: Shutdown never creates, or re-attempts to create, an AWS client. Pagination is linear rather than quadratic, and the EC2 chain allocates one list instead of three. Tests: - sbt "discovery-aws-api/test" - 8 succeeded, 0 failed - sbt "discovery-aws-api/testOnly *Ec2ClientShutdownSpec" against the pre-fix Ec2TagBasedServiceDiscovery - "should not re-attempt EC2 client creation during shutdown when creation previously failed" FAILED (2 attempts, not 1) - sbt "+discovery-aws-api/mimaReportBinaryIssues" - success - sbt "discovery-aws-api/scalafmt" "discovery-aws-api/Test/scalafmt", sbt headerCreateAll - clean. `sortImports` is not a task in this build. - Not run against live AWS - no account available; the specs stub AmazonECS and ClientConfiguration instead. References: Refs apache#907
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Follow-up to #907.
The
ec2ClientUsed/ecsClientUsedflags added in #907 are set at the top of the lazy client initialiser, before the client exists:A Scala
lazy valwhose initialiser throws stays uninitialised and is re-evaluated on the next access. So ifbuilder.build()fails, or the customclient-configFQCN cannot be instantiated (throw new Exception(s"Could not create instance of '$fqcn'", ex)), we end up withec2ClientUsed == trueand no client. ThePhaseServiceUnbindtask then callsec2Client.shutdown(), re-enters the initialiser and throws again — which is precisely the shutdown-time client creation the flag was added to avoid (see @He-Pin's review comment on #907). Setting the flag beforebuilder.build()returns also leaves a window where shutdown observestruewhile another thread is still constructing the client.Separately, the two collection changes in #907 are not the optimisations the description claimed:
taskArns.grouped(100).toSeq—IterableOnceOps.toSeqisimmutable.Seq.from(this), i.e. strict, andListis the defaultimmutable.Seq. This materialises exactly like the.toListit replaced.getInstances—java.util.List.asScalayields a strictmutable.Buffer, not a lazy view, soflatMapthenmapbuild two intermediate buffers before.toList. No fewer allocations than the original.Both paginated loops also do
accumulator ++ pageper page (Listin EC2,immutable.Seqin ECS), which is O(pages²) copying.Modification
@volatile varflags with anAtomicReferencethat is populated afterbuilder.build()succeeds. The shutdown task closes whatever client it finds there and returnsDonewhen it is null, so it never touches the lazy val.getInstances: use.viewsoflatMap/mapare genuinely lazy, and accumulate into aVector.describeTasks:flatMapstraight off thegroupediterator instead of materialising the groups first.listTaskArns: accumulate into aVectorso appends are amortised constant time.Ec2ClientShutdownSpecandEcsResolveTasksSpec.EcsServiceDiscovery.resolveTasksis relaxed fromprivatetoprivate[ecs]so the latter can drive it — a new method in bytecode, so MiMa-clean.Result
Shutdown never creates, or re-attempts to create, an AWS client — including after a failed creation. Pagination is linear in the number of pages rather than quadratic, and the EC2 chain allocates one list instead of three.
Tests
sbt "discovery-aws-api/test"— 8 succeeded, 0 failed.sbt "discovery-aws-api/testOnly *Ec2ClientShutdownSpec"withEc2TagBasedServiceDiscovery.scalareverted tomain— "should not re-attempt EC2 client creation during shutdown when creation previously failed" FAILED (2 configuration attempts instead of 1). Passes with the fix.sbt "+discovery-aws-api/mimaReportBinaryIssues"— success.sbt "discovery-aws-api/scalafmt" "discovery-aws-api/Test/scalafmt"andsbt headerCreateAll— clean.sortImportsis not a task in this build (onlyjavafmtSortImports); no Java sources changed.AbstractAmazonECSandClientConfigurationso no credentials, region or network access are needed.References
Refs #907