WIP: rebuild cache on MapRef, fix cancellation defects by stasimus · Pull Request #369 · evolution-gaming/scache

stasimus · 2026-07-31T17:15:08Z

Rebuilds LoadingCache on cats.effect.std.MapRef: per-key CAS instead of whole-map Ref, cancelable loads with cleanup, ExpiringCache evicts stale Loading entries. Fixes four proven defects; defect spec now asserts expected behavior and passes. Adds load-test bench (Test/runMain CacheLoadTest). Creation bounds widened Concurrent to Async.

Bench: 8 fibers x 100k ops/fiber, keySpace 10k, median of 3 runs, ops/s.

LoadingCache (single partition)	old (Ref[Map])	new (MapRef)	ratio
getOrUpdate, insert distinct keys	709k	996k	1.40x
getOrUpdate, hit random keys	1.79M	2.37M	1.32x
getOrUpdate, hit single hot key	11.1M	15.7M	1.41x
put, replace random keys	4.42M	5.56M	1.26x
mixed get/put/remove	1.59M	1.90M	1.19x

Cache.loading (partitioned)	old	new	ratio
getOrUpdate, insert distinct keys	1.31M	1.55M	1.18x
getOrUpdate, hit random keys	9.64M	22.2M	2.31x
getOrUpdate, hit single hot key	9.76M	11.3M	1.15x
put, replace random keys	6.67M	8.99M	1.35x
mixed get/put/remove	3.90M	5.88M	1.51x

Cache.expiring (partitioned)	old	new	ratio
getOrUpdate, insert distinct keys	892k	1.34M	1.50x
getOrUpdate, hit random keys	7.62M	10.3M	1.35x
getOrUpdate, hit single hot key	9.88M	14.0M	1.42x
put, replace random keys	6.30M	8.00M	1.27x
mixed get/put/remove	3.59M	4.49M	1.25x

mr-git · 2026-08-04T18:40:04Z

Fixes four proven defects

are the following the above-mentioned defects:

claim 1: loads are cancelable and cancellation cleans up the Loading entry;

claim 2: entries stuck in Loading state are evicted by the expiration routine;

claim 3: waiters on a Loading entry are unblocked when the load is cancelled;

claim 4: operations on distinct keys are independent, no shared-state CAS retries.

stasimus · 2026-08-04T20:34:11Z

Yes, exactly those four, declared and asserted in CacheDefectsSpec.scala, one test per claim. They weren't filed as separate GitHub issues, just documented there.

mr-git · 2026-08-05T07:36:44Z

I am at the beginning of review and I understood that I fail to comprehend what happens in new functionality.

ScalaDocs are really required! With explicitly explaining why? part, overall description of envisioned algorithm would be very welcome too!

stasimus · 2026-08-05T10:32:05Z

Added scaladocs on LoadingCache, EntryMap, EntryState, the cache operations and the ExpiringCache eviction routine, describing the algorithm and the four defects this change fixes.

P1, contention on shared state. The state used to be one Ref[F, Map[K, EntryRef]], so every insert or removal of any key CAS-ed the same Ref, and a getOrUpdate of one key lost its CAS whenever an unrelated key was written. Sustained writes elsewhere starved it, which is what MaxRetries = 10000 and IllegalStateException("extreme contention") were guarding, i.e. contention turned into a user visible failure. Every write also copied the whole map. Now each key has its own Ref over a ConcurrentHashMap, unrelated keys never interfere and the retry limit is gone.

P2, cancellation poisoned the key. The value computation ran unmasked inside the retry loop with no onCancel, so cancelling getOrUpdate left Loading(deferred) in the map with the deferred never completed: the key stayed unusable and every waiter blocked forever, and a value computed just as cancellation hit was leaked. Now state transitions are masked, and cancellation unlinks the key, completes the deferred with CancelledError and releases the value if one was produced.

P3, expiration ignored Loading. removeExpiredAndCheckSize only inspected Value states, so an entry whose load never completes was never evicted. Now loads that run longer than the expiration interval are evicted and their waiters get ExpiredError.

P4, a stuck load blocked finalization, since clear runs on resource release and waits on Loading entries. P2 and P3 remove both ways of getting stuck there.

CacheDefectsSpec covers all four.

mr-git · 2026-08-05T11:08:40Z

Bench: 8 fibers x 100k ops/fiber, keySpace 10k, median of 3 runs, ops/s.

where is the code for benchmarks?

mr-git · 2026-08-05T11:22:12Z

@stasimus, could we get the PR with failing (and ignored or explicitly waiting for failures with corresponding comments and verbose printouts) unit-tests against "current" (as in master branch)?

I tried to do the quick rollback of implementation in the branch, but new unit-tests do not compile against old implementation.

It also means that this PR might change public API in incompatible way - we must provide the instructions on how to migrate from "old" to "new" APIs!

mr-git · 2026-08-05T07:20:24Z

   *   resources stored in a cache are also released.
   */
-  def loading[F[_]: Concurrent: Parallel: Runtime, K, V]: Resource[F, Cache[F, K, V]] = {
+  def loading[F[_]: Async: Parallel: Runtime, K, V]: Resource[F, Cache[F, K, V]] = {


this will require major version bump

mr-git · 2026-08-05T07:21:52Z

+
+import scala.util.control.NoStackTrace
+
+case object ExpiredError extends RuntimeException with NoStackTrace


do we need RuntimeException part?

would following work?:

Suggested change

case object ExpiredError extends RuntimeException with NoStackTrace

case object ExpiredError extends NoStackTrace

mr-git · 2026-08-05T07:25:31Z

+    def removeExpiredAndCheckSize(
+      entryMap: LoadingCache.EntryMap[F, K, E],
+      cache: Cache[F, K, E],
+      loadingSince: Ref[F, Map[K, (DeferredE, Timestamp)]],


I cannot grasp why we need (Deferred[F, Either[Throwable, LoadingCache.Entry[F, E]]], Timestamp)

what is the point in having "failure reason AND timestamp"? shouldn't we keep only Either[Throwable, Timestamp]? I guess that we can assume failed loading as "expired", thus due for removal.

mr-git · 2026-08-05T07:42:27Z

+      entryMap: LoadingCache.EntryMap[F, K, E],
+      cache: Cache[F, K, E],
+      loadingSince: Ref[F, Map[K, (DeferredE, Timestamp)]],


Why cannot we use full names, like Key and Entry in type-parameters?

We have 3 structures here: entryMap, cache and loadingSince - does it mean that we have to manually keep their content in sync? Why do we need them all?

mr-git · 2026-08-05T08:47:31Z

-          entryRefs <- ref.get
-          result <- if (entryRefs.size > maxSize) drop(entryRefs) else ().pure[F]
+          size <- entryMap.size
+          result <- if (size > maxSize) entryMap.entries.flatMap(drop) else ().pure[F]


somehow I like the form: Async[F].whenA() more, like:

Suggested change

result <- if (size > maxSize) entryMap.entries.flatMap(drop) else ().pure[F]

result <- Async[F].whenA(size > maxSize) { entryMap.entries.flatMap(drop) }

mr-git · 2026-08-05T12:40:15Z

+      ),
+    )
+    for {
+      _ <- IO.println(f"fibers=$fibers, ops/fiber=$opsPerFiber, keySpace=$keySpace")


fibers is not a number of available CPU cores :)

mr-git · 2026-08-05T12:48:52Z

+ * sbt "scache/Test/runMain com.evolution.scache.CacheLoadTest"
+ * }}}
+ */
+object CacheLoadTest extends IOApp.Simple {


feels like this should be in benchmark module, ideally with copy of previous version, otherwise the generated "report" is just a bunch of random values

mr-git · 2026-08-05T12:49:48Z

+          }
+        }
+        _ <- measure("getOrUpdate, hit single hot key") {
+          parRun { (_, _) => cache.getOrUpdate(0)(0.pure[IO]).void }


here are on line :49, why test .getOrUpdate against populated cache, where we know that it is the same as .get (ok, it would return an Option, but it doesn't affect the semantics, does it?)?

mr-git · 2026-08-05T12:56:08Z

+              case 0 => cache.put(key, i).flatten.void
+              case 1 => cache.remove(key).flatten.void
+              case _ => cache.getOrUpdate(key)(i.pure[IO]).void


why we use only these 3 APIs, while Cache exposes more than 3?

mr-git · 2026-08-05T13:01:31Z

+  private def scramble(i: Int): Int = {
+    val h = i * 0x9e3775cd
+    (h ^ (h >>> 16)) & Int.MaxValue
+  }


how is this better than Random.nextInt(n + 1)?

stasimus marked this pull request as draft July 31, 2026 18:23

stasimus changed the title ~~Add failing defect tests for LoadingCache and ExpiringCache~~ WIP: rebuild cache on MapRef, fix cancellation defects Jul 31, 2026

Rebuild cache on MapRef, fix cancellation defects

d47eaa5

stasimus force-pushed the experimenting branch from 226f2e4 to d47eaa5 Compare July 31, 2026 20:02

stasimus closed this Jul 31, 2026

stasimus reopened this Jul 31, 2026

stasimus self-assigned this Aug 4, 2026

stasimus requested a review from edubrovski August 4, 2026 20:24

stasimus marked this pull request as ready for review August 4, 2026 20:24

Document cache algorithm and defects it fixes

df170ad

mr-git reviewed Aug 5, 2026

View reviewed changes


		import scala.util.control.NoStackTrace

		case object ExpiredError extends RuntimeException with NoStackTrace

	case object ExpiredError extends RuntimeException with NoStackTrace
	case object ExpiredError extends NoStackTrace

	result <- if (size > maxSize) entryMap.entries.flatMap(drop) else ().pure[F]
	result <- Async[F].whenA(size > maxSize) { entryMap.entries.flatMap(drop) }

Uh oh!

Conversation

stasimus commented Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mr-git commented Aug 4, 2026

Uh oh!

stasimus commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mr-git commented Aug 5, 2026

Uh oh!

stasimus commented Aug 5, 2026

Uh oh!

mr-git commented Aug 5, 2026

Uh oh!

mr-git commented Aug 5, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

stasimus commented Jul 31, 2026 •

edited

Loading

stasimus commented Aug 4, 2026 •

edited

Loading