Warmpool & Deletion Benchmarks - #6899
Conversation
f66a61f to
a524471
Compare
|
/cc @pmandewalkar |
|
|
||
| # Drain to 0 for clean measurement (moved from Prepare for sweep compatibility) | ||
| utils.DrainWarmPool(ns, warmpool_name, label, timeout=int(drain_timeout)) | ||
| time.sleep(2) |
There was a problem hiding this comment.
Saw these a couple times in the test, could we maybe use a more deterministic method to avoid waiting too long/too little if the system resolves earlier/later? Maybe a polling loop or kubectl command checking the state of what we are waiting for.
| logging.info("Provisioning... %d/%d (%.0f%%)", running, batch_size, pct) | ||
| if running >= batch_size: | ||
| break | ||
| time.sleep(3) |
There was a problem hiding this comment.
Since we have a configurable poll_interval that is used during the deletion loop on line 258, should we replace this hardcoded time.sleep(3) with time.sleep(poll_interval) for consistency?
| # Pods we never saw disappear (stuck) get the full drain time | ||
| for pn in pod_names_before: | ||
| if pn not in pod_gone_times: | ||
| pod_gone_times[pn] = total_drain_time |
There was a problem hiding this comment.
Is there a way we are differentiating pods in the edge case that take the whole drain time to successfully delete vs pods that are stuck and are assigned the same drain time? Maybe we could track a count of stuck pods and push it through as an additional metric (like k8s_deletion_stuck_pods_count) so we can separate that?
|
|
||
| ip_after = _CountAllocatedIPs(ns, label) | ||
| deletion_rate = ( | ||
| (len(pod_names_before) / total_drain_time) if total_drain_time > 0 else 0 |
There was a problem hiding this comment.
If pods are left stuck, when we use len(pod_names_before) will it assume all pods were successfully deleted and inflate the deletion_rate? Since we are already tracking the exact number of deleted pods inside the loop on line 244, could we use the deleted variable as the numerator here?
| provision_start = time.monotonic() | ||
| _PatchReplicas(ns, warmpool_name, batch_size) | ||
|
|
||
| deadline = time.time() + provision_timeout |
There was a problem hiding this comment.
Would it be better to stick to time.monotonic() rather than using and comparing different measures? Since time.time() could be affected by system clock adjustments, it could theoretically make the timeout weird.
| deadline = t_scale + threshold_s | ||
| first_pod_time = None | ||
|
|
||
| while time.time() < deadline: |
There was a problem hiding this comment.
Same question as before regarding monotonic() vs time()
pmandewalkar
left a comment
pmandewalkar
left a comment
There was a problem hiding this comment.
Left a few comments regarding clock and hardcoded sleeps.
3 participants
Files: 2 new files
perfkitbenchmarker/linux_benchmarks/kubernetes/agentic/k8s_warmpool_benchmark.pyperfkitbenchmarker/linux_benchmarks/kubernetes/agentic/k8s_deletion_benchmark.pyDescription: Adds two infrastructure lifecycle benchmarks:
Warmpool Scale-Up: Measures how quickly N sandbox pods can be provisioned from zero via the SandboxWarmPool controller. Scrapes pod lifecycle timestamps (creationTimestamp, PodScheduled condition, Ready condition) for time-to-created/scheduled/running percentiles. No agent API needed — interacts directly with the Kubernetes API.
Deletion & Cleanup: Measures bulk deletion efficiency and IP address reclamation. Provisions N pods via SandboxWarmPool, bulk-deletes them by scaling to zero, and tracks per-pod deletion latency and IPAM release timing. Tests whether "Terminating" pods hold IP addresses longer than expected.