Eviction candidates take a long time to identify in large `FlowTable`s

[FelixMcFelix]

At high levels of table occupancy, we're spending more time than expected trying to identify evictable table entries in eviction_priority and related. To some extent this is expected as we're performing a linear scan over all the resident entries -- we haven't identified a valid strategy for dealing with freshness/correctness of e.g. a small secondary eviction likelihood pool per FlowTable. However, the cost (and impact on pps rate) appears to suddenly increase when the occupancy goes past a table-dependent threshold.

A hypothesis I had on this is covered by #1039 -- I had originally thought that we could be tracking (many) more children than anticipated per LFT entry, which would greatly inflate the work done for each. But from racklette testing that does not appear to be the case. Other than that I'm not convinced yet that there's anything accidentally quadratic going on under the hood that could explain a greater-than-linear cost in $N$.

We've been seeing this degradation in XDE. I'll see whether similar spikes in performance present themselves in userland as a next step.

[FelixMcFelix]

opte/lib/opte/src/engine/flow_table.rs

Lines 180 to 188 in 16ef8c6

	for maybe_child in &*self.lifetime.children.read() {
	if let Some(child) = maybe_child.0.upgrade() {
	match (best_prio, child.eviction_priority(now)) {
	(None, a) => best_prio = a,
	(Some(_), None) => {}
	(Some(old), Some(new)) => best_prio = Some(new.min(old)),
	}
	}
	}

This presence of this loop appears to make up for ~85% of the cost of all packet processing, which is a little surprising. Mainly because we see at most 2/3 parents per entry, and we don't have any recursion by design. That leaves us with two points to think on:

• Bounding the number of eviction candidates we're willing to consider per table.
• Reducing the cost of calculating the true eviction priority for each candidate.