Repository navigation
Optimize: apply loop with backup and undo - #2372
Merged
Merged
Conversation
…nch (#2356) * Shared shader cache: record pipelines with Fossilize, upload on exit, replay the pool before launch Bionic containers get the Fossilize capture layer as an implicit layer. Each launch records into a fresh session database (skipping entries already in the downloaded pool and gamescope's own instance), exit uploads the new databases, and the pre-launch hook downloads the merged pool for the game, dxvk and vkd3d versions and replays it with fossilize-replay so the driver cache is warm. Gated on a new Shared shader cache setting (default on). The layer and replayer binaries land separately; until then every step is a logged no-op. * Shader cache: bundle the bionic Fossilize layer and replayer Built from Fossilize 556da9ff with tools/build-fossilize.sh (NDK r27d, android28, static libc++, 16K pages). The patch makes the layer take the Linux code path on bionic so it honours FOSSILIZE_DUMP_PATH and the app-info filter, and enables the crash-isolated replayer. * Shader cache: fix in-app replay on bionic, filter zink/Xwayland, normalise DXVK/VKD3D keys - Fossilize patch: export replay metadata through memfd_create on bionic so the multi-process replayer starts instead of exiting with 246 - Bundle the layer as libfossilize_layer.so so Android's loader does not scan it; it is still installed into the container as libVkLayer_fossilize.so - Replay succeeds only on exit 0; a failed multi-process run retries once in a single process, and a failed retry is stamped so launches are not delayed again until the pool changes - Blacklist Xwayland and the mesa zink engine in the recording filter - Normalise DXVK/VKD3D keys to the leading version number - Fall back to pool/local.foz when the server has no pool * Shader cache: log the replay skip and stop writing the unused pipeline cache blob The replayer's --on-disk-pipeline-cache left ~4 MB per game that nothing loads; the driver's own disk cache is the warm-up. * Shader cache: name the toggle Upload shader caches and say that off also stops downloads * Restore CRLF line endings in the Turkish strings file * Shader cache: show replay progress as a percentage * Shader cache: key recordings per session, bound transfers, stop on exit - Store the pool key with each recording session and upload the session under that key; sessions without one are dropped. The session being recorded is skipped by uploads and pruning. - Resolve Epic and Amazon games to their store ids (appName, product uuid) instead of the local database ids in the container name. - Keep a trailing letter on vkd3d versions (3.0b). - Cap pool downloads and per-file uploads at 256 MiB. - Cancel the shader cache API calls when the caller is cancelled. - Join the exit upload to CloseSyncTracker so an uninstall waits for it. - Stop the replay and skip the launch when the game is exited while shaders are being prepared. - Treat an unreadable runtime marker as a reinstall. - Build only the layer and replayer in build-fossilize.sh. - French subtitle wording; ShaderCacheKey tests. * Shader cache: per-launch cancel flag, cancellable pool refresh, refresh on 401 - Each launch gets its own cancel flag, set by exit(). The replay, the pool lookup and download, and the launcher's final check before starting the guest all read it, and setup tears the environment down if the flag was set while it was being built. - A 401 from the shader cache API refreshes the account token and retries authorized, falling back to anonymous only when the account is gone. - Retention pruning leaves the session being recorded alone. --------- Co-authored-by: Utkarsh Dalal <utkarsh.dalal@toptal.com>
* Upload FEX code maps after a session * Code maps: stop the upload pass on the first network error On the RP6 each unreachable upload waited the full 30 s connect timeout and the pass retried the whole backlog, so one session's cleanup took 1.5 minutes. * Debug builds can point the API at a real server via debug.gamenative.api_base The debug BASE_URL is the emulator's host loopback, so a debug build on a physical device cannot reach any API; the RP6 code map run timed out on every upload. * Code maps are not analytics: capture on every supported FEX build * Download hosted FEX code caches before launch A launch dependency asks /api/code-caches for the caches compiled for this game, the container's FEX build and this SoC, and puts any that are missing or changed into the prefix's fex-emu/cache/ (verified by size and sha256, written through a temp file). FEX then loads translated code at launch instead of compiling it. The list is re-checked at most every six hours per FEX version, and a failure never blocks the launch. * Code caches are served as gzip: inflate while downloading * Code caches: take a zstd patch when the device holds the previous version A cache for the same FEX build grows as coverage grows. When the hosted entry names the sha256 the device already has as its patch base, only the zstd patch is fetched and applied with the old file as dictionary; a different or missing local version still gets the full gzip. The result is always inflated through a temp file and verified before it replaces the old cache. * Code maps: upload only the blocks this device has not sent before A session on a game uploaded the same map every time. The uploader now keeps a ledger of every (file id, block offset) it has sent for the container and writes each session's map as a delta: the main-executable and library records stay (the compiler resolves file ids through them), only new blocks go up, and a map with no new blocks is dropped. With a hosted cache loaded FEX records only the misses anyway, so those deltas are the new coverage the next cache build needs. * Code maps: count blocks sent earlier in the same pass as sent On the RP6 a backlog of identical maps all went up as fully new because the in-memory set was read once and only the file was appended to. Each successful upload now adds its block keys to the set, and pruning the backlog past ten files logs what it drops. * Code caches: a 200 without a cache list is a failure, not an empty list An unrouted path on api.gamenative.app is answered by the update-check worker, and the dependency read that as zero hosted caches. * FEX code caches: a settings toggle, and support detected from the pack Settings > Emulation gains "FEX code caches" (16 locales), on by default; off means no code-map capture, no upload and no cache download. Whether a container's FEXCore can do any of this is no longer read from its version string: a pack that ships FEXOfflineCompiler64.exe next to its DLLs is one that writes usable maps and loads caches, so that file's presence in the installed pack is the test. * Cache download honours the FEX code cache toggle and the pack capability check * FEX code caches: look up hosted caches by image sha256 instead of by game * FEX code caches: key hosted caches by code identity (codeSha256) instead of file hash * Code caches: match the Steam client DLLs case-insensitively GameOverlayRenderer.dll keeps Steam's casing in the prefix, so the overlay caches were never looked up. * Code caches: review fixes Capture gate applied after every env source is merged, and the flag cleared when capture is off. Uploader: per-map failures no longer abort the batch, temp files are unique, the feature toggle is checked before each request, only a transport failure ends the pass, library records with cut-off padding are rejected, the sent-block file is compacted when it carries duplicates, and the game name lookup cannot throw. Downloader: file names with "." or NUL are refused, a failed rename is logged as such, the Steam DLL list is sorted. Image hashing refuses section headers larger than the file. The debug API override must parse as an HTTP URL. Lookup responses without codeSha256 no longer fail the parse. The capability check reuses the launcher's ContentsManager. --------- Co-authored-by: Utkarsh Dalal <utkarsh.dalal@toptal.com>
Co-authored-by: Utkarsh Dalal <utkarsh.dalal@toptal.com>
Contributor
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the
|
| throw e | ||
| } catch (e: Exception) { | ||
| Timber.tag("Optimize").w(e, "Lever $i of ${rung.rungId} failed, taking the others off") | ||
| OptimizeReverter.revertLevers(context, container, applied) |
| OptimizeLocalState.fromJson(JSONObject(file.readText())) | ||
| } catch (e: Exception) { | ||
| Timber.tag("Optimize").w(e, "state.json unreadable, starting fresh") | ||
| OptimizeLocalState() |
| val factory = DocumentBuilderFactory.newInstance().apply { | ||
| isNamespaceAware = false | ||
| isExpandEntityReferences = false | ||
| runCatching { setFeature("http://apache.org/xml/features/disallow-doctype-decl", true) } |
|
|
||
| val flagged = state.pending?.takeIf { it.revert != null } | ||
| if (flagged != null) { | ||
| if (!OptimizeReverter.revertLevers(context, container, flagged.levers)) { |
| fun sha256(bytes: ByteArray): String = | ||
| MessageDigest.getInstance("SHA-256").digest(bytes).joinToString("") { "%02x".format(it) } | ||
|
|
||
| fun atomicWrite(target: File, bytes: ByteArray) { |
| fun encodeRegistryValue(type: String, data: String): String = when (type) { | ||
| "dword" -> { | ||
| val v = parseInteger(data) | ||
| require(v in Int.MIN_VALUE.toLong()..0xFFFFFFFFL) { "dword out of range" } |
| <string name="optimize_vc_less_grass_view_distance">moins d\'herbe et une distance d\'affichage réduite</string> | ||
| <string name="optimize_vc_textures_softer">textures plus douces</string> | ||
| <string name="optimize_vc_effects_simpler">effets plus simples</string> | ||
| <string name="optimize_vc_fps_held">images par seconde maintenues un peu plus bas</string> |
| ) | ||
| } | ||
|
|
||
| if (optimizeRequests[appId] == true) { |
| <string name="optimize_vc_less_grass_view_distance">menos hierba y menor distancia de visión</string> | ||
| <string name="optimize_vc_textures_softer">texturas más suaves</string> | ||
| <string name="optimize_vc_effects_simpler">efectos más simples</string> | ||
| <string name="optimize_vc_fps_held">fotogramas limitados un poco más bajo</string> |
| if (flagged != null) { | ||
| if (!OptimizeReverter.revertLevers(context, container, flagged.levers)) { | ||
| Timber.tag(TAG).w("Taking ${flagged.rung.rungId} off failed; trying again next launch") | ||
| store.write(state) |
| @Test | ||
| fun longSessionsAreCompared() { | ||
| assertEquals( | ||
| OptimizeJudge.Decision.Final(OptimizeJudge.HELPED), |
| } | ||
| } | ||
| } | ||
| for (kept in s.kept.asReversed()) { |
| if (range != null) { | ||
| val at = find(lines, range, op.key) | ||
| if (at != null) { | ||
| lines[at] = replaceValue(lines[at], value) |
| for (op in finalOps.values) { | ||
| val now = format.read(out, op) | ||
| if (op.isDelete) { | ||
| if (now != null) throw FormatException("${op.key} is still present") |
| } | ||
|
|
||
| private fun OptimizeLocalState.queued(report: JSONObject): OptimizeLocalState = | ||
| copy(outcomeQueue = (outcomeQueue + report).takeLast(OptimizeLocalState.MAX_QUEUE)) |
| override fun read(text: String, op: FileOp): String? { | ||
| val lines = text.split(ConfigFormats.eolOf(text)) | ||
| val range = sectionRange(lines, op.section) ?: return null | ||
| return find(lines, range, op.key)?.let { valueOf(lines[it]).removeSuffix("\r").trim() } |
| val levers: List<Lever>, | ||
| val expiresAtMillis: Long?, | ||
| ) { | ||
| fun isExpired(nowMillis: Long): Boolean = expiresAtMillis != null && nowMillis > expiresAtMillis |
| priors, | ||
| System.currentTimeMillis(), | ||
| ) | ||
| OptimizeFiles.writeHive(hive, text) |
| throttle: Int? = null, | ||
| reason: String = "quick_menu", | ||
| window: Boolean = true, | ||
| ) = Metrics(gameplaySec = gameplay, steadyFps = fps, stutterShare = stutter, throttleOnsetSec = throttle, exitReason = reason, windowSeen = window) |
| put("gpu_name", gpuName(context)) | ||
| put("ram_mb", ramMb(context)) | ||
| put("android_sdk", Build.VERSION.SDK_INT) | ||
| put("proc_stat_real", baseline?.procStatReal ?: false) |
| } | ||
| } | ||
| } | ||
| for (kept in s.kept.asReversed()) { |
|
|
||
| /** Returns null when the server did not answer within [timeoutMs]; the request is left to finish on its own. */ | ||
| suspend fun nextRung(body: JSONObject, timeoutMs: Long): NextRungResponse? { | ||
| val call = scope.async { |
| val response = OptimizeApi.nextRung(nextRungBody(context, appId, container, state, baseline), NEXT_RUNG_TIMEOUT_MS) | ||
| if (response == null) { | ||
| Timber.tag(TAG).i("No next-rung answer; launching unchanged") | ||
| if (flagged != null) store.write(state) |
| return@withLock ChangeResult.FAILED | ||
| } | ||
| store.write( | ||
| OptimizeLocalState(profile = profile, bad = left.bad, outcomeQueue = left.outcomeQueue), |
| val optimizeDeviceId: String | ||
| get() = optimizeDeviceIdCache ?: synchronized(OPTIMIZE_DEVICE_ID) { | ||
| optimizeDeviceIdCache ?: getPref(OPTIMIZE_DEVICE_ID, "").ifEmpty { | ||
| java.util.UUID.randomUUID().toString().also { setPref(OPTIMIZE_DEVICE_ID, it) } |
| val after = line.substring(eq + 1) | ||
| val lead = after.takeWhile { it == ' ' || it == '\t' } | ||
| val cr = if (line.endsWith("\r")) "\r" else "" | ||
| return line.substring(0, eq + 1) + lead + value + cr |
| <string name="optimize_profile_balanced_description">Smoother play while keeping most of the detail</string> | ||
| <string name="optimize_profile_performance_description">Higher, steadier frame rates first</string> | ||
| <string name="optimize_profile_stability_description">Fewer stutters, freezes and crashes</string> | ||
| <string name="optimize_profile_power_description">Fewer slowdowns from heat and longer battery</string> |
added 2 commits
November 8, 2026 11:45
| baseline = Metrics.fromJson(o.optJSONObject("baseline")), | ||
| visualCostId = o.optStringOrNull("visual_cost_id"), | ||
| visualCostEn = o.optStringOrNull("visual_cost_en"), | ||
| rung = o.optJSONObject("rung"), |
| else -> MaterialTheme.colorScheme.onSurface | ||
| }, | ||
| ) | ||
| option.content?.invoke() |
| } | ||
| val request = GameNativeApi.buildPostRequest(GameNativeApi.BASE_URL + path, body) | ||
| client.newCall(request).execute().use { response -> | ||
| if (response.code == 401) throw Unauthorized() |
| val throttleOnsetSec: Int? = null, | ||
| val cls: String? = null, | ||
| val confidence: Double? = null, | ||
| val confident: Boolean? = null, |
| } | ||
| store.dropRungDir(profile, pending.rung.rungId) | ||
| val baseline = if (pending.revert != null) pending.baseline ?: ladder.baselineOverride else ladder.baselineOverride | ||
| s = s.withLadder(profile, ladder.copy(pending = null, baselineOverride = baseline)) |
| OptimizeProfile.BALANCED -> | ||
| fps != null && stutter != null && fps >= BALANCED_MIN_FPS && stutter < BALANCED_MAX_STUTTER && after.throttleOnsetSec == null | ||
| OptimizeProfile.STABILITY -> stutter != null && stutter < STABILITY_MAX_STUTTER | ||
| OptimizeProfile.POWER -> after.gameplaySec >= POWER_MIN_GAMEPLAY_SEC && after.throttleOnsetSec == null |
| }, | ||
| ) | ||
|
|
||
| OptimizeCardHost( |
| baselineOverride = Metrics.fromJson(o.optJSONObject("baseline_override")), | ||
| lastStatus = lastStatus, | ||
| ) | ||
| return OptimizeLocalState(activeProfile = profile, profiles = mapOf(profile to ladder), outcomeQueue = queue) |
|
|
||
| private val LIMITING_CLASSES = setOf("gpu", "cpu_single", "cpu_multi", "thermal_gpu") | ||
|
|
||
| /** True when the session after a step already reached what the profile asks for; inconclusive sessions are false. */ |
Comment on lines
+2114
to
+2115
| <string name="optimize_measured_fps">+%1$d %% frame rate</string> | ||
| <string name="optimize_measured_stutter">Stutter down %1$d %%</string> |
added 2 commits
November 8, 2026 12:12
…umCompilerThreads to dxvk.conf
| is Lever.ContainerSet -> applyContainer(i, lever, context, container, onProgress) | ||
| // The bionic Turnip and wrapper are built without driconf files; Mesa still reads each option | ||
| // from an env var of the same name. | ||
| is Lever.Drirc -> if (Container.GLIBC.equals(container.containerVariant, ignoreCase = true)) { |
| val factory = DocumentBuilderFactory.newInstance().apply { | ||
| isNamespaceAware = false | ||
| isExpandEntityReferences = false | ||
| runCatching { setFeature("http://xml.org/sax/features/external-general-entities", false) } |
| is Lever.Drirc -> if (Container.GLIBC.equals(container.containerVariant, ignoreCase = true)) { | ||
| applyDrirc(i, lever, container, rungDir) | ||
| } else { | ||
| val env = Lever.ContainerSet(lever.options.map { "envVars.${it.first}" to it.second }) |
| } | ||
|
|
||
| String numCompilerThreads = config.get("numCompilerThreads"); | ||
| if (!numCompilerThreads.isEmpty() && !numCompilerThreads.equals("0")) { |
| /** Sets or, for a null [value], removes [name] in a DXVK_CONFIG value, keeping every other entry. */ | ||
| internal fun dxvkConfigPut(config: String, name: String, value: String?): String { | ||
| val (parts, globalEnd) = dxvkGlobalParts(config) | ||
| val at = parts.subList(0, globalEnd).indexOfFirst { dxvkKey(it) == name } |
| */ | ||
| fun merge(text: String, executable: String?, changes: List<Pair<String, String?>>): Merged { | ||
| val source = text.ifBlank { EMPTY } | ||
| if (!DRICONF_END.containsMatchIn(source)) throw FormatException("not a driconf file") |
| key.getString("name") to (if (key.isNull("prior")) null else key.optString("prior")) | ||
| } | ||
| val text = Driconf.merge(current.toString(Charsets.UTF_8), record.optStringOrNull("executable"), priors).text | ||
| if (!existed && Driconf.applicationCount(text) == 0) { |
| val raw = payload.optJSONObject("options") ?: return Unknown(type) | ||
| val options = raw.keys().asSequence().toList().map { name -> | ||
| if (!DRIRC_OPTION.matches(name) || raw.isNull(name)) return Unknown(type) | ||
| val value = raw.optString(name) |
added 4 commits
November 8, 2026 12:54
Owner
Author
|
Folded into #2363 |
6 of 8 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The client half of the Optimize loop, stacked on #2363. The app measures a session, sends it to the server, applies the one change the server returns before the next launch with a backup, reports how the next session went, and takes the change off when asked or when it crashed. The client holds no rules about which change fits which game.
What the user sees
entitlementanswer.What the client does
.gamenative/optimize/, and restores the backup on undo. Anti-cheat games never get file or registry changes. Component downloads show in the launch dialog.outcome), queued and retried on the next launch; a crash takes the change off.optimize_stateongame_exited, and the last card answer (card_action) is sent with the next request.Recording
None yet; device proof pending.
Type of Change
Checklist
#code-changes, I have discussed this change there and it has been green-lighted. If I do not have access, I have still provided clear context in this PR. If I skip both, I accept that this change may face delays in review, may not be reviewed at all, or may be closed.CONTRIBUTING.md.