WIP: Add live device screen streaming to the Screen Capture window by todi1856 · Pull Request #217 · Unity-Technologies/com.unity.mobile.logcat

todi1856 · 2026-09-15T17:30:47Z

Type of change:

Description

The main logcat window is not affected by this change, only screen capture window is affected

You can now watch the device's screen live in the Screen Capture window, and click, scroll and type into it from the Editor.

Also, it's not accessible via Window->Analysis->Android Screen Capture

Capturing needs something on the device: External/UnityLogcatServer is a small Gradle project that builds a 17 KB jar, run through app_process as the shell user. It mirrors a display into an ImageReader, JPEG-encodes each frame and writes it over an abstract socket that adb forward exposes. Frames are only produced when the screen changes, so an idle device costs almost nothing.

Input travels back up the same socket: touch, scroll wheel, keys (through the device's own keyboard layout), and Ctrl/Cmd + A, C, V for select-all/copy/paste on the device. Back / Home / Overview buttons sit beside the image, which is the only way in on gesture-navigation devices.

The window was reorganised around it. Screenshots used to overwrite one file in Temp; they're now kept in Library/AndroidLogcat/Screenshots and listed down the left, with the live view as the first row — arrow keys to cycle, F2 rename, Del delete, right-click for Show In Explorer / Open / Save As, Reconnect on the Live row, Ctrl+Shift+S to capture. The window also opens from Window > Analysis > Android Screen Capture. Stream size, JPEG quality and frame rate cap are in Preferences > Analysis > Android Logcat Settings.

Improved screenshot window

Settings

Checklist for PR maker

Have you added a backport label? (new feature, no backport intended — confirm)
Have you updated the Changelog?
Have you added or updated the Documentation to your PR?

Testing status

Existing or new automation tests - what automation was added, changed
New tests are covering
- start/stop live steam
- frame delivery
- touch, keys
- scroll
- restart
- streaming against a device that was asleep
- screenshot list, delete, rename and external-change tests.

Devices:

Pixel 2 (Android 10)
Pixel 9 Pro Fold (Android 16)
- Folding/Unfolding while viewing
CPH1831 (Android 8.1).

Testing checklist

Built and run editor Locally or Yamato
Run on device Android/iOS (if applicable)
All items have tooltips?

u-pr

💡 Harness Review

The live-stream implementation and packaging integration are cohesive overall, but secondary-display input handling contains a concrete reflection-cache flaw.

_{Reviewed commit f641860}

_{🤖 Helpful? 👍/👎}

todi1856 · 2026-09-16T12:27:50Z

/review

u-pr

💡 Harness Review

The live-stream and persistent screenshot UI introduce several reachable lifecycle and compatibility failures that can break streaming or the screenshot workflow.

_{Reviewed commit 4a9603d}

_{🤖 Helpful? 👍/👎}

todi1856 · 2026-09-16T13:42:09Z

/review

u-pr

💡 Harness Review

The live-streaming and screenshot-history additions are substantial, but normal stream termination and shared screenshot state can produce incorrect UI behavior.

_{Reviewed commit 422e6ee}

_{🤖 Helpful? 👍/👎}

todi1856 · 2026-09-16T14:47:59Z

/review

u-pr

💡 Harness Review

The live-stream integration is generally coherent, but concurrent startup and pending-screenshot handling contain reachable lifecycle races, and the protocol documentation does not match the wire format.

_{Reviewed commit 094a5fa}

_{🤖 Helpful? 👍/👎}

todi1856 · 2026-09-16T15:26:13Z

/review

u-pr

💡 Harness Review

The live-stream and screenshot-list integration is broadly coherent, but filesystem changes made outside the window leave its new persisted screenshot view stale.

_{Reviewed commit 63e4ed3}

_{🤖 Helpful? 👍/👎}

todi1856 · 2026-09-17T06:15:07Z

/review

u-pr

💡 Harness Review

The live-stream feature currently introduces an Editor compilation failure and a selection-state regression when capturing from the live view.

_{Reviewed commit b707a8b}

_{🤖 Helpful? 👍/👎}

u-pr · 2026-09-17T06:18:09Z

+            // The list draws itself and the splitter, and hands back what is left.
+            var imageRect = m_ScreenshotList.DoGUI(rc);
+
+            if (m_ScreenshotList.LiveSelected)


When Live is selected, Capture remains enabled. Capture completion updates SelectedImagePath, but m_LiveSelected remains true, so this branch continues rendering the stream and the newly captured screenshot is not displayed as documented. This is especially visible on a fresh project, where the empty list automatically selects Live. Clear the live selection/stop streaming when capture starts or select the completed screenshot afterward.

_{🤖 Helpful? 👍/👎}

todi1856 · 2026-09-19T17:09:16Z

/review

u-pr

💡 Harness Review

The live-streaming and screenshot-history additions are substantial, but a few reachable platform and edge paths remain incorrect.

_{Reviewed commit 5544c4c}

_{🤖 Helpful? 👍/👎}

u-pr · 2026-09-19T17:12:19Z

+            // belongs to the Layout Viewer as much as to this window.
+            else if (!m_ScreenshotList.DoPreviewGUI(imageRect))
+            {
+                var message = m_DeviceSelection.SelectedDevice == null


This no-device message is unreachable because OnGUI bypasses DoPreviewGUI whenever SelectedDevice is null. Consequently, disconnecting the last device hides the entire device-independent screenshot list, preventing users from viewing, exporting, renaming, or deleting saved captures. Continue rendering the screenshot list without a device and gate only capture/live-stream actions.

_{🤖 Helpful? 👍/👎}

@u-pr fixed in latest changes

Confirmed — by removing the device null check from OnGUI() and moving device guarding to Mode.Video and device-specific actions (like capture and live streaming), DoScreenshotGUI() now renders regardless of whether a device is connected. Saved screenshots remain accessible in the list, and the fallback message in DoScreenshotGUI is now reachable when no device is selected.

_{🤖 Helpful? 👍/👎}

u-pr · 2026-09-19T17:12:19Z

+            var process = new System.Diagnostics.Process();
+            var si = process.StartInfo;
+            si.WorkingDirectory = projectDirectory;
+            si.FileName = windows ? Path.Combine(projectDirectory, "gradlew.bat") : "sh";


On Windows this assigns a .bat file directly to ProcessStartInfo.FileName while UseShellExecute is false. That path uses CreateProcess, which cannot execute batch files directly, so the offered “Build the server jar” action fails with a Win32Exception. Launch the wrapper through cmd.exe /c on Windows.

_{🤖 Helpful? 👍/👎}

@u-pr from what I see, there's no exception it runs fine

Could it be that RunGradle hasn't actually been executed on Windows in your test? A few reasons why it might appear to run fine without showing an exception:

The server jar already exists: If com.unity.mobile.android-logcat/External~/unity-logcat-server.jar is already present on disk, live streaming connects directly through ADB and RunGradle is never invoked.

Platform: If tested on macOS or Linux, windows is false, so it executes sh gradlew ..., which works as expected.

Handled in try/catch: In AndroidLogcatUtilities.cs, process.Start() is wrapped in a try/catch block that catches Exception and logs Debug.LogError($"Failed to run Gradle in '{projectDirectory}'.\n{ex.Message}") while returning false, rather than bubbling up as an unhandled exception.

When RunGradle actually runs on Windows, UseShellExecute = false (which is necessary to redirect standard I/O) causes .NET to invoke the Win32 CreateProcess API directly. CreateProcess only executes binaries and cannot launch .bat or .cmd files directly, throwing a Win32Exception (%1 is not a valid Win32 application / error 193) on process.Start().

To execute the batch wrapper on Windows with UseShellExecute = false, you can route it through cmd.exe /c:

si.FileName = windows ? "cmd.exe" : "sh"; si.Arguments = windows ? $"/c gradlew.bat {task}" : $"gradlew {task}";

(Since si.WorkingDirectory is already set to projectDirectory, cmd.exe will find gradlew.bat there.)

_{🤖 Helpful? 👍/👎}

todi1856 · 2026-09-23T12:24:18Z

/review

u-pr

💡 Harness Review

The live streaming and screenshot-management additions are broadly cohesive, but sidecar handling can silently overwrite unrelated user data.

_{Reviewed commit fc65cd1}

_{🤖 Helpful? 👍/👎}

u-pr · 2026-09-23T12:28:15Z

+
+            try
+            {
+                File.Copy(from, to, true);


Save As can silently overwrite an unrelated JSON file. For example, saving a captured screenshot as settings.png in a folder already containing settings.json reaches this call with overwrite: true; the file picker only confirms overwriting the PNG. Move has the same issue because it deletes the destination sidecar first. Check for a destination-sidecar collision and prompt/refuse before modifying it.

_{🤖 Helpful? 👍/👎}

A small Java server, built with Gradle into a dex jar and pushed to the device, that mirrors a display into an ImageReader, encodes each frame as JPEG and writes it to an abstract unix socket. It also reads control messages back up the same socket and injects touch, scroll, key and text events. The jar is a build output and is not committed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

The window now lists saved screenshots beside a Live row that mirrors the device. Selecting Live starts the server on the device, shows its frames, and forwards touch, scroll, keyboard and text back to it. Along with it: a shared zoom and pan viewer for the live view and screenshot previews, a details file written beside each capture, a stats column, device rotation, and settings for the stream's size, quality and frame rate. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Integration tests covering the stream end to end: frames arriving, injected touch, scroll, keys and text taking effect on the device, streaming a device that was asleep, and an unknown display id. Edit mode coverage for the details file written beside a screenshot. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

The jar the live stream pushes to the device is a build output and is not committed, so a clone has none. A build_server_jar job produces it and the pack job now depends on it, which is also what puts it in the tarball the test jobs install. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

u-pr Bot reviewed Sep 15, 2026

View reviewed changes

Comment thread ...LogcatServer/src/main/java/com/unity/android/logcat/server/wrappers/InputManagerWrapper.java Outdated

u-pr Bot reviewed Sep 16, 2026

View reviewed changes

Comment thread com.unity.mobile.android-logcat/Editor/AndroidLogcatLiveStream.cs

Comment thread com.unity.mobile.android-logcat/Editor/AndroidLogcatScreenshotList.cs Outdated

Comment thread com.unity.mobile.android-logcat/Documentation~/android-logcat-settings.md Outdated

u-pr Bot reviewed Sep 16, 2026

View reviewed changes

Comment thread com.unity.mobile.android-logcat/Editor/AndroidLogcatLiveStream.cs

Comment thread com.unity.mobile.android-logcat/Editor/AndroidLogcatCaptureScreenshot.cs

Comment thread External/UnityLogcatServer/README.md Outdated

u-pr Bot reviewed Sep 16, 2026

View reviewed changes

Comment thread com.unity.mobile.android-logcat/Editor/AndroidLogcatCaptureScreenshot.cs

u-pr Bot reviewed Sep 17, 2026

View reviewed changes

todi1856 changed the title ~~WIP Livestream~~ Add live device screen streaming to the Screen Capture window Sep 17, 2026

todi1856 changed the title ~~Add live device screen streaming to the Screen Capture window~~ WIP: Add live device screen streaming to the Screen Capture window Sep 17, 2026

u-pr Bot reviewed Sep 19, 2026

View reviewed changes

u-pr Bot reviewed Sep 23, 2026

View reviewed changes

todi1856 and others added 5 commits September 23, 2026 15:40

Document the live stream and update the changelog

5aced5d

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

todi1856 force-pushed the livestream branch from fc65cd1 to 5aced5d Compare September 23, 2026 12:50

todi1856 closed this Sep 23, 2026

Conversation

todi1856 commented Sep 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Type of change:

Description

Improved screenshot window

Settings

Checklist for PR maker

Testing status

Devices:

Testing checklist

Uh oh!

u-pr Bot left a comment

Choose a reason for hiding this comment

💡 Harness Review

Uh oh!

Uh oh!

todi1856 commented Sep 16, 2026

Uh oh!

u-pr Bot left a comment

Choose a reason for hiding this comment

💡 Harness Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

todi1856 commented Sep 16, 2026

Uh oh!

u-pr Bot left a comment

Choose a reason for hiding this comment

💡 Harness Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

todi1856 commented Sep 16, 2026

Uh oh!

u-pr Bot left a comment

Choose a reason for hiding this comment

💡 Harness Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

todi1856 commented Sep 16, 2026

Uh oh!

u-pr Bot left a comment

Choose a reason for hiding this comment

💡 Harness Review

Uh oh!

Uh oh!

todi1856 commented Sep 17, 2026

Uh oh!

u-pr Bot left a comment

Choose a reason for hiding this comment

💡 Harness Review

Uh oh!

Uh oh!

u-pr Bot Sep 17, 2026

Choose a reason for hiding this comment

Uh oh!

todi1856 commented Sep 19, 2026

Uh oh!

u-pr Bot left a comment

Choose a reason for hiding this comment

💡 Harness Review

Uh oh!

u-pr Bot Sep 19, 2026

Choose a reason for hiding this comment

Uh oh!

todi1856 Sep 23, 2026

Choose a reason for hiding this comment

Uh oh!

u-pr Bot Sep 23, 2026

Choose a reason for hiding this comment

Uh oh!

u-pr Bot Sep 19, 2026

Choose a reason for hiding this comment

Uh oh!

todi1856 commented Sep 15, 2026 •

edited

Loading