feat: stream initial package installations for WASM - #9702
Open
dmadisetti wants to merge 7 commits into
Open
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
4 issues found across 8 files
Architecture diagram
sequenceDiagram
participant UI as Client UI
participant useInstall as useInstallPackage hook
participant WS as WebSocket
participant Kernel as Kernel Runtime
participant PkgCallbacks as PackagesCallbacks
participant PkgMgr as PackageManager
participant MicropipStream as MicropipStreamInstall
participant DB as Database
participant Ntf as Broadcast Notification
Note over UI,Ntf: WASM Package Installation Flow
UI->>useInstall: installPackages(["numpy", "pandas", "scipy"])
useInstall->>useInstall: join packages with space separator
useInstall->>WS: addPackage({ package: "numpy pandas scipy" })
WS->>Kernel: enqueue InstallPackagesCommand
Kernel->>PkgCallbacks: _handle_install()
PkgCallbacks->>PkgCallbacks: _notebook_index_urls() - read PEP 723 config
Note over PkgCallbacks: Mark all packages as "installing" up-front
PkgCallbacks->>Ntf: broadcast InstallingPackageAlertNotification (installing)
PkgCallbacks->>PkgMgr: stream_install(["numpy","pandas","scipy"], versions, index_urls)
alt Micropip available (Pyodide)
PkgMgr->>MicropipStream: stream_transaction_install()
Note over MicropipStream: Resolve all requirements via micropip Transaction
MicropipStream->>MicropipStream: gather_requirements(flat_requirements)
alt Resolution failures
MicropipStream-->>PkgMgr: yield (package, False) per failed
end
Note over MicropipStream: Install all wheels in parallel via asyncio.as_completed
par Each wheel completes
MicropipStream->>MicropipStream: wheel.install()
MicropipStream-->>PkgMgr: yield (package, True/False) as completed
end
Note over MicropipStream: Handle pyodide-native packages via loadPackage
MicropipStream->>MicropipStream: loadPackage()
MicropipStream-->>PkgMgr: yield (package, True) per pyodide package
MicropipStream->>MicropipStream: Check remaining packages via importlib.metadata
MicropipStream-->>PkgMgr: yield (package, True/False)
else Micropip unavailable or Transaction API shifted
Note over PkgMgr: Fallback to sequential base install
loop Each package sequentially
PkgMgr->>PkgMgr: install(package, version)
PkgMgr-->>PkgMgr: yield (package, True/False)
end
end
PkgMgr-->>PkgCallbacks: async iterator of (package, success)
loop Each result from stream
PkgCallbacks->>PkgCallbacks: Mark package as installed/failed
alt Success
PkgCallbacks->>Ntf: broadcast InstallingPackageAlertNotification (installed, log "done")
else Failure
PkgCallbacks->>DB: add to excluded_modules
PkgCallbacks->>Ntf: broadcast InstallingPackageAlertNotification (failed, log "done")
end
end
PkgCallbacks->>Ntf: broadcast CompletedRunNotification
Ntf-->>WS: notification to client
WS-->>useInstall: response.success = True
useInstall->>UI: showAddPackageToast for each package
useInstall->>UI: onSuccess callback
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
dmadisetti
force-pushed
the
dm/micropip-streaming
branch
from
June 15, 2026 21:23
4307799 to
9ace65b
Compare
Contributor
There was a problem hiding this comment.
1 issue found across 4 files (changes from recent commits).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="frontend/src/core/packages/__tests__/useInstallPackage.test.tsx">
<violation number="1" location="frontend/src/core/packages/__tests__/useInstallPackage.test.tsx:91">
P1: onSuccess fires even on install failure, not only on success. The test only covers the happy path, so this bug is masked.</violation>
</file>
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
| const { result } = renderHook(() => useInstallPackages()); | ||
|
|
||
| await act(async () => { | ||
| await result.current.handleInstallPackages(["numpy"], onSuccess); |
Contributor
There was a problem hiding this comment.
P1: onSuccess fires even on install failure, not only on success. The test only covers the happy path, so this bug is masked.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At frontend/src/core/packages/__tests__/useInstallPackage.test.tsx, line 91:
<comment>onSuccess fires even on install failure, not only on success. The test only covers the happy path, so this bug is masked.</comment>
<file context>
@@ -0,0 +1,96 @@
+ const { result } = renderHook(() => useInstallPackages());
+
+ await act(async () => {
+ await result.current.handleInstallPackages(["numpy"], onSuccess);
+ });
+
</file context>
Contributor
There was a problem hiding this comment.
Pull request overview
Adds streaming/batched package installation support (targeting Pyodide/micropip) so initial installs can resolve/download in parallel and report progress incrementally.
Changes:
- Introduces a marimo-free
stream_transaction_installengine that uses micropip Transaction internals to stream per-package completion. - Adds a
PackageManager.stream_install(...)API and wires the runtime package-install callback to use it (including notebook-derivedindex_urls). - Updates the frontend to batch multi-package installs into a single request and adjusts toast messaging/tests accordingly.
Reviewed changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/_runtime/test_manage_script_metadata.py | Updates mocks to include stream_install to preserve existing expectations. |
| tests/_runtime/packages/test_micropip_streaming.py | New unit tests for the micropip streaming engine + manager wrapper/fallback behavior. |
| packages/openapi/src/api.ts | Adds indexUrls to generated TS OpenAPI types. |
| packages/openapi/api.yaml | Adds indexUrls to the OpenAPI schema for install commands/requests. |
| marimo/_runtime/packages/package_manager.py | Adds default sequential stream_install API to the base package manager. |
| marimo/_runtime/packages/pypi_package_manager.py | Implements MicropipPackageManager.stream_install using the streaming engine with fallback to sequential installs. |
| marimo/_runtime/packages/_micropip_streaming.py | New marimo-free streaming installer built on micropip Transaction internals. |
| marimo/_runtime/commands.py | Extends InstallPackagesCommand with index_urls. |
| marimo/_runtime/callbacks/packages.py | Reads notebook index config and switches installs to use stream_install for incremental progress. |
| frontend/src/core/packages/useInstallPackage.ts | Batches multi-package installs into a single backend call. |
| frontend/src/core/packages/toast-components.tsx | Updates toast copy/formatting to support singular/plural packages. |
| frontend/src/core/packages/tests/useInstallPackage.test.tsx | Adds tests validating batching + aggregate toast behavior. |
Comment on lines
+184
to
+188
| try: | ||
| importlib.metadata.version(original) | ||
| yield (original, True) | ||
| except importlib.metadata.PackageNotFoundError: | ||
| yield (original, False) |
Comment on lines
18
to
22
| const handleInstallPackages = async ( | ||
| packages: string[], | ||
| onSuccess?: () => void, | ||
| ) => { | ||
| setLoading(true); |
mchav
previously approved these changes
Jun 24, 2026
| current: list[str] = [] | ||
| in_marker = False | ||
|
|
||
| for part in package.split(): |
Contributor
There was a problem hiding this comment.
Is there a standard grammar for this that we can link to?
Contributor
There was a problem hiding this comment.
Let's add a case for mixed success and failure.
mchav
approved these changes
Jul 4, 2026
Contributor
|
This pull request has been automatically marked as stale because it has not had activity in 30 days. It will be closed in 14 days if no further activity occurs. If this PR is still relevant, please leave a comment or push new changes to keep it open. Thank you for your contribution! |
dmadisetti
force-pushed
the
dm/micropip-streaming
branch
from
August 12, 2026 22:28
f4745b1 to
6c30403
Compare
Contributor
|
All contributors have signed the CLA ✍️ ✅ |
Bundle ReportChanges will increase total bundle size by 90.84kB (0.36%) ⬆️. This is within the configured threshold ✅ Detailed changes
Affected Assets, Files, and Routes:view changes for bundle: marimo-esmAssets Changed:
Files in
|
Contributor
Coverage Report for ./frontend
File Coverage
|
||||||||||||||||||||||||||||||||||||||||||||
6 tasks
for more information, see https://pre-commit.ci
- Track requested packages by parsed PEP 508 base name (via packaging.requirements.Requirement), so versioned / URL specs like `foo==1.0` and `foo @ git+…@ref` correctly match the wheel names micropip yields back instead of getting reported as failed. - Replace `# type: ignore` micropip imports with importlib.import_module so the module loads cleanly outside Pyodide and mypy doesn't need to see the optional dep at all. - Isolate loadPackage failures so one pyodide-batch error yields per-package False rather than terminating the generator. - Fallback path now retries only packages the engine didn't yield, avoiding double-yield + (pkg, False) reports for packages that already succeeded. - Defensive isinstance(url, str) checks when reading PEP 723 index config from the notebook (guards malformed pyproject entries). - Regenerate packages/openapi/api.yaml for the new index_urls field.
…sts) The WASM streaming PR batched all packages into a single addPackage call but then looped, emitting one toast per package keyed on a single aggregate success/error. PackageOperationResponse only carries an aggregate success+error (no per-package detail), so this implied a per-package outcome that doesn't exist: one failure showed N error toasts, one success showed N success toasts even if some silently failed. Show a single aggregate toast covering all requested packages instead, with truthful plural/singular wording. Batching behavior is preserved. Also adds tests for MicropipPackageManager.stream_install covering the fallback path: when the engine raises after partial yields, already-yielded packages are not re-installed and each package is yielded exactly once.
dmadisetti
force-pushed
the
dm/micropip-streaming
branch
from
August 12, 2026 22:56
6c30403 to
29dda4e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
📝 Summary
Streams initial package installation in WASM/Pyodide notebooks so packages install in parallel and surface progress as each one lands, instead of resolving one-at-a-time behind an opaque blocking call.
Adds:
_micropip_streaming.py): a marimo-free wrapper around micropip'sTransactionAPIKept the engine module intentionally marimo-free so it can be contributed upstream to pyodide/micropip as a streaming-install primitive.