fix(wasm): make the streaming micropip install engine actually run - #10522
fix(wasm): make the streaming micropip install engine actually run#10522dmadisetti wants to merge 1 commit into
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
All contributors have signed the CLA ✍️ ✅ |
There was a problem hiding this comment.
All reported issues were addressed across 3 files
Architecture diagram
sequenceDiagram
participant UI as WASM Notebook UI
participant Runtime as Marimo Runtime
participant Engine as Stream Transaction Install Engine
participant Micropip as Micropip Module (Pyodide)
participant PkgMgr as Package Manager Singleton
participant Txn as Transaction
participant Packaging as Packaging Library
participant PyPI as PyPI Server
Note over UI,PyPI: Streaming Micropip Install Flow (micropip >= 0.11)
UI->>Runtime: Request package installation (batch)
Runtime->>Engine: stream_transaction_install(packages)
Engine->>Micropip: Import micropip module
Engine->>Packaging: Import packaging.markers.default_environment
alt Current micropip (>= 0.11)
Engine->>Micropip: Access _package_manager_singleton
Micropip-->>Engine: PackageManager instance
else Legacy micropip (< 0.11)
Engine->>Micropip: Access _micropip
Micropip-->>Engine: PackageManager instance
end
Engine->>PkgMgr: Get compat layer & defaults
PkgMgr-->>Engine: index_urls, constraints, compat
Engine->>Packaging: Get default_environment()
Packaging-->>Engine: PEP 508 markers
Engine->>Txn: Create Transaction(requirements, ctx, compat)
Txn-->Txn: Resolve dependencies & fetch metadata
loop Per-package resolution
Txn->>PyPI: Fetch wheel metadata
PyPI-->>Txn: Package metadata
Txn->>Txn: Collect wheels (success) or Requirement objects (failure)
end
Engine->>Txn: Install collected wheels (parallel)
Txn->>PyPI: Download wheels
PyPI-->>Txn: Wheel content
alt All packages resolved
Engine->>Runtime: Yield (package_name, True) per package
else Some packages failed
Txn-->>Engine: failed list contains Requirement objects
Engine->>Engine: Extract .name attribute from each Requirement
Engine->>Runtime: Yield (original_pkg_name, False) per failure
end
Runtime-->>UI: Installation results (success/failure per package)
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
6f6802a to
d9f6f1f
Compare
There was a problem hiding this comment.
Pull request overview
Updates marimo’s WASM/Pyodide micropip streaming installer to match newer micropip internals so the parallel “Transaction”-based install path actually runs (instead of silently falling back to sequential installs), and adds coverage to detect future API drift.
Changes:
- Switches the streaming engine to use micropip’s renamed global package-manager singleton and
packaging’s marker environment helper. - Fixes resolution-failure handling by interpreting
Transaction.failedentries asRequirementobjects (and adds tests for versioned specs). - Extends Pyodide acceptance coverage by installing real packages via
stream_transaction_installand asserting they import.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
marimo/_runtime/packages/_micropip_streaming.py |
Updates private micropip integration points and failure handling for streaming installs. |
tests/_runtime/packages/test_micropip_streaming.py |
Updates stubs/tests to reflect the new singleton name and Requirement-based failure entries; adds new assertions. |
tests/_pyodide/test_pyodide_acceptance.mjs |
Adds an acceptance step that exercises the streaming engine against real micropip in Pyodide. |
Suppressed comments (1)
tests/_runtime/packages/test_micropip_streaming.py:152
- The test name says it “reports_base_name”, but the behavior/assertion is that a versioned request is yielded back using the caller’s original spec string (
"foo==1.2.3"). Renaming the test to match the actual contract will avoid confusion when maintaining failures.
async def test_failed_requirement_with_specifier_reports_base_name(
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
d9f6f1f to
6fedd81
Compare
The streaming installer looks up two names that micropip does not have: `micropip._micropip` and `micropip._utils.default_environment`. Neither exists in any micropip from 0.4.0 (2023) onward. `_micropip` was a module name up to 0.2.2, the package-manager singleton has long been `_package_manager_singleton`, and micropip vendors `packaging`, so `default_environment` comes from `packaging.markers`. Both lookups raise AttributeError. The caller reads that as a changed interface and installs one package at a time instead, so the batched install never ran. The unit tests stubbed micropip with the same names micropip does not have, so they passed, and no test exercised the installer against real micropip. This is what happens when a notebook opens and declares no dependencies in an inline script-metadata header (PEP 723). marimo finds nothing to pre-install, the cells run, and their failing imports surface the missing packages. For 8 packages the install takes 0.5s instead of 2.4s, median of 4 runs. Also read the entries of a failed resolution as requirement objects rather than strings. When a package failed to resolve, reading them as strings raised AttributeError into the same fallback. The Pyodide acceptance test now installs two packages through the installer against real micropip. That step fails on the parent commit.
5b4a945 to
18bfe36
Compare
📝 Summary
Follow-up to #9702, with fixes for the bump to micropip 0.11 from 0.9.
Installing 8 packages from that prompt, median of 4 runs on one machine against
the Python Package Index (PyPI):
Note, times vary with network conditions.
Each run uses a fresh browser profile, so no package comes from a previous run. marimo fetches package metadata prior to installation click, so both rows understate the cost of resolving packages against a cold cache.