Skip to content

fix(wasm): make the streaming micropip install engine actually run - #10522

Draft
dmadisetti wants to merge 1 commit into
dm/micropip-streamingfrom
dm/micropip-parallel
Draft

fix(wasm): make the streaming micropip install engine actually run#10522
dmadisetti wants to merge 1 commit into
dm/micropip-streamingfrom
dm/micropip-parallel

Conversation

@dmadisetti

@dmadisetti dmadisetti commented Aug 12, 2026

Copy link
Copy Markdown
Member

📝 Summary

Follow-up to #9702, with fixes for the bump to micropip 0.11 from 0.9.

Installing 8 packages from that prompt, median of 4 runs on one machine against
the Python Package Index (PyPI):

click to last package installed
before 2.4 s (range 1.9–3.9)
after 0.5 s (range 0.3–0.8)

Note, times vary with network conditions.
Each run uses a fresh browser profile, so no package comes from a previous run. marimo fetches package metadata prior to installation click, so both rows understate the cost of resolving packages against a cold cache.

@vercel

vercel Bot commented Aug 12, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
marimo-docs Ready Ready Preview Aug 12, 2026 10:58pm

Request Review

@github-actions

Copy link
Copy Markdown
Contributor

All contributors have signed the CLA ✍️ ✅
Posted by the CLA Assistant Lite bot.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 3 files

Architecture diagram
sequenceDiagram
    participant UI as WASM Notebook UI
    participant Runtime as Marimo Runtime
    participant Engine as Stream Transaction Install Engine
    participant Micropip as Micropip Module (Pyodide)
    participant PkgMgr as Package Manager Singleton
    participant Txn as Transaction
    participant Packaging as Packaging Library
    participant PyPI as PyPI Server

    Note over UI,PyPI: Streaming Micropip Install Flow (micropip >= 0.11)

    UI->>Runtime: Request package installation (batch)
    Runtime->>Engine: stream_transaction_install(packages)
    
    Engine->>Micropip: Import micropip module
    Engine->>Packaging: Import packaging.markers.default_environment
    
    alt Current micropip (>= 0.11)
        Engine->>Micropip: Access _package_manager_singleton
        Micropip-->>Engine: PackageManager instance
    else Legacy micropip (< 0.11)
        Engine->>Micropip: Access _micropip
        Micropip-->>Engine: PackageManager instance
    end

    Engine->>PkgMgr: Get compat layer & defaults
    PkgMgr-->>Engine: index_urls, constraints, compat
    Engine->>Packaging: Get default_environment()
    Packaging-->>Engine: PEP 508 markers
    
    Engine->>Txn: Create Transaction(requirements, ctx, compat)
    Txn-->Txn: Resolve dependencies & fetch metadata
    
    loop Per-package resolution
        Txn->>PyPI: Fetch wheel metadata
        PyPI-->>Txn: Package metadata
        Txn->>Txn: Collect wheels (success) or Requirement objects (failure)
    end
    
    Engine->>Txn: Install collected wheels (parallel)
    Txn->>PyPI: Download wheels
    PyPI-->>Txn: Wheel content
    
    alt All packages resolved
        Engine->>Runtime: Yield (package_name, True) per package
    else Some packages failed
        Txn-->>Engine: failed list contains Requirement objects
        Engine->>Engine: Extract .name attribute from each Requirement
        Engine->>Runtime: Yield (original_pkg_name, False) per failure
    end
    
    Runtime-->>UI: Installation results (success/failure per package)
Loading

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread marimo/_runtime/packages/_micropip_streaming.py Outdated
Comment thread marimo/_runtime/packages/_micropip_streaming.py Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates marimo’s WASM/Pyodide micropip streaming installer to match newer micropip internals so the parallel “Transaction”-based install path actually runs (instead of silently falling back to sequential installs), and adds coverage to detect future API drift.

Changes:

  • Switches the streaming engine to use micropip’s renamed global package-manager singleton and packaging’s marker environment helper.
  • Fixes resolution-failure handling by interpreting Transaction.failed entries as Requirement objects (and adds tests for versioned specs).
  • Extends Pyodide acceptance coverage by installing real packages via stream_transaction_install and asserting they import.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File Description
marimo/_runtime/packages/_micropip_streaming.py Updates private micropip integration points and failure handling for streaming installs.
tests/_runtime/packages/test_micropip_streaming.py Updates stubs/tests to reflect the new singleton name and Requirement-based failure entries; adds new assertions.
tests/_pyodide/test_pyodide_acceptance.mjs Adds an acceptance step that exercises the streaming engine against real micropip in Pyodide.
Suppressed comments (1)

tests/_runtime/packages/test_micropip_streaming.py:152

  • The test name says it “reports_base_name”, but the behavior/assertion is that a versioned request is yielded back using the caller’s original spec string ("foo==1.2.3"). Renaming the test to match the actual contract will avoid confusion when maintaining failures.
async def test_failed_requirement_with_specifier_reports_base_name(

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread marimo/_runtime/packages/_micropip_streaming.py
Comment thread marimo/_runtime/packages/_micropip_streaming.py
Comment thread tests/_runtime/packages/test_micropip_streaming.py
The streaming installer looks up two names that micropip does not have:
`micropip._micropip` and `micropip._utils.default_environment`. Neither exists in
any micropip from 0.4.0 (2023) onward. `_micropip` was a module name up to 0.2.2,
the package-manager singleton has long been `_package_manager_singleton`, and
micropip vendors `packaging`, so `default_environment` comes from
`packaging.markers`.

Both lookups raise AttributeError. The caller reads that as a changed interface
and installs one package at a time instead, so the batched install never ran.
The unit tests stubbed micropip with the same names micropip does not have, so
they passed, and no test exercised the installer against real micropip.

This is what happens when a notebook opens and declares no dependencies in an
inline script-metadata header (PEP 723). marimo finds nothing to pre-install,
the cells run, and their failing imports surface the missing packages. For 8
packages the install takes 0.5s instead of 2.4s, median of 4 runs.

Also read the entries of a failed resolution as requirement objects rather than
strings. When a package failed to resolve, reading them as strings raised
AttributeError into the same fallback.

The Pyodide acceptance test now installs two packages through the installer
against real micropip. That step fails on the parent commit.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants