Live2DKit

A Swift package wrapping the Live2D Cubism SDK for Native, rendering through Metal. Load a .model3.json, drive its parameters (lipsync, blink, gaze, anything the rig exposes), play its authored motions, and draw it into an MTKView, from Swift, on iOS and macOS.

Live2DRuntime.start()
let model = try Live2DModel(contentsOf: hiyoriURL)      // .model3.json
model.startMotion(group: "Idle", index: 0)              // authored clip
model.setValue(0.6, for: "ParamMouthOpenY")             // your lipsync

How you consume this (read first)

Clone it, stage the SDK yourself, and reference it by path. This package can never be a normal .package(url:…, from:…) dependency, for two structural reasons:

1. It carries none of Live2D's software. Cubism Core is under the Live2D Proprietary Software License and the Framework under the Live2D Open Software License; neither is mine to redistribute. Everything Live2D-owned lives under the gitignored Vendor/, staged locally from your own SDK download.
2. A C++ target needs unsafeFlags, and SwiftPM forbids unsafeFlags in version-pinned dependencies. Path (or branch) dependencies are the only shape SwiftPM allows.

So the flow is:

git clone <this repo>
# download CubismSdkForNative from https://www.live2d.com/en/sdk/download/native/
cd Live2DKit
Scripts/bootstrap.sh                              # defaults to ~/Downloads/CubismSdkForNative-5-r.5
Scripts/build-metallibs.sh                        # compile the Metal shader libraries
swift build && swift test                         # verify

…then, in your app's Package.swift or Xcode project, add Live2DKit as a local path dependency pointing at your clone.

bootstrap.sh assembles Vendor/Live2DCubismCore.xcframework from the Release slices (iOS device, universal simulator, universal macOS) and copies the Framework sources. It records the SDK version in Vendor/CUBISM_VERSION so a stale checkout is diagnosable. build-metallibs.sh compiles Cubism's shaders into Vendor/FrameworkMetallibs-<sdk>/, one directory per SDK (iphoneos, iphonesimulator, macosx).

What your app must bundle

Two things ship in the app's bundle, not in this package:

1. The shader libraries. Copy Vendor/FrameworkMetallibs-<sdk>/ (the slice matching the SDK you're building for) into your app bundle as a directory named FrameworkMetallibs/. A build phase keyed on $PLATFORM_NAME does this cleanly:

   rsync -a --delete \
     "$LIVE2DKIT/Vendor/FrameworkMetallibs-$PLATFORM_NAME/" \
     "$TARGET_BUILD_DIR/$UNLOCALIZED_RESOURCES_FOLDER_PATH/FrameworkMetallibs/"

   Why: Cubism loads its shaders by file, at runtime, from the hosting app's main bundle. See the shader story below. Without this directory no model will ever render (Live2DRenderer will refuse with isReady == false rather than let Cubism abort the process).

2. A model. Cubism's sample models (Hiyori et al.) are in the SDK download under their own Free Material License. Bundle one for testing, or ship your own. A model directory holds Name.model3.json plus the .moc3, textures, and optional physics3.json / pose3.json / motions/.

Usage

The pieces

type	role
Live2DRuntime	process-wide Cubism lifecycle (start() before anything else)
Live2DModel	one loaded model: parameters, groups, physics, pose, motions
Live2DRenderer	draws a model into a Metal render pass

A frame goes: evaluateMotions(deltaTime:) → your parameter writes → update(deltaTime:) → draw(…). That order is a contract: motion evaluation discards writes made before it (Cubism's LoadParameters sandwich), and update runs physics/pose and pushes everything into vertex state.

UIKit

import Live2DKit
import MetalKit

final class CharacterViewController: UIViewController, MTKViewDelegate {
    private var model: Live2DModel!
    private var renderer: Live2DRenderer!
    private var queue: MTLCommandQueue!
    private var lastFrame: CFTimeInterval?

    override func viewDidLoad() {
        super.viewDidLoad()
        let device = MTLCreateSystemDefaultDevice()!
        let mtkView = MTKView(frame: view.bounds, device: device)
        mtkView.autoresizingMask = [.flexibleWidth, .flexibleHeight]
        Live2DRenderer.configure(mtkView)   // Cubism hard-codes its formats; this sets them
        mtkView.delegate = self
        view.addSubview(mtkView)

        Live2DRuntime.start()
        let url = Bundle.main.url(forResource: "Hiyori", withExtension: "model3.json",
                                  subdirectory: "Hiyori")!
        model = try! Live2DModel(contentsOf: url)

        let size = mtkView.drawableSize
        renderer = Live2DRenderer(model: model, device: device,
                                  width: Int(size.width), height: Int(size.height))!
        precondition(renderer.isReady, "FrameworkMetallibs/ missing from the app bundle")

        // Textures come from the model's own manifest, bound in order, loaded
        // AS STORED (no flip; Cubism's UVs are authored against the atlas as-is)
        // with straight alpha (the loader does not premultiply; nor should you).
        let loader = MTKTextureLoader(device: device)
        let manifest = try! JSONSerialization.jsonObject(
            with: Data(contentsOf: url)) as! [String: Any]
        let refs = manifest["FileReferences"] as! [String: Any]
        for (index, name) in (refs["Textures"] as! [String]).enumerated() {
            let textureURL = url.deletingLastPathComponent().appendingPathComponent(name)
            renderer.bindTexture(try! loader.newTexture(URL: textureURL, options: [:]), at: index)
        }

        queue = device.makeCommandQueue()
        model.startMotion(group: "Idle", index: 0)
    }

    func draw(in view: MTKView) {
        let now = CACurrentMediaTime()
        let dt = lastFrame.map { now - $0 } ?? 0
        lastFrame = now

        if model.motionsFinished {                       // rotate idle clips
            model.startMotion(group: "Idle", index: Int.random(in: 0..<model.motionCount(inGroup: "Idle")))
        }
        model.evaluateMotions(deltaTime: dt)             // 1. authored motion
        model.setValue(mouthOpenness, for: "ParamMouthOpenY")   // 2. your writes
        model.update(deltaTime: dt)                      // 3. physics + pose + commit

        guard let descriptor = view.currentRenderPassDescriptor,
              let drawable = view.currentDrawable,
              let commandBuffer = queue.makeCommandBuffer() else { return }
        renderer.draw(commandBuffer: commandBuffer,
                      renderPassDescriptor: descriptor,
                      drawableSize: view.drawableSize)
        commandBuffer.present(drawable)
        commandBuffer.commit()
    }

    func mtkView(_ view: MTKView, drawableSizeWillChange size: CGSize) {
        renderer.resize(width: Int(size.width), height: Int(size.height))
    }

    var mouthOpenness: Float { 0 }   // feed your lipsync here
}

SwiftUI

Wrap the same machinery in a representable. Build everything once in makeUIView; per-frame state is pushed to the object, never rebuilt from body:

import Live2DKit
import MetalKit
import SwiftUI

struct Live2DView: UIViewRepresentable {
    let modelURL: URL
    var mouthOpen: Float = 0

    func makeCoordinator() -> Renderer { Renderer(modelURL: modelURL) }

    func makeUIView(context: Context) -> MTKView {
        let view = MTKView(frame: .zero, device: MTLCreateSystemDefaultDevice())
        Live2DRenderer.configure(view)
        view.clearColor = MTLClearColorMake(0, 0, 0, 0)   // transparent stage
        view.isOpaque = false
        view.delegate = context.coordinator
        return view
    }

    func updateUIView(_ uiView: MTKView, context: Context) {
        context.coordinator.mouthOpen = mouthOpen          // push, don't rebuild
    }

    final class Renderer: NSObject, MTKViewDelegate {
        var mouthOpen: Float = 0
        // model / renderer / queue built lazily on first draw, exactly as in the
        // UIKit example; omitted here for brevity.
        init(modelURL: URL) { /* … */ }
        func draw(in view: MTKView) { /* the same 1-2-3 frame as above */ }
        func mtkView(_ view: MTKView, drawableSizeWillChange size: CGSize) { /* resize */ }
    }
}

A complete, runnable integration (model load, texture binding, motion playback, offscreen render) lives in Tools/snapshot/main.swift, which is also the package's headless verification tool:

Scripts/snapshot.sh /tmp/frame.png Vendor/Models/Hiyori/Hiyori.model3.json 2 Idle:0

The demo app

Demo/ is a complete SwiftUI app: Hiyori full-bleed on the system background, rotating her authored Idle clips (never the same twice in a row) with an asymmetric procedural blink multiplied into the clip's eyelid curves and gaze saccades composed over its eye curves. Tap her and she plays her TapBody clip. Reduce Motion parks the clips and damps the gaze; blink stays.

# after bootstrap.sh + build-metallibs.sh:
cd Demo && xcodegen generate && open Live2DKitDemo.xcodeproj

It consumes the package by path (..), exactly as the section above prescribes, so it doubles as a working reference for the build phase and the frame loop. Requires XcodeGen; the generated project is not committed.

Beyond the basics

• Groups, not guesses: models disagree on parameter names (Hiyori lipsyncs via ParamMouthOpenY, Mao via ParamA). Read model.lipSyncParameterIDs / eyeBlinkParameterIDs instead of hard-coding. Unknown IDs are safely ignored.
• Framing: renderer.framingScale / framingOffsetY zoom and shift the fitted model in screen space. This is how a full-figure canvas becomes an upper-body portrait without touching the model.
• Pose fade: model.setPoseFadeDuration(_:) overrides pose3.json's part-swap cross-dissolve. Models with swappable part groups (alternate arm sets, say) ghost both variants for the authored fade time when motions switch between them.
• Motions are one-shot even when authored Loop: true, so motionsFinished actually fires and you own the rotation policy. Starting a new clip mid-play cross-fades.

Why shaders live in YOUR app's bundle

The most surprising integration detail, and it is not optional. Cubism does not use a compiled-in default Metal library. At init, CubismShader_Metal::GenerateShaders looks its shaders up by file, at runtime, in the hosting app's main bundle:

[[NSBundle mainBundle] URLForResource:@"MetalShaders"
                        withExtension:@"metallib"
                         subdirectory:@"FrameworkMetallibs"]

SwiftPM cannot produce that shape: it compiles a target's .metal sources into one default.metallib inside the package's resource bundle, where Cubism never looks. So the .metal files are excluded from the SwiftPM target and built by Scripts/build-metallibs.sh into Vendor/FrameworkMetallibs-<sdk>/, which your app embeds (see above).

Two further traps in the same area:

• Six *Blend.metal files only compile with CSM_COLOR_BLEND_MODE / CSM_ALPHA_BLEND_MODE defined; Cubism builds each once per blend-mode pair, 474 variants in total. FragShaderSrcColorBlend.metal and FragShaderSrcAlphaBlend.metal are #include fragments and must never be compiled standalone. Compiling any of these bare is what a naive SwiftPM build does, and it fails loudly.
• When the library is missing, the process aborts. GenerateShaders null-checks the library and returns, which looks like a graceful degrade, but that check is unreachable: with no resource the URL is nil and -newLibraryWithURL: asserts (url must not be nil.) first. Live2DRenderer therefore checks for the library itself and refuses to build, so isReady reports false instead of the app dying. Check isReady: false means the model will never appear.

Layers

target	kind	why
Live2DCubismCore	binary	the proprietary Core, staged not committed
CubismFramework	C++	Live2D's sources + the Metal renderer
CubismBridge	ObjC++	a narrow ObjC surface over the C++ API
Live2DKit	Swift	what callers use

Cubism's public API is heavyweight C++ (its own csmVector/csmString containers, CubismUserModel, CubismRenderer_Metal), which Swift's C++ interop does not consume cleanly. Hence the ObjC++ shim. Keep it narrow: every symbol crossing that boundary is one Swift can't see through, and a wide bridge becomes a second SDK to maintain.

Notes

• Cubism is process-global and not thread-safe. The framework, allocator, and id manager are shared mutable state; this package serializes load and teardown, and you should drive a given model from one thread (the main thread is fine).
• ARC is off for CubismFramework only. Cubism's Metal renderer sends autorelease directly, which ARC forbids. CubismBridge is this package's code and keeps ARC.
• The allocator is load-bearing. Cubism reads model data through aligned vector loads, so AllocateAligned must genuinely align; a plain malloc faults on some payloads rather than merely being slower.
• Delta times are clamped to 0.1 s so a resume-from-background hitch can't slingshot physics pendulums or fast-forward a motion.

License

The wrapper code in this repository is MIT-licensed (see LICENSE). The Live2D Cubism SDK is Live2D Inc.'s, under their own terms: the Live2D Proprietary Software License (Core), the Live2D Open Software License (Framework), and the Free Material License (sample models). Business users above Live2D's revenue threshold need a Cubism SDK Release License. Nothing in this repository grants any right to Live2D's software, and none of it is included here.