FreeVideo 0.4.12

.github/ISSUE_TEMPLATE/feedback.yml

@@ -0,0 +1,18 @@
+name: Feedback / 反馈
+description: Tell us what could be better / 说说哪里可以做得更好
+labels: ["feedback"]
+body:
+  - type: textarea
+    id: feedback
+    attributes:
+      label: What could be better? / 哪里可以做得更好？
+    validations:
+      required: true
+  - type: input
+    id: version
+    attributes:
+      label: Version / 版本
+  - type: textarea
+    id: diagnostics
+    attributes:
+      label: Diagnostic info (optional) / 诊断信息（选填）

.gitignore

@@ -34,6 +34,12 @@ resource-history.sqlite3*
 setup-runs/
 envs/
 vendor/
+!web/vendor/
+web/vendor/*
+!web/vendor/three/
+web/vendor/three/*
+!web/vendor/three/three.module.min.js
+!web/vendor/three/LICENSE
 downloads/
 prepared/
 kernel-cache/

MANIFEST.in

@@ -4,7 +4,8 @@ include __init__.py README.zh-CN.md
 include NOTICE THIRD_PARTY_NOTICES.md CONTRIBUTING.md SECURITY.md CITATION.cff
 recursive-include example_workflows *.json
 recursive-include web *.js *.css
-recursive-include web/assets *.svg *.png
+include web/vendor/three/LICENSE
+recursive-include web/assets *.svg *.png *.webp
 recursive-include web/fonts *.woff2
 recursive-include freevideo_engine/assets *.png *.ico
 recursive-include freevideo_engine/launcher *.qml

THIRD_PARTY_NOTICES.md

@@ -4,6 +4,7 @@ FreeVideo code uses [Apache-2.0](LICENSE). Its dependencies and models retain th
 
 | Component | License |
 | --- | --- |
+| [Three.js](https://threejs.org/) r160, © 2010–2023 three.js authors (bundled whitebox renderer) | [MIT](web/vendor/three/LICENSE) |
 | Qt, PySide6 and Shiboken6 | LGPL-3.0; [component notices](freevideo_engine/launcher/licenses/NOTICE.txt) |
 | PyInstaller bootloader | GPL-2.0 with the bootloader exception |
 | CPython / psutil | PSF / BSD-3-Clause |

docs/zh-CN/使用说明.md

@@ -0,0 +1,53 @@
+# 使用说明
+
+这一页说明创作面板里三件常被问到的事：用参考图生成视频、首帧尾帧和参考图的区别、做更长的视频。
+
+## 参考图生成视频
+
+1. 打开创作面板，展开“素材”。
+2. 点“添加”，选择图片。上传的每张图片可以指定用途：首帧、尾帧或参考。要让画面里出现这张图里的人物、物品或场景，选“参考”。
+3. 在描述里输入 `@`，从列表里选一张素材，描述里会出现 `<Picture 1>` 这样的标记。视频和音频素材对应 `<Video 1>`、`<Audio 1>`，各类素材分别编号。
+4. 在描述里说明这张图在画面中的角色，例如“`<Picture 1>` 里的女孩在海边奔跑”，然后点“生成视频”。
+
+素材可以暂时关闭而不删除：取消勾选的素材不参与这次生成，其余素材的编号会随之重排，描述里的标记会同步更新。
+
+参考图生成视频目前仍是实验能力。能够运行不等于每次都能保持人物外观一致，画面不满意时可以换一张更清晰的正面图再试。
+
+## 首帧、尾帧和参考的区别
+
+| 用途 | 作用 | 适合 |
+| --- | --- | --- |
+| 首帧 | 视频从这张图开始 | 让一张画好的图动起来 |
+| 尾帧 | 视频在这张图结束 | 指定结束时的画面，可以只给尾帧 |
+| 参考 | 图里的内容出现在视频中，但不规定哪一帧 | 保持人物、物品、场景的样子 |
+
+- 首帧和尾帧可以同时使用。
+- 首帧、尾帧与参考暂时不能混用。同时给了两类，先保留其中一类。
+- 首帧和尾帧会按输出画幅居中裁切；参考图保持原有比例，按输出像素面积缩小。
+
+## 在描述里用 @ 引用素材
+
+- 输入 `@` 后会列出当前可用的素材，选中后插入对应的标记。
+- 也可以直接写标记，如 `<Picture 2>`、`<Video 1>`、`<Audio 1>`。
+- 编号顺序是：先是上传的素材，再是从上到下连接的图片，最后是音频。参考视频的声音也占一个音频编号。
+- 删除或关闭某张素材后，描述里指向它的标记会改成“[素材已移除]”，其余标记按新编号更新。
+
+## 做更长的视频
+
+有三种做法，可以单独用，也可以搭配。
+
+1. **直接把时长填长。** 超过一次能生成的长度时，FreeVideo 自动分成几段依次生成，再接成一条视频。
+2. **接着生成。** 在生成好的视频结果卡上点“接着生成”，继续下一段；也可以到“剪辑视图”里逐段修改。适合边看边调。
+3. **用 AI 全自动。** 打开“AI 全自动”，把想法交给 Agent，它会按镜头拆分、逐镜生成并拼接。耗时随镜头数和每镜时长而定，每镜按普通生成计时。
+
+### 在剪辑视图里加参考图
+
+在剪辑视图里延长时，选中那一段，点生成条描述框旁的“＋ 参考图”，上传或生成图片。接续段最多用 3 张参考图，和创作面板共用同一份素材。剪辑视图里只能加图片，参考视频和音频要在创作面板的“素材”里添加。
+
+### 限制
+
+- 两种做法一样，一条视频最多 5 段（第一段加 4 次接着生成），所以创作面板里时长最多填 60 秒。接续的每一段至少 10 秒，整条约 70 秒。剪辑视图里逐段生成同样适用。接得越多，衔接处越容易出现变化，能切镜头的地方建议直接切镜头，用对应的参考图开始新的一段。
+- 接着生成使用轻量档，需要 NVIDIA 显卡，也不能和“继续二采”同时使用。
+- 单次生成的最长时长随画幅和显卡而不同，在创作面板的“范围”里可以查看每种尺寸能生成多长、大约需要多久。
+- 分段生成同样需要 NVIDIA 显卡，Mac 上不支持。
+- 分段生成过程中取消，只会停止当前这一段，已经完成的段落会保留，之后可以从第一个未完成的段落继续。

freevideo_engine/adaln.py

@@ -4,6 +4,7 @@
 known before sampling, so retain the small modulation outputs instead of 26 GB
 of immutable projection weights. FL2VA has a separate keyframe timestep schedule.
 """
+import contextlib
 import hashlib
 import json
 from pathlib import Path
@@ -12,6 +13,24 @@
 import torch
 
 
+@contextlib.contextmanager
+def serial_ops():
+    """Run the small CPU tensor work of a table on this thread alone.
+
+    A table is a few hundred kilobytes. Spread over the full intra-op pool each
+    operation waits for its workers, and on a machine whose cores are busy that
+    wait is hundreds of times the work (about 0.3 s per block for a cat that
+    takes 0.2 ms alone). The thread count is a setting of the whole process, so
+    it is restored on exit and other threads running torch ops meanwhile see 1.
+    """
+    previous = torch.get_num_threads()
+    torch.set_num_threads(1)
+    try:
+        yield
+    finally:
+        torch.set_num_threads(previous)
+
+
 def all_finite(tensor):
     """Whether every element is finite; reads half-precision bits instead of calling isfinite.
 
@@ -50,8 +69,9 @@ def __init__(self, values, cursor):
         super().__init__()
         self.cursor = cursor
         self.steps = len(values)
-        for index, values_at_step in enumerate(values):
-            self.register_buffer(f'step_{index}', torch.cat(values_at_step, dim=-1))
+        with serial_ops():
+            for index, values_at_step in enumerate(values):
+                self.register_buffer(f'step_{index}', torch.cat(values_at_step, dim=-1))
         # Steps differ in how many distinct timesteps they carry: the first H3
         # step has video and audio at the same t, so its rows are fewer. The
         # fused pre/post kernels compile for static shapes and recompiled in the
@@ -188,7 +208,9 @@ def load(self, index, steps):
         # These are small immutable constants; their producer's GPU/runtime is
         # deliberately irrelevant. Validate before transferring to this device.
         state = load_file(path, device=self.device)
-        if any(not all_finite(tensor) for tensor in state.values()):
+        with serial_ops():
+            finite = all(all_finite(tensor) for tensor in state.values())
+        if not finite:
             raise ValueError('Nonfinite AdaLN model asset')
         return [state[f'step_{i}'].chunk(6, dim=-1) for i in range(steps)]
 

freevideo_engine/agent/cast.py

@@ -40,6 +40,18 @@
 CLOSE_UP = dict(aspect='1:1', seconds=4)
 SCENE = dict(aspect='16:9', seconds=6)
 
+# The sheet of every view a shot carries for each person, put together here from the card's own pictures: up to six
+# equal cells (three to a row) on a light grey ground, each picture scaled whole and centred, no text.
+SHEET_VERSION = 1
+SHEET_ORDER = ('front', 'three_quarter', 'side', 'back', 'face')  # then any other label, in card order
+SHEET_CELL = (736, 828)
+SHEET_GAP = 48
+SHEET_GROUND = (238, 238, 238)
+SHEET_COLUMNS = 3
+SHEET_MAX = 6
+SHEET_SIZE = (SHEET_COLUMNS * SHEET_CELL[0] + (SHEET_COLUMNS + 1) * SHEET_GAP, 2 * SHEET_CELL[1] + 3 * SHEET_GAP)  # 2400 x 1800
+_SHEETS_SEEN = set()
+
 
 def reference_prompt(card, *, face=False):
     """The one place the reference clip prompts are written."""
@@ -67,13 +79,13 @@ def turn_label(progress):
 class CastError(JobError):
     """A card error; conflicts carry the current card so the caller can retry from it."""
 
-    def __init__(self, code, message, next_action, *, card=None, job=None):
+    def __init__(self, code, message, next_action, *, card=None, job=None, details=None):
         super().__init__(code, message)
         self.status = errors.ERRORS[code][0]
-        self.next_action, self.card, self.job = next_action, card, job
+        self.next_action, self.card, self.job, self.details = next_action, card, job, details or []
 
     def view(self):
-        value = errors.build(self.code, validation=True, details=[], message=self.message,
+        value = errors.build(self.code, validation=True, details=deepcopy(self.details), message=self.message,
                              next_action=self.next_action)
         if self.code in errors.IMAGE_REASONS:
             value.update(errors.image_failure(self.code))
@@ -99,6 +111,8 @@ def view(card):
         value['voice'] = dict(deepcopy(card['voice']), media=media_view(card['voice']['media']),
                               clip=assets.view(card['voice']['clip']['file']))
     value['generate_error'] = errors.with_reason(value.get('generate_error'))
+    if card.get('_sheet'):
+        value['sheet'] = media_view(card['_sheet'])  # the sheet made from the pictures, when the card has none of its own
     return value
 
 
@@ -262,6 +276,15 @@ def _now(self):
         return iso(self.jobs.clock())
 
     # Media --------------------------------------------------------------------------------------------
+    def _picture_at(self, identity, item, by, index):
+        """`_picture` for the request's images[index]; a refused picture names its place, so the dialog marks it."""
+        try:
+            return self._picture(identity, item, by)
+        except CastError as error:
+            if error.code == 'invalid_media' and not error.details:
+                error.details = [dict(field='/images/%d/media' % index)]
+            raise
+
     def _picture(self, identity, item, by):
         """Resolve one picture to an input file in this card's folder (frames, imports, edits)."""
         from ..comfy_assets import media_kind
@@ -276,6 +299,15 @@ def _picture(self, identity, item, by):
         if kind != 'image':
             raise CastError('invalid_media', 'A card picture must be an image.',
                             'Upload an image, or name a video frame with from_job or library and frame.')
+        # The extension alone lets a renamed or cut-off file in; decode it once so a broken picture is refused here,
+        # not when a video first uses the card.
+        try:
+            from PIL import Image
+            with Image.open(self.jobs.port.input_path(relative)) as image:
+                image.load()
+        except Exception:  # noqa: BLE001 - any decoder failure means the picture cannot be used
+            raise CastError('invalid_media', 'A card picture must be a readable image.',
+                            'Upload another image.') from None
         entry = dict(media=dict(file=relative), by=by)
         if isinstance(source, dict) and source.get('from_job'):
             entry['from_job'] = source['from_job']
@@ -340,8 +372,8 @@ def create(self, value, *, by='builtin', ref=None, made_for=None, local=True, cl
                 return replay
         identity = 'c' + uuid.uuid4().hex[:12]
         images = []
-        for row in value.get('images', []):
-            entry = self._picture(identity, row['media'], by)
+        for index, row in enumerate(value.get('images', [])):
+            entry = self._picture_at(identity, row['media'], by, index)
             images.append(dict(entry, label=row.get('label', 'other')))
         voice = self._voice(identity, value['voice']) if value.get('voice') else None
         with self.changed:
@@ -364,7 +396,12 @@ def create(self, value, *, by='builtin', ref=None, made_for=None, local=True, cl
 
     def get(self, identity):
         with self.changed:
-            return view(self._record(identity)['card'])
+            card = deepcopy(self._record(identity)['card'])
+        if card.get('kind') == 'character':
+            sheet = self.sheet(card)
+            if sheet is not None:
+                card['_sheet'] = sheet
+        return view(card)
 
     def list(self, *, q=None, kind=None, origin=None, production=None, limit=50):
         with self.changed:
@@ -410,8 +447,8 @@ def update(self, identity, value, *, by='builtin', ref=None, revision=None, loca
         resolved = {}
         for key, rows in changes.items():
             if key == 'images':
-                resolved['images'] = [entry if entry is not None else dict(self._picture(identity, row['media'], by),
-                                      label=row.get('label', 'other')) for row, entry in rows]
+                resolved['images'] = [entry if entry is not None else dict(self._picture_at(identity, row['media'], by, index),
+                                      label=row.get('label', 'other')) for index, (row, entry) in enumerate(rows)]
             elif key == 'voice':
                 resolved['voice'] = None if rows is None else self._voice(identity, rows)
             else:
@@ -544,10 +581,49 @@ def _unfinished(self, film):
 
     # Filming ------------------------------------------------------------------------------------------
     def snapshot(self, identity):
-        """The current card for a submission, or None once it is deleted or unknown."""
+        """The current card for a submission, or None once it is deleted or unknown. A card with several pictures
+        and no sheet of its own carries the one made from them as `_sheet` (never part of the public card)."""
         with self.changed:
             record = self.records.get(identity)
-            return deepcopy(record['card']) if record else None
+            card = deepcopy(record['card']) if record else None
+        if card is not None and card.get('kind') == 'character':
+            sheet = self.sheet(card)
+            if sheet is not None:
+                card['_sheet'] = sheet
+        return card
+
+    def sheet(self, card):
+        """The sheet of every view made from this card's pictures, as a media item, or None when it has no sheet to
+        make (fewer than two pictures, or one cannot be read). The file is named by what it is made from, so a
+        picture changed gives a new one and the same pictures reuse the old."""
+        if any(row['label'] == 'sheet' for row in card.get('images') or []):
+            return None
+        try:
+            port = self.jobs.port
+            rank = {label: i for i, label in enumerate(SHEET_ORDER)}
+            rows = sorted(enumerate(card.get('images') or []), key=lambda r: (rank.get(r[1]['label'], len(rank)), r[0]))
+            files = [port.input_path(row['media']['file']) for _, row in rows[:SHEET_MAX]
+                     if isinstance(row['media'].get('file'), str)]
+            if len(files) < 2:
+                return None
+            pictures = [path.read_bytes() for path in files]  # the same bytes are named and drawn, so a picture
+            digest = hashlib.sha256(str(SHEET_VERSION).encode())  # changed meanwhile cannot sit under the old name
+            for data in pictures:
+                digest.update(hashlib.sha256(data).digest())
+            relative = 'freevideo/agent/%s/sheet-%s.png' % (card['id'], digest.hexdigest()[:16])
+            target = port.input_path(relative, create=True)
+            if not _sheet_intact(target):
+                temporary = target.with_name(target.name + '.' + uuid.uuid4().hex[:8] + '.tmp')
+                try:
+                    compose_sheet(pictures, temporary)
+                    os.replace(temporary, target)
+                finally:
+                    temporary.unlink(missing_ok=True)
+                # A sheet is never deleted here: a queued shot or an earlier answer may still name it.
+            return dict(file=relative)
+        except Exception:  # A sheet is a help: without one the person carries the face alone.
+            logging.getLogger(__name__).warning('no sheet for card %s', card.get('id'), exc_info=True)
+            return None
 
     def used(self, identities, film):
         """Record that a film uses these cards; a film never leaves used_by once it is there."""
@@ -881,6 +957,61 @@ def _fail(self, identity, hidden, failure):
             self.changed.notify_all()
 
 
+def _sheet_intact(path):
+    """Whether this is a whole sheet: a file left half written or cut short is made again. A file already checked
+    and unchanged is not decoded twice."""
+    try:
+        status = path.stat()
+        key = (str(path), status.st_size, status.st_mtime_ns)
+        if key in _SHEETS_SEEN:
+            return True
+        from PIL import Image
+        with Image.open(path) as opened:
+            if opened.size != SHEET_SIZE:
+                return False
+            opened.load()
+    except Exception:
+        return False
+    if len(_SHEETS_SEEN) > 256:
+        _SHEETS_SEEN.clear()
+    _SHEETS_SEEN.add(key)
+    return True
+
+
+def compose_sheet(pictures, out):
+    """Write one PNG of these pictures (two to six, as file bytes or paths): always three columns by two rows of
+    equal cells, a light grey ground (cells with no picture stay empty), each picture scaled whole to fit its cell
+    and centred, nothing cropped, no text."""
+    import io
+    from PIL import Image, ImageOps
+    (width, height), gap = SHEET_CELL, SHEET_GAP
+    sheet = Image.new('RGB', SHEET_SIZE, SHEET_GROUND)
+    for index, source in enumerate(pictures[:SHEET_MAX]):
+        with Image.open(io.BytesIO(source) if isinstance(source, bytes) else source) as opened:
+            picture = ImageOps.exif_transpose(opened)
+            if picture.mode in ('RGBA', 'LA', 'P'):
+                flat = Image.new('RGB', picture.size, SHEET_GROUND)
+                picture = picture.convert('RGBA')
+                flat.paste(picture, mask=picture.getchannel('A'))
+                picture = flat
+            picture = ImageOps.contain(picture.convert('RGB'), (width, height), Image.LANCZOS)
+        row, column = divmod(index, SHEET_COLUMNS)
+        sheet.paste(picture, (gap + column * (width + gap) + (width - picture.width) // 2,
+                              gap + row * (height + gap) + (height - picture.height) // 2))
+    sheet.save(out, format='PNG')
+
+
+def continued_pictures(cards):
+    """The pictures a shot that continues the previous one carries for these cards (the current cards, from
+    `Cast.snapshot`), at most three: a single person carries the sheet then the face close-up; two or three people
+    carry one each, the face (else the front view, the three-quarter view, the first other picture), no sheet.
+    Each is a media item."""
+    from . import drama
+    cards = list(cards)[:spec.CONTINUED_PICTURES]
+    return [media for card in cards
+            for media in drama.person_pictures(card, alone=len(cards) == 1, continues=True)][:spec.CONTINUED_PICTURES]
+
+
 def _unreadable(error):
     """A clip that is gone or cannot be decoded, rather than a fault of ours."""
     if isinstance(error, (JobError, ValueError, OSError)):