Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -132,6 +132,11 @@
and index/platform/config tuple. Its hosted acquisition and filesystem
qualification must succeed before wrapper metadata/inventory pins are adopted;
release publication alone does not establish those inventories.
Docker archive views may mutate root directory size/mtime while switching
roots: validate those fields under the exact `/` schema, retain exact stable
root fields and bounded UID/mode ownership headers before and after protected
reads. Keep full stat equality for every nonroot path; see
`tools/RELEASED_SDK_INVENTORY.md` for the root fence contract.
- Validate Dockerfiles with `docker build --check`. Build and smoke-test each
affected image, including `classic-validation` before `classic-final` and the
Windows `classic-check` target plus native bundle; note that a cold
Expand Down
45 changes: 45 additions & 0 deletions tools/RELEASED_SDK_INVENTORY.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,51 @@ file, 8 GiB aggregate payload and a shared 900-second deadline; single-file
reads use the same deadline and aggregate accounting. The existing collector's
operation alarm and socket timeout remain required outer bounds.

On a failed whole-path-stat comparison, `inventory_failure.path_stat_drift`
records the fixed comparison site (`directory-header`, `producer-header`,
`regular-header` or `recheck`), a fixed subject class (`root-directory`,
`directory` or `file`), changed known field names (`name`, `size`, `mode`,
`mtime`, `linkTarget`), and booleans for field-set and unknown-field changes.
It never retains stat values, arbitrary field names, paths or exception text.
Every nonroot path retains full stat equality, including timestamps, sizes and
unknown fields, before inventories or scanner qualification can succeed.

Moby Docker 29.4.2 opens a new filesystem view for each
[`HEAD` or `GET` archive operation](https://github.com/moby/moby/blob/docker-v29.4.2/daemon/archive_unix.go).
Its [`filesystem view`](https://github.com/moby/moby/blob/docker-v29.4.2/daemon/containerfs_linux.go)
calls [`SwitchRoot`](https://github.com/moby/moby/blob/docker-v29.4.2/daemon/internal/mounttree/switchroot_linux.go),
which can create and remove a temporary directory directly beneath the root.
The resulting root `mtime` is specific to that filesystem view. Root `size` is
the raw `lstat.Size()` of the directory: Go documents nonregular-file size as
[system-dependent](https://go.dev/src/io/fs/fs.go), and Linux directories store
[directory entries in filesystem data blocks](https://docs.kernel.org/filesystems/ext4/directory.html).
Creating/removing a directory entry does not guarantee restoring that storage
size. This differs from the root tar header's required zero payload size.

Only the exact `/` path uses a separate metadata fence. Both root stats must
have exactly `name`, `size`, `mode`, `mtime` and `linkTarget`; unknown or absent
fields fail even when both snapshots agree. Name must be `/`, link target must
be empty, and mode must be a true integer containing only the Go directory bit
and safe permissions. These fields remain exactly equal across views. Size must
be a true integer in `0..2^63-1`; mtime must be a valid calendar timestamp in
RFC3339Nano form (years 1–9999, at most nine fractional digits, explicit UTC or
numeric offset). These two view-local fields are validated but not compared for
equality at `/`. This exception never applies to any other path.

Protected reads retain the exact created-container identity, selected image,
mount, read-only and network isolation fences before and after reading bytes.
Root ownership is checked through a bounded GET header both before and after
the protected read: exact `/` name, directory type, matching mode, UID 0 and
zero tar payload size. Each stream closes after the header. The existing GID
policy is unchanged. All provider/tool bytes and nonroot ancestor stats remain
fenced, and the separate scanner must still prove complete inventory equality.

The Moby source proves that root timestamp equality is an invalid cross-view
invariant. It does not establish the actual comparison site or changed field
in a hosted failure whose stats were not retained. Source-derived fixtures and
diagnostics do not qualify the released SDK; actual hosted collection and
filesystem equality remain required.

Each producer record is `relative_path -> {sha256, size, executable}`, including
zero-byte files. Canonical sorted compact JSON uses the wrapper's
`linux_portable.inventory_digest` format. Common licenses use exactly the
Expand Down
2 changes: 1 addition & 1 deletion tools/collect-released-sdk-metadata.py
Original file line number Diff line number Diff line change
Expand Up @@ -938,7 +938,7 @@ def collect_final_inventory(container, data, file_records, llvm_tools, policy, *
progress['stage'] = 'metadata'
protocol = SimpleNamespace(BINARY_LIMIT=BINARY_LIMIT, stat_matches=stat_matches, regular_mode=regular_mode,
path_stat=path_stat, single_file_archive=single_file_archive, pax_times=pax_times)
reader = INVENTORY.InstalledInventory(container, protocol)
reader = INVENTORY.InstalledInventory(container, protocol, progress=progress)
raw = {path.removeprefix(INVENTORY.ROOT + '/'): value for path, value in data.items()
if path.startswith(INVENTORY.ROOT + '/')}
records = {name: file_records[INVENTORY.ROOT + '/' + name] for name in raw}
Expand Down
76 changes: 67 additions & 9 deletions tools/released_sdk_inventory.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,12 @@
# SPDX-License-Identifier: MIT
"""Internal fixed-boundary inventory reader; no CLI or acquisition authority.

The caller must supply the collector's authenticated, created-state StoppedImage
and its archive protocol. This foundation is deliberately not wired to a release
target: final producer/consumer/index/platform/config pins are still required.
The fixed final-target collector supplies its authenticated, created-state
StoppedImage and archive protocol. This helper grants no acquisition or release
trust; target pins and hosted filesystem qualification belong to the caller.
"""
from abc import ABC, abstractmethod
import datetime
import hashlib
import json
from pathlib import PurePosixPath
Expand Down Expand Up @@ -528,9 +529,58 @@ def read(self, size):

class InstalledInventory(ProviderReader):
"""Use only with an authenticated StoppedImage; does not grant release trust."""
def __init__(self, image, protocol):
def __init__(self, image, protocol, *, progress=None):
self.image, self.protocol = image, protocol
self.budget = Budget()
self.progress = progress

def _same_stat(self, actual, expected, *, site, path):
"""Fence stable root fields and full nonroot stats; retain no values."""
require(type(actual) is dict and type(expected) is dict, 'inventory-path-drift')
matches = actual == expected
if path == '/':
# Moby's separate filesystem views create/remove a pivot directory
# beneath /. Its mtime and directory storage size are view-local.
# Only these two validated fields are excluded from root equality.
matches = (self._valid_root_stat(actual) and self._valid_root_stat(expected)
and {key: value for key, value in actual.items() if key not in ('size', 'mtime')} ==
{key: value for key, value in expected.items() if key not in ('size', 'mtime')})
if not matches and self.progress is not None:
fields = ('name', 'size', 'mode', 'mtime', 'linkTarget')
mode = expected.get('mode')
subject = ('root-directory' if path == '/' else
'directory' if type(mode) is int and mode & GO_DIRECTORY else 'file')
self.progress['path_stat_drift'] = {
'site': site,
'subject': subject,
'changed_fields': [key for key in fields
if (key in actual) != (key in expected)
or actual.get(key) != expected.get(key)],
'field_set_changed': actual.keys() != expected.keys(),
'unknown_fields_changed':
{key: value for key, value in actual.items() if key not in fields} !=
{key: value for key, value in expected.items() if key not in fields},
}
require(matches, 'inventory-path-drift')

@staticmethod
def _valid_root_stat(value):
if type(value) is not dict or set(value) != {'name', 'size', 'mode', 'mtime', 'linkTarget'}:
return False
mode, size, mtime = value['mode'], value['size'], value['mtime']
if not (value['name'] == '/' and value['linkTarget'] == ''
and type(mode) is int and mode & ~0o777 == GO_DIRECTORY
and mode & 0o022 == 0 and mode & 0o555 == 0o555
and type(size) is int and 0 <= size <= 2**63 - 1
and type(mtime) is str and len(mtime) <= 35
and re.fullmatch(r'[0-9]{4}-[0-9]{2}-[0-9]{2}T(?:[01][0-9]|2[0-3]):[0-5][0-9]:[0-5][0-9]'
r'(?:\.[0-9]{1,9})?(?:Z|[+-](?:[01][0-9]|2[0-3]):[0-5][0-9])', mtime)):
return False
try:
datetime.datetime.fromisoformat(mtime)
except ValueError:
return False
return True

def _fence(self):
self.budget.check()
Expand All @@ -545,7 +595,8 @@ def _directory_owner(self, path, expected):
"""
conn, response = self._get(path)
try:
require(self.protocol.path_stat(response) == expected, 'inventory-path-drift')
self._same_stat(self.protocol.path_stat(response), expected,
site='directory-header', path=path)
framing(response)
info = next_member(response, self.budget, self.protocol)
names = {'/'} if path == '/' else {PurePosixPath(path).name}
Expand All @@ -564,7 +615,7 @@ def _parents(self, path, *, protected=False):
self.budget.check()
value = self.image.stat(str(parent))
if str(parent) == '/':
require(value.get('name') == '/', 'inventory-directory-entry')
require(self._valid_root_stat(value), 'inventory-directory-entry')
self.protocol.stat_matches({**value, 'name': ''}, '/', directory=True)
else:
self.protocol.stat_matches(value, str(parent), directory=True)
Expand All @@ -576,7 +627,12 @@ def _parents(self, path, *, protected=False):
def _recheck(self, observations):
for path, before in observations.items():
self.budget.check()
require(self.image.stat(path) == before, 'inventory-path-drift')
after = self.image.stat(path)
self._same_stat(after, before, site='recheck', path=path)
if path == '/':
# HEAD cannot prove ownership. Recheck the bounded UID/mode
# header after protected bytes as well as before them.
self._directory_owner(path, after)
self._fence()

def _get(self, path):
Expand All @@ -592,7 +648,8 @@ def producer(self):
observations[ROOT] = expected
conn, response = self._get(ROOT)
try:
require(self.protocol.path_stat(response) == expected, 'inventory-path-drift')
self._same_stat(self.protocol.path_stat(response), expected,
site='producer-header', path=ROOT)
result = tree_archive(response, self.protocol, expected, self.budget)
finally:
conn.close()
Expand Down Expand Up @@ -627,7 +684,8 @@ def _regular(self, path, *, executable=False, provider=False):
self.budget.member(expected['size'])
conn, response = self._get(path)
try:
require(self.protocol.path_stat(response) == expected, 'inventory-path-drift')
self._same_stat(self.protocol.path_stat(response), expected,
site='regular-header', path=path)
identity, _ = self.protocol.single_file_archive(
BudgetResponse(response, self.budget), path, expected, retain=False, executable=executable, root_owned=protected)
self.budget.check()
Expand Down
17 changes: 15 additions & 2 deletions tools/tests/test_released_sdk_diagnostics.py
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ def __hash__(self):
with self.subTest(kind=type(error).__name__):
self.assertEqual(I.failure_code(error), 'inventory-operation-failed')

def collect_failure(self, stage, error):
def collect_failure(self, stage, error, details=None):
fixture = FIXTURES.OfflineTests()
data, binary = fixture.portable_fixture()
shader_path = I.ROOT + '/shader-generation.json'
Expand Down Expand Up @@ -90,7 +90,7 @@ def collect_failure(self, stage, error):
for operation in mocked[index + 1:]:
operation.assert_not_called()
self.assertEqual(result['status'], 'stopped')
self.assertEqual(result['inventory_failure'], {'stage': stage})
self.assertEqual(result['inventory_failure'], {'stage': stage, **(details or {})})
self.assertTrue(result['owned_container_removed'])
self.assertFalse(result['image_code_executed'])
self.assertFalse(result['final_portable_cohort'])
Expand All @@ -101,6 +101,19 @@ def collect_failure(self, stage, error):
self.assertNotIn(secret, json.dumps(result))
return result

def test_actual_stat_comparison_reaches_receipt_without_values_or_execution(self):
def drift(_records, _runtime, _contract, *, reader):
expected = {'name': '/', 'size': 4096, 'mode': I.GO_DIRECTORY | 0o755,
'mtime': 'before', 'linkTarget': ''}
actual = {**expected, 'mtime': SECRET, SECRET: SECRET}
with patch.object(reader.image, 'stat', return_value=actual):
reader._recheck({'/': expected})
receipt = self.collect_failure('source-archives-and-providers', drift, {
'path_stat_drift': {'site': 'recheck', 'subject': 'root-directory',
'changed_fields': ['mtime'], 'field_set_changed': True,
'unknown_fields_changed': True}})
self.assertEqual(receipt['reason'], 'inventory-path-drift')

def test_each_inventory_stage_retains_fixed_code_and_stops_before_scanner(self):
for stage, code in (
('producer-tree', 'inventory-extension-or-link'),
Expand Down
2 changes: 1 addition & 1 deletion tools/tests/test_released_sdk_filesystem.py
Original file line number Diff line number Diff line change
Expand Up @@ -431,7 +431,7 @@ def reader_check(producer, runtime, contract, *, reader):
return source_check(producer, runtime, contract, reader=reader)
expected_reader = reader
reader_class = C.INVENTORY.InstalledInventory
def new_reader(cls, *args):
def new_reader(cls, *args, **kwargs):
return expected_reader
with patch.object(reader_class, '__new__', new_reader), \
patch.object(C.INVENTORY, 'crosscheck_archives', side_effect=reader_check) as crosscheck:
Expand Down
Loading
Loading