Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
{
"changes": [
{
"comment": "Add the `useProjectDependencyGraph` experiment. When enabled, `rush install` writes a `dependency-graph.bin` file into each project's `.rush/temp` folder containing the slice of the workspace dependency graph reachable from that project, and that file replaces `shrinkwrap-deps.json` when computing whether a project's dependencies have changed.",
"type": "none",
"packageName": "@microsoft/rush"
}
],
"packageName": "@microsoft/rush"
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
{
"changes": [
{
"comment": "Initial release. Provides the `RRC1` binary resolver cache format, lockfile-derived Merkle hashing of dependency contexts, per-project graph slicing, and a parser that transparently accepts either the binary format or the legacy monolithic JSON cache.",
"type": "minor",
"packageName": "@rushstack/resolver-cache"
}
],
"packageName": "@rushstack/resolver-cache"
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
{
"changes": [
{
"comment": "Add `loadResolverCacheAsync` and `loadResolverCache`, which read the first available resolver cache file from a list of candidates, transparently accepting either the new binary format or the legacy monolithic JSON cache. The `ISerializedResolveContext` and `IResolverCacheFile` types are now re-exported from the new `@rushstack/resolver-cache` package.",
"type": "minor",
"packageName": "@rushstack/webpack-workspace-resolve-plugin"
}
],
"packageName": "@rushstack/webpack-workspace-resolve-plugin"
}
36 changes: 36 additions & 0 deletions libraries/resolver-cache/.npmignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
# THIS IS A STANDARD TEMPLATE FOR .npmignore FILES IN THIS REPO.

# Ignore all files by default, to avoid accidentally publishing unintended files.
*

# Use negative patterns to bring back the specific things we want to publish.
!/bin/**
!/lib/**
!/lib-*/**
!/dist/**
!/includes/**

!CHANGELOG.md
!CHANGELOG.json
!heft-plugin.json
!rush-plugin-manifest.json
!ThirdPartyNotice.txt

# Ignore certain patterns that should not get published.
/dist/*.stats.*
/lib/**/test/
/lib-*/**/test/
*.test.js
*.test.[cm]js
*.test.d.ts
*.test.d.[cm]ts

# NOTE: These don't need to be specified, because NPM includes them automatically.
#
# package.json
# README.md
# LICENSE

# ---------------------------------------------------------------------------
# DO NOT MODIFY ABOVE THIS LINE! Add any project-specific overrides below.
# ---------------------------------------------------------------------------
24 changes: 24 additions & 0 deletions libraries/resolver-cache/LICENSE
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
@rushstack/lookup-by-path

Copyright (c) Microsoft Corporation. All rights reserved.

MIT License

Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
"Software"), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:

The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
50 changes: 50 additions & 0 deletions libraries/resolver-cache/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
# @rushstack/resolver-cache

A dedicated binary format for the Rush resolver cache, plus helpers for hashing and slicing the
dependency graph that it describes.

This package intentionally has no runtime dependencies, so that a consumer such as a Webpack plugin
can decode a cache without taking a dependency on `@microsoft/rush-lib`.

## The format

The format is optimized for a single linear decode; full decode is cheap enough that random access
is not worth the extra complexity.

- **Strings** are stored once, in a lexicographically sorted, front-coded table. Each entry is a
`[prefixIndexDelta, suffixLengthInCharacters]` pair, where the prefix is the complete value of the
entry `prefixIndexDelta` positions earlier. The table is closed under branch-point prefixes, so a
suitable prefix always exists. All suffixes share one UTF-8 blob, which the decoder decodes exactly
once and then indexes with `substring`.
- **Suffix lengths are measured in UTF-16 code units**, not bytes and not code points. Byte lengths
would force a separate decode per entry; code points would disagree with `substring`. An encoder
written in another language must match this definition.
- **Integers** are LEB128 varints, and are stored as deltas against a nearby value (the previous
dependency key, or the ordinal of the context being decoded) so that they almost always fit in a
single byte.

## Context identity

A context is identified solely by its root path. Contexts that share a package name and version but
differ in root path are distinct and must never be merged. In particular, a PNPM injected dependency
resolves its own dependencies within the consuming subspace, so it has a different dependency graph
than the workspace project it was copied from, and the same project may be injected more than once
with different peer resolutions.

## Hashes

`computeContextHashes` produces a Merkle hash for each context. The preimage is purely
lockfile-derived: the root path and name of the context, its dependency keys, and the hashes of the
contexts those keys resolve to. File contents, build outputs, and timestamps are excluded, because
Rush tracks those separately through the build graph. The hashes are therefore computable from a
checkout that has never been installed.

Contexts that participate in a dependency cycle are condensed into a strongly connected component
and hashed as a unit.

## Links

- [CHANGELOG.md](https://github.com/microsoft/rushstack/blob/main/libraries/resolver-cache/CHANGELOG.md) - Find
out what's new in the latest version

`@rushstack/resolver-cache` is part of the [Rush Stack](https://rushstack.io/) family of projects.
4 changes: 4 additions & 0 deletions libraries/resolver-cache/config/api-extractor.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
{
"$schema": "https://developer.microsoft.com/json-schemas/api-extractor/v7/api-extractor.schema.json",
"extends": "local-node-rig/profiles/default/config/api-extractor-base.json"
}
3 changes: 3 additions & 0 deletions libraries/resolver-cache/config/jest.config.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{
"extends": "local-node-rig/profiles/default/config/jest.config.json"
}
7 changes: 7 additions & 0 deletions libraries/resolver-cache/config/rig.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
{
// The "rig.json" file directs tools to look for their config files in an external package.
// Documentation for this system: https://www.npmjs.com/package/@rushstack/rig-package
"$schema": "https://developer.microsoft.com/json-schemas/rig-package/rig.schema.json",

"rigPackageName": "local-node-rig"
}
20 changes: 20 additions & 0 deletions libraries/resolver-cache/eslint.config.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license.
// See LICENSE in the project root for license information.

const nodeProfile = require('local-node-rig/profiles/default/includes/eslint/flat/profile/node');
const friendlyLocalsMixin = require('local-node-rig/profiles/default/includes/eslint/flat/mixins/friendly-locals');
const tsdocMixin = require('local-node-rig/profiles/default/includes/eslint/flat/mixins/tsdoc');

module.exports = [
...nodeProfile,
...friendlyLocalsMixin,
...tsdocMixin,
{
files: ['**/*.ts', '**/*.tsx'],
languageOptions: {
parserOptions: {
tsconfigRootDir: __dirname
}
}
}
];
67 changes: 67 additions & 0 deletions libraries/resolver-cache/package.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
{
"name": "@rushstack/resolver-cache",
"version": "0.1.0",
"description": "Codec and resolution helpers for the Rush resolver cache binary format.",
"main": "./lib-commonjs/index.js",
"module": "./lib-esm/index.js",
"types": "./dist/resolver-cache.d.ts",
"exports": {
".": {
"types": "./dist/resolver-cache.d.ts",
"node": "./lib-commonjs/index.js",
"import": "./lib-esm/index.js",
"require": "./lib-commonjs/index.js"
},
"./lib/*": {
"types": "./lib-dts/*.d.ts",
"node": "./lib-commonjs/*.js",
"import": "./lib-esm/*.js",
"require": "./lib-commonjs/*.js"
},
"./package.json": "./package.json"
},
"typesVersions": {
"*": {
"lib/*": [
"lib-dts/*"
]
}
},
"keywords": [
"rush",
"resolver",
"cache",
"binary",
"lockfile"
],
"license": "MIT",
"repository": {
"url": "https://github.com/microsoft/rushstack.git",
"type": "git",
"directory": "libraries/resolver-cache"
},
"engines": {
"node": ">=20.9.0"
},
"scripts": {
"build": "heft build --clean",
"_phase:build": "heft run --only build -- --clean",
"_phase:pack": "rush-pnpm pack",
"_phase:test": "heft run --only test -- --clean"
},
"devDependencies": {
"@rushstack/heft": "workspace:*",
"@types/node": "20.17.19",
"eslint": "~9.37.0",
"local-node-rig": "workspace:*"
},
"peerDependencies": {
"@types/node": "*"
},
"peerDependenciesMeta": {
"@types/node": {
"optional": true
}
},
"sideEffects": false
}
128 changes: 128 additions & 0 deletions libraries/resolver-cache/src/BinaryReader.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,128 @@
// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license.
// See LICENSE in the project root for license information.

/**
* A forward-only cursor over a buffer produced by `BinaryWriter`.
*
* @internal
*/
export class BinaryReader {
readonly #buffer: Uint8Array;
#offset: number;

public constructor(buffer: Uint8Array) {
this.#buffer = buffer;
this.#offset = 0;
}

/**
* The current read position.
*/
public get offset(): number {
return this.#offset;
}

/**
* True if every byte has been consumed.
*/
public get atEnd(): boolean {
return this.#offset >= this.#buffer.length;
}

#require(additionalBytes: number): void {
if (this.#offset + additionalBytes > this.#buffer.length) {
throw new Error(
`Unexpected end of buffer: needed ${additionalBytes} byte(s) at offset ${this.#offset} ` +
`but only ${this.#buffer.length - this.#offset} remain`
);
}
}

/**
* Reads a varint that describes how many items follow, rejecting values that could not possibly
* be satisfied by the remaining bytes.
*
* @remarks
* Decoders frequently preallocate storage from a count, so an implausible count read from a
* corrupt or hostile file would otherwise request an arbitrarily large allocation before the
* first missing byte is noticed.
*
* @param minimumBytesPerItem - The smallest number of bytes any single item can occupy.
*/
public readCount(minimumBytesPerItem: number): number {
const count: number = this.readVarint();
const maximumCount: number = Math.floor(
(this.#buffer.length - this.#offset) / Math.max(minimumBytesPerItem, 1)
);
if (count > maximumCount) {
throw new Error(
`Declared item count ${count} exceeds the ${maximumCount} item(s) that the remaining ` +
`${this.#buffer.length - this.#offset} byte(s) could contain`
);
}
return count;
}

/**
* Reads a single byte.
*/
public readUint8(): number {
this.#require(1);
return this.#buffer[this.#offset++];
}

/**
* Reads a 16-bit little-endian integer.
*/
public readUint16(): number {
this.#require(2);
const value: number = this.#buffer[this.#offset] | (this.#buffer[this.#offset + 1] << 8);
this.#offset += 2;
return value;
}

/**
* Reads an unsigned LEB128 variable-length integer.
*/
public readVarint(): number {
let result: number = 0;
let scale: number = 1;
for (;;) {
const byte: number = this.readUint8();
result += (byte & 0x7f) * scale;
if ((byte & 0x80) === 0) {
break;
}
scale *= 0x80;
if (!Number.isSafeInteger(result + scale)) {
throw new RangeError('Varint exceeds the safe integer range');
}
}
return result;
}

/**
* Reads a zigzag-encoded signed integer.
*/
public readSignedVarint(): number {
const raw: number = this.readVarint();
return raw % 2 === 0 ? raw / 2 : -(raw + 1) / 2;
}

/**
* Reads `byteLength` raw bytes. The result is a view over the original buffer, not a copy.
*/
public readBytes(byteLength: number): Uint8Array {
this.#require(byteLength);
const bytes: Uint8Array = this.#buffer.subarray(this.#offset, this.#offset + byteLength);
this.#offset += byteLength;
return bytes;
}

/**
* Reads a varint byte length followed by that many bytes.
*/
public readLengthPrefixedBytes(): Uint8Array {
return this.readBytes(this.readVarint());
}
}
Loading
Loading