Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ jobs:
with:
python-version: ${{ matrix.python }}
- run: python -m pip install ".[dev]"
- run: python -m black --check src tests examples scripts
- run: python -m black --check --diff src tests examples scripts
- run: python -m threadmark setup-tokenizer
- run: python -m pytest -q
- run: python -m threadmark demo
Expand Down
26 changes: 25 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,30 @@
# Changelog

## Unreleased
## Unreleased: budgeted active context

- Add a provider-independent `ActiveContext` compiler for incrementally maintained
workflow memory, with explicit request rendering and token budgets.
- Preserve current facts, open/active-approved proposals, unresolved references
and complete source evidence; select optional history, claims and original
records as whole bundles and expose coverage. Essential overflow raises an
explicit error without model calls or truncation.
- Export immutable compiled requests, with model-specific renderer/tokenizer
support. Include offline context-retention/token tests and an integration
example. These checks do not establish model accuracy or production readiness.

## 0.2.0 — draft (unreleased)

- Preserve distinct ordered assignments, proposals, and notes for the same target within one event; continue rejecting identical duplicate operations.
- Give additional same-event/same-target proposals indexed IDs (`::2`, `::3`, …), retaining existing first-proposal identifiers and scoped alias resolution.
- Add transactional `WorkflowMemory.apply_operations` for externally supplied typed operations without model or tokenizer calls. Validation does not establish semantic correctness.
- Add opt-in `reference_history_entries=0..256` (default zero), bounded by the updater reference limit without pruning the archive or full memory packet.
- Write workflow snapshot schema 2 with the history setting; continue restoring schema 1 with its original no-history behavior.
- Add explicit local LoRA adapter loading, pinned file identities, structural compatibility checks, and separate optional dependencies. The caller must supply the intended local base; adapter metadata cannot choose a remote base.
- Add optional structured workflow decoding for supported local updater calls, with bounded JSON/schema checks and no unconstrained fallback. Format compliance does not verify extracted claims.
- Add a source-snapshotted offline integrity runner: 9/12 cases pass on the prior reducer versus 12/12 on the improved reducer, including 1,000 unique updates with an explicit 2 MB packet cap. These use recorded operations, not live extraction.
- Expose `WorkflowMemory.updater_instructions` and public default workflow instruction constants for exact role routing, including the opt-in history suffix.
- Retain the failed first local trial as development evidence. A separate fresh constrained-decoding trial commits 17/32 events and completes 2/4 workflows versus 2/32 and 0/4, but both arms have 0/4 exact final current states. No model retraining, RAG comparison, or production-readiness claim follows.
- Record a separate unadapted-base diagnostic on the reused development fixture: constrained decoding commits 18/32 events and completes 2/4 workflows, but both arms still have 0/4 exact final states. This is not fresh replication or evidence of a general adapter effect.

- Added `WorkflowMemory` for historical facts, explicit proposal acceptance/rejection, uniquely resolved external proposal labels, and source-linked reasons.
- Added free-form answers from bounded structured memory and portable snapshots restored by validated operation replay without model calls.
Expand Down
2 changes: 2 additions & 0 deletions MANIFEST.in
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@ include scripts/build_research_pdf.py
include scripts/compare_local.py
include scripts/download_models.py
include scripts/prepare_comparison_evidence.py
include scripts/evaluate_workflow_integrity.py
include scripts/evaluate_active_context.py
recursive-include docs *.md
recursive-include examples *.py *.json
recursive-include benchmarks *.md *.json *.zip
Expand Down
76 changes: 76 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,76 @@ The existing `Threadmark` API remains available for its current-fact register an

This is a Python library and comparison harness. Its workflow memory methods use fixed provider weights. A separate research component trains LoRA event classifiers on Qwen and SmolLM2; that narrow component does not establish a new foundation model, a learned decoder memory layer, or general replacement of retrieval-augmented generation.

## Budgeted context for different models

The goal is to process new information once, keep useful state outside the
model, and send a bounded context on each subsequent request. `ActiveContext`
compiles that context from `WorkflowMemory` without another model call. It
preserves selected current facts, pending decisions and their original evidence;
adds history and relevant records when they fit; and reports omissions. Essential
context that cannot fit raises `ContextBudgetExceeded` instead of being cut off.

```python
from threadmark import ActiveContext
from threadmark.context import token_counter

# memory is an existing WorkflowMemory, updated with new events only.
request = ActiveContext(memory).compile(
"What is approved now, and what is still pending?",
max_input_tokens=4096,
count_tokens=token_counter(), # requires the explicitly prepared local cache
)
# Pass request.instructions and request.content to a compatible reader adapter.
# request.rendered_request is the exact text measured by the supplied counter.
```

The default JSON-envelope count excludes provider-specific chat framing. Supply
the target model's complete request renderer and tokenizer for a native token
budget, including tools and any additional messages. This is portable external
memory; model weights stay unchanged. Compression does not establish extraction
or answer accuracy, and ingesting new text can still require model calls.

See the [active-context guide](docs/active-context.md) for the architecture,
equations, incremental use, token accounting, and limits. Its offline evaluation
uses application-supplied operations and is separate from the failed November 10
extractor experiment; no trained checkpoint was promoted by this change.

## 0.2.0: workflow integrity and local extraction

Version 0.2.0 adds ordered distinct changes, proposals, and notes within one
record; indexed proposal identities; atomic `apply_operations` without an LLM
call; and an optional recent-history window for the updater. New workflow
snapshots use schema 2 and still read schema 1. Existing defaults preserve the
current-facts/open-proposals reference unless history is explicitly enabled.

Local inference can now load an explicitly supplied LoRA adapter alongside a
separately supplied base model. Optional structured workflow decoding constrains
the JSON operation format. It does not determine whether an operation is true.
See [workflow memory](docs/workflow-memory.md) and
[local adapters and decoding](docs/local-and-retrieval.md#use-an-explicit-local-lora-adapter)
for API examples, optional dependencies, and failure behavior.

These changes are **experimental, not a production-readiness claim**. Offline
integrity checks improved from 9/12 to 12/12 cases with recorded operations,
including 1,000 unique updates under an explicit 2 MB packet cap. They do not
measure language-model accuracy. The first local-adapter trial completed 0/4
eight-event workflows in both software arms and is now development evidence.

A separate fresh four-workflow comparison used the same fixed adapter and
improved memory in both arms. Constrained decoding committed 17/32 events and
completed 2/4 workflows, versus 2/32 and 0/4 with ordinary decoding. **Both
produced 0/4 exact final current states.** Committing a valid operation can
still preserve the wrong fact or omit an update. No model was retrained in
these trials, no RAG comparison was run, and local GPU/electricity costs remain
unpriced. Historical benchmarks below do not measure these APIs or decoder
settings. See the [engineering evaluation notes](docs/workflow-memory.md#evaluation).

A follow-up reused those same cases with the unadapted base model. Constraints
committed 18/32 events and completed 2/4 workflows, but both decoding arms again
had 0/4 exact final states. Removing the adapter did not resolve the observed
failures. This is a separate development diagnostic, not fresh validation or
a general conclusion about adapter quality.

## Research and mathematical design

Read the [November 1 long-conversation evaluation](research/evaluation-2026-11-01.md) for the history and proposal improvements, the separate 54-question study and 18-question follow-up, costs, and limitations. The packaged API adds persistence and validation around the tested algorithm; it has not undergone another paid model evaluation.
Expand Down Expand Up @@ -46,6 +116,7 @@ Then install and run:

```bash
python -m pip install -e .
python examples/use_application_events.py
python examples/use_workflow_memory.py
python -m threadmark demo
python -m threadmark setup-tokenizer
Expand All @@ -55,6 +126,11 @@ python -m threadmark compare --input examples/workflow.json --output runs/prepar

The workflow-memory example and demo use recorded responses and a conservative byte counter to demonstrate the data flow. They are not fresh model evaluations. The example includes an external proposal label, its acceptance, an owner handoff, a free-form answer, and snapshot restoration. `setup-tokenizer` downloads official tokenizer data once; it does not make a paid API call. Subsequent inspection and preparation run offline. Preparation validates the fixture and writes a request plan without calling OpenAI. See [workflow-memory usage](docs/workflow-memory.md) for the new API, and [usage](docs/usage.md) for the existing CLI, paid comparison, input format, and output files.

The [application-events example](examples/use_application_events.py) uses typed
operations supplied by the application. It demonstrates ordered handoffs,
multiple proposals, acceptance/rejection, and snapshot replay without model or
tokenizer calls.

## Latest: long-conversation memory improvement

The November 1 follow-up used **three fresh synthetic histories** in the 32K, 64K and 96K bands, with six questions per history. All methods used the same Luna reader. The largest full-history request had **99,495 input tokens**.
Expand Down
Loading
Loading