Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,19 @@
# Changelog

## 1.15.0

- **A later results review can supersede an earlier one, by name.** A reviewed row that had to be read another
way, to the same result (a command that read the run by a fixed path, rewritten to read `$RUN` so the gate
passes), was held as it was: `lab check` said "a row that a review had passed in ... is gone or changed", a
second passing review did not lift it, and putting the old row back brought back the failure it was changed
to fix. `lab review <experiment> results pass "<why>" --supersedes <of>` now names the earlier passing results
review by its `of`; it ends the note with "Supersedes the results review of <of>.", and the rows that review
read are held no more once it is committed together with the rows it read, which are held in their place. A review that does not name the earlier one
still supersedes nothing, and a note may not end with that sentence by hand. `lab review` refuses it on a plan or a failing review, for an `of` no earlier passing
results review has, and when the earlier review read the rows as they are now; `lab check` refuses a note that
names no earlier passing results review of other rows.
- The skill `lab-review` changed: run `lab init` in a lab after upgrading.

## 1.14.0

- **A home map has Current storylines.** When `lab.toml` names active storylines, one of the library's home
Expand Down
17 changes: 16 additions & 1 deletion docs/model.md
Original file line number Diff line number Diff line change
Expand Up @@ -199,6 +199,20 @@ Each arrow is a command that refuses when the step before it is missing. You see
problem `lab check` reports at every step: a different result is a new experiment. A review is
bound to the rows it read by its `of`, so only those are held: a draft row committed after a
review, which that review never read, may still be corrected before the next review reads it.
**A review that supersedes another.** A reviewed row that must be read another way, to the same
result (a command that read the run by a fixed path and now reads `$RUN`), is not a new
experiment. A later results review that passes the new rows says so by name: `lab review
<experiment> results pass "<why the rows changed, and what was re-read>" --supersedes <of>`, where
`<of>` is the earlier passing results review's `of` (eight to sixteen of its first hex digits).
The command ends the note with "Supersedes the results review of <of>.". Once that review is
committed together with the rows it read, the rows the earlier review read are held no more, and
the new rows are held in their place; until then it supersedes nothing. A note cannot end with that
sentence except through `--supersedes`.
A later review that passes changed rows without naming the earlier one supersedes nothing, so the
old rows are still held. `lab review` refuses `--supersedes` on a plan review or a failing one, an
`of` that no earlier passing results review has, and a review that read the rows as they are now;
`lab check` refuses a note that names a review that is not an earlier passing results review of
other rows.
**The history that is read.** The check reads the history of the record under the lab's root as
that root is now. Moving the lab's root inside its repository, or to another repository, starts
again what is held: what was committed under the old root is not compared. And it needs the
Expand Down Expand Up @@ -822,7 +836,8 @@ with each quote inside written twice, as a spreadsheet writes it, is read withou
quote is a character like any other.

**`reviews.tsv`** is written by `lab review`: `date`, `stage` (plan, results), `verdict` (pass, fail),
`by`, `of` (a hash of what was read), `note`.
`by`, `of` (a hash of what was read), `note`. A results review written with `--supersedes` ends its
note with "Supersedes the results review of <of>." (section 3, Reviewed rows stay).

**`resources.toml`** has one table per resource: `what`, `key` (where it is kept), `rules` (a list of
sentences), `retired` (why, once it is no longer used), and for a metered resource `unit`, `granted`,
Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ build-backend = "hatchling.build"
[project]
dynamic = ["readme"]
name = "lab-kit-cli"
version = "1.14.0"
version = "1.15.0"
description = "A research lab's record, its method, and the command that keeps both."
license = "MIT"
authors = [{ name = "Piergiuseppe Mallozzi" }]
Expand Down
5 changes: 5 additions & 0 deletions skills/lab-review/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,11 @@ A failed plan is fixed by its author and read again.
can read, saying what you conclude ("The numbers are what the rows say, and the bars were fixed
before the run."). What you re-ran and what you could not verify comes after it: the page shows
the first sentence and folds the rest.
5. When rows that an earlier results review passed have changed since (`lab check` says "a row that a
review had passed in ... is gone or changed"), read what changed (`git diff` of `results.tsv` from
the commit it names) as well as the rows. Pass it only when each changed row still reports the same
result, read another way, and add `--supersedes <of>`, naming that earlier review's `of` from
`reviews.tsv`: the note says why the rows changed. A different result is a new experiment: fail it.

A failed results review is final for that experiment. Say in the note what a replacement must do
differently. Its numbers can then be cited nowhere: if a storyline page lists it under included, `lab check`
Expand Down
2 changes: 1 addition & 1 deletion src/lab_kit/__init__.py
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
"""lab-kit: a research lab's record, its method, and the command that keeps both."""
__version__ = "1.14.0"
__version__ = "1.15.0"
4 changes: 4 additions & 0 deletions src/lab_kit/checks.py
Original file line number Diff line number Diff line change
Expand Up @@ -305,6 +305,10 @@ def see(where, page, who=""):
if r["stage"] not in ("plan", "results") or r["verdict"] not in ("pass", "fail") or not R.day(r["date"]) or not r["of"]:
bad("reviews", p, f"a row of reviews.tsv is not one `lab review` wrote: {R.flat(' '.join(r.values()))[:60]!r}")
break
for i, r in enumerate(x["reviews"]):
m = R.SUPERSEDING.search((r.get("note") or "").rstrip()) if r.get("stage") == "results" and r.get("verdict") == "pass" else None
if m and not any(e.get("stage") == "results" and e.get("verdict") == "pass" and e.get("of") == m.group(1) and e.get("of") != r.get("of") for e in x["reviews"][:i]):
bad("reviews", p, f"a results review supersedes {m.group(1)}, which is no earlier results review of other rows that passed")
if x["results"]:
if not x["lock"] or not x["finished"]:
bad("results", p, "has results with no locked plan and finished run behind them")
Expand Down
9 changes: 6 additions & 3 deletions src/lab_kit/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,9 @@
lab runs close <run> "<why>" [--used <resource>=<amount>]... a run whose process died; --used for one that wrote no used.json
lab runs close <run> "<why>" --used <resource>=<amount>... --evidence <folder> an ended run that did not record what it used of a resource it ran on
lab runs lost <run> "<why>" [--used <resource>=<amount>]... a run that started and whose folder is gone for good
lab review <experiment> plan|results pass|fail "<note>" [--by <who>]
lab review <experiment> plan|results pass|fail "<note>" [--by <who>] [--supersedes <of>]
--supersedes: a results review that passes rows changed after the review
whose `of` it names had passed them; the note says why they changed
lab check [--no-rederive] [--strict] runs every command in results.tsv: only on a lab you trust. --strict fails when
history or the store could not be checked, as in CI
Once
Expand Down Expand Up @@ -191,10 +193,11 @@ def flag(name, many=False, switch=False):
else:
raise SystemExit(USAGE)
elif cmd == "review":
by = flag("--by")
named = "--supersedes" in rest
by, supersedes = flag("--by"), flag("--supersedes")
if len(rest) != 4:
raise SystemExit(USAGE)
print(f"recorded in {write.review(lab, rest[0], rest[1], rest[2], rest[3], by or None)}")
print(f"recorded in {write.review(lab, rest[0], rest[1], rest[2], rest[3], by or None, supersedes if named else None)}")
elif cmd == "check":
if set(rest) - {"--no-rederive", "--strict"}:
raise SystemExit(USAGE)
Expand Down
8 changes: 8 additions & 0 deletions src/lab_kit/history.py
Original file line number Diff line number Diff line change
Expand Up @@ -382,6 +382,7 @@ def check(lab):
rows, read = where.rsplit("/", 1)[0] + "/results.tsv", set()
if rows not in now:
continue # never committed, so nothing is held; or committed and gone, which is said above
held_at = [] # (commit, the rows it holds, the last review that passed them there), newest first
for c, path in reversed(commits + versions.get(rows, [])): # every commit that touched the reviews or the rows
folder = os.path.dirname(path)
if c in read:
Expand All @@ -395,6 +396,13 @@ def check(lab):
seals = [l[5] for l in listed if len(l) >= 6 and l[1] == "run" and l[2] == "ended" and l[4] == "0"] # the finished run, as that commit lists it
if passed[-1].get("of") not in {R.text_hash(R.text_hash(was) + str(seal))[:16] for seal in seals}:
continue # the last review that passed there read other rows than these
held_at.append((c, was, passed[-1]))
# A passing results review that names an earlier one by its `of` (`lab review ... --supersedes`) supersedes it
# only once it is committed together with the rows it read: then those rows are held instead of the earlier ones.
gone = R.superseded([r for _, _, r in held_at])
for c, was, review in held_at:
if review.get("of") in gone:
continue # a later review, committed with the rows it read, superseded it and said why
lost = subsequence(was.splitlines(), R.read(os.path.join(root, now[rows][0])).splitlines())
if lost is not None:
bad(now[rows][0], f"a row that a review had passed in {c[:8]} is gone or changed: {lost[:70]!r}. Reviewed rows stay; a different result is a new experiment")
Expand Down
12 changes: 12 additions & 0 deletions src/lab_kit/record.py
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,18 @@
# it lost (what it used, and why). A file written before `used` and `note` existed has the first six columns.
LEDGER = ("run", "kind", "what", "at", "exit", "seal", "used", "note")
COLUMNS = {"results.tsv": ("name", "label", "value", "reads", "versus", "command"), "reviews.tsv": ("date", "stage", "verdict", "by", "of", "note")}
# A results review that passed rows changed after an earlier one passed them ends its note with this, naming the
# earlier review by its `of` (`lab review ... --supersedes`): the rows the earlier one read are then held no more.
SUPERSEDES = "Supersedes the results review of {}."
SUPERSEDING = re.compile(r"Supersedes the results review of ([0-9a-f]{16})\.$")


def superseded(reviews):
"""The `of` of each results review that a later passing results review supersedes."""
found = (SUPERSEDING.search((r.get("note") or "").rstrip()) for r in reviews if r.get("stage") == "results" and r.get("verdict") == "pass")
return {m.group(1) for m in found if m}


HASH = 2 # how a folder is hashed in the locks and runs made now; one without the marker was made with the first way
NUMBER = re.compile(r"[+-]?(?:[0-9]+(?:\.[0-9]*)?|\.[0-9]+)(?:[eE][+-]?[0-9]+)?")
LISTS = {"goal": ("see",), "question": ("needs", "see"), "experiment": ("needs", "uses", "see"), "build": ("needs", "commits", "tests", "see"),
Expand Down
21 changes: 19 additions & 2 deletions src/lab_kit/write.py
Original file line number Diff line number Diff line change
Expand Up @@ -317,22 +317,39 @@ def lock(lab, ident):
return lab.rel(path)


def review(lab, ident, stage, verdict, note, by=None):
"""One independent reading of a plan or of an experiment's results, bound to exactly what it read."""
def review(lab, ident, stage, verdict, note, by=None, supersedes=None):
"""One independent reading of a plan or of an experiment's results, bound to exactly what it read. With
`supersedes`, a passing results review of rows that were changed after an earlier results review passed them: it
names that review by its `of`, and the note says why the rows changed. The rows that review read are held no more;
these are, once committed."""
x = lab.nodes.get(ident)
if not x or x["type"] != "experiment":
stop(f"no experiment {ident}")
if stage not in ("plan", "results") or verdict not in ("pass", "fail"):
stop("lab review <experiment> plan|results pass|fail \"<note>\"")
if supersedes is not None and (stage != "results" or verdict != "pass"):
stop("only a results review that passes supersedes another")
if not R.flat(note):
stop("a review says what it read and what it found")
if R.SUPERSEDING.search(R.flat(note).rstrip()):
stop("a note does not end by superseding a review: name the earlier review with --supersedes <of>, and the command writes that sentence")
if stage == "plan" and x["lock"]:
stop(f"{ident} is locked: a plan is reviewed before its lock")
if stage == "results":
if not x["finished"] or not x["results"]:
stop(f"{ident} has no finished run with results to review")
if lab.reviewed(x, "results")[0] == "fail":
stop(f"{ident} already failed its results review, and that is final: replace the experiment")
if supersedes is not None:
if not re.fullmatch(r"[0-9a-f]{8,16}", supersedes or ""):
stop("--supersedes names the earlier review by its `of`, eight to sixteen hex digits")
earlier = {r["of"] for r in x["reviews"] if r.get("stage") == "results" and r.get("verdict") == "pass" and (r.get("of") or "").startswith(supersedes)}
if len(earlier) != 1:
stop(f"{ident} has {'no' if not earlier else 'more than one'} results review that passed whose `of` begins {supersedes}")
earlier = earlier.pop()
if earlier in (lab.content(x, stage), lab.content(x, stage, 1)):
stop(f"the review {earlier} read the rows as they are now: there is nothing to supersede, so review them without --supersedes")
note = f"{R.flat(note).rstrip()} {R.SUPERSEDES.format(earlier)}"
path = os.path.join(lab.root, x["dir"], "reviews.tsv")
if not os.path.exists(path):
open(path, "w").write("date\tstage\tverdict\tby\tof\tnote\n")
Expand Down
61 changes: 61 additions & 0 deletions tests/test_held.py
Original file line number Diff line number Diff line change
Expand Up @@ -406,3 +406,64 @@ def test_a_page_of_another_library_is_looked_for_where_that_library_is(tmp_path)
# a library the charter does not name is a problem wherever the check runs
front(f"{qdir}/question.md", see="[elsewhere:/content/readings/the-law.html]")
assert "which is not a page of a library the charter names" in lab(root, "check", ok=False)


def test_a_later_review_supersedes_an_earlier_one_only_by_naming_it(tmp_path):
root = under(tmp_path, "lab")
ident, d = finished(root) # its rows reviewed, passed and committed
rows, reviews, held = f"{d}/results.tsv", f"{d}/reviews.tsv", "a row that a review had passed in"
was = open(rows).read()
first = [l.split("\t") for l in open(reviews).read().splitlines() if "\tresults\tpass\t" in l][-1][4]

# a reviewed row read again another way, to the same value: held, and a second review alone does not change that
open(rows, "w").write(was.replace("cat $RUN/ratio.txt", "python3 -c \"print(open('$RUN/ratio.txt').read().strip())\""))
assert open(rows).read() != was
assert held in lab(root, "check", "--no-rederive", ok=False)
lab(root, "review", ident, "results", "pass", "Read the new command; it prints the same value.")
commit(root, "the row read another way, reviewed again")
assert held in lab(root, "check", "--no-rederive", ok=False)

# what --supersedes refuses: a plan review, a fail, a review that is not there, one that read the rows as they are now
second = [l.split("\t") for l in open(reviews).read().splitlines() if "\tresults\tpass\t" in l][-1][4]
assert "only a results review that passes" in lab(root, "review", ident, "plan", "pass", "x", "--supersedes", first, ok=False)
assert "only a results review that passes" in lab(root, "review", ident, "results", "fail", "x", "--supersedes", first, ok=False)
assert "eight to sixteen hex" in lab(root, "review", ident, "results", "pass", "x", "--supersedes", "zz", ok=False)
assert "no results review that passed" in lab(root, "review", ident, "results", "pass", "x", "--supersedes", "0123abcd", ok=False)
assert "nothing to supersede" in lab(root, "review", ident, "results", "pass", "x", "--supersedes", second, ok=False)
assert "eight to sixteen hex" in lab(root, "review", ident, "results", "pass", "x", "--supersedes", "", ok=False)
assert "name the earlier review with --supersedes" in lab(root, "review", ident, "results", "pass", f"x. Supersedes the results review of {first}.", ok=False)

# named, with why: the rows the first review read are held no more, and these are
out = lab(root, "review", ident, "results", "pass", "The row now reads the value through $RUN, as the gate asks; the value is the same.", "--supersedes", first[:8])
assert "recorded in" in out
assert open(reviews).read().rstrip().endswith(f"Supersedes the results review of {first}.")
commit(root, "the first review superseded")
assert "the record is sound" in lab(root, "check", "--no-rederive", "--strict")
open(rows, "w").write(was)
assert held in lab(root, "check", "--no-rederive", ok=False) # the superseding review's rows are held in their turn


def test_a_review_that_names_no_earlier_pass_supersedes_nothing(tmp_path):
root = under(tmp_path, "lab")
ident, d = finished(root)
reviews = f"{d}/reviews.tsv"
line = [l for l in open(reviews).read().splitlines() if "\tresults\tpass\t" in l][-1]
open(reviews, "a").write(line + " Supersedes the results review of 0123456789abcdef.\n")
assert "which is no earlier results review of other rows that passed" in lab(root, "check", "--no-rederive", ok=False)


def test_a_superseding_review_frees_nothing_until_its_own_rows_are_committed_with_it(tmp_path):
root = under(tmp_path, "lab")
ident, d = finished(root)
rows, reviews, held = f"{d}/results.tsv", f"{d}/reviews.tsv", "a row that a review had passed in"
was = open(rows).read()
first = [l.split("\t") for l in open(reviews).read().splitlines() if "\tresults\tpass\t" in l][-1][4]
other = was.replace("cat $RUN/ratio.txt", "cat \"$RUN/ratio.txt\"")
open(rows, "w").write(other)
lab(root, "review", ident, "results", "pass", "The command quotes $RUN now; same value.", "--supersedes", first)
git(root, "add", reviews)
git(root, "commit", "-qm", "the review, without the rows it read")
open(rows, "w").write(other.replace("cat \"$RUN/ratio.txt\"", "cat $RUN/ratio.txt # a third way"))
lab(root, "review", ident, "results", "pass", "Read the rows again.")
commit(root, "a third way, reviewed without naming anything")
assert held in lab(root, "check", "--no-rederive", ok=False)
Loading