diff --git a/CHANGELOG.md b/CHANGELOG.md index 206ab30..8c0f90a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,19 @@ # Changelog +## 1.15.0 + +- **A later results review can supersede an earlier one, by name.** A reviewed row that had to be read another + way, to the same result (a command that read the run by a fixed path, rewritten to read `$RUN` so the gate + passes), was held as it was: `lab check` said "a row that a review had passed in ... is gone or changed", a + second passing review did not lift it, and putting the old row back brought back the failure it was changed + to fix. `lab review results pass "" --supersedes ` now names the earlier passing results + review by its `of`; it ends the note with "Supersedes the results review of .", and the rows that review + read are held no more once it is committed together with the rows it read, which are held in their place. A review that does not name the earlier one + still supersedes nothing, and a note may not end with that sentence by hand. `lab review` refuses it on a plan or a failing review, for an `of` no earlier passing + results review has, and when the earlier review read the rows as they are now; `lab check` refuses a note that + names no earlier passing results review of other rows. +- The skill `lab-review` changed: run `lab init` in a lab after upgrading. + ## 1.14.0 - **A home map has Current storylines.** When `lab.toml` names active storylines, one of the library's home diff --git a/docs/model.md b/docs/model.md index b35e857..1b822ac 100644 --- a/docs/model.md +++ b/docs/model.md @@ -199,6 +199,20 @@ Each arrow is a command that refuses when the step before it is missing. You see problem `lab check` reports at every step: a different result is a new experiment. A review is bound to the rows it read by its `of`, so only those are held: a draft row committed after a review, which that review never read, may still be corrected before the next review reads it. + **A review that supersedes another.** A reviewed row that must be read another way, to the same + result (a command that read the run by a fixed path and now reads `$RUN`), is not a new + experiment. A later results review that passes the new rows says so by name: `lab review + results pass "" --supersedes `, where + `` is the earlier passing results review's `of` (eight to sixteen of its first hex digits). + The command ends the note with "Supersedes the results review of .". Once that review is + committed together with the rows it read, the rows the earlier review read are held no more, and + the new rows are held in their place; until then it supersedes nothing. A note cannot end with that + sentence except through `--supersedes`. + A later review that passes changed rows without naming the earlier one supersedes nothing, so the + old rows are still held. `lab review` refuses `--supersedes` on a plan review or a failing one, an + `of` that no earlier passing results review has, and a review that read the rows as they are now; + `lab check` refuses a note that names a review that is not an earlier passing results review of + other rows. **The history that is read.** The check reads the history of the record under the lab's root as that root is now. Moving the lab's root inside its repository, or to another repository, starts again what is held: what was committed under the old root is not compared. And it needs the @@ -822,7 +836,8 @@ with each quote inside written twice, as a spreadsheet writes it, is read withou quote is a character like any other. **`reviews.tsv`** is written by `lab review`: `date`, `stage` (plan, results), `verdict` (pass, fail), -`by`, `of` (a hash of what was read), `note`. +`by`, `of` (a hash of what was read), `note`. A results review written with `--supersedes` ends its +note with "Supersedes the results review of ." (section 3, Reviewed rows stay). **`resources.toml`** has one table per resource: `what`, `key` (where it is kept), `rules` (a list of sentences), `retired` (why, once it is no longer used), and for a metered resource `unit`, `granted`, diff --git a/pyproject.toml b/pyproject.toml index c1cf7e8..65766a7 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -5,7 +5,7 @@ build-backend = "hatchling.build" [project] dynamic = ["readme"] name = "lab-kit-cli" -version = "1.14.0" +version = "1.15.0" description = "A research lab's record, its method, and the command that keeps both." license = "MIT" authors = [{ name = "Piergiuseppe Mallozzi" }] diff --git a/skills/lab-review/SKILL.md b/skills/lab-review/SKILL.md index 5071531..99ed4ff 100644 --- a/skills/lab-review/SKILL.md +++ b/skills/lab-review/SKILL.md @@ -42,6 +42,11 @@ A failed plan is fixed by its author and read again. can read, saying what you conclude ("The numbers are what the rows say, and the bars were fixed before the run."). What you re-ran and what you could not verify comes after it: the page shows the first sentence and folds the rest. +5. When rows that an earlier results review passed have changed since (`lab check` says "a row that a + review had passed in ... is gone or changed"), read what changed (`git diff` of `results.tsv` from + the commit it names) as well as the rows. Pass it only when each changed row still reports the same + result, read another way, and add `--supersedes `, naming that earlier review's `of` from + `reviews.tsv`: the note says why the rows changed. A different result is a new experiment: fail it. A failed results review is final for that experiment. Say in the note what a replacement must do differently. Its numbers can then be cited nowhere: if a storyline page lists it under included, `lab check` diff --git a/src/lab_kit/__init__.py b/src/lab_kit/__init__.py index 399196c..d184348 100644 --- a/src/lab_kit/__init__.py +++ b/src/lab_kit/__init__.py @@ -1,2 +1,2 @@ """lab-kit: a research lab's record, its method, and the command that keeps both.""" -__version__ = "1.14.0" +__version__ = "1.15.0" diff --git a/src/lab_kit/checks.py b/src/lab_kit/checks.py index b47703d..3f585c9 100644 --- a/src/lab_kit/checks.py +++ b/src/lab_kit/checks.py @@ -305,6 +305,10 @@ def see(where, page, who=""): if r["stage"] not in ("plan", "results") or r["verdict"] not in ("pass", "fail") or not R.day(r["date"]) or not r["of"]: bad("reviews", p, f"a row of reviews.tsv is not one `lab review` wrote: {R.flat(' '.join(r.values()))[:60]!r}") break + for i, r in enumerate(x["reviews"]): + m = R.SUPERSEDING.search((r.get("note") or "").rstrip()) if r.get("stage") == "results" and r.get("verdict") == "pass" else None + if m and not any(e.get("stage") == "results" and e.get("verdict") == "pass" and e.get("of") == m.group(1) and e.get("of") != r.get("of") for e in x["reviews"][:i]): + bad("reviews", p, f"a results review supersedes {m.group(1)}, which is no earlier results review of other rows that passed") if x["results"]: if not x["lock"] or not x["finished"]: bad("results", p, "has results with no locked plan and finished run behind them") diff --git a/src/lab_kit/cli.py b/src/lab_kit/cli.py index 3f70f52..ac1bb38 100644 --- a/src/lab_kit/cli.py +++ b/src/lab_kit/cli.py @@ -34,7 +34,9 @@ lab runs close "" [--used =]... a run whose process died; --used for one that wrote no used.json lab runs close "" --used =... --evidence an ended run that did not record what it used of a resource it ran on lab runs lost "" [--used =]... a run that started and whose folder is gone for good - lab review plan|results pass|fail "" [--by ] + lab review plan|results pass|fail "" [--by ] [--supersedes ] + --supersedes: a results review that passes rows changed after the review + whose `of` it names had passed them; the note says why they changed lab check [--no-rederive] [--strict] runs every command in results.tsv: only on a lab you trust. --strict fails when history or the store could not be checked, as in CI Once @@ -191,10 +193,11 @@ def flag(name, many=False, switch=False): else: raise SystemExit(USAGE) elif cmd == "review": - by = flag("--by") + named = "--supersedes" in rest + by, supersedes = flag("--by"), flag("--supersedes") if len(rest) != 4: raise SystemExit(USAGE) - print(f"recorded in {write.review(lab, rest[0], rest[1], rest[2], rest[3], by or None)}") + print(f"recorded in {write.review(lab, rest[0], rest[1], rest[2], rest[3], by or None, supersedes if named else None)}") elif cmd == "check": if set(rest) - {"--no-rederive", "--strict"}: raise SystemExit(USAGE) diff --git a/src/lab_kit/history.py b/src/lab_kit/history.py index 193d54b..7431d6d 100644 --- a/src/lab_kit/history.py +++ b/src/lab_kit/history.py @@ -382,6 +382,7 @@ def check(lab): rows, read = where.rsplit("/", 1)[0] + "/results.tsv", set() if rows not in now: continue # never committed, so nothing is held; or committed and gone, which is said above + held_at = [] # (commit, the rows it holds, the last review that passed them there), newest first for c, path in reversed(commits + versions.get(rows, [])): # every commit that touched the reviews or the rows folder = os.path.dirname(path) if c in read: @@ -395,6 +396,13 @@ def check(lab): seals = [l[5] for l in listed if len(l) >= 6 and l[1] == "run" and l[2] == "ended" and l[4] == "0"] # the finished run, as that commit lists it if passed[-1].get("of") not in {R.text_hash(R.text_hash(was) + str(seal))[:16] for seal in seals}: continue # the last review that passed there read other rows than these + held_at.append((c, was, passed[-1])) + # A passing results review that names an earlier one by its `of` (`lab review ... --supersedes`) supersedes it + # only once it is committed together with the rows it read: then those rows are held instead of the earlier ones. + gone = R.superseded([r for _, _, r in held_at]) + for c, was, review in held_at: + if review.get("of") in gone: + continue # a later review, committed with the rows it read, superseded it and said why lost = subsequence(was.splitlines(), R.read(os.path.join(root, now[rows][0])).splitlines()) if lost is not None: bad(now[rows][0], f"a row that a review had passed in {c[:8]} is gone or changed: {lost[:70]!r}. Reviewed rows stay; a different result is a new experiment") diff --git a/src/lab_kit/record.py b/src/lab_kit/record.py index 82250d6..eb003c6 100644 --- a/src/lab_kit/record.py +++ b/src/lab_kit/record.py @@ -52,6 +52,18 @@ # it lost (what it used, and why). A file written before `used` and `note` existed has the first six columns. LEDGER = ("run", "kind", "what", "at", "exit", "seal", "used", "note") COLUMNS = {"results.tsv": ("name", "label", "value", "reads", "versus", "command"), "reviews.tsv": ("date", "stage", "verdict", "by", "of", "note")} +# A results review that passed rows changed after an earlier one passed them ends its note with this, naming the +# earlier review by its `of` (`lab review ... --supersedes`): the rows the earlier one read are then held no more. +SUPERSEDES = "Supersedes the results review of {}." +SUPERSEDING = re.compile(r"Supersedes the results review of ([0-9a-f]{16})\.$") + + +def superseded(reviews): + """The `of` of each results review that a later passing results review supersedes.""" + found = (SUPERSEDING.search((r.get("note") or "").rstrip()) for r in reviews if r.get("stage") == "results" and r.get("verdict") == "pass") + return {m.group(1) for m in found if m} + + HASH = 2 # how a folder is hashed in the locks and runs made now; one without the marker was made with the first way NUMBER = re.compile(r"[+-]?(?:[0-9]+(?:\.[0-9]*)?|\.[0-9]+)(?:[eE][+-]?[0-9]+)?") LISTS = {"goal": ("see",), "question": ("needs", "see"), "experiment": ("needs", "uses", "see"), "build": ("needs", "commits", "tests", "see"), diff --git a/src/lab_kit/write.py b/src/lab_kit/write.py index 345b302..138f05f 100644 --- a/src/lab_kit/write.py +++ b/src/lab_kit/write.py @@ -317,15 +317,22 @@ def lock(lab, ident): return lab.rel(path) -def review(lab, ident, stage, verdict, note, by=None): - """One independent reading of a plan or of an experiment's results, bound to exactly what it read.""" +def review(lab, ident, stage, verdict, note, by=None, supersedes=None): + """One independent reading of a plan or of an experiment's results, bound to exactly what it read. With + `supersedes`, a passing results review of rows that were changed after an earlier results review passed them: it + names that review by its `of`, and the note says why the rows changed. The rows that review read are held no more; + these are, once committed.""" x = lab.nodes.get(ident) if not x or x["type"] != "experiment": stop(f"no experiment {ident}") if stage not in ("plan", "results") or verdict not in ("pass", "fail"): stop("lab review plan|results pass|fail \"\"") + if supersedes is not None and (stage != "results" or verdict != "pass"): + stop("only a results review that passes supersedes another") if not R.flat(note): stop("a review says what it read and what it found") + if R.SUPERSEDING.search(R.flat(note).rstrip()): + stop("a note does not end by superseding a review: name the earlier review with --supersedes , and the command writes that sentence") if stage == "plan" and x["lock"]: stop(f"{ident} is locked: a plan is reviewed before its lock") if stage == "results": @@ -333,6 +340,16 @@ def review(lab, ident, stage, verdict, note, by=None): stop(f"{ident} has no finished run with results to review") if lab.reviewed(x, "results")[0] == "fail": stop(f"{ident} already failed its results review, and that is final: replace the experiment") + if supersedes is not None: + if not re.fullmatch(r"[0-9a-f]{8,16}", supersedes or ""): + stop("--supersedes names the earlier review by its `of`, eight to sixteen hex digits") + earlier = {r["of"] for r in x["reviews"] if r.get("stage") == "results" and r.get("verdict") == "pass" and (r.get("of") or "").startswith(supersedes)} + if len(earlier) != 1: + stop(f"{ident} has {'no' if not earlier else 'more than one'} results review that passed whose `of` begins {supersedes}") + earlier = earlier.pop() + if earlier in (lab.content(x, stage), lab.content(x, stage, 1)): + stop(f"the review {earlier} read the rows as they are now: there is nothing to supersede, so review them without --supersedes") + note = f"{R.flat(note).rstrip()} {R.SUPERSEDES.format(earlier)}" path = os.path.join(lab.root, x["dir"], "reviews.tsv") if not os.path.exists(path): open(path, "w").write("date\tstage\tverdict\tby\tof\tnote\n") diff --git a/tests/test_held.py b/tests/test_held.py index 40dd711..3b44bd6 100644 --- a/tests/test_held.py +++ b/tests/test_held.py @@ -406,3 +406,64 @@ def test_a_page_of_another_library_is_looked_for_where_that_library_is(tmp_path) # a library the charter does not name is a problem wherever the check runs front(f"{qdir}/question.md", see="[elsewhere:/content/readings/the-law.html]") assert "which is not a page of a library the charter names" in lab(root, "check", ok=False) + + +def test_a_later_review_supersedes_an_earlier_one_only_by_naming_it(tmp_path): + root = under(tmp_path, "lab") + ident, d = finished(root) # its rows reviewed, passed and committed + rows, reviews, held = f"{d}/results.tsv", f"{d}/reviews.tsv", "a row that a review had passed in" + was = open(rows).read() + first = [l.split("\t") for l in open(reviews).read().splitlines() if "\tresults\tpass\t" in l][-1][4] + + # a reviewed row read again another way, to the same value: held, and a second review alone does not change that + open(rows, "w").write(was.replace("cat $RUN/ratio.txt", "python3 -c \"print(open('$RUN/ratio.txt').read().strip())\"")) + assert open(rows).read() != was + assert held in lab(root, "check", "--no-rederive", ok=False) + lab(root, "review", ident, "results", "pass", "Read the new command; it prints the same value.") + commit(root, "the row read another way, reviewed again") + assert held in lab(root, "check", "--no-rederive", ok=False) + + # what --supersedes refuses: a plan review, a fail, a review that is not there, one that read the rows as they are now + second = [l.split("\t") for l in open(reviews).read().splitlines() if "\tresults\tpass\t" in l][-1][4] + assert "only a results review that passes" in lab(root, "review", ident, "plan", "pass", "x", "--supersedes", first, ok=False) + assert "only a results review that passes" in lab(root, "review", ident, "results", "fail", "x", "--supersedes", first, ok=False) + assert "eight to sixteen hex" in lab(root, "review", ident, "results", "pass", "x", "--supersedes", "zz", ok=False) + assert "no results review that passed" in lab(root, "review", ident, "results", "pass", "x", "--supersedes", "0123abcd", ok=False) + assert "nothing to supersede" in lab(root, "review", ident, "results", "pass", "x", "--supersedes", second, ok=False) + assert "eight to sixteen hex" in lab(root, "review", ident, "results", "pass", "x", "--supersedes", "", ok=False) + assert "name the earlier review with --supersedes" in lab(root, "review", ident, "results", "pass", f"x. Supersedes the results review of {first}.", ok=False) + + # named, with why: the rows the first review read are held no more, and these are + out = lab(root, "review", ident, "results", "pass", "The row now reads the value through $RUN, as the gate asks; the value is the same.", "--supersedes", first[:8]) + assert "recorded in" in out + assert open(reviews).read().rstrip().endswith(f"Supersedes the results review of {first}.") + commit(root, "the first review superseded") + assert "the record is sound" in lab(root, "check", "--no-rederive", "--strict") + open(rows, "w").write(was) + assert held in lab(root, "check", "--no-rederive", ok=False) # the superseding review's rows are held in their turn + + +def test_a_review_that_names_no_earlier_pass_supersedes_nothing(tmp_path): + root = under(tmp_path, "lab") + ident, d = finished(root) + reviews = f"{d}/reviews.tsv" + line = [l for l in open(reviews).read().splitlines() if "\tresults\tpass\t" in l][-1] + open(reviews, "a").write(line + " Supersedes the results review of 0123456789abcdef.\n") + assert "which is no earlier results review of other rows that passed" in lab(root, "check", "--no-rederive", ok=False) + + +def test_a_superseding_review_frees_nothing_until_its_own_rows_are_committed_with_it(tmp_path): + root = under(tmp_path, "lab") + ident, d = finished(root) + rows, reviews, held = f"{d}/results.tsv", f"{d}/reviews.tsv", "a row that a review had passed in" + was = open(rows).read() + first = [l.split("\t") for l in open(reviews).read().splitlines() if "\tresults\tpass\t" in l][-1][4] + other = was.replace("cat $RUN/ratio.txt", "cat \"$RUN/ratio.txt\"") + open(rows, "w").write(other) + lab(root, "review", ident, "results", "pass", "The command quotes $RUN now; same value.", "--supersedes", first) + git(root, "add", reviews) + git(root, "commit", "-qm", "the review, without the rows it read") + open(rows, "w").write(other.replace("cat \"$RUN/ratio.txt\"", "cat $RUN/ratio.txt # a third way")) + lab(root, "review", ident, "results", "pass", "Read the rows again.") + commit(root, "a third way, reviewed without naming anything") + assert held in lab(root, "check", "--no-rederive", ok=False)