Skip to content

Optionally reduce test transcript verbosity with {include:silent} #29

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
stormi wants to merge 3 commits into
base: master
Choose a base branch
from
Open

Optionally reduce test transcript verbosity with {include:silent} #29

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
da08273
Add ** include: directive for external file inclusion
stormi Jul 26, 2026
3ddcf16
Add a {include:silent} modifier for {include}
stormi Jul 30, 2026
c8f9faf
Silence the setup that precedes a silent first action
stormi Jul 30, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 40 additions & 1 deletion regtest.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -124,6 +124,18 @@ <h3>The Test File</h3>
A line beginning with <code>"** checkclass:"</code> specifies a (Python) file containing extra check classes. I won't get into the details here, but see <a href="extracc.py">this sample file</a>.
</p>

<p>
A line beginning with <code>"** include:"</code> specifies a file to include at that point in the test file. Like all <code>**</code> directives, it must appear before the first test definition. The included file is parsed recursively; its tests, directives, and nested includes take effect as if the content were pasted inline. File paths are resolved relative to the directory of the file containing the <code>** include:</code> directive. You can use multiple <code>** include:</code> lines to include several files.
</p>

<p>
Because included content is parsed inline, directives in included files follow the same rules as in the main file: <code>** game:</code> overrides any prior value, <code>** precommand:</code> appends to the precommand list, and so on. Test names must be unique across all files. A duplicate test name will cause an error. Circular includes (A includes B which includes A) are also an error.
</p>

<p>
An included file can contain anything the main file can: top-level directives, test definitions, comments, and even more <code>** include:</code> lines.
</p>

<p>
The rest of the test file is a set of <em>tests</em>. Each test is a separate run through the game. A test contains a sequence of <em>commands</em>. A command can contain various <em>checks</em>, validating the output of that command.
</p>
Expand Down Expand Up @@ -264,7 +276,31 @@ <h3>Partial Tests</h3>
</p>

<p>
(No space between the "&gt;" and the "{". Checks after an <code>&gt;{include}</code> line are meaningless; they are ignored.)
To silence the output of an included test in verbose mode, use the <code>:<i>silent</i></code> modifier:
</p>

<p>
<code>&gt;{include:silent} TESTNAME</code>
</p>

<p>
This executes all of the included test's commands normally, but suppresses their output from the verbose transcript. The included test's checks still run and will still report failures. To override the silencing and see everything, pass <code>-vv</code> instead of <code>-v</code>.
</p>

<p>
In <code>-v</code> mode, a marker line <code>[silently included: TESTNAME]</code> is printed in place of the hidden commands, so the transcript still shows where the included test ran. This marker is not printed at <code>-vv</code>, since everything is already visible there.
</p>

<p>
When the first thing a test does is silenced, whatever comes before it is silenced with it: the game's initial output (before any command has been sent), and any <code>** precommand:</code> lines or <code>-p</code> command-line precommands running ahead of it. That whole span is the boilerplate setup you already saw when the included test ran on its own, so the transcript of each test starts where the test itself does. This happens automatically; there's no separate way to mark it as silent, and <code>-vv</code> still shows everything.
</p>

<p>
A test starts silently when its first command is <code>&gt;{include:silent}</code>, when its first command is a plain <code>&gt;{include}</code> of a "wrapper" test which itself starts silently (RegTest follows the chain of leading includes), or when the silent include is a precommand (<code>** precommand: {include:silent} TESTNAME</code>, or the same through <code>-p</code>) and therefore the first thing every test in the file does. If something you can see runs first instead, such as a plain precommand ahead of the silent include, the initial output stays visible along with it.
</p>

<p>
(No space between the "&gt;" and the "{". Checks after an <code>&gt;{include}</code> line are meaningless; they are ignored.)
</p>

<p>
Expand Down Expand Up @@ -377,6 +413,9 @@ <h3>Dictionary of Inputs</h3>
<dt>&gt;{include} testname
<dd>Performs all the commands and checks in the named test.

<dt>&gt;{include:silent} testname
<dd>Same as <code>{include}</code>, but suppresses the included commands' output from the verbose transcript. Their checks still run. Use <code>-vv</code> to override this silencing. In <code>-v</code> mode, a <code>[silently included: testname]</code> marker line is printed in place of the hidden commands.

</dl>

<h3>Dictionary of Check Modifiers</h3>
Expand Down
134 changes: 107 additions & 27 deletions regtest.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -96,8 +96,9 @@ class RegTest:
A test is one session of the game, from the beginning. (Not necessarily
to the end.) After every game command, tests can be run.
"""
def __init__(self, name):
def __init__(self, name, testfile):
self.name = name
self.testfile = testfile
self.gamefile = None # use global gamefile
self.terp = None # global terppath, terpargs
self.precmd = None
Expand Down Expand Up @@ -125,14 +126,13 @@ class Command:
def __init__(self, cmd, type=None):
if type is None:
# Peel off the "{...}" prefix, if found.
match = re.match('{([a-z_]*)}', cmd)
match = re.match('{([a-z_:]*)}', cmd)
if not match:
type = 'line'
cmd = cmd.strip()
else:
type = match.group(1)
cmd = cmd[match.end() : ].strip()

self.type = type
if self.type == 'line':
self.cmd = cmd
Expand Down Expand Up @@ -183,7 +183,9 @@ def __init__(self, cmd, type=None):
self.height = int(ls[1])
except:
pass
elif self.type == 'include':
elif self.type in ['include', 'include:silent']:
self.silent = self.type.endswith(':silent')
self.type = 'include'
self.cmd = cmd
elif self.type == 'fileref_prompt':
self.cmd = cmd
Expand Down Expand Up @@ -582,7 +584,7 @@ def initialize(self):
def perform_input(self, cmd):
raise Exception('perform_input not implemented')

def accept_output(self):
def accept_output(self, silenced=False):
raise Exception('accept_output not implemented')

class GameStateCheap(GameState):
Expand All @@ -597,7 +599,7 @@ def perform_input(self, cmd):
self.infile.write((cmd.cmd+'\n').encode())
self.infile.flush()

def accept_output(self):
def accept_output(self, silenced=False):
self.storywin = []
output = bytearray()

Expand All @@ -616,7 +618,7 @@ def accept_output(self):

dat = output.decode('utf-8')
res = dat.split('\n')
if (opts.verbose):
if opts.verbose and not silenced:
for ln in res:
if (ln == '>'):
continue
Expand Down Expand Up @@ -714,7 +716,7 @@ def perform_input(self, cmd):
self.infile.write((cmd+'\n').encode())
self.infile.flush()

def accept_output(self):
def accept_output(self, silenced=False):
import json
output = bytearray()
update = None
Expand Down Expand Up @@ -748,7 +750,7 @@ def accept_output(self):
if time.time() >= timeout_time:
raise Exception('Timed out awaiting output')

self.parse_remglk_update(update)
self.parse_remglk_update(update, silenced=silenced)

def construct_remglk_input(self, cmd):
if cmd.type == 'line':
Expand Down Expand Up @@ -804,7 +806,7 @@ def construct_remglk_input(self, cmd):
print()
return update

def parse_remglk_update(self, update):
def parse_remglk_update(self, update, silenced=False):
# Parse the update object. This is complicated. For the format,
# see http://eblong.com/zarf/glk/glkote/docs.html

Expand Down Expand Up @@ -851,7 +853,7 @@ def parse_remglk_update(self, update):
if text:
for line in text:
dat = self.extract_text(line)
if (opts.verbose == 1):
if opts.verbose == 1 and not silenced:
if (dat != '>'):
print(dat)
if line.get('append') and len(self.storywin):
Expand Down Expand Up @@ -955,14 +957,14 @@ def perform_input(self, cmd):
(outdat, errdat) = proc.communicate((cmd+'\n').encode(), timeout=opts.timeout_secs)
self.pendingupdate = outdat.decode()

def accept_output(self):
def accept_output(self, silenced=False):
import json
dat = self.pendingupdate
self.assert_json(dat)
update = json.loads(dat)
self.pendingupdate = None

self.parse_remglk_update(update)
self.parse_remglk_update(update, silenced=silenced)

class ObjPrint:
NoneType = type(None)
Expand Down Expand Up @@ -1111,12 +1113,42 @@ def parse_checkfile(filename):
finally:
fl.close()

def parse_tests(filename):
"""Parse the test file. This fills out the testls array, and the
other globals which will be used during testing.
def parse_file(filename, base_dir, testfile, visiting):
"""Parse a test file, handling ** include directives recursively.

filename: the file path to parse
base_dir: parent directory — used to resolve the given filename if it
is not absolute (for the top-level file this is CWD, for an
included file this is the directory of the including file)
testfile: file name shown in error messages
visiting: ordered list of (filename, absolute-path) tuples of all
files currently in the include chain, from outermost to most
recent. Used to detect circular includes: if the file we're
about to parse is already in the chain, we raise an error
showing the full cycle path.

Note on path resolution:
- The file being parsed (filename) is resolved relative to base_dir.
- Files listed in that file's ** include: are resolved relative to
the including file's directory (abspath below), not base_dir.
This lets you nest includes in subdirectories without having to
rewrite paths on every level.
"""
global gamefile, terppath, terpargs, terpformat

if not os.path.isabs(filename):
filename = os.path.join(base_dir, filename)
abspath = os.path.abspath(filename)

# Check for circular includes
visited_paths = [p for _, p in visiting]
if abspath in visited_paths:
cycle = ' -> '.join([sf for sf, _ in visiting] + [testfile])
raise Exception('Circular include chain: ' + cycle)

# Push self onto the chain so nested includes can detect cycles
visiting.append((testfile, abspath))

fl = open(filename)
curtest = None
curcmd = None
Expand Down Expand Up @@ -1153,8 +1185,14 @@ def parse_tests(filename):
terpformat = 'rem' if (val.lower() > 'og') else 'cheap'
elif (key == 'checkclass'):
parse_checkfile(val)
elif (key == 'include'):
# Resolve included file relative to THIS file's
# directory (not the outer base_dir), so that nested
# includes in subdirectories work naturally.
inc_base = os.path.dirname(abspath)
parse_file(val, inc_base, val, list(visiting))
else:
raise Exception('Unknown option: ** ' + key)
raise Exception('Unknown option: ** ' + key + ' (in ' + testfile + ')')
else:
if (key == 'game'):
curtest.gamefile = val
Expand All @@ -1168,8 +1206,11 @@ def parse_tests(filename):
if (ln.startswith('*')):
ln = ln[1:].strip()
if (ln in testmap):
existing = testmap[ln]
if existing.testfile != testfile:
raise Exception('Test name used twice: ' + ln + ' (in ' + testfile + ' and ' + existing.testfile + ')')
raise Exception('Test name used twice: ' + ln)
curtest = RegTest(ln)
curtest = RegTest(ln, testfile=testfile)
testls.append(curtest)
testmap[curtest.name] = curtest
curcmd = Command('(init)')
Expand All @@ -1186,9 +1227,20 @@ def parse_tests(filename):
fl.close()


def list_commands(ls, res=None, nested=()):
def parse_tests(filename):
"""Parse the test file. This fills out the testls array, and the
other globals which will be used during testing.
"""
base_dir = os.getcwd()
parse_file(filename, base_dir, filename, [])


def list_commands(ls, res=None, nested=(), silenced=False):
"""Given a list of commands, replace any {include} commands with the
commands in the named subtests. This works recursively.
commands in the named subtests. This works recursively. The result is
a list of (command, silenced) pairs; a command is silenced when it
comes from a {include:silent}, which also appends the include command
itself (marked silenced) to stand in for the hidden block.
"""
if res is None:
res = []
Expand All @@ -1199,11 +1251,21 @@ def list_commands(ls, res=None, nested=()):
test = testmap.get(cmd.cmd)
if not test:
raise Exception('Included test not found: %s' % (cmd.cmd,))
list_commands(test.cmds, res, nested+(cmd.cmd,))
if cmd.silent:
res.append((cmd, True))
list_commands(test.cmds, res, nested+(cmd.cmd,), silenced=silenced or cmd.silent)
continue
res.append(cmd)
res.append((cmd, silenced))
return res

def starts_silenced(cmdlist):
"""Whether the first action of a flattened command list is silenced.
Whatever gets printed before that action (the game's initial output,
or precommands running ahead of it) is part of the same silenced
span, since it is all setup leading up to it.
"""
return bool(cmdlist) and cmdlist[0][1]

class VitalCheckException(Exception):
pass
class NotJSONException(Exception):
Expand Down Expand Up @@ -1240,11 +1302,21 @@ def run(test):
else:
raise Exception('Unrecognized format: %s' % (terpformat,))

cmdlist = list_commands(precommands + test.cmds)
# The test's own commands, flattened. When its first action turns out to
# be silenced (a leading {include:silent}, possibly reached through a
# chain of plain {include}s), the global precommands running ahead of it
# are just as much setup you have already seen, so silence those too.
tail = list_commands(test.cmds)
cmdlist = list_commands(precommands, silenced=starts_silenced(tail)) + tail

try:
gamestate.initialize()
gamestate.accept_output()
# The initial output comes before the first action of all, so it is
# silenced whenever that action is, whether the silence originates
# in the test's own commands or in a precommand which is itself a
# silent include.
initsuppressed = starts_silenced(cmdlist) and opts.verbose < 2
gamestate.accept_output(silenced=initsuppressed)
if (test.precmd):
for check in test.precmd.checks:
res = check.eval(gamestate)
Expand All @@ -1255,8 +1327,16 @@ def run(test):
if check.vital:
raise VitalCheckException()

for cmd in cmdlist:
if (opts.verbose):
for cmd, silenced in cmdlist:
# An 'include' Command only ever reaches this list as a marker for a
# silenced block (list_commands() always continues past it otherwise).
if cmd.type == 'include':
if opts.verbose == 1:
print('[silently included: %s]' % (cmd.cmd,))
print()
continue
suppressed = silenced and opts.verbose < 2
if opts.verbose and not suppressed:
if cmd.type == 'line':
if terpformat == 'cheap':
print('> %s' % (cmd.cmd,))
Expand All @@ -1266,7 +1346,7 @@ def run(test):
else:
print('> {%s} %s' % (cmd.type, repr(cmd.cmd),))
gamestate.perform_input(cmd)
gamestate.accept_output()
gamestate.accept_output(silenced=suppressed)
for check in cmd.checks:
res = check.eval(gamestate)
if (res):
Expand Down