valis - Working with the Code

valis is grown against a live image, not a cold edit-compile-run cycle. You keep an SBCL image loaded with the system and its state, prototype a form at the REPL, persist it to its file, re-evaluate to confirm, then run the tests, all without restarting. The editor and the running image see the same world.

EXPLORE -> EXPERIMENT -> PERSIST -> VERIFY
   ^                              |
   +----------- REFINE -----------+

The one discipline this trades for: warm images hide missing dependency declarations. valis is a package-inferred-system where each file's defpackage declares its own :import-from, with no central component list. A form that resolves in your warm REPL because some other file already loaded the symbol will fail a cold build. So the authoritative gate is always a cold run: make test in a fresh SBCL (see Testing).

• SBCL 2.x and ASDF 3.x.
• The seven sibling 9P library, version 1.2.0 or later. The floor whose session close reports whether every thread inside a session has left, which valis waits on before it releases an owner session's endpoint.
• The mercer sibling, version 0.7.0 or later. The factotum: the Noise_XX transport authenticator, the post-attach session cipher, and the ACME certificate custody the :443 edge draws on.
• The runciter sibling, version 0.8.0 or later. The authoritative-DNS answer logic the :53 serving path binds to.
• A PostgreSQL instance for the operator-state tests. Without VALIS_PG_DSN the suite stands those tests down and tells you so; scripts/pg-dev.sh brings a development one up.
• The remaining dependencies, resolvable through $LISP_WORKSPACE/ or Quicklisp.

valis resolves dependencies in a strict order (see Conventions): the project's own code first, then a local fork under $LISP_WORKSPACE/, then Quicklisp as a last resort. Point $LISP_WORKSPACE at your local-projects tree (default ~/SourceCode/lisp/) so the build picks up seven and the other local checkouts.

Bring up a live image with a Slynk listener and attach your editor (Sly, SLIME, or any Swank/Slynk client) to it:

./scripts/dev-boot.sh               # foreground
./scripts/dev-boot.sh --background  # detach; logs to /tmp/valis-dev.log

The listener honours SLYNK_PORT (default 20159 from this script; the built binary's own --dev defaults to 4005), SLYNK_HOST (default 127.0.0.1), LISP_WORKSPACE, and QUICKLISP_SETUP. The boot loads the valis system, so by the time you connect the world is ready.

From a bare SBCL the equivalent is:

(asdf:load-system :valis)

The minimal cycle is prototype → persist → re-load → test:

1. Prototype a form by evaluating it at the REPL (in the right package: each file has its own, e.g. valis/src/listener, valis/src/fabric).
2. Persist it: edit the form in its source file. Prefer structure-aware editing (your editor's "compile defun / send form" command) so indentation and surrounding comments are preserved.
3. Re-load to pick up file changes. Edited files do not auto-reload; the reliable way is (asdf:load-system :valis) (it recompiles only what changed). Compiling a single form from the editor is fine for a tight loop, but reload the system before you trust a result.
4. Verify with the test suite (see below).

• The package-per-file shape means symbols are namespaced. When a symbol is not visible, either the system is not loaded, or you are in the wrong package: qualify it (valis/src/fabric::symbol) or switch packages.
• Once the system is loaded, the editor's "find definition", "who-calls", and "describe symbol" commands are the fastest way to move; they read the live image, so they are precise.
• For a filesystem-level search that does not need the system loaded, grep the src/ tree.

Tests are written against zebra (define-test, true / false / fail / is), which each test package =:use=s. The authoritative gate is the cold suite:

make test          # the whole suite in a fresh SBCL
make test-one TEST=valis/tests/<name>   # one suite, for the inner loop
make interop-test  # drive a real Linux v9fs mount against the namespace
                   # (requires sudo / CAP_SYS_ADMIN)

make test runs the linter first (see Linting), so a lint finding fails the run before the suite starts. It reports its own counts as it runs, and it distinguishes three outcomes that a single exit code would flatten: tests ran and some failed, a named test package was never interned so nothing was judged at all, or the image died. Read what it prints rather than quoting a count from here, because the count moves daily and a number written down in a document is wrong the moment it is written.

Every gate test runs under a deadline, 120 seconds by default (*test-deadline*, defined in tests/support/test-deadline.lisp; rebind it to change the limit, or bind it to nil to turn it off). A test that runs past it is failed, and the run prints every thread's backtrace and the owner of every lock a thread is waiting on, then moves on to the next test, so a hang costs you one failed test with the evidence beside it. A test that cannot finish its own cleanup within a further grace period ends the run with exit status 12, because grading later tests against an image still stuck on that one would prove nothing. Wrap work that carries its own bound, such as building the saved image a child process starts from, in off-the-clock: the deadline neither counts that time nor fires during it, and the clock runs again once the work finishes, so a test that hangs afterwards is still caught.

⚠ Without VALIS_PG_DSN the operator-state tests stand down. That is a success, not a failure: they say why they stood down, and going red because Postgres is not running would put a false entry beside the real failures and make those easy to skim past. The run prints a banner loud enough to survive a scrollback. Bring a database up with eval "$(scripts/pg-dev.sh dsn)" before you trust a green on anything they cover.

During the inner loop you can run the test system from the REPL for fast feedback:

(asdf:test-system :valis/tests)

That runs the gate, the same tests make test runs, and holds back the few listed for the nightly tier in tests/support/test-tiers.lisp. Run make test-nightly for the whole suite, those included, and run it before every deploy, since the gate alone never exercises the tests it holds back.

…but a green warm REPL is not sufficient. Re-run make test cold before you consider a change done, so a missing :import-from cannot hide.

make interop-test stands up an in-process fabric, publishes a sample item under /pub, performs an anonymous v9fs attach (-t 9p ... access=any), and asserts the reconciled model: the mount names /pub and traverses the item while the credential-gated base tree is absent, the same published floor an unauthenticated HTTP GET sees. Run it in the foreground: its readiness fifo read blocks if the server child dies, so a backgrounded run can hang.

The real-NIC steering acceptance gate is not in this repo: it lives in fulcrum (the privileged sibling) as make steer-test, a netns target proving a client dialing :80 lands on valis's single steered socket with the original port recovered.

The extension surface valis is built for. A new wire protocol is a new file src/<proto>.lisp that:

1. Subclasses protocol (src/protocol.lisp), the class that carries the per-port contract.
2. Specialises handle-connection, the per-connection entry point. The recovered destination port has already selected your module; you receive the connection with its principal and capabilities resolved.
3. Registers the protocol against the port it claims, so dispatch-connection (src/registry.lisp) routes matching connections to it.
4. Projects over a semantic subtree through the capability-mounted namespace, rather than owning state directly. Read through the published /pub view for an anonymous caller and the fuller frame for a proven owner; never reach around the namespace to touch irreplaceable state.

Registering a protocol wires dispatch: it routes the connections that arrive on your port to your module, and step 2's "the recovered destination port has already selected your module" assumes the traffic reached valis. Whether the traffic gets that far is a separate question with a separate answer, below.

On a deployed host the privileged agent's catchall steer fans every port to valis, and a separate, privileged firewall selects which are reachable from off-host. What puts your port in front of that firewall is loading the module: valis binds it, declares the whole set it serves, and the agent admits what the declaration names. The firewall's rules stay the agent's alone and nothing a module does can reach them, so a module can never widen the ruleset; what your declaration reaches is set membership. A UDP protocol needs one step more: the steer cannot carry a UDP reply, so the agent binds the port and hands its descriptor down, as :53 already does. The public surface chapter is the full treatment.

The HTTP adapter is the worked reference: a conformant-but-bounded HTTP/1.1 server that serves two identity-selected views over one URL space. It is no longer in this tree: the HTTP and Gopher engines are the c3po-http and c3po-gopher modules from the c3po protocol sibling, adopted as content-addressed edge-adapter modules that self-register onto the edge-adapter registry (valis/src/plugin/edge-adapter) and bind their ports through the edge controller. Read them there and model a new adapter on them. The Architecture document explains the edge↔core seam your adapter sits behind, and the API reference lists the exported entry points you specialise.

A wire-protocol adapter is one of two extension surfaces; the other is a keyed, authenticated service: a 9P responder a remote peer reaches by proving a key, its session sealed end to end. Both, and the mount-not-guard authority model they share, are written up for the operator building their own system in Building on valis. This section is the in-tree adapter walk-through; that guide is the from-outside one.

make build          # the shipped binary at bin/valis
make dist           # stage the versioned delivery tarball for a host

make build drives asdf:make :valis/delivery, which composes the additive DNS serving stack into the image so the resident can answer public :53. A binary built from the plain :valis system cannot, and fails closed at boot saying so. The older build.sh still works and is superseded.

Process lifecycle lives in src/main.lisp. The resident steered boot (--resident, run-daemon) adopts the inherited :53 descriptors from fulcrum and serves over them without a privileged bind. Beyond that the binary carries --dev for a Slynk listener, --backup and --restore for the backup-critical set, and the zone, obtain, publish, publisher, config and restart verbs, which drive a running resident owner-keyed over its loopback fabric rather than editing anything on the host (config also changes a stopped node's own settings, and zone has routes for a node that is not serving yet). The rest do address the host: apply puts a delivery archive already on the host into place and confirms the node answers, dist registers the private dependency distribution as this node's dependency source, assess reads the host and reports whether it is fit to run a node, changing nothing, and module tries a module install or supersede in a process of its own, binding nothing. valis --help prints the grammar; --daemon survives as a no-op.

make dist builds the valis binary from committed state, in a throwaway worktree at the current commit, never from your working tree. That commit is recorded and need not be published. Every other repository that supplied an input to either binary in the tarball, valis or the fulcrum resident, must be clean, and its HEAD must be on origin/main or origin/master, or the target refuses; where the remote's HEAD points never counts. The input set is what the build actually loaded. To ship something unpublished on purpose, name its directory in VALIS_DIST_ALLOW_UNPUBLISHED; a modified tracked file is never admitted. To build against clones of the sibling repositories pinned to published commits rather than against your workspace checkouts, name their directories in VALIS_DIST_SOURCE_DIRS (colon separated): they are searched ahead of the workspace and checked like any other input. The tarball carries share/valis/build-provenance.txt, so you can tell from a deployed node which commit of each repository it runs. scripts/dist-provenance.sh and scripts/dist-source-registry.sh say exactly what is checked and how the registry is assembled.

Standing a node up on a real Debian host (the bring-up order, provisioning, and fail-closed invariants) is written up in the host-deployment contract (deploy = condense-from-genesis). The =proving-ground/ tree is its regression harness: a Nix devshell fabric, a Terraform-libvirt two-location Debian topology, cloud-init provisioning, and an in-image Lisp/SSH assertion layer (valis-proving-ground, its own ASDF system) that drives the deploy / evacuate / restore scenarios. Load and test the harness like any system: (asdf:test-system :valis-proving-ground/tests).

These are not suggestions; they are how the substrate stays coherent.

• Dependency hierarchy. When you need a capability, look in strict order: (1) the project's own code: search before reaching for a library; (2) a local fork under $LISP_WORKSPACE/: fork the upstream, keep it current, add it as a local ASDF dependency so it stays patchable; (3) Quicklisp, only for stable deps with no local checkout. Prefer com.inuoe.jzon over yason for JSON. Adding a dependency is a durable decision. When in doubt, ask.
• System shape. valis.asd is a package-inferred-system: each file's defpackage declares its own dependencies; there is no central component list. Always cold-verify: a warm REPL hides a missing :import-from.
• Licensing. Every source file opens with an SPDX header (;;;; SPDX-License-Identifier: AGPL-3.0-or-later).
• Naming. Name a symbol (function, variable, class, macro, test) for the behaviour it implements. Never embed planning or ticket indices in a symbol.
• Comments. Write the what and the why. The code is the canonical how; prose narration of the implementation rots silently when the code changes.
• Handling failure. Define a condition, bind a handler that can discriminate causes, offer a named restart. A handler that cannot tell two causes apart converts a defect into a plausible value. See Condition handling. It is enforced, not merely advised.

scripts/lint-lisp.sh runs the mallet linter over the tree; --staged limits it to staged files, which is how the pre-commit hook invokes it. Configuration is .mallet.lisp at the repo root.

scripts/lint-lisp.sh              # whole tree
scripts/lint-lisp.sh --staged     # staged files only
scripts/lint-lisp.sh src/mail     # a subtree

The linter lives at $LISP_WORKSPACE/mallet. Beware a name collision: a mallet on the system PATH is an unrelated topic-modeling toolkit, so the runner resolves the path explicitly rather than by PATH: resolving by PATH would lint nothing and report success. Override with MALLET_BIN if your checkout is elsewhere.

make lint runs it over src/ and tests/, and make test runs make lint first. Without the linter checkout, scripts/lint-lisp.sh exits 127 and says where it looked, so make test stops there too. To gate commits locally, have .git/hooks/pre-commit call scripts/lint-lisp.sh --staged, which lints the staged content of each staged file rather than the working copy.

What it enforces, and why each is set where it is:

The baseline names each declined site by file and top-level form, with its reason, so an unrelated edit moves nothing. When you convert a site to a condition class, lower or delete its line in the same change. Condition handling says when a site belongs in the baseline at all.

A genuinely ignorable failure is suppressed at the site with a written reason (; mallet:suppress no-ignore-errors -- why). The reason is the point: the linter reports suppressions that have gone stale, so they do not accumulate silently.

Treat a rule's count as a floor until you have checked what it matches. error-without-custom-condition fires only when the message string contains no colon (format arguments are irrelevant), so (error "prefix: message") goes unreported. It sees 13 sites in src/ where (error " appears 77 times, because 65 of them carry a colon. The rule is blinded by good message hygiene, which is a defect in the linter rather than a design choice.

Before trusting any rule's number, write a file containing the shapes you believe it catches and confirm which report, and if the tool is one of ours, read its source rather than probing it. Do not strip colons from error messages to make this rule's count move: the prefixes are worth more than the metric, and the defect is the linter's. That predicate is six lines; black-box probing of it produced three different confidently-held theories before anyone opened the file.

Two shell-level versions of the same trap, both encountered in practice: a mallet resolved from PATH lints nothing and exits 0, and a file list built with mapfile under zsh lints an unintended set while still printing a confident number. Assert your file count if you glob it.

• "Symbol not found": the system is not loaded ((asdf:load-system :valis)), or you are in the wrong package (qualify with pkg::symbol).
• Green warm, red cold: a missing :import-from in some file's defpackage. The warm image had the symbol from another file; the cold build does not. Add the declaration to the file that uses the symbol.
• make interop-test hangs: it was backgrounded, or the server child died while the harness waited on the readiness fifo. Run it in the foreground; ensure passwordless sudo and the 9p kernel modules are present.