From bbe505c8d48e822c012c9820dfe93b83dc61bcab Mon Sep 17 00:00:00 2001 From: Ralph Amissah Date: Thu, 24 Sep 2026 16:33:53 -0400 Subject: --validate, and a second reader, in Gleam --validate: the specification, executed. A validator written from ssp-format-specification.md and from nothing else, so that a file can be held against the document rather than against an implementation. Each finding names the specification section it comes from, so a disagreement can be taken back to the text. It checks the version banner and its major part, the file shape and the order of the preamble blocks and the nine sections, the two spaces of a preamble pair, the object declaration and its sixteen kinds and eight heading levels, that a property is one of the 42 and in the emitted order, that text lines carry their prefix, the ten keys of doc_has, and that no line is of a shape the specification does not define. Every .ssp of the markup document samples validates clean. Five classes of deliberate breakage are caught: an unknown major version, a property that is not one of the 42, a property out of order, a kind that is not one of the sixteen, and a block that is neither preamble nor section. Writing it found two rules the specification implied and did not state (repeated properties are consecutive; no property follows the text), and both were written into it. One repository, a reader per language. The readers are not separate tools that share a name. They are the same specification implemented more than once, and the interesting thing about them is that they agree, so they are published as one thing with that claim at the top of it. The D reader moves to dlang/; the Gleam reader arrives at gleam/. The language directory is named for the language and not for the tool: gleam refuses a hyphenated package name, so ssp-reader cannot be a gleam package and must be ssp_reader. A .gitignore for the repository, denying everything and whitelisting what belongs, covering both languages. Both build under nix, sandboxed and without network. There is no gleam builder in nixpkgs and gleam build fetches from Hex, so the Gleam build uses nix-gleam's builder, kept in gleam/nix-gleam/ (MIT), which fetches each Hex package by the checksum manifest.toml already carries, with no hash to maintain; the result is an escript wrapped with erlang. The D reader depends on nothing beyond phobos, so --skip-registry=all is the whole of what it takes: ldc by default, dmd available as ssp-reader-dmd (given gcc14, GCC 15 having introduced nullptr in headers dmd's ImportC does not yet parse). (assisted by Claude-Code) --- README.md | 202 +++++++++++++++++--------------------------------------------- 1 file changed, 56 insertions(+), 146 deletions(-) (limited to 'README.md') diff --git a/README.md b/README.md index 0b25341..eba8c42 100644 --- a/README.md +++ b/README.md @@ -1,151 +1,61 @@ # ssp-reader -Reader for the **`.ssp` document-abstraction text format** emitted -by [sisudoc-spine](https://git.sisudoc.org/projects/sisudoc-spine). +Readers for the **`.ssp`** document abstraction, the object-centric +document abstraction written as text, which `sisudoc-spine` emits and +reads. -Reads a `.ssp` file (spine's PEG-parsable serialisation of its -internal document abstraction) and renders it into a chosen output: -an outline tree, scroll HTML, plain text with heading breadcrumbs, -JSONL chunks suitable for LLM / embedding pipelines, or an image -manifest for verifying the images a document embeds. - -## What this is for - -- Inspecting and debugging the abstraction stage of spine without - rebuilding spine itself. -- Round-trip experiments: markup -> spine -> `.ssp` -> ssp-reader - -> downstream artefact. -- Re-using spine's parsed document structure outside the spine - binary: feeding it to an indexer, an LLM pipeline, or a custom - renderer. -- A worked example of consuming `.ssp` for anyone implementing - another reader in another language. - -## Audience - -Developers building tools that consume spine's output: search -indexers, LLM ingestion pipelines, custom renderers, alternative -output formats, and anyone studying the abstraction stage of -spine itself. - -End-user document authors do **not** need this tool to publish a -document. Run spine directly for that. - -## Relationship to spine - -Spine's processing pipeline has three stages: - -1. **Input:** parse `.sst` / `.ssm` markup. -2. **Abstraction:** build a numbered object model (the "spine - abstraction"; one of spine's named outputs is `.ssp`, a - PEG-parsable text representation of this model). -3. **Output:** render to HTML / EPUB / PDF / SQLite / etc. - -`ssp-reader` picks up the abstraction at the end of stage 2 (the -`.ssp` file) and runs alternative stage-3 renderings on top of it. -The `.ssp` format itself is documented in -[`doc-abstraction-format`](https://git.sisudoc.org/tools/doc-abstraction-format). - -## Quickstart - -### Build - -Inside the project's nix dev environment (the same one spine uses): - -```sh -nix develop ".#dsh-nixpkgs-ldc" --print-build-logs -c $SHELL -dub build --compiler=ldmd2 -``` - -The binary lands at `./bin/ssp-reader`. - -### Run - -```sh -# generate a .ssp from a sample document via spine, then: -ssp-reader --tree doc.ssp # outline view -ssp-reader --html doc.ssp # HTML scroll output -ssp-reader --llm-text doc.ssp # plain text with breadcrumbs -ssp-reader --llm-jsonl doc.ssp # JSONL chunks for embeddings -ssp-reader --images doc.ssp # image manifest (digest, size, pixels) -ssp-reader --verify --image-dir=pod/media/image doc.ssp # check images against the records -ssp-reader --report-unknown doc.ssp # what it read and did not understand -ssp-reader --tree --output=out.txt doc.ssp -ssp-reader --help -``` - -## Format version - -The reader knows abstraction format **2.x**. Every `.ssp` declares its -version on its first line, and the reader checks the **major** part and -refuses what it does not know: an unknown major means a property may have -changed meaning or gone away, and loading such a file half populated in -silence is the failure the check exists to prevent. `--any-version` reads -it regardless, for looking at a file rather than trusting one. - -## What it does not understand - -Within a major version a reader **should** ignore properties it does not -know, since the minor part is raised precisely by adding them. This reader -does, and `--report-unknown` says what it ignored and how often: - -``` -read and NOT understood: - properties ignored (15 distinct): - .is_of_type 328 - .segment_html_is 138 - ... -``` - -### What it ignores, and why - -Twelve properties are ignored across the reference collection, and each is -ignored for a reason rather than by oversight: - -| ignored | why | -|---------|-----| -| `.ancestors_collapsed`, `.parent_lev`, `.heading_lev_collapsed`, `.dom_status`, `.dom_status_collapsed` | level bookkeeping; this reader works from `.ancestors` and the heading level | -| `.segment_epub`, `.segment_epub_is`, `.segment_lv4_is`, `.heading_lev_anchor`, `.anchor_tag`, `.lev4_subtoc` | naming for segmented HTML and EPUB output, which this reader does not produce | -| `.attrib` | one occurrence in the whole collection, and no output uses it | - -The list was longer. `--report-unknown` found that `.is_of_type` was -ignored, and **that was a defect**: objects of type `comment` are markup -commentary, carried by the abstraction and not put into output, and three -of the four outputs were rendering them. A markup comment was appearing in -the HTML, in the plain text, and as an embedding chunk in the JSONL. -Fixed, and the reason the report exists. - -That is deliberate and is what makes the reader usable as a check on the -[format specification](https://git.sisudoc.org/specs/doc-abstraction-format): -a reader that drops what it does not know is conforming, but one that -cannot say what it dropped cannot tell you whether the specification it was -written from was complete. Silence and understanding look alike otherwise. +**One reader per language, in a directory named for the language.** -## Documentation +| | | +|--------------------|-------------------------------------------------------------------------------------------------------------------------------------------| +| [`dlang/`](dlang/) | D. Outline, HTML scroll, breadcrumbed text, JSONL chunks for embedding pipelines, an image manifest, image verification, and `--validate` | +| [`gleam/`](gleam/) | Gleam. Summary, outline, and what it did not understand | +| `ruby/` | to come | -- The `.ssp` format itself: see - [doc-abstraction-format](https://git.sisudoc.org/tools/doc-abstraction-format). -- Source: [`src/`](src/) - one module per output format - (`output_tree.d`, `output_html.d`, `output_llm_breadcrumbs.d`, - `output_llm_jsonl.d`, `output_images.d`, `verify_images.d`), plus - `ssp_parser.d` and `ssp_types.d`. - -## License - -AGPL-3.0-or-later. See [`dub.json`](dub.json) and the parent -[sisudoc-spine](https://git.sisudoc.org/projects/sisudoc-spine) -project for full licence text. - -## Repository - -- **Canonical URL:** https://git.sisudoc.org/tools/ssp-reader -- **Clone:** `git clone --depth=1 git://git.sisudoc.org/tools/ssp-reader` - -## Related projects - -- [sisudoc-spine](https://git.sisudoc.org/projects/sisudoc-spine) - - produces the `.ssp` files this tool consumes. -- [doc-abstraction-format](https://git.sisudoc.org/tools/doc-abstraction-format) - - specification of the `.ssp` text format. -- [tree-sitter-sisu](https://git.sisudoc.org/tools/tree-sitter-sisu) - - tree-sitter grammar for the upstream `.sst` / `.ssm` markup. +## Why more than one + +These are not three tools that happen to share a name. They are **the same +specification implemented more than once**, and the interesting thing about +them is that they agree. + +Each was written from the format specification, `doc-abstraction-format`, +rather than from spine's emitter or from each other. Run over the 36 +document reference collection, **the D and Gleam readers agree on the +section counts of all 36**. + +That is the point of the exercise. A specification that one implementation +follows is a description of that implementation; a specification that two +independent implementations follow, in languages with nothing in common, is +a description of the format. + +## What a reader must do + +From the specification's conformance section, and both readers do it: + +- **check the major version** declared on the first line and refuse a major + it does not know, rather than loading an unfamiliar artefact half + populated in silence; +- **tolerate unknown properties** within a major version, since the minor + part is raised precisely by adding them; +- **say what it ignored**, on request. A reader that drops what it does not + know is conforming; one that cannot say what it dropped cannot show a + specification to be complete, silence and understanding being alike from + outside. + +The last is why both carry `--report-unknown`, and it has earned its place: +it found that the D reader was rendering markup comments into its HTML, +text and JSONL output, because it was ignoring the property that says an +object is commentary. + +## Building + +Each language directory builds in its own way and says how in its own +README. Neither depends on the other. + +## The format + +Specified in **doc-abstraction-format**: `ocda-model.md` for the model both +serialisations share, `ssp-format-specification.md` for this one, and +`ocda-db-format-specification.md` for the sqlite form. + +The readers here are at format **2.x**. -- cgit v1.2.3