diff options
| author | Ralph Amissah <ralph.amissah@gmail.com> | 2026-09-24 11:12:28 -0400 |
|---|---|---|
| committer | Ralph Amissah <ralph.amissah@gmail.com> | 2026-09-24 11:12:28 -0400 |
| commit | da833f8b157ae501c9f6eec827dd4a965bfa7265 (patch) | |
| tree | 9368bbfe07dc130867dc8241d20bc3d125f3e510 /README.md | |
| parent | commencement: a reader for the .ssp (diff) | |
format 2.0: the version gate
The reader as it stood when the format reached 2.0, in D: a parser of
the .ssp with no dependency beyond phobos, and outputs from it - a
heading tree, an html scroll, breadcrumbed text, chunked jsonl for
embedding pipelines, an image manifest, and a check of image records
against the images on disk.
Brought to abstraction format 2.0, and made able to say what it does
not understand, so a reader can check that its format specification is
complete. Previously the reader ignored what it did not recognise and
reported nothing, which made it seem it had understood a 2.0 file (and
drop things in silence).
Now:
- the banner is read, and the major part checked. A major this
reader does not know is refused, exit 3, because an unknown major
means a property may have changed meaning or gone away and loading
it half populated in silence is the failure the check exists to
prevent. --any-version reads it regardless, for looking at a file
rather than trusting one. A file with no banner warns and is read.
- @source is a preamble block beside @meta, @make and @doc_has, and
the summary line shows the language and the markup digest.
- tail is a section like the rest.
- --report-unknown lists every property, block and line shape met and
not understood, with counts. Ignoring them remains the conforming
behaviour within a major version; the counting is so that it is not
silent.
Over the markup document samples that report names 18 properties,
from .is_of_type at 56,882 occurrences down to .attrib at
1. Several are legitimately of no interest to a renderer and several
are not. Deciding which is which is the next pass, and it now has a
measured work list rather than a guess.
(assisted by Claude-Code)
Diffstat (limited to 'README.md')
| -rw-r--r-- | README.md | 138 |
1 files changed, 131 insertions, 7 deletions
@@ -1,9 +1,133 @@ -# sisudoc-ssp-reader +# ssp-reader -A reader for the `.ssp`, the text form of the object-centric document -abstraction that [sisudoc-spine](https://git.sisudoc.org/sisudoc-spine) -writes, to show that the abstraction carries what is needed to present a -document without the markup it was made from. +Reader for the **`.ssp` document-abstraction text format** emitted +by [sisudoc-spine](https://git.sisudoc.org/projects/sisudoc-spine). -Work on the reader begins here. The first version of it to be published -reads format 2.0, which follows. +Reads a `.ssp` file (spine's PEG-parsable serialisation of its +internal document abstraction) and renders it into a chosen output: +an outline tree, scroll HTML, plain text with heading breadcrumbs, +JSONL chunks suitable for LLM / embedding pipelines, or an image +manifest for verifying the images a document embeds. + +## What this is for + +- Inspecting and debugging the abstraction stage of spine without + rebuilding spine itself. +- Round-trip experiments: markup -> spine -> `.ssp` -> ssp-reader + -> downstream artefact. +- Re-using spine's parsed document structure outside the spine + binary: feeding it to an indexer, an LLM pipeline, or a custom + renderer. +- A worked example of consuming `.ssp` for anyone implementing + another reader in another language. + +## Audience + +Developers building tools that consume spine's output: search +indexers, LLM ingestion pipelines, custom renderers, alternative +output formats, and anyone studying the abstraction stage of +spine itself. + +End-user document authors do **not** need this tool to publish a +document. Run spine directly for that. + +## Relationship to spine + +Spine's processing pipeline has three stages: + +1. **Input:** parse `.sst` / `.ssm` markup. +2. **Abstraction:** build a numbered object model (the "spine + abstraction"; one of spine's named outputs is `.ssp`, a + PEG-parsable text representation of this model). +3. **Output:** render to HTML / EPUB / PDF / SQLite / etc. + +`ssp-reader` picks up the abstraction at the end of stage 2 (the +`.ssp` file) and runs alternative stage-3 renderings on top of it. +The `.ssp` format itself is documented in +[`doc-abstraction-format`](https://git.sisudoc.org/tools/doc-abstraction-format). + +## Quickstart + +### Build + +Inside the project's nix dev environment (the same one spine uses): + +```sh +nix develop ".#dsh-nixpkgs-ldc" --print-build-logs -c $SHELL +dub build --compiler=ldmd2 +``` + +The binary lands at `./bin/ssp-reader`. + +### Run + +```sh +# generate a .ssp from a sample document via spine, then: +ssp-reader --tree doc.ssp # outline view +ssp-reader --html doc.ssp # HTML scroll output +ssp-reader --llm-text doc.ssp # plain text with breadcrumbs +ssp-reader --llm-jsonl doc.ssp # JSONL chunks for embeddings +ssp-reader --images doc.ssp # image manifest (digest, size, pixels) +ssp-reader --verify --image-dir=pod/media/image doc.ssp # check images against the records +ssp-reader --report-unknown doc.ssp # what it read and did not understand +ssp-reader --tree --output=out.txt doc.ssp +ssp-reader --help +``` + +## Format version + +The reader knows abstraction format **2.x**. Every `.ssp` declares its +version on its first line, and the reader checks the **major** part and +refuses what it does not know: an unknown major means a property may have +changed meaning or gone away, and loading such a file half populated in +silence is the failure the check exists to prevent. `--any-version` reads +it regardless, for looking at a file rather than trusting one. + +## What it does not understand + +Within a major version a reader **should** ignore properties it does not +know, since the minor part is raised precisely by adding them. This reader +does, and `--report-unknown` says what it ignored and how often: + +``` +read and NOT understood: + properties ignored (15 distinct): + .is_of_type 328 + .segment_html_is 138 + ... +``` + +That is deliberate and is what makes the reader usable as a check on the +[format specification](https://git.sisudoc.org/specs/doc-abstraction-format): +a reader that drops what it does not know is conforming, but one that +cannot say what it dropped cannot tell you whether the specification it was +written from was complete. Silence and understanding look alike otherwise. + +## Documentation + +- The `.ssp` format itself: see + [doc-abstraction-format](https://git.sisudoc.org/tools/doc-abstraction-format). +- Source: [`src/`](src/) - one module per output format + (`output_tree.d`, `output_html.d`, `output_llm_breadcrumbs.d`, + `output_llm_jsonl.d`, `output_images.d`, `verify_images.d`), plus + `ssp_parser.d` and `ssp_types.d`. + +## License + +AGPL-3.0-or-later. See [`dub.json`](dub.json) and the parent +[sisudoc-spine](https://git.sisudoc.org/projects/sisudoc-spine) +project for full licence text. + +## Repository + +- **Canonical URL:** https://git.sisudoc.org/tools/ssp-reader +- **Clone:** `git clone --depth=1 git://git.sisudoc.org/tools/ssp-reader` + +## Related projects + +- [sisudoc-spine](https://git.sisudoc.org/projects/sisudoc-spine) - + produces the `.ssp` files this tool consumes. +- [doc-abstraction-format](https://git.sisudoc.org/tools/doc-abstraction-format) - + specification of the `.ssp` text format. +- [tree-sitter-sisu](https://git.sisudoc.org/tools/tree-sitter-sisu) - + tree-sitter grammar for the upstream `.sst` / `.ssm` markup. |
