diff options
Diffstat (limited to 'dlang/README.md')
| -rw-r--r-- | dlang/README.md | 162 |
1 files changed, 162 insertions, 0 deletions
diff --git a/dlang/README.md b/dlang/README.md new file mode 100644 index 0000000..fa39eab --- /dev/null +++ b/dlang/README.md @@ -0,0 +1,162 @@ +# ssp-reader + +Reader for the **`.ssp` document-abstraction text format** emitted +by [sisudoc-spine](https://git.sisudoc.org/projects/sisudoc-spine). + +Reads a `.ssp` file (spine's PEG-parsable serialisation of its +internal document abstraction) and renders it into a chosen output: +an outline tree, scroll HTML, plain text with heading breadcrumbs, +JSONL chunks suitable for LLM / embedding pipelines, or an image +manifest for verifying the images a document embeds. + +## What this is for + +- Inspecting and debugging the abstraction stage of spine without + rebuilding spine itself. +- Round-trip experiments: markup -> spine -> `.ssp` -> ssp-reader + -> downstream artefact. +- Re-using spine's parsed document structure outside the spine + binary: feeding it to an indexer, an LLM pipeline, or a custom + renderer. +- A worked example of consuming `.ssp` for anyone implementing + another reader in another language. + +## Audience + +Developers building tools that consume spine's output: search +indexers, LLM ingestion pipelines, custom renderers, alternative +output formats, and anyone studying the abstraction stage of +spine itself. + +End-user document authors do **not** need this tool to publish a +document. Run spine directly for that. + +## Relationship to spine + +Spine's processing pipeline has three stages: + +1. **Input:** parse `.sst` / `.ssm` markup. +2. **Abstraction:** build a numbered object model (the "spine + abstraction"; one of spine's named outputs is `.ssp`, a + PEG-parsable text representation of this model). +3. **Output:** render to HTML / EPUB / PDF / SQLite / etc. + +`ssp-reader` picks up the abstraction at the end of stage 2 (the +`.ssp` file) and runs alternative stage-3 renderings on top of it. +The `.ssp` format itself is documented in +[`doc-abstraction-format`](https://git.sisudoc.org/tools/doc-abstraction-format). + +## Quickstart + +### Build + +```sh +nix build # the reader, with ldc +./result/bin/ssp-reader <file.ssp> +``` + +The reader depends on nothing beyond phobos, so `nix build` needs no +network and works in a sandbox. `nix build .#ssp-reader-dmd` builds it with +dmd instead, which is given gcc14 for the same reason spine gives it: GCC +15 introduced `nullptr` in its headers and dmd's ImportC parser does not +yet handle it. + +For development, or to build without nix: + +```sh +nix develop # ldc, dub, dtools +dub build --compiler=ldmd2 +``` + +The binary then lands at `./bin/ssp-reader`. + +### Run + +```sh +# generate a .ssp from a sample document via spine, then: +ssp-reader --tree doc.ssp # outline view +ssp-reader --html doc.ssp # HTML scroll output +ssp-reader --llm-text doc.ssp # plain text with breadcrumbs +ssp-reader --llm-jsonl doc.ssp # JSONL chunks for embeddings +ssp-reader --images doc.ssp # image manifest (digest, size, pixels) +ssp-reader --verify --image-dir=pod/media/image doc.ssp # check images against the records +ssp-reader --report-unknown doc.ssp # what it read and did not understand +ssp-reader --tree --output=out.txt doc.ssp +ssp-reader --help +``` + +## Format version + +The reader knows abstraction format **2.x**. Every `.ssp` declares its +version on its first line, and the reader checks the **major** part and +refuses what it does not know: an unknown major means a property may have +changed meaning or gone away, and loading such a file half populated in +silence is the failure the check exists to prevent. `--any-version` reads +it regardless, for looking at a file rather than trusting one. + +## What it does not understand + +Within a major version a reader **should** ignore properties it does not +know, since the minor part is raised precisely by adding them. This reader +does, and `--report-unknown` says what it ignored and how often: + +``` +read and NOT understood: + properties ignored (15 distinct): + .is_of_type 328 + .segment_html_is 138 + ... +``` + +### What it ignores, and why + +Twelve properties are ignored across the reference collection, and each is +ignored for a reason rather than by oversight: + +| ignored | why | +|--------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------| +| `.ancestors_collapsed`, `.parent_lev`, `.heading_lev_collapsed`, `.dom_status`, `.dom_status_collapsed` | level bookkeeping; this reader works from `.ancestors` and the heading level | +| `.segment_epub`, `.segment_epub_is`, `.segment_lv4_is`, `.heading_lev_anchor`, `.anchor_tag`, `.lev4_subtoc` | naming for segmented HTML and EPUB output, which this reader does not produce | +| `.attrib` | one occurrence in the whole collection, and no output uses it | + +The list was longer. `--report-unknown` found that `.is_of_type` was +ignored, and **that was a defect**: objects of type `comment` are markup +commentary, carried by the abstraction and not put into output, and three +of the four outputs were rendering them. A markup comment was appearing in +the HTML, in the plain text, and as an embedding chunk in the JSONL. +Fixed, and the reason the report exists. + +That is deliberate and is what makes the reader usable as a check on the +[format specification](https://git.sisudoc.org/specs/doc-abstraction-format): +a reader that drops what it does not know is conforming, but one that +cannot say what it dropped cannot tell you whether the specification it was +written from was complete. Silence and understanding look alike otherwise. + +## Documentation + +- The `.ssp` format itself: see + [doc-abstraction-format](https://git.sisudoc.org/tools/doc-abstraction-format). +- Source: [`src/`](src/) - one module per output format + (`output_tree.d`, `output_html.d`, `output_llm_breadcrumbs.d`, + `output_llm_jsonl.d`, `output_images.d`, `verify_images.d`), plus + `ssp_parser.d` and `ssp_types.d`. + +## License + +AGPL-3.0-or-later. See [`dub.json`](dub.json) and the parent +[sisudoc-spine](https://git.sisudoc.org/projects/sisudoc-spine) +project for full licence text. + +## Repository + +- **Canonical URL:** https://git.sisudoc.org/tools/ssp-reader +- **Clone:** `git clone --depth=1 git://git.sisudoc.org/tools/ssp-reader` + +## Related projects + +- [sisudoc-spine](https://git.sisudoc.org/projects/sisudoc-spine) - + produces the `.ssp` files this tool consumes. +- [doc-abstraction-format](https://git.sisudoc.org/tools/doc-abstraction-format) - + specification of the `.ssp` text format. +- [tree-sitter-sisu](https://git.sisudoc.org/tools/tree-sitter-sisu) - + tree-sitter grammar for the upstream `.sst` / `.ssm` markup. |
