diff options
Diffstat (limited to 'README.md')
| -rw-r--r-- | README.md | 138 |
1 files changed, 131 insertions, 7 deletions
@@ -1,9 +1,133 @@ -# sisudoc-ssp-reader +# ssp-reader -A reader for the `.ssp`, the text form of the object-centric document -abstraction that [sisudoc-spine](https://git.sisudoc.org/sisudoc-spine) -writes, to show that the abstraction carries what is needed to present a -document without the markup it was made from. +Reader for the **`.ssp` document-abstraction text format** emitted +by [sisudoc-spine](https://git.sisudoc.org/projects/sisudoc-spine). -Work on the reader begins here. The first version of it to be published -reads format 2.0, which follows. +Reads a `.ssp` file (spine's PEG-parsable serialisation of its +internal document abstraction) and renders it into a chosen output: +an outline tree, scroll HTML, plain text with heading breadcrumbs, +JSONL chunks suitable for LLM / embedding pipelines, or an image +manifest for verifying the images a document embeds. + +## What this is for + +- Inspecting and debugging the abstraction stage of spine without + rebuilding spine itself. +- Round-trip experiments: markup -> spine -> `.ssp` -> ssp-reader + -> downstream artefact. +- Re-using spine's parsed document structure outside the spine + binary: feeding it to an indexer, an LLM pipeline, or a custom + renderer. +- A worked example of consuming `.ssp` for anyone implementing + another reader in another language. + +## Audience + +Developers building tools that consume spine's output: search +indexers, LLM ingestion pipelines, custom renderers, alternative +output formats, and anyone studying the abstraction stage of +spine itself. + +End-user document authors do **not** need this tool to publish a +document. Run spine directly for that. + +## Relationship to spine + +Spine's processing pipeline has three stages: + +1. **Input:** parse `.sst` / `.ssm` markup. +2. **Abstraction:** build a numbered object model (the "spine + abstraction"; one of spine's named outputs is `.ssp`, a + PEG-parsable text representation of this model). +3. **Output:** render to HTML / EPUB / PDF / SQLite / etc. + +`ssp-reader` picks up the abstraction at the end of stage 2 (the +`.ssp` file) and runs alternative stage-3 renderings on top of it. +The `.ssp` format itself is documented in +[`doc-abstraction-format`](https://git.sisudoc.org/tools/doc-abstraction-format). + +## Quickstart + +### Build + +Inside the project's nix dev environment (the same one spine uses): + +```sh +nix develop ".#dsh-nixpkgs-ldc" --print-build-logs -c $SHELL +dub build --compiler=ldmd2 +``` + +The binary lands at `./bin/ssp-reader`. + +### Run + +```sh +# generate a .ssp from a sample document via spine, then: +ssp-reader --tree doc.ssp # outline view +ssp-reader --html doc.ssp # HTML scroll output +ssp-reader --llm-text doc.ssp # plain text with breadcrumbs +ssp-reader --llm-jsonl doc.ssp # JSONL chunks for embeddings +ssp-reader --images doc.ssp # image manifest (digest, size, pixels) +ssp-reader --verify --image-dir=pod/media/image doc.ssp # check images against the records +ssp-reader --report-unknown doc.ssp # what it read and did not understand +ssp-reader --tree --output=out.txt doc.ssp +ssp-reader --help +``` + +## Format version + +The reader knows abstraction format **2.x**. Every `.ssp` declares its +version on its first line, and the reader checks the **major** part and +refuses what it does not know: an unknown major means a property may have +changed meaning or gone away, and loading such a file half populated in +silence is the failure the check exists to prevent. `--any-version` reads +it regardless, for looking at a file rather than trusting one. + +## What it does not understand + +Within a major version a reader **should** ignore properties it does not +know, since the minor part is raised precisely by adding them. This reader +does, and `--report-unknown` says what it ignored and how often: + +``` +read and NOT understood: + properties ignored (15 distinct): + .is_of_type 328 + .segment_html_is 138 + ... +``` + +That is deliberate and is what makes the reader usable as a check on the +[format specification](https://git.sisudoc.org/specs/doc-abstraction-format): +a reader that drops what it does not know is conforming, but one that +cannot say what it dropped cannot tell you whether the specification it was +written from was complete. Silence and understanding look alike otherwise. + +## Documentation + +- The `.ssp` format itself: see + [doc-abstraction-format](https://git.sisudoc.org/tools/doc-abstraction-format). +- Source: [`src/`](src/) - one module per output format + (`output_tree.d`, `output_html.d`, `output_llm_breadcrumbs.d`, + `output_llm_jsonl.d`, `output_images.d`, `verify_images.d`), plus + `ssp_parser.d` and `ssp_types.d`. + +## License + +AGPL-3.0-or-later. See [`dub.json`](dub.json) and the parent +[sisudoc-spine](https://git.sisudoc.org/projects/sisudoc-spine) +project for full licence text. + +## Repository + +- **Canonical URL:** https://git.sisudoc.org/tools/ssp-reader +- **Clone:** `git clone --depth=1 git://git.sisudoc.org/tools/ssp-reader` + +## Related projects + +- [sisudoc-spine](https://git.sisudoc.org/projects/sisudoc-spine) - + produces the `.ssp` files this tool consumes. +- [doc-abstraction-format](https://git.sisudoc.org/tools/doc-abstraction-format) - + specification of the `.ssp` text format. +- [tree-sitter-sisu](https://git.sisudoc.org/tools/tree-sitter-sisu) - + tree-sitter grammar for the upstream `.sst` / `.ssm` markup. |
