diff options
Diffstat (limited to 'README.md')
| -rw-r--r-- | README.md | 202 |
1 files changed, 56 insertions, 146 deletions
@@ -1,151 +1,61 @@ # ssp-reader -Reader for the **`.ssp` document-abstraction text format** emitted -by [sisudoc-spine](https://git.sisudoc.org/projects/sisudoc-spine). +Readers for the **`.ssp`** document abstraction, the object-centric +document abstraction written as text, which `sisudoc-spine` emits and +reads. -Reads a `.ssp` file (spine's PEG-parsable serialisation of its -internal document abstraction) and renders it into a chosen output: -an outline tree, scroll HTML, plain text with heading breadcrumbs, -JSONL chunks suitable for LLM / embedding pipelines, or an image -manifest for verifying the images a document embeds. - -## What this is for - -- Inspecting and debugging the abstraction stage of spine without - rebuilding spine itself. -- Round-trip experiments: markup -> spine -> `.ssp` -> ssp-reader - -> downstream artefact. -- Re-using spine's parsed document structure outside the spine - binary: feeding it to an indexer, an LLM pipeline, or a custom - renderer. -- A worked example of consuming `.ssp` for anyone implementing - another reader in another language. - -## Audience - -Developers building tools that consume spine's output: search -indexers, LLM ingestion pipelines, custom renderers, alternative -output formats, and anyone studying the abstraction stage of -spine itself. - -End-user document authors do **not** need this tool to publish a -document. Run spine directly for that. - -## Relationship to spine - -Spine's processing pipeline has three stages: - -1. **Input:** parse `.sst` / `.ssm` markup. -2. **Abstraction:** build a numbered object model (the "spine - abstraction"; one of spine's named outputs is `.ssp`, a - PEG-parsable text representation of this model). -3. **Output:** render to HTML / EPUB / PDF / SQLite / etc. - -`ssp-reader` picks up the abstraction at the end of stage 2 (the -`.ssp` file) and runs alternative stage-3 renderings on top of it. -The `.ssp` format itself is documented in -[`doc-abstraction-format`](https://git.sisudoc.org/tools/doc-abstraction-format). - -## Quickstart - -### Build - -Inside the project's nix dev environment (the same one spine uses): - -```sh -nix develop ".#dsh-nixpkgs-ldc" --print-build-logs -c $SHELL -dub build --compiler=ldmd2 -``` - -The binary lands at `./bin/ssp-reader`. - -### Run - -```sh -# generate a .ssp from a sample document via spine, then: -ssp-reader --tree doc.ssp # outline view -ssp-reader --html doc.ssp # HTML scroll output -ssp-reader --llm-text doc.ssp # plain text with breadcrumbs -ssp-reader --llm-jsonl doc.ssp # JSONL chunks for embeddings -ssp-reader --images doc.ssp # image manifest (digest, size, pixels) -ssp-reader --verify --image-dir=pod/media/image doc.ssp # check images against the records -ssp-reader --report-unknown doc.ssp # what it read and did not understand -ssp-reader --tree --output=out.txt doc.ssp -ssp-reader --help -``` - -## Format version - -The reader knows abstraction format **2.x**. Every `.ssp` declares its -version on its first line, and the reader checks the **major** part and -refuses what it does not know: an unknown major means a property may have -changed meaning or gone away, and loading such a file half populated in -silence is the failure the check exists to prevent. `--any-version` reads -it regardless, for looking at a file rather than trusting one. - -## What it does not understand - -Within a major version a reader **should** ignore properties it does not -know, since the minor part is raised precisely by adding them. This reader -does, and `--report-unknown` says what it ignored and how often: - -``` -read and NOT understood: - properties ignored (15 distinct): - .is_of_type 328 - .segment_html_is 138 - ... -``` - -### What it ignores, and why - -Twelve properties are ignored across the reference collection, and each is -ignored for a reason rather than by oversight: - -| ignored | why | -|---------|-----| -| `.ancestors_collapsed`, `.parent_lev`, `.heading_lev_collapsed`, `.dom_status`, `.dom_status_collapsed` | level bookkeeping; this reader works from `.ancestors` and the heading level | -| `.segment_epub`, `.segment_epub_is`, `.segment_lv4_is`, `.heading_lev_anchor`, `.anchor_tag`, `.lev4_subtoc` | naming for segmented HTML and EPUB output, which this reader does not produce | -| `.attrib` | one occurrence in the whole collection, and no output uses it | - -The list was longer. `--report-unknown` found that `.is_of_type` was -ignored, and **that was a defect**: objects of type `comment` are markup -commentary, carried by the abstraction and not put into output, and three -of the four outputs were rendering them. A markup comment was appearing in -the HTML, in the plain text, and as an embedding chunk in the JSONL. -Fixed, and the reason the report exists. - -That is deliberate and is what makes the reader usable as a check on the -[format specification](https://git.sisudoc.org/specs/doc-abstraction-format): -a reader that drops what it does not know is conforming, but one that -cannot say what it dropped cannot tell you whether the specification it was -written from was complete. Silence and understanding look alike otherwise. +**One reader per language, in a directory named for the language.** -## Documentation +| | | +|--------------------|-------------------------------------------------------------------------------------------------------------------------------------------| +| [`dlang/`](dlang/) | D. Outline, HTML scroll, breadcrumbed text, JSONL chunks for embedding pipelines, an image manifest, image verification, and `--validate` | +| [`gleam/`](gleam/) | Gleam. Summary, outline, and what it did not understand | +| `ruby/` | to come | -- The `.ssp` format itself: see - [doc-abstraction-format](https://git.sisudoc.org/tools/doc-abstraction-format). -- Source: [`src/`](src/) - one module per output format - (`output_tree.d`, `output_html.d`, `output_llm_breadcrumbs.d`, - `output_llm_jsonl.d`, `output_images.d`, `verify_images.d`), plus - `ssp_parser.d` and `ssp_types.d`. - -## License - -AGPL-3.0-or-later. See [`dub.json`](dub.json) and the parent -[sisudoc-spine](https://git.sisudoc.org/projects/sisudoc-spine) -project for full licence text. - -## Repository - -- **Canonical URL:** https://git.sisudoc.org/tools/ssp-reader -- **Clone:** `git clone --depth=1 git://git.sisudoc.org/tools/ssp-reader` - -## Related projects - -- [sisudoc-spine](https://git.sisudoc.org/projects/sisudoc-spine) - - produces the `.ssp` files this tool consumes. -- [doc-abstraction-format](https://git.sisudoc.org/tools/doc-abstraction-format) - - specification of the `.ssp` text format. -- [tree-sitter-sisu](https://git.sisudoc.org/tools/tree-sitter-sisu) - - tree-sitter grammar for the upstream `.sst` / `.ssm` markup. +## Why more than one + +These are not three tools that happen to share a name. They are **the same +specification implemented more than once**, and the interesting thing about +them is that they agree. + +Each was written from the format specification, `doc-abstraction-format`, +rather than from spine's emitter or from each other. Run over the 36 +document reference collection, **the D and Gleam readers agree on the +section counts of all 36**. + +That is the point of the exercise. A specification that one implementation +follows is a description of that implementation; a specification that two +independent implementations follow, in languages with nothing in common, is +a description of the format. + +## What a reader must do + +From the specification's conformance section, and both readers do it: + +- **check the major version** declared on the first line and refuse a major + it does not know, rather than loading an unfamiliar artefact half + populated in silence; +- **tolerate unknown properties** within a major version, since the minor + part is raised precisely by adding them; +- **say what it ignored**, on request. A reader that drops what it does not + know is conforming; one that cannot say what it dropped cannot show a + specification to be complete, silence and understanding being alike from + outside. + +The last is why both carry `--report-unknown`, and it has earned its place: +it found that the D reader was rendering markup comments into its HTML, +text and JSONL output, because it was ignoring the property that says an +object is commentary. + +## Building + +Each language directory builds in its own way and says how in its own +README. Neither depends on the other. + +## The format + +Specified in **doc-abstraction-format**: `ocda-model.md` for the model both +serialisations share, `ssp-format-specification.md` for this one, and +`ocda-db-format-specification.md` for the sqlite form. + +The readers here are at format **2.x**. |
