summaryrefslogtreecommitdiffhomepage
path: root/dlang/README.md
diff options
context:
space:
mode:
Diffstat (limited to 'dlang/README.md')
-rw-r--r--dlang/README.md162
1 files changed, 162 insertions, 0 deletions
diff --git a/dlang/README.md b/dlang/README.md
new file mode 100644
index 0000000..fa39eab
--- /dev/null
+++ b/dlang/README.md
@@ -0,0 +1,162 @@
+# ssp-reader
+
+Reader for the **`.ssp` document-abstraction text format** emitted
+by [sisudoc-spine](https://git.sisudoc.org/projects/sisudoc-spine).
+
+Reads a `.ssp` file (spine's PEG-parsable serialisation of its
+internal document abstraction) and renders it into a chosen output:
+an outline tree, scroll HTML, plain text with heading breadcrumbs,
+JSONL chunks suitable for LLM / embedding pipelines, or an image
+manifest for verifying the images a document embeds.
+
+## What this is for
+
+- Inspecting and debugging the abstraction stage of spine without
+ rebuilding spine itself.
+- Round-trip experiments: markup -> spine -> `.ssp` -> ssp-reader
+ -> downstream artefact.
+- Re-using spine's parsed document structure outside the spine
+ binary: feeding it to an indexer, an LLM pipeline, or a custom
+ renderer.
+- A worked example of consuming `.ssp` for anyone implementing
+ another reader in another language.
+
+## Audience
+
+Developers building tools that consume spine's output: search
+indexers, LLM ingestion pipelines, custom renderers, alternative
+output formats, and anyone studying the abstraction stage of
+spine itself.
+
+End-user document authors do **not** need this tool to publish a
+document. Run spine directly for that.
+
+## Relationship to spine
+
+Spine's processing pipeline has three stages:
+
+1. **Input:** parse `.sst` / `.ssm` markup.
+2. **Abstraction:** build a numbered object model (the "spine
+ abstraction"; one of spine's named outputs is `.ssp`, a
+ PEG-parsable text representation of this model).
+3. **Output:** render to HTML / EPUB / PDF / SQLite / etc.
+
+`ssp-reader` picks up the abstraction at the end of stage 2 (the
+`.ssp` file) and runs alternative stage-3 renderings on top of it.
+The `.ssp` format itself is documented in
+[`doc-abstraction-format`](https://git.sisudoc.org/tools/doc-abstraction-format).
+
+## Quickstart
+
+### Build
+
+```sh
+nix build # the reader, with ldc
+./result/bin/ssp-reader <file.ssp>
+```
+
+The reader depends on nothing beyond phobos, so `nix build` needs no
+network and works in a sandbox. `nix build .#ssp-reader-dmd` builds it with
+dmd instead, which is given gcc14 for the same reason spine gives it: GCC
+15 introduced `nullptr` in its headers and dmd's ImportC parser does not
+yet handle it.
+
+For development, or to build without nix:
+
+```sh
+nix develop # ldc, dub, dtools
+dub build --compiler=ldmd2
+```
+
+The binary then lands at `./bin/ssp-reader`.
+
+### Run
+
+```sh
+# generate a .ssp from a sample document via spine, then:
+ssp-reader --tree doc.ssp # outline view
+ssp-reader --html doc.ssp # HTML scroll output
+ssp-reader --llm-text doc.ssp # plain text with breadcrumbs
+ssp-reader --llm-jsonl doc.ssp # JSONL chunks for embeddings
+ssp-reader --images doc.ssp # image manifest (digest, size, pixels)
+ssp-reader --verify --image-dir=pod/media/image doc.ssp # check images against the records
+ssp-reader --report-unknown doc.ssp # what it read and did not understand
+ssp-reader --tree --output=out.txt doc.ssp
+ssp-reader --help
+```
+
+## Format version
+
+The reader knows abstraction format **2.x**. Every `.ssp` declares its
+version on its first line, and the reader checks the **major** part and
+refuses what it does not know: an unknown major means a property may have
+changed meaning or gone away, and loading such a file half populated in
+silence is the failure the check exists to prevent. `--any-version` reads
+it regardless, for looking at a file rather than trusting one.
+
+## What it does not understand
+
+Within a major version a reader **should** ignore properties it does not
+know, since the minor part is raised precisely by adding them. This reader
+does, and `--report-unknown` says what it ignored and how often:
+
+```
+read and NOT understood:
+ properties ignored (15 distinct):
+ .is_of_type 328
+ .segment_html_is 138
+ ...
+```
+
+### What it ignores, and why
+
+Twelve properties are ignored across the reference collection, and each is
+ignored for a reason rather than by oversight:
+
+| ignored | why |
+|--------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------|
+| `.ancestors_collapsed`, `.parent_lev`, `.heading_lev_collapsed`, `.dom_status`, `.dom_status_collapsed` | level bookkeeping; this reader works from `.ancestors` and the heading level |
+| `.segment_epub`, `.segment_epub_is`, `.segment_lv4_is`, `.heading_lev_anchor`, `.anchor_tag`, `.lev4_subtoc` | naming for segmented HTML and EPUB output, which this reader does not produce |
+| `.attrib` | one occurrence in the whole collection, and no output uses it |
+
+The list was longer. `--report-unknown` found that `.is_of_type` was
+ignored, and **that was a defect**: objects of type `comment` are markup
+commentary, carried by the abstraction and not put into output, and three
+of the four outputs were rendering them. A markup comment was appearing in
+the HTML, in the plain text, and as an embedding chunk in the JSONL.
+Fixed, and the reason the report exists.
+
+That is deliberate and is what makes the reader usable as a check on the
+[format specification](https://git.sisudoc.org/specs/doc-abstraction-format):
+a reader that drops what it does not know is conforming, but one that
+cannot say what it dropped cannot tell you whether the specification it was
+written from was complete. Silence and understanding look alike otherwise.
+
+## Documentation
+
+- The `.ssp` format itself: see
+ [doc-abstraction-format](https://git.sisudoc.org/tools/doc-abstraction-format).
+- Source: [`src/`](src/) - one module per output format
+ (`output_tree.d`, `output_html.d`, `output_llm_breadcrumbs.d`,
+ `output_llm_jsonl.d`, `output_images.d`, `verify_images.d`), plus
+ `ssp_parser.d` and `ssp_types.d`.
+
+## License
+
+AGPL-3.0-or-later. See [`dub.json`](dub.json) and the parent
+[sisudoc-spine](https://git.sisudoc.org/projects/sisudoc-spine)
+project for full licence text.
+
+## Repository
+
+- **Canonical URL:** https://git.sisudoc.org/tools/ssp-reader
+- **Clone:** `git clone --depth=1 git://git.sisudoc.org/tools/ssp-reader`
+
+## Related projects
+
+- [sisudoc-spine](https://git.sisudoc.org/projects/sisudoc-spine) -
+ produces the `.ssp` files this tool consumes.
+- [doc-abstraction-format](https://git.sisudoc.org/tools/doc-abstraction-format) -
+ specification of the `.ssp` text format.
+- [tree-sitter-sisu](https://git.sisudoc.org/tools/tree-sitter-sisu) -
+ tree-sitter grammar for the upstream `.sst` / `.ssm` markup.