summaryrefslogtreecommitdiffhomepage
path: root/README.md
diff options
context:
space:
mode:
Diffstat (limited to 'README.md')
-rw-r--r--README.md202
1 files changed, 56 insertions, 146 deletions
diff --git a/README.md b/README.md
index 0b25341..eba8c42 100644
--- a/README.md
+++ b/README.md
@@ -1,151 +1,61 @@
# ssp-reader
-Reader for the **`.ssp` document-abstraction text format** emitted
-by [sisudoc-spine](https://git.sisudoc.org/projects/sisudoc-spine).
+Readers for the **`.ssp`** document abstraction, the object-centric
+document abstraction written as text, which `sisudoc-spine` emits and
+reads.
-Reads a `.ssp` file (spine's PEG-parsable serialisation of its
-internal document abstraction) and renders it into a chosen output:
-an outline tree, scroll HTML, plain text with heading breadcrumbs,
-JSONL chunks suitable for LLM / embedding pipelines, or an image
-manifest for verifying the images a document embeds.
-
-## What this is for
-
-- Inspecting and debugging the abstraction stage of spine without
- rebuilding spine itself.
-- Round-trip experiments: markup -> spine -> `.ssp` -> ssp-reader
- -> downstream artefact.
-- Re-using spine's parsed document structure outside the spine
- binary: feeding it to an indexer, an LLM pipeline, or a custom
- renderer.
-- A worked example of consuming `.ssp` for anyone implementing
- another reader in another language.
-
-## Audience
-
-Developers building tools that consume spine's output: search
-indexers, LLM ingestion pipelines, custom renderers, alternative
-output formats, and anyone studying the abstraction stage of
-spine itself.
-
-End-user document authors do **not** need this tool to publish a
-document. Run spine directly for that.
-
-## Relationship to spine
-
-Spine's processing pipeline has three stages:
-
-1. **Input:** parse `.sst` / `.ssm` markup.
-2. **Abstraction:** build a numbered object model (the "spine
- abstraction"; one of spine's named outputs is `.ssp`, a
- PEG-parsable text representation of this model).
-3. **Output:** render to HTML / EPUB / PDF / SQLite / etc.
-
-`ssp-reader` picks up the abstraction at the end of stage 2 (the
-`.ssp` file) and runs alternative stage-3 renderings on top of it.
-The `.ssp` format itself is documented in
-[`doc-abstraction-format`](https://git.sisudoc.org/tools/doc-abstraction-format).
-
-## Quickstart
-
-### Build
-
-Inside the project's nix dev environment (the same one spine uses):
-
-```sh
-nix develop ".#dsh-nixpkgs-ldc" --print-build-logs -c $SHELL
-dub build --compiler=ldmd2
-```
-
-The binary lands at `./bin/ssp-reader`.
-
-### Run
-
-```sh
-# generate a .ssp from a sample document via spine, then:
-ssp-reader --tree doc.ssp # outline view
-ssp-reader --html doc.ssp # HTML scroll output
-ssp-reader --llm-text doc.ssp # plain text with breadcrumbs
-ssp-reader --llm-jsonl doc.ssp # JSONL chunks for embeddings
-ssp-reader --images doc.ssp # image manifest (digest, size, pixels)
-ssp-reader --verify --image-dir=pod/media/image doc.ssp # check images against the records
-ssp-reader --report-unknown doc.ssp # what it read and did not understand
-ssp-reader --tree --output=out.txt doc.ssp
-ssp-reader --help
-```
-
-## Format version
-
-The reader knows abstraction format **2.x**. Every `.ssp` declares its
-version on its first line, and the reader checks the **major** part and
-refuses what it does not know: an unknown major means a property may have
-changed meaning or gone away, and loading such a file half populated in
-silence is the failure the check exists to prevent. `--any-version` reads
-it regardless, for looking at a file rather than trusting one.
-
-## What it does not understand
-
-Within a major version a reader **should** ignore properties it does not
-know, since the minor part is raised precisely by adding them. This reader
-does, and `--report-unknown` says what it ignored and how often:
-
-```
-read and NOT understood:
- properties ignored (15 distinct):
- .is_of_type 328
- .segment_html_is 138
- ...
-```
-
-### What it ignores, and why
-
-Twelve properties are ignored across the reference collection, and each is
-ignored for a reason rather than by oversight:
-
-| ignored | why |
-|---------|-----|
-| `.ancestors_collapsed`, `.parent_lev`, `.heading_lev_collapsed`, `.dom_status`, `.dom_status_collapsed` | level bookkeeping; this reader works from `.ancestors` and the heading level |
-| `.segment_epub`, `.segment_epub_is`, `.segment_lv4_is`, `.heading_lev_anchor`, `.anchor_tag`, `.lev4_subtoc` | naming for segmented HTML and EPUB output, which this reader does not produce |
-| `.attrib` | one occurrence in the whole collection, and no output uses it |
-
-The list was longer. `--report-unknown` found that `.is_of_type` was
-ignored, and **that was a defect**: objects of type `comment` are markup
-commentary, carried by the abstraction and not put into output, and three
-of the four outputs were rendering them. A markup comment was appearing in
-the HTML, in the plain text, and as an embedding chunk in the JSONL.
-Fixed, and the reason the report exists.
-
-That is deliberate and is what makes the reader usable as a check on the
-[format specification](https://git.sisudoc.org/specs/doc-abstraction-format):
-a reader that drops what it does not know is conforming, but one that
-cannot say what it dropped cannot tell you whether the specification it was
-written from was complete. Silence and understanding look alike otherwise.
+**One reader per language, in a directory named for the language.**
-## Documentation
+| | |
+|--------------------|-------------------------------------------------------------------------------------------------------------------------------------------|
+| [`dlang/`](dlang/) | D. Outline, HTML scroll, breadcrumbed text, JSONL chunks for embedding pipelines, an image manifest, image verification, and `--validate` |
+| [`gleam/`](gleam/) | Gleam. Summary, outline, and what it did not understand |
+| `ruby/` | to come |
-- The `.ssp` format itself: see
- [doc-abstraction-format](https://git.sisudoc.org/tools/doc-abstraction-format).
-- Source: [`src/`](src/) - one module per output format
- (`output_tree.d`, `output_html.d`, `output_llm_breadcrumbs.d`,
- `output_llm_jsonl.d`, `output_images.d`, `verify_images.d`), plus
- `ssp_parser.d` and `ssp_types.d`.
-
-## License
-
-AGPL-3.0-or-later. See [`dub.json`](dub.json) and the parent
-[sisudoc-spine](https://git.sisudoc.org/projects/sisudoc-spine)
-project for full licence text.
-
-## Repository
-
-- **Canonical URL:** https://git.sisudoc.org/tools/ssp-reader
-- **Clone:** `git clone --depth=1 git://git.sisudoc.org/tools/ssp-reader`
-
-## Related projects
-
-- [sisudoc-spine](https://git.sisudoc.org/projects/sisudoc-spine) -
- produces the `.ssp` files this tool consumes.
-- [doc-abstraction-format](https://git.sisudoc.org/tools/doc-abstraction-format) -
- specification of the `.ssp` text format.
-- [tree-sitter-sisu](https://git.sisudoc.org/tools/tree-sitter-sisu) -
- tree-sitter grammar for the upstream `.sst` / `.ssm` markup.
+## Why more than one
+
+These are not three tools that happen to share a name. They are **the same
+specification implemented more than once**, and the interesting thing about
+them is that they agree.
+
+Each was written from the format specification, `doc-abstraction-format`,
+rather than from spine's emitter or from each other. Run over the 36
+document reference collection, **the D and Gleam readers agree on the
+section counts of all 36**.
+
+That is the point of the exercise. A specification that one implementation
+follows is a description of that implementation; a specification that two
+independent implementations follow, in languages with nothing in common, is
+a description of the format.
+
+## What a reader must do
+
+From the specification's conformance section, and both readers do it:
+
+- **check the major version** declared on the first line and refuse a major
+ it does not know, rather than loading an unfamiliar artefact half
+ populated in silence;
+- **tolerate unknown properties** within a major version, since the minor
+ part is raised precisely by adding them;
+- **say what it ignored**, on request. A reader that drops what it does not
+ know is conforming; one that cannot say what it dropped cannot show a
+ specification to be complete, silence and understanding being alike from
+ outside.
+
+The last is why both carry `--report-unknown`, and it has earned its place:
+it found that the D reader was rendering markup comments into its HTML,
+text and JSONL output, because it was ignoring the property that says an
+object is commentary.
+
+## Building
+
+Each language directory builds in its own way and says how in its own
+README. Neither depends on the other.
+
+## The format
+
+Specified in **doc-abstraction-format**: `ocda-model.md` for the model both
+serialisations share, `ssp-format-specification.md` for this one, and
+`ocda-db-format-specification.md` for the sqlite form.
+
+The readers here are at format **2.x**.