1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
|
# ssp-reader
Reader for the **`.ssp` document-abstraction text format** emitted
by [sisudoc-spine](https://git.sisudoc.org/projects/sisudoc-spine).
Reads a `.ssp` file (spine's PEG-parsable serialisation of its
internal document abstraction) and renders it into a chosen output:
an outline tree, scroll HTML, plain text with heading breadcrumbs,
JSONL chunks suitable for LLM / embedding pipelines, or an image
manifest for verifying the images a document embeds.
## What this is for
- Inspecting and debugging the abstraction stage of spine without
rebuilding spine itself.
- Round-trip experiments: markup -> spine -> `.ssp` -> ssp-reader
-> downstream artefact.
- Re-using spine's parsed document structure outside the spine
binary: feeding it to an indexer, an LLM pipeline, or a custom
renderer.
- A worked example of consuming `.ssp` for anyone implementing
another reader in another language.
## Audience
Developers building tools that consume spine's output: search
indexers, LLM ingestion pipelines, custom renderers, alternative
output formats, and anyone studying the abstraction stage of
spine itself.
End-user document authors do **not** need this tool to publish a
document. Run spine directly for that.
## Relationship to spine
Spine's processing pipeline has three stages:
1. **Input:** parse `.sst` / `.ssm` markup.
2. **Abstraction:** build a numbered object model (the "spine
abstraction"; one of spine's named outputs is `.ssp`, a
PEG-parsable text representation of this model).
3. **Output:** render to HTML / EPUB / PDF / SQLite / etc.
`ssp-reader` picks up the abstraction at the end of stage 2 (the
`.ssp` file) and runs alternative stage-3 renderings on top of it.
The `.ssp` format itself is documented in
[`doc-abstraction-format`](https://git.sisudoc.org/tools/doc-abstraction-format).
## Quickstart
### Build
```sh
nix build # the reader, with ldc
./result/bin/ssp-reader <file.ssp>
```
The reader depends on nothing beyond phobos, so `nix build` needs no
network and works in a sandbox. `nix build .#ssp-reader-dmd` builds it with
dmd instead, which is given gcc14 for the same reason spine gives it: GCC
15 introduced `nullptr` in its headers and dmd's ImportC parser does not
yet handle it.
For development, or to build without nix:
```sh
nix develop # ldc, dub, dtools
dub build --compiler=ldmd2
```
The binary then lands at `./bin/ssp-reader`.
### Run
```sh
# generate a .ssp from a sample document via spine, then:
ssp-reader --tree doc.ssp # outline view
ssp-reader --html doc.ssp # HTML scroll output
ssp-reader --llm-text doc.ssp # plain text with breadcrumbs
ssp-reader --llm-jsonl doc.ssp # JSONL chunks for embeddings
ssp-reader --images doc.ssp # image manifest (digest, size, pixels)
ssp-reader --verify --image-dir=pod/media/image doc.ssp # check images against the records
ssp-reader --report-unknown doc.ssp # what it read and did not understand
ssp-reader --tree --output=out.txt doc.ssp
ssp-reader --help
```
## Format version
The reader knows abstraction format **2.x**. Every `.ssp` declares its
version on its first line, and the reader checks the **major** part and
refuses what it does not know: an unknown major means a property may have
changed meaning or gone away, and loading such a file half populated in
silence is the failure the check exists to prevent. `--any-version` reads
it regardless, for looking at a file rather than trusting one.
## What it does not understand
Within a major version a reader **should** ignore properties it does not
know, since the minor part is raised precisely by adding them. This reader
does, and `--report-unknown` says what it ignored and how often:
```
read and NOT understood:
properties ignored (15 distinct):
.is_of_type 328
.segment_html_is 138
...
```
### What it ignores, and why
Twelve properties are ignored across the reference collection, and each is
ignored for a reason rather than by oversight:
| ignored | why |
|--------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------|
| `.ancestors_collapsed`, `.parent_lev`, `.heading_lev_collapsed`, `.dom_status`, `.dom_status_collapsed` | level bookkeeping; this reader works from `.ancestors` and the heading level |
| `.segment_epub`, `.segment_epub_is`, `.segment_lv4_is`, `.heading_lev_anchor`, `.anchor_tag`, `.lev4_subtoc` | naming for segmented HTML and EPUB output, which this reader does not produce |
| `.attrib` | one occurrence in the whole collection, and no output uses it |
The list was longer. `--report-unknown` found that `.is_of_type` was
ignored, and **that was a defect**: objects of type `comment` are markup
commentary, carried by the abstraction and not put into output, and three
of the four outputs were rendering them. A markup comment was appearing in
the HTML, in the plain text, and as an embedding chunk in the JSONL.
Fixed, and the reason the report exists.
That is deliberate and is what makes the reader usable as a check on the
[format specification](https://git.sisudoc.org/specs/doc-abstraction-format):
a reader that drops what it does not know is conforming, but one that
cannot say what it dropped cannot tell you whether the specification it was
written from was complete. Silence and understanding look alike otherwise.
## Documentation
- The `.ssp` format itself: see
[doc-abstraction-format](https://git.sisudoc.org/tools/doc-abstraction-format).
- Source: [`src/`](src/) - one module per output format
(`output_tree.d`, `output_html.d`, `output_llm_breadcrumbs.d`,
`output_llm_jsonl.d`, `output_images.d`, `verify_images.d`), plus
`ssp_parser.d` and `ssp_types.d`.
## License
AGPL-3.0-or-later. See [`dub.json`](dub.json) and the parent
[sisudoc-spine](https://git.sisudoc.org/projects/sisudoc-spine)
project for full licence text.
## Repository
- **Canonical URL:** https://git.sisudoc.org/tools/ssp-reader
- **Clone:** `git clone --depth=1 git://git.sisudoc.org/tools/ssp-reader`
## Related projects
- [sisudoc-spine](https://git.sisudoc.org/projects/sisudoc-spine) -
produces the `.ssp` files this tool consumes.
- [doc-abstraction-format](https://git.sisudoc.org/tools/doc-abstraction-format) -
specification of the `.ssp` text format.
- [tree-sitter-sisu](https://git.sisudoc.org/tools/tree-sitter-sisu) -
tree-sitter grammar for the upstream `.sst` / `.ssm` markup.
|