The library's reader, ocda-read: where it lives and why (src/read/, kept out of the library in D by dub configuration, in Gleam because nothing outside src/ is compiled), its modes in each language, and the two properties it holds, the same output from every form and D == Gleam byte for byte. And the one defect it found in the library, the database loader unable to say what it had not read. The module tables gain ocda.inventory and the reader, the build notes the three programs in one result. (assisted by Claude-Code)
ocda-lib read, installed also as ocda-read: the modes the Gleam readers had (the summary, --tree/--outline, --languages, --files, --report-unknown), over the library, in src/read/, which is where Gleam must put it. A D-only mode asked of it says so. Its output is the D ocda-read's. For that the D reader's closing hint no longer lists its modes, which this one has fewer of: both say that --help lists them. Checked over the document sample set, on the .ssp and the .ocda.db: D == Gleam byte for byte in all five modes, 36 of 36. And the same from the .ssp, the .ocda.db and the .zip (a .sisupod unwrapped, the Gleam library having no zstd): the summary, --tree and --report-unknown 36 of 36; --languages of a database == of its pod 27 of 27, and of a single-language .ssp == its database; --files answers for a database and says why not for the others, in the same words in both languages. (assisted by Claude-Code)
The mirror of the D commit before it. ocda/inventory gives the languages of any form, in the manifest's order, each with its markup file and the sha256 of its .ssp (stored by the database, taken from the bytes for a .ssp or a pod), and what a .ocda.db carries. The sha256 comes from OTP's crypto through a four line Erlang FFI file, since the standard library has none and a Hex package would be one more fixed output hash; the whitelist admits src/*.erl for it. The database loader counts the columns it does not read, on the conditions the .ssp writer uses, so that it can say what it did not understand; it could not before. The pod loader's archive, language and text helpers become public for the inventory. Checked: the four dumps, D and Gleam of .ssp and .ocda.db, identical 36 of 36, and with --unknown also 36 of 36; --languages of a database == of its pod, 27 of 27. (assisted by Claude-Code)
One reader where there were two. sisudoc-ssp-reader read a .ssp and
sisudoc-ocda-db-reader a .ocda.db, each with its own parser; this one
is built on the library, so every mode reads a .ssp, a .ocda.db and a
.sisupod, and says the same thing of each. Its output modules are the
.ssp reader's, copied (the reader repositories are unchanged) and
moved onto the library's model, which was the reader's types renamed
and added to. From the database reader, --languages and --files.
ocda-read [--tree|--outline|--html|--llm-text|--llm-jsonl|--images|
--verify --image-dir=<d>|--languages|--files|
--report-unknown|--validate] <file> [lang]
Two modes are about the file and say so: --files answers for a
.ocda.db, and --validate checks a .ssp as text and refuses anything
else. Where a file holds several languages and none is named, every
form reads the first and says so, where the database used to take one
and the pod refuse.
In src/read/ beside the library in src/ocda/; dub.json keeps it out of
the library (configurations "library" and "ocda-read"), and the flake
builds it as a package. The flake's default is now all three programs
in one bin/: built separately into the same result link, each replaced
the one before, and the pipeline's dump looked for ocda-dump where
only ocda-agree was left.
Checked over the document sample set:
- against the .ssp reader, on a .ssp, byte for byte after the first
line (Parsed: is Read:): --tree, --html, --llm-text, --llm-jsonl,
--images, --verify, --validate, 36 of 36; the summary differs only
in the closing hint, which names the two new modes;
- against the database reader, on a .ocda.db: --files 27 of 27, and
--languages 27 of 27 on language, source and digest (its id column,
the database's own key, is left out);
- the same output from the .ssp, the .ocda.db and the .sisupod: the
summary and --tree, --html, --llm-text, --llm-jsonl, --images and
--report-unknown, 36 of 36; --languages, .ocda.db == .sisupod 27 of
27, and a single-language .ssp == its database, 26 of 26.
Not reproduced, deliberately: the database reader's --outline, which
ordered the headings by their position within their own section, so
that the toc and the "(skip)" headings of other sections fell among
the body's. --outline here is --tree.
(assisted by Claude-Code)
ocda.inventory answers two questions about the file rather than the document. languages(path), for every form alike: the languages in the manifest's order (which every .ssp records in @source languages), each with the markup it was made from and the sha256 of its .ssp - stored by the database, taken from the bytes for a .ssp or a pod, and the same. carried(path), for a .ocda.db: what it holds besides the abstraction, by role and storage. And the database loader now counts what it holds and the model does not. The text loader meets every property line and counts those it does not keep; this one asks for columns by name and so never saw the others, and reported that it had understood everything, which is the silence the count exists to prevent. It counts them on the conditions under which the .ssp writes the same property, and the report is now the same from a .ssp, a .ocda.db and a .sisupod: 36 of 36. The pod loader's archive and language-name helpers become package visible for the inventory. (assisted by Claude-Code)
The format now states, for a container, the first and last ocn of what it holds: .ocn_range in the .ssp, ocn_range_first and ocn_range_last in the .ocda.db. A poem is the one container, and since 2.1 it carries ocn 0 and comes before its verses. Both fields in the model, read by both loaders, in both languages, and written by the dump. A 2.0 database has neither column, so each loader asks sqlite whether it is there and reads a 2.0 file as having no ranges, which is what 2.0 meant; checked on a published 2.0 database, D and Gleam giving the same dump. Over the document sample set written by spine at 2.1, the four dumps (D and Gleam, .ssp and .ocda.db) are byte identical. (assisted by Claude-Code)
Two additions needed by the epub writer.
ocda.images: an image's bytes, from whichever input the document came
from. The model names images, with digest, byte size and pixel size, and
does not hold them (right as an abstraction is not a bundle of assets)
so something has to know where they are, and the answer depends on the
input form rather than on the output. Hence the library.
.ocda.db the files table, role 'image', by name
.sisupod the archive entry whose path ends in /image/<name>
.zip the same
.ssp beside it: a pod puts the abstraction in media/abstraction/
and its images in media/image/, so the path is documented
rather than guessed
canGiveImages says whether a source can answer at all, because a writer
has to know before it emits an <img>: a manifest item whose file is
absent and an <img> whose file is not in the manifest are both epubcheck
errors. An .ssp outside a pod has nowhere to look, and that is absence
rather than error - the document is still readable.
Reading a blob needed sqlite3_column_blob rather than _column_text: an
image read as text stops at the first zero byte, and a png's ninth byte
is zero.
And two more model fields: segment_epub and segment_epub_is. The epub
segments at heading levels 1 to 4 where the html segments at level 4
alone, so the two are different files and the abstraction records both.
Nine of the 36 documents have images; the four-way agreement still holds
at 36 of 36 each way.
(assisted by Claude-Code)
anchor_tags and heading_lev_anchor. Both are in the specification's property table and neither was read, so they went to the unknown counts and a writer over the library had no way to emit a link target. The cost was visible the moment an html writer existed: 154 of its internal links pointed at nothing, because a note reference points at the endnote's anchor and the endnote's anchor is an anchor_tag. heading_lev_anchor is the name a document's own cross references use, 1,854 of them in the reference collection. The .ssp carries them as repeated .anchor_tag and as .heading_lev_anchor; the database as the object_anchors child table and the heading_lev_anchor column. Both loaders read both now, the dump reports them, and the four-way agreement still holds: 36 of 36 each way. One thing to watch when adding a column to the database query: the read is by position, so a column inserted in the middle shifts every one after it. Adding heading_lev_anchor after segment_html_is broke all 36 documents until it moved to the end of the select. (assisted by Claude-Code)
A pod built with --pod2 carries the abstraction inside it, at media/abstraction/<uid>.<lang>.ssp. So ocda.load gains .sisupod and .zip, and because the dispatch is in one place a writer over the library gained a third input form without being touched. Over the reference collection a .sisupod gives the same model as the .ssp beside it: all document (& prog-language) outputs, byte identical dumps. And the text writer, unchanged, renders a document from its .sisupod byte identically to spine --text, 36 of 36. Two small readers, neither of them a library: ocda.zstd declares the four libzstd entry points needed to unwrap a frame, as ocda.sqlite declares sqlite3; ocda.zip reads a central directory and one stored entry. Deflate is refused rather than mishandled, since spine writes stored entries and a library that says it read what it skipped is worse than one that says it cannot. A frame that declares an expansion over 512 MiB is refused before anything is allocated: a reader of an artefact it did not make declines to be told how much memory to take. The Gleam library reads the zip and not the frame. The only zstd on the BEAM is a NIF that clones zstd from github in its own build, so it cannot be built sandboxed without patching it. A .zip pod is read in both languages, a .sisupod in D only, and the Gleam refusal says why and says "zstd -d" rather than claiming the file is malformed. It is a packaging limit, not anything about the format, and the only place the two libraries differ in what they accept. A pod without an abstraction is refused with a message naming the flag that puts one in, because that is a fact about the pod. (assisted by Claude-Code)
incorrect fixture property line indentation resulted in unread lines A `.ssp` property line is `.name: value` at the start of the line. The Gleam test fixture indented them by two spaces, which makes each a line of no known shape, so every property in the fixture was being counted as malformed and ignored. The tests around it passed for the wrong reason. Corrected, and a test added that reads a property, a digest and a parent, and asserts that the fixture now holds no malformed line. Only the fixture was wrong: the loaders agree with each other and with the D library over the whole reference collection, which is real files. (assisted by Claude-Code)