summaryrefslogtreecommitdiffhomepage
path: root/gleam/src
AgeCommit message (Collapse)AuthorFilesLines
2026-10-02trackRalph Amissah2-2/+2
2026-10-02gleam sisudoc-ocda-libRalph Amissah12-0/+588
2026-10-02Gleam: ocda-read, byte for byte the D reader'sRalph Amissah2-3/+470
ocda-lib read, installed also as ocda-read: the modes the Gleam readers had (the summary, --tree/--outline, --languages, --files, --report-unknown), over the library, in src/read/, which is where Gleam must put it. A D-only mode asked of it says so. Its output is the D ocda-read's. For that the D reader's closing hint no longer lists its modes, which this one has fewer of: both say that --help lists them. Checked over the 36 document-languages, on the .ssp and the .ocda.db: D == Gleam byte for byte in all five modes, 36 of 36. And the same from the .ssp, the .ocda.db and the .zip (a .sisupod unwrapped, the Gleam library having no zstd): the summary, --tree and --report-unknown 36 of 36; --languages of a database == of its pod 27 of 27, and of a single-language .ssp == its database; --files answers for a database and says why not for the others, in the same words in both languages. (assisted by Claude-Code)
2026-10-02Gleam: what a source holds, and what a database holds unreadRalph Amissah4-3/+254
The mirror of the D commit before it. ocda/inventory gives the languages of any form, in the manifest's order, each with its markup file and the sha256 of its .ssp (stored by the database, taken from the bytes for a .ssp or a pod), and what a .ocda.db carries. The sha256 comes from OTP's crypto through a four line Erlang FFI file, since the standard library has none and a Hex package would be one more fixed output hash; the whitelist admits src/*.erl for it. The database loader counts the columns it does not read, on the conditions the .ssp writer uses, so that it can say what it did not understand; it could not before. The pod loader's archive, language and text helpers become public for the inventory. Checked: the four dumps, D and Gleam of .ssp and .ocda.db, identical 36 of 36, and with --unknown also 36 of 36; --languages of a database == of its pod, 27 of 27. (assisted by Claude-Code)
2026-10-02model 2.1: ocn_range, the verses a poem holdsRalph Amissah4-3/+48
The format now states, for a container, the first and last ocn of what it holds: .ocn_range in the .ssp, ocn_range_first and ocn_range_last in the .ocda.db. A poem is the one container, and since 2.1 it carries ocn 0 and comes before its verses. Both fields in the model, read by both loaders, in both languages, and written by the dump. A 2.0 database has neither column, so each loader asks sqlite whether it is there and reads a 2.0 file as having no ranges, which is what 2.0 meant; checked on a published 2.0 database, D and Gleam giving the same dump. Over the 36 document-languages written by spine at 2.1, the four dumps (D and Gleam, .ssp and .ocda.db) are byte identical, 36 of 36. (assisted by Claude-Code)
2026-10-02images from the source, and the epub segment fieldsRalph Amissah5-1/+149
Two additions the epub writer needed and could not have without. ocda.images: an image's bytes, from whichever input form the document came from. The model names images, with digest, byte size and pixel size, and does not hold them - which is right, an abstraction is not a bundle of assets - so something has to know where they are, and the answer depends on the input form rather than on the output. Hence the library. .ocda.db the files table, role 'image', by name .sisupod the archive entry whose path ends in /image/<name> .zip the same .ssp beside it: a pod puts the abstraction in media/abstraction/ and its images in media/image/, so the path is documented rather than guessed canGiveImages says whether a source can answer at all, because a writer has to know before it emits an <img>: a manifest item whose file is absent and an <img> whose file is not in the manifest are both epubcheck errors. A .ssp outside a pod has nowhere to look, and that is absence rather than error - the document is still readable. Reading a blob needed sqlite3_column_blob rather than _column_text: an image read as text stops at the first zero byte, and a png's ninth byte is zero. And two more model fields: segment_epub and segment_epub_is. The epub segments at heading levels 1 to 4 where the html segments at level 4 alone, so the two are different files and the abstraction records both. Nine of the 36 documents have images; the four-way agreement still holds at 36 of 36 each way. (assisted by Claude-Code)
2026-10-02two fields the model was missing, found by writing an html writerRalph Amissah4-2/+44
anchor_tags and heading_lev_anchor. Both are in the specification's property table and neither was read, so they went to the unknown counts and a writer over the library had no way to emit a link target. The cost was visible the moment an html writer existed: 154 of its internal links pointed at nothing, because a note reference points at the endnote's anchor and the endnote's anchor is an anchor_tag. heading_lev_anchor is the name a document's own cross references use, 1,854 of them in the reference collection. The .ssp carries them as repeated .anchor_tag and as .heading_lev_anchor; the database as the object_anchors child table and the heading_lev_anchor column. Both loaders read both now, the dump reports them, and the four-way agreement still holds: 36 of 36 each way. One thing to watch when adding a column to the database query: the read is by position, so a column inserted in the middle shifts every one after it. Adding heading_lev_anchor after segment_html_is broke all 36 documents until it moved to the end of the select. (assisted by Claude-Code)
2026-10-02a third input form: the abstraction out of a pod archiveRalph Amissah3-14/+332
A pod built with --pod2 carries the abstraction inside it, at media/abstraction/<uid>.<lang>.ssp. So ocda.load gains .sisupod and .zip, and because the dispatch is in one place a writer over the library gained a third input form without being touched. Over the reference collection a .sisupod gives the same model as the .ssp beside it: 36 of 36 document-languages, byte identical dumps. And the text writer, unchanged, renders a document from its .sisupod byte identically to spine --text, 36 of 36. Two small readers, neither of them a library: ocda.zstd declares the four libzstd entry points needed to unwrap a frame, as ocda.sqlite declares sqlite3; ocda.zip reads a central directory and one stored entry. Deflate is refused rather than mishandled, since spine writes stored entries and a library that says it read what it skipped is worse than one that says it cannot. A frame that declares an expansion over 512 MiB is refused before anything is allocated: a reader of an artefact it did not make declines to be told how much memory to take. The Gleam library reads the zip and not the frame. The only zstd on the BEAM is a NIF that clones zstd from github in its own build, so it cannot be built sandboxed without patching it. A .zip pod is read in both languages, a .sisupod in D only, and the Gleam refusal says why and says "zstd -d" rather than claiming the file is malformed. It is a packaging limit, not anything about the format, and the only place the two libraries differ in what they accept. A pod without an abstraction is refused with a message naming the flag that puts one in, because that is a fact about the pod. (assisted by Claude-Code)
2026-10-02ocda-lib in Gleam, and four agreements of 36 of 36Ralph Amissah5-0/+1415
The model, both loaders, the dispatch and the canonical dump, in Gleam. A deliberate field-for-field mirror of the D library, including where that is not the most idiomatic Gleam: text is one string with the lines joined by newline rather than a list of lines, and the properties are typed fields rather than a list of pairs. A difference that existed in one language and not the other would show in every dump and mean nothing, so the two lose the same distinctions in the same places. The dump is what makes the agreement testable. Both languages write it byte for byte alike, so four questions are diff, and over the reference collection all four hold: D: .ssp == .ocda.db 36 of 36 Gleam: .ssp == .ocda.db 36 of 36 .ssp: D == Gleam 36 of 36 .ocda.db: D == Gleam 36 of 36 and a fifth, that the two report exactly the same ignored properties with the same counts. The Gleam .ssp loader agreed with the D one at the first attempt. The Gleam .ocda.db loader had one defect of its own, found by the sweep and worth recording because of its shape: a JSON element decoder handled strings and integers and fell through to a fallback that yielded "". table_widths is an integer array except where the columns divide evenly, when it is [33.3333,...], and one table in the collection does that. Its widths came back as three empty strings, so a value that could not be read turned into one that looked read. That is the failure the unknown counts exist to prevent, and a decoder fallback quietly reintroduces it. Shipped as an Erlang shipment rather than an escript, as the .ocda.db reader is and for the same reason: sqlite on the BEAM is a NIF and a NIF cannot load from inside an escript archive. gleam.toml and the model came in with the previous commit, which was about the dump; they belong here. (assisted by Claude-Code)
2026-10-02ocda-dump, and the four disagreements it foundRalph Amissah1-0/+270
The agreement check compared ten fields of each object. That was enough to find the blank-first-line defect and not enough for anything subtler, so this adds ocda-dump: the whole loaded model written out canonically, every field, sorted dictionaries, sections in format order, defaults omitted. It is an oracle, not an output format. Two implementations of this model can now be compared with diff. Run against the reference collection it found four disagreements between the two loaders, none of which the ten-field check could see. 1. .identifier and .anchor. The .ssp emits a property only when it carries information, so the identifier is absent when it is simply the ocn and the anchor is absent when it is the segment name; the database stores both unconditionally. Over one document that is 1,314 identifiers and 69 anchors. The loader now derives them, which is what the model means: an object's identifier is its ocn unless it says otherwise. 2. make and doc_has. The database has one metadata table where the text form has three blocks, and tells them apart by a key prefix. The loader now unprefixes them, so a writer asks doc.make["footer"] whichever serialisation it was handed and never doc.metadata["make.footer"]. 3. The JSON columns were read by a character scan that stripped brackets and quotes, which is wrong the moment a value holds an escape: JSONValue.toString escapes "/" as "\/", so every heading containing a slash came back with a backslash in it. All 36 document-languages differed on heading_ancestors_text. Now parsed as JSON. 4. translation.ocn_aligned is in the database and not in the .ssp, legitimately: it says whether the languages share one numbering, and a .ssp is one language and cannot know. It is a fact about the file rather than the document, so the loader leaves it out rather than letting a writer tell which serialisation it was given. Metadata values are also stripped now, as the text loader strips them: one topic_register value carried surrounding spaces in the database that the text form cannot represent. After all four: 36 of 36 document-languages give byte identical dumps from their .ssp and from their .ocda.db. (assisted by Claude-Code)