# sisudoc-ocda-lib The **object-centric document abstraction as a library**: the model, and loaders for both of the forms it is written in. **One reader per language, in a directory named for the language.** | | | |--------------------|---------| | [`dlang/`](dlang/) | D | | [`gleam/`](gleam/) | Gleam | | `ruby/` | to come | ## What it is for The readers in `sisudoc-ssp-reader` and `sisudoc-ocda-db-reader` each read one serialisation and print. A **writer** needs neither: it needs the document. ``` .ssp ─┐ ├─> the model ─> a writer .ocda.db ─┘ ``` So this library holds the model and both loaders, and **a writer cannot tell which serialisation it was handed** - exactly as spine's own writers cannot, since spine 0.25.0. That is the whole of the idea. The writers are functions from the model to bytes, and the reader is a program over the model. ## The reader ```sh ocda-read [--tree|--html|--llm-text|--llm-jsonl|--images|--verify --image-dir=| --languages|--files|--report-unknown|--validate] .ssp | .ocda.db | .sisupod [lang] ``` One reader, in each language, where `sisudoc-ssp-reader` and `sisudoc-ocda-db-reader` were two, each with a parser of its own. Built on this library, **every mode reads every form and says the same thing of a document's `.ssp`, `.ocda.db` and `.sisupod`**, 36 of 36 document-languages. The D reader has every mode of both old readers; the Gleam one the modes the Gleam readers had (the summary, `--tree`, `--languages`, `--files`, `--report-unknown`), and in those **its output is the D one's, byte for byte**, 36 of 36. It is in each language's `src/read/`, and not part of the library: a program using the library does not build it (in D, by dub configuration; in Gleam a dependant compiles it and does not use it, Gleam building everything in `src/`). Writing it found one defect in the library: **the database loader could not say what it had not understood**. It asks for columns by name, and so never saw the ones it did not ask for, and reported that it had understood everything. It now counts them on the conditions under which the `.ssp` writes the same property, and the report is the same from either form. ## The checks that matter The claim above is only true if the loaders agree, so the library asks. Two programs, in each language: ```sh ocda-agree .ssp .ocda.db [lang] # do the two loaders agree? ocda-dump .ssp|.ocda.db [lang] # the whole model, canonically ``` `agree` compares two models within one language. **`dump` is the stronger of the two**, and the reason it exists: it writes the entire loaded model out in a fully determined form, and the D and the Gleam library write it *byte for byte alike*. So four questions can be asked, all of them with `diff`: | | | |-----------------------------|---------------------------------| | D: `.ssp` == `.ocda.db` | **36 of 36** document-languages | | Gleam: `.ssp` == `.ocda.db` | **36 of 36** | | `.ssp`: D == Gleam | **36 of 36** | | `.ocda.db`: D == Gleam | **36 of 36** | None of those held at first. ## What the checks found ### the ten-field check: a blank first text line An object whose **first text line is blank** lost it. The loader decided whether to start a new line by asking whether the accumulated text was empty, which is not the same question as whether a line has been seen. Of the sample collection **seven documents have verse opening on a blank line**, and in every one of them the blank was dropped. Eight of thirty-six documents disagreed. Nothing else had caught it. The two `.ssp` readers had been compared on *section counts*, which this does not change. It took loading the same document two ways and comparing the objects. Fixed here and in `sisudoc-ssp-reader/dlang`, which carried the same code. ### the whole-model dump: four more The ten-field check was enough for that and not enough for anything subtler. Dumping the whole model found four further disagreements between the two loaders: 1. **`.identifier` and `.anchor`.** The `.ssp` emits a property only when it carries information, so the identifier is absent when it is simply the ocn and the anchor is absent when it is the segment name; the database stores both unconditionally. Over one document that is **1,314 identifiers and 69 anchors**. The loaders now derive them, which is what the model means. 2. **`make` and `doc_has`.** The database has one metadata table where the text form has three blocks, and tells them apart by a key prefix. The loaders unprefix them, so a writer asks `make["footer"]` whichever serialisation it was handed. 3. **The JSON columns were scanned rather than parsed.** Phobos writes `/` as `\/`, so every heading containing a slash came back with a backslash in it: **all 36 document-languages** differed on `heading_ancestors_text` and nothing else had shown it. 4. **`translation.ocn_aligned`** is in the database and cannot be in a `.ssp`, which is one language and cannot know whether the languages share a numbering. It is a fact about the *file*, so the loaders leave it out rather than letting a writer tell which serialisation it was given. The first three are defects. The fourth is a real asymmetry between the two serialisations, and naming it is the useful part. ## What it reads, and what it does not | | | | |-----------------------|------------------------------------------------------------------|-------------------| | `.ssp` | yes | | | `.ocda.db` | yes, any language of it | | | `.sisupod` | yes, any language it carries | D only, see below | | `.zip` pod | yes, any language it carries | | | `.sst`, `.ssm` markup | **no, and not planned**: that is a parser, which is spine's work | | | a url | no | | A pod carries the abstraction only when spine was asked for it: `--pod2` puts it in, `--source` does not. A pod without one is refused with a message saying which flag makes a pod this library can read, because that is a fact about the pod rather than about the library. **Over the reference collection, a `.sisupod` gives the same model as the `.ssp` beside it: 36 of 36 document-languages, byte identical dumps.** A writer over the library therefore gained a third input form without being touched, which is the point of the dispatch being in one place. ### the one thing the Gleam library cannot do A `.sisupod` is **one zstd frame wrapping** the zip. The D library reads both through a declared `libzstd`. The Gleam library reads the **zip and not the frame**, because the only zstd on the BEAM is a NIF that clones zstd from github in its own build and so cannot be built in a sandbox without patching it. So a `.zip` pod is read in both languages, a `.sisupod` in D only, and the Gleam refusal says why and says `zstd -d` rather than claiming the file is malformed. It is a packaging limit, not anything about the format, and it is the only place the two libraries differ in what they accept. `ocda/load` is the one place in each language that decides what a path is, so a new input form is added once and every reader and writer gains it. ## The format Specified in **sisudoc-abstraction-format**: `ocda-model.md` for the model this library implements, and the two format specifications for the loaders. Section 4 of `ocda-db-format-specification.md` gives the column by column correspondence the `load_db` of each language follows. At format **2.x**.