diff options
Diffstat (limited to 'README.md')
| -rw-r--r-- | README.md | 150 |
1 files changed, 150 insertions, 0 deletions
diff --git a/README.md b/README.md new file mode 100644 index 0000000..5e328c0 --- /dev/null +++ b/README.md @@ -0,0 +1,150 @@ +# sisudoc-ocda-lib + +The **object-centric document abstraction as a library**: the model, and +loaders for both of the forms it is written in. + +Written by Claude (Anthropic), working in Claude Code from the specification, +at the direction of and reviewed by Ralph Amissah. + +**One reader per language, in a directory named for the language.** + +| | | +|--------------------|---------| +| [`dlang/`](dlang/) | D | +| [`gleam/`](gleam/) | Gleam | +| `ruby/` | to come | + +## What it is for + +The readers in `sisudoc-ssp-reader` and `sisudoc-ocda-db-reader` each read +one serialisation and print. A **writer** needs neither: it needs the document. + +``` + .ssp ─┐ + ├─> the model ─> a writer + .ocda.db ─┘ +``` + +So this library holds the model and both loaders, and **a writer cannot +tell which serialisation it was handed** - exactly as spine's own writers +cannot, since spine 0.25.0. + +That is the whole of the idea. The readers become thin programs over this, +and the writers are functions from the model to bytes. + +## The checks that matter + +The claim above is only true if the loaders agree, so the library asks. Two +programs, in each language: + +```sh +ocda-agree <doc>.ssp <doc>.ocda.db [lang] # do the two loaders agree? +ocda-dump <file>.ssp|<file>.ocda.db [lang] # the whole model, canonically +``` + +`agree` compares two models within one language. **`dump` is the stronger +of the two**, and the reason it exists: it writes the entire loaded model +out in a fully determined form, and the D and the Gleam library write it +*byte for byte alike*. So four questions can be asked, all of them with +`diff`: + +| | | +|-----------------------------|---------------------------------| +| D: `.ssp` == `.ocda.db` | **36 of 36** document-languages | +| Gleam: `.ssp` == `.ocda.db` | **36 of 36** | +| `.ssp`: D == Gleam | **36 of 36** | +| `.ocda.db`: D == Gleam | **36 of 36** | + +None of those held at first. + +## What the checks found + +### the ten-field check: a blank first text line + +An object whose **first text line is blank** lost it. The loader decided +whether to start a new line by asking whether the accumulated text was +empty, which is not the same question as whether a line has been seen. Of +the sample collection **seven documents have verse opening on a blank +line**, and in every one of them the blank was dropped. Eight of +thirty-six documents disagreed. + +Nothing else had caught it. The two `.ssp` readers had been compared on +*section counts*, which this does not change. It took loading the same +document two ways and comparing the objects. + +Fixed here and in `sisudoc-ssp-reader/dlang`, which carried the same code. + +### the whole-model dump: four more + +The ten-field check was enough for that and not enough for anything +subtler. Dumping the whole model found four further disagreements between +the two loaders: + +1. **`.identifier` and `.anchor`.** The `.ssp` emits a property only when it + carries information, so the identifier is absent when it is simply the + ocn and the anchor is absent when it is the segment name; the database + stores both unconditionally. Over one document that is **1,314 + identifiers and 69 anchors**. The loaders now derive them, which is what + the model means. +2. **`make` and `doc_has`.** The database has one metadata table where the + text form has three blocks, and tells them apart by a key prefix. The + loaders unprefix them, so a writer asks `make["footer"]` whichever + serialisation it was handed. +3. **The JSON columns were scanned rather than parsed.** Phobos writes `/` + as `\/`, so every heading containing a slash came back with a backslash + in it: **all 36 document-languages** differed on + `heading_ancestors_text` and nothing else had shown it. +4. **`translation.ocn_aligned`** is in the database and cannot be in a + `.ssp`, which is one language and cannot know whether the languages + share a numbering. It is a fact about the *file*, so the loaders leave + it out rather than letting a writer tell which serialisation it was + given. + +The first three are defects. The fourth is a real asymmetry between the two +serialisations, and naming it is the useful part. + +## What it reads, and what it does not + +| | | | +|-----------------------|------------------------------------------------------------------|-------------------| +| `.ssp` | yes | | +| `.ocda.db` | yes, any language of it | | +| `.sisupod` | yes, any language it carries | D only, see below | +| `.zip` pod | yes, any language it carries | | +| `.sst`, `.ssm` markup | **no, and not planned**: that is a parser, which is spine's work | | +| a url | no | | + +A pod carries the abstraction only when spine was asked for it: `--pod2` +puts it in, `--source` does not. A pod without one is refused with a message +saying which flag makes a pod this library can read, because that is a fact +about the pod rather than about the library. + +**Over the reference collection, a `.sisupod` gives the same model as the +`.ssp` beside it: 36 of 36 document-languages, byte identical dumps.** A +writer over the library therefore gained a third input form without being +touched, which is the point of the dispatch being in one place. + +### the one thing the Gleam library cannot do + +A `.sisupod` is **one zstd frame wrapping** the zip. The D library reads both +through a declared `libzstd`. The Gleam library reads the **zip and not the +frame**, because the only zstd on the BEAM is a NIF that clones zstd from +github in its own build and so cannot be built in a sandbox without patching +it. + +So a `.zip` pod is read in both languages, a `.sisupod` in D only, and the +Gleam refusal says why and says `zstd -d` rather than claiming the file is +malformed. It is a packaging limit, not anything about the format, and it is +the only place the two libraries differ in what they accept. + +`ocda/load` is the one place in each language that decides what a path is, +so a new input form is added once and every reader and writer gains it. + +## The format + +Specified in **sisudoc-abstraction-format**: `ocda-model.md` for the model +this library implements, and the two format specifications for the loaders. +Section 4 of `ocda-db-format-specification.md` gives the column by column +correspondence the `load_db` of each language follows. + +At format **2.x**. |
