summaryrefslogtreecommitdiffhomepage
path: root/README.md
diff options
context:
space:
mode:
authorRalph Amissah <ralph.amissah@gmail.com>2026-09-26 09:47:07 -0400
committerRalph Amissah <ralph.amissah@gmail.com>2026-09-26 09:47:07 -0400
commitfda48e417435e31a3122bfa7ca563333628aed99 (patch)
treec91f50006de299a9db42247bc21df4f59f9c7a82 /README.md
format 2.0: abstraction as library, in D and Gleam
The object-centric document abstraction that sisudoc-spine builds from SiSU markup, read as a library: the model, and a loader for each form it is written in - the .ssp (text), the .ocda.db (sqlite), and a pod archive carrying a .ssp (.sisupod, .zip). Begun 2026-09-25; this is the library as it stood at format 2.0. The readers (sisudoc-ssp-reader, sisudoc-ocda-db-reader) each read one form and print. A writer needs neither: it needs the document. So the model and the loaders live here, ocda.model imports nothing, and a program built on the library cannot tell which form it was handed. ocda.load is the one place that decides what a path is, so a new input form is added once and every program gains it. The model and the .ssp loader began as the .ssp reader's code, moved and renamed to the level they belong to; the .ocda.db loader is new. The Gleam library is a deliberate field-for-field mirror of the D one, including where that is not the most idiomatic Gleam: a difference in one language and not the other would show in every comparison and mean nothing. The claim holds only if the loaders agree, so the library asks: ocda-agree <doc>.ssp <doc>.ocda.db [lang] ten fields, per object ocda-dump <file> [lang] the whole model, canonically The dump is an oracle, not an output format, and the D and Gleam dumps are byte identical, so four questions are a diff: D and Gleam, each from the .ssp and from the .ocda.db. Over the markup document samples all four hold, every document-language; a .sisupod gives the same model as the .ssp beside it. None of that held at first, and what the checks found is the record of how the specification became enough: - an object whose first text line is blank lost it (the loader asked whether the text was empty, not whether a line had been seen); seven sample documents open verse that way. Found by agree; fixed here and in the .ssp reader, which carried the same code. - found by the dump: the identifier and the anchor are omitted by the text form when they can be derived and stored by the database always, so the loaders derive them, and the rule went into the specification (derived values); make and doc_has share one table in the database under a key prefix, which the loaders strip; the JSON columns were scanned, not parsed, and Phobos writes "/" as "\/"; and translation.ocn_aligned is a fact about the file, not the document, so neither loader puts it in the model. - in Gleam, a JSON element decoder fell through to a fallback giving "", so a float column read as empty strings: a value that could not be read turned into one that looked read. - found by the writers: anchor_tags, heading_lev_anchor, segment_epub and segment_epub_is were in the specification and read by nothing. ocda.images gives an image's bytes from whichever form the document came from (the database's files table, the archive entry, or beside a .ssp at media/image/), and says beforehand whether it can. The Gleam library reads a .zip pod but not a .sisupod: the zstd frame around it has no implementation on the BEAM that builds sandboxed. It says so, and says zstd -d, rather than calling the file malformed. gleam/ builds with nix-gleam's builder, kept in gleam/nix-gleam/ (MIT) Use: dlang/ nix build ocda-dump, ocda-agree; or a dub path dependency on dlang/ gleam/ nix build ocda-lib dump | agree See README.md, and dlang/README.md and gleam/README.md. (assisted by Claude-Code)
Diffstat (limited to 'README.md')
-rw-r--r--README.md150
1 files changed, 150 insertions, 0 deletions
diff --git a/README.md b/README.md
new file mode 100644
index 0000000..5e328c0
--- /dev/null
+++ b/README.md
@@ -0,0 +1,150 @@
+# sisudoc-ocda-lib
+
+The **object-centric document abstraction as a library**: the model, and
+loaders for both of the forms it is written in.
+
+Written by Claude (Anthropic), working in Claude Code from the specification,
+at the direction of and reviewed by Ralph Amissah.
+
+**One reader per language, in a directory named for the language.**
+
+| | |
+|--------------------|---------|
+| [`dlang/`](dlang/) | D |
+| [`gleam/`](gleam/) | Gleam |
+| `ruby/` | to come |
+
+## What it is for
+
+The readers in `sisudoc-ssp-reader` and `sisudoc-ocda-db-reader` each read
+one serialisation and print. A **writer** needs neither: it needs the document.
+
+```
+ .ssp ─┐
+ ├─> the model ─> a writer
+ .ocda.db ─┘
+```
+
+So this library holds the model and both loaders, and **a writer cannot
+tell which serialisation it was handed** - exactly as spine's own writers
+cannot, since spine 0.25.0.
+
+That is the whole of the idea. The readers become thin programs over this,
+and the writers are functions from the model to bytes.
+
+## The checks that matter
+
+The claim above is only true if the loaders agree, so the library asks. Two
+programs, in each language:
+
+```sh
+ocda-agree <doc>.ssp <doc>.ocda.db [lang] # do the two loaders agree?
+ocda-dump <file>.ssp|<file>.ocda.db [lang] # the whole model, canonically
+```
+
+`agree` compares two models within one language. **`dump` is the stronger
+of the two**, and the reason it exists: it writes the entire loaded model
+out in a fully determined form, and the D and the Gleam library write it
+*byte for byte alike*. So four questions can be asked, all of them with
+`diff`:
+
+| | |
+|-----------------------------|---------------------------------|
+| D: `.ssp` == `.ocda.db` | **36 of 36** document-languages |
+| Gleam: `.ssp` == `.ocda.db` | **36 of 36** |
+| `.ssp`: D == Gleam | **36 of 36** |
+| `.ocda.db`: D == Gleam | **36 of 36** |
+
+None of those held at first.
+
+## What the checks found
+
+### the ten-field check: a blank first text line
+
+An object whose **first text line is blank** lost it. The loader decided
+whether to start a new line by asking whether the accumulated text was
+empty, which is not the same question as whether a line has been seen. Of
+the sample collection **seven documents have verse opening on a blank
+line**, and in every one of them the blank was dropped. Eight of
+thirty-six documents disagreed.
+
+Nothing else had caught it. The two `.ssp` readers had been compared on
+*section counts*, which this does not change. It took loading the same
+document two ways and comparing the objects.
+
+Fixed here and in `sisudoc-ssp-reader/dlang`, which carried the same code.
+
+### the whole-model dump: four more
+
+The ten-field check was enough for that and not enough for anything
+subtler. Dumping the whole model found four further disagreements between
+the two loaders:
+
+1. **`.identifier` and `.anchor`.** The `.ssp` emits a property only when it
+ carries information, so the identifier is absent when it is simply the
+ ocn and the anchor is absent when it is the segment name; the database
+ stores both unconditionally. Over one document that is **1,314
+ identifiers and 69 anchors**. The loaders now derive them, which is what
+ the model means.
+2. **`make` and `doc_has`.** The database has one metadata table where the
+ text form has three blocks, and tells them apart by a key prefix. The
+ loaders unprefix them, so a writer asks `make["footer"]` whichever
+ serialisation it was handed.
+3. **The JSON columns were scanned rather than parsed.** Phobos writes `/`
+ as `\/`, so every heading containing a slash came back with a backslash
+ in it: **all 36 document-languages** differed on
+ `heading_ancestors_text` and nothing else had shown it.
+4. **`translation.ocn_aligned`** is in the database and cannot be in a
+ `.ssp`, which is one language and cannot know whether the languages
+ share a numbering. It is a fact about the *file*, so the loaders leave
+ it out rather than letting a writer tell which serialisation it was
+ given.
+
+The first three are defects. The fourth is a real asymmetry between the two
+serialisations, and naming it is the useful part.
+
+## What it reads, and what it does not
+
+| | | |
+|-----------------------|------------------------------------------------------------------|-------------------|
+| `.ssp` | yes | |
+| `.ocda.db` | yes, any language of it | |
+| `.sisupod` | yes, any language it carries | D only, see below |
+| `.zip` pod | yes, any language it carries | |
+| `.sst`, `.ssm` markup | **no, and not planned**: that is a parser, which is spine's work | |
+| a url | no | |
+
+A pod carries the abstraction only when spine was asked for it: `--pod2`
+puts it in, `--source` does not. A pod without one is refused with a message
+saying which flag makes a pod this library can read, because that is a fact
+about the pod rather than about the library.
+
+**Over the reference collection, a `.sisupod` gives the same model as the
+`.ssp` beside it: 36 of 36 document-languages, byte identical dumps.** A
+writer over the library therefore gained a third input form without being
+touched, which is the point of the dispatch being in one place.
+
+### the one thing the Gleam library cannot do
+
+A `.sisupod` is **one zstd frame wrapping** the zip. The D library reads both
+through a declared `libzstd`. The Gleam library reads the **zip and not the
+frame**, because the only zstd on the BEAM is a NIF that clones zstd from
+github in its own build and so cannot be built in a sandbox without patching
+it.
+
+So a `.zip` pod is read in both languages, a `.sisupod` in D only, and the
+Gleam refusal says why and says `zstd -d` rather than claiming the file is
+malformed. It is a packaging limit, not anything about the format, and it is
+the only place the two libraries differ in what they accept.
+
+`ocda/load` is the one place in each language that decides what a path is,
+so a new input form is added once and every reader and writer gains it.
+
+## The format
+
+Specified in **sisudoc-abstraction-format**: `ocda-model.md` for the model
+this library implements, and the two format specifications for the loaders.
+Section 4 of `ocda-db-format-specification.md` gives the column by column
+correspondence the `load_db` of each language follows.
+
+At format **2.x**.