summaryrefslogtreecommitdiffhomepage
path: root/README.md
diff options
context:
space:
mode:
Diffstat (limited to 'README.md')
-rw-r--r--README.md150
1 files changed, 150 insertions, 0 deletions
diff --git a/README.md b/README.md
new file mode 100644
index 0000000..5e328c0
--- /dev/null
+++ b/README.md
@@ -0,0 +1,150 @@
+# sisudoc-ocda-lib
+
+The **object-centric document abstraction as a library**: the model, and
+loaders for both of the forms it is written in.
+
+Written by Claude (Anthropic), working in Claude Code from the specification,
+at the direction of and reviewed by Ralph Amissah.
+
+**One reader per language, in a directory named for the language.**
+
+| | |
+|--------------------|---------|
+| [`dlang/`](dlang/) | D |
+| [`gleam/`](gleam/) | Gleam |
+| `ruby/` | to come |
+
+## What it is for
+
+The readers in `sisudoc-ssp-reader` and `sisudoc-ocda-db-reader` each read
+one serialisation and print. A **writer** needs neither: it needs the document.
+
+```
+ .ssp ─┐
+ ├─> the model ─> a writer
+ .ocda.db ─┘
+```
+
+So this library holds the model and both loaders, and **a writer cannot
+tell which serialisation it was handed** - exactly as spine's own writers
+cannot, since spine 0.25.0.
+
+That is the whole of the idea. The readers become thin programs over this,
+and the writers are functions from the model to bytes.
+
+## The checks that matter
+
+The claim above is only true if the loaders agree, so the library asks. Two
+programs, in each language:
+
+```sh
+ocda-agree <doc>.ssp <doc>.ocda.db [lang] # do the two loaders agree?
+ocda-dump <file>.ssp|<file>.ocda.db [lang] # the whole model, canonically
+```
+
+`agree` compares two models within one language. **`dump` is the stronger
+of the two**, and the reason it exists: it writes the entire loaded model
+out in a fully determined form, and the D and the Gleam library write it
+*byte for byte alike*. So four questions can be asked, all of them with
+`diff`:
+
+| | |
+|-----------------------------|---------------------------------|
+| D: `.ssp` == `.ocda.db` | **36 of 36** document-languages |
+| Gleam: `.ssp` == `.ocda.db` | **36 of 36** |
+| `.ssp`: D == Gleam | **36 of 36** |
+| `.ocda.db`: D == Gleam | **36 of 36** |
+
+None of those held at first.
+
+## What the checks found
+
+### the ten-field check: a blank first text line
+
+An object whose **first text line is blank** lost it. The loader decided
+whether to start a new line by asking whether the accumulated text was
+empty, which is not the same question as whether a line has been seen. Of
+the sample collection **seven documents have verse opening on a blank
+line**, and in every one of them the blank was dropped. Eight of
+thirty-six documents disagreed.
+
+Nothing else had caught it. The two `.ssp` readers had been compared on
+*section counts*, which this does not change. It took loading the same
+document two ways and comparing the objects.
+
+Fixed here and in `sisudoc-ssp-reader/dlang`, which carried the same code.
+
+### the whole-model dump: four more
+
+The ten-field check was enough for that and not enough for anything
+subtler. Dumping the whole model found four further disagreements between
+the two loaders:
+
+1. **`.identifier` and `.anchor`.** The `.ssp` emits a property only when it
+ carries information, so the identifier is absent when it is simply the
+ ocn and the anchor is absent when it is the segment name; the database
+ stores both unconditionally. Over one document that is **1,314
+ identifiers and 69 anchors**. The loaders now derive them, which is what
+ the model means.
+2. **`make` and `doc_has`.** The database has one metadata table where the
+ text form has three blocks, and tells them apart by a key prefix. The
+ loaders unprefix them, so a writer asks `make["footer"]` whichever
+ serialisation it was handed.
+3. **The JSON columns were scanned rather than parsed.** Phobos writes `/`
+ as `\/`, so every heading containing a slash came back with a backslash
+ in it: **all 36 document-languages** differed on
+ `heading_ancestors_text` and nothing else had shown it.
+4. **`translation.ocn_aligned`** is in the database and cannot be in a
+ `.ssp`, which is one language and cannot know whether the languages
+ share a numbering. It is a fact about the *file*, so the loaders leave
+ it out rather than letting a writer tell which serialisation it was
+ given.
+
+The first three are defects. The fourth is a real asymmetry between the two
+serialisations, and naming it is the useful part.
+
+## What it reads, and what it does not
+
+| | | |
+|-----------------------|------------------------------------------------------------------|-------------------|
+| `.ssp` | yes | |
+| `.ocda.db` | yes, any language of it | |
+| `.sisupod` | yes, any language it carries | D only, see below |
+| `.zip` pod | yes, any language it carries | |
+| `.sst`, `.ssm` markup | **no, and not planned**: that is a parser, which is spine's work | |
+| a url | no | |
+
+A pod carries the abstraction only when spine was asked for it: `--pod2`
+puts it in, `--source` does not. A pod without one is refused with a message
+saying which flag makes a pod this library can read, because that is a fact
+about the pod rather than about the library.
+
+**Over the reference collection, a `.sisupod` gives the same model as the
+`.ssp` beside it: 36 of 36 document-languages, byte identical dumps.** A
+writer over the library therefore gained a third input form without being
+touched, which is the point of the dispatch being in one place.
+
+### the one thing the Gleam library cannot do
+
+A `.sisupod` is **one zstd frame wrapping** the zip. The D library reads both
+through a declared `libzstd`. The Gleam library reads the **zip and not the
+frame**, because the only zstd on the BEAM is a NIF that clones zstd from
+github in its own build and so cannot be built in a sandbox without patching
+it.
+
+So a `.zip` pod is read in both languages, a `.sisupod` in D only, and the
+Gleam refusal says why and says `zstd -d` rather than claiming the file is
+malformed. It is a packaging limit, not anything about the format, and it is
+the only place the two libraries differ in what they accept.
+
+`ocda/load` is the one place in each language that decides what a path is,
+so a new input form is added once and every reader and writer gains it.
+
+## The format
+
+Specified in **sisudoc-abstraction-format**: `ocda-model.md` for the model
+this library implements, and the two format specifications for the loaders.
+Section 4 of `ocda-db-format-specification.md` gives the column by column
+correspondence the `load_db` of each language follows.
+
+At format **2.x**.