summaryrefslogtreecommitdiffhomepage
path: root/README.md
diff options
context:
space:
mode:
authorRalph Amissah <ralph.amissah@gmail.com>2026-09-25 21:18:06 -0400
committerRalph Amissah <ralph.amissah@gmail.com>2026-09-25 21:18:06 -0400
commit04285c2f7079ee21d91c66b97a24e042d40dbb33 (patch)
tree25a481a48c662d9e7abd0074771d650422a17359 /README.md
parentocda-dump, and the four disagreements it found (diff)
ocda-lib in Gleam, and four agreements of 36 of 36
The model, both loaders, the dispatch and the canonical dump, in Gleam. A deliberate field-for-field mirror of the D library, including where that is not the most idiomatic Gleam: text is one string with the lines joined by newline rather than a list of lines, and the properties are typed fields rather than a list of pairs. A difference that existed in one language and not the other would show in every dump and mean nothing, so the two lose the same distinctions in the same places. The dump is what makes the agreement testable. Both languages write it byte for byte alike, so four questions are diff, and over the reference collection all four hold: D: .ssp == .ocda.db 36 of 36 Gleam: .ssp == .ocda.db 36 of 36 .ssp: D == Gleam 36 of 36 .ocda.db: D == Gleam 36 of 36 and a fifth, that the two report exactly the same ignored properties with the same counts. The Gleam .ssp loader agreed with the D one at the first attempt. The Gleam .ocda.db loader had one defect of its own, found by the sweep and worth recording because of its shape: a JSON element decoder handled strings and integers and fell through to a fallback that yielded "". table_widths is an integer array except where the columns divide evenly, when it is [33.3333,...], and one table in the collection does that. Its widths came back as three empty strings, so a value that could not be read turned into one that looked read. That is the failure the unknown counts exist to prevent, and a decoder fallback quietly reintroduces it. Shipped as an Erlang shipment rather than an escript, as the .ocda.db reader is and for the same reason: sqlite on the BEAM is a NIF and a NIF cannot load from inside an escript archive. gleam.toml and the model came in with the previous commit, which was about the dump; they belong here. (assisted by Claude-Code) Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Diffstat (limited to 'README.md')
-rw-r--r--README.md77
1 files changed, 58 insertions, 19 deletions
diff --git a/README.md b/README.md
index 39da9f9..efa9e70 100644
--- a/README.md
+++ b/README.md
@@ -8,7 +8,7 @@ loaders for both of the forms it is written in.
| | |
|---|---|
| [`dlang/`](dlang/) | D |
-| `gleam/` | to come |
+| [`gleam/`](gleam/) | Gleam |
| `ruby/` | to come |
## What it is for
@@ -29,38 +29,77 @@ cannot, since spine 0.25.0.
That is the whole of the idea. The readers become thin programs over this,
and the writers are functions from the model to bytes.
-## The check that matters
+## The checks that matter
-The claim above is only true if the two loaders agree, so the library asks:
+The claim above is only true if the loaders agree, so the library asks. Two
+programs, in each language:
```sh
-nix build && ./result/bin/ocda-agree <doc>.ssp <doc>.ocda.db [lang]
+ocda-agree <doc>.ssp <doc>.ocda.db [lang] # do the two loaders agree?
+ocda-dump <file>.ssp|<file>.ocda.db [lang] # the whole model, canonically
```
-It compares the two models object by object: citation number, kind, type,
-part, parent, heading level, digest, text, ancestors and image count.
+`agree` compares two models within one language. **`dump` is the stronger
+of the two**, and the reason it exists: it writes the entire loaded model
+out in a fully determined form, and the D and the Gleam library write it
+*byte for byte alike*. So four questions can be asked, all of them with
+`diff`:
-**Over the reference collection the two loaders agree on 36 of 36
-document-languages.** They did not at first: see below.
+| | |
+|---|---|
+| D: `.ssp` == `.ocda.db` | **36 of 36** document-languages |
+| Gleam: `.ssp` == `.ocda.db` | **36 of 36** |
+| `.ssp`: D == Gleam | **36 of 36** |
+| `.ocda.db`: D == Gleam | **36 of 36** |
-## What the check found, first time it was run
+None of those held at first.
-Eight of thirty-six disagreed, and the cause was a defect in the `.ssp`
-loader rather than a difference between the formats.
+## What the checks found
+
+### the ten-field check: a blank first text line
An object whose **first text line is blank** lost it. The loader decided
whether to start a new line by asking whether the accumulated text was
empty, which is not the same question as whether a line has been seen. Of
the sample collection **seven documents have verse opening on a blank
-line**, and in every one of them the blank was dropped.
+line**, and in every one of them the blank was dropped. Eight of
+thirty-six documents disagreed.
-Nothing else had caught it. The two `.ssp` readers were compared on
-*section counts*, which this does not change; spine's own output was never
-diffed against the reader's. It took loading the same document two ways and
-comparing the objects.
+Nothing else had caught it. The two `.ssp` readers had been compared on
+*section counts*, which this does not change. It took loading the same
+document two ways and comparing the objects.
Fixed here and in `sisudoc-ssp-reader/dlang`, which carried the same code.
+### the whole-model dump: four more
+
+The ten-field check was enough for that and not enough for anything
+subtler. Dumping the whole model found four further disagreements between
+the two loaders:
+
+1. **`.identifier` and `.anchor`.** The `.ssp` emits a property only when it
+ carries information, so the identifier is absent when it is simply the
+ ocn and the anchor is absent when it is the segment name; the database
+ stores both unconditionally. Over one document that is **1,314
+ identifiers and 69 anchors**. The loaders now derive them, which is what
+ the model means.
+2. **`make` and `doc_has`.** The database has one metadata table where the
+ text form has three blocks, and tells them apart by a key prefix. The
+ loaders unprefix them, so a writer asks `make["footer"]` whichever
+ serialisation it was handed.
+3. **The JSON columns were scanned rather than parsed.** Phobos writes `/`
+ as `\/`, so every heading containing a slash came back with a backslash
+ in it: **all 36 document-languages** differed on
+ `heading_ancestors_text` and nothing else had shown it.
+4. **`translation.ocn_aligned`** is in the database and cannot be in a
+ `.ssp`, which is one language and cannot know whether the languages
+ share a numbering. It is a fact about the *file*, so the loaders leave
+ it out rather than letting a writer tell which serialisation it was
+ given.
+
+The first three are defects. The fourth is a real asymmetry between the two
+serialisations, and naming it is the useful part.
+
## What it reads, and what it does not
| | |
@@ -71,14 +110,14 @@ Fixed here and in `sisudoc-ssp-reader/dlang`, which carried the same code.
| `.sst`, `.ssm` markup | **no, and not planned**: that is a parser, which is spine's work |
| a url | no |
-`ocda/load.d` is the one place that decides what a path is, so a new input
-type is added once and every reader and writer gains it.
+`ocda/load` is the one place in each language that decides what a path is,
+so a new input form is added once and every reader and writer gains it.
## The format
Specified in **sisudoc-abstraction-format**: `ocda-model.md` for the model
this library implements, and the two format specifications for the loaders.
Section 4 of `ocda-db-format-specification.md` gives the column by column
-correspondence that `load_db.d` follows.
+correspondence the `load_db` of each language follows.
At format **2.x**.