ocda-dump, and the four disagreements it found

The agreement check compared ten fields of each object. That was enough
to find the blank-first-line defect and not enough for anything
subtler, so this adds ocda-dump: the whole loaded model written out
canonically, every field, sorted dictionaries, sections in format
order, defaults omitted. It is an oracle, not an output format. Two
implementations of this model can now be compared with diff.

Run against the reference collection it found four disagreements
between the two loaders, none of which the ten-field check could see.

1. .identifier and .anchor. The .ssp emits a property only when it
   carries information, so the identifier is absent when it is simply
   the ocn and the anchor is absent when it is the segment name; the
   database stores both unconditionally. Over one document that is
   1,314 identifiers and 69 anchors. The loader now derives them, which
   is what the model means: an object's identifier is its ocn unless it
   says otherwise.

2. make and doc_has. The database has one metadata table where the
   text form has three blocks, and tells them apart by a key prefix.
   The loader now unprefixes them, so a writer asks doc.make["footer"]
   whichever serialisation it was handed and never
   doc.metadata["make.footer"].

3. The JSON columns were read by a character scan that stripped
   brackets and quotes, which is wrong the moment a value holds an
   escape: JSONValue.toString escapes "/" as "\/", so every heading
   containing a slash came back with a backslash in it. All 36
   document-languages differed on heading_ancestors_text. Now parsed as
   JSON.

4. translation.ocn_aligned is in the database and not in the .ssp,
   legitimately: it says whether the languages share one numbering, and
   a .ssp is one language and cannot know. It is a fact about the file
   rather than the document, so the loader leaves it out rather than
   letting a writer tell which serialisation it was given.

Metadata values are also stripped now, as the text loader strips them:
one topic_register value carried surrounding spaces in the database
that the text form cannot represent.

After all four: 36 of 36 document-languages give byte identical dumps
from their .ssp and from their .ocda.db.

(assisted by Claude-Code)

2 files changed, 283 insertions, 0 deletions

diff --git a/gleam/gleam.toml b/gleam/gleam.toml
new file mode 100644
index 0000000..d1a7a57
--- /dev/null
+++ b/gleam/gleam.toml
@@ -0,0 +1,13 @@
+name = "ocda_lib"
+version = "0.1.0"
+description = "the object-centric document abstraction as a library: the model, and loaders for its two serialisations"
+licences = ["AGPL-3.0-or-later"]
+
+[dependencies]
+gleam_stdlib = ">= 1.0.0 and < 2.0.0"
+argv = ">= 1.1.0 and < 2.0.0"
+simplifile = ">= 2.7.0 and < 3.0.0"
+sqlight = ">= 1.0.0 and < 2.0.0"
+
+[dev-dependencies]
+gleeunit = ">= 1.0.0 and < 2.0.0"
diff --git a/gleam/src/ocda/model.gleam b/gleam/src/ocda/model.gleam
new file mode 100644
index 0000000..20d7fba
--- /dev/null
+++ b/gleam/src/ocda/model.gleam
@@ -0,0 +1,270 @@
+//// The object-centric document abstraction, as Gleam types.
+////
+//// A deliberate mirror of the D library's `ocda.model`, field for field.
+//// The two are meant to agree, and the cheapest way to make that testable
+//// is for them to hold the same things under the same names: the dump of a
+//// loaded document is then identical by construction rather than by
+//// negotiation.
+////
+//// Written against specs/sisudoc-abstraction-format: `ocda-model.md` for
+//// the model, the two format specifications for what each serialisation
+//// says about it.
+
+import gleam/dict.{type Dict}
+import gleam/list
+import gleam/string
+
+/// The format major version this library knows.
+///
+/// The `.ssp` and the `.ocda.db` carry one version between them, declared
+/// in band. The minor part rises when a property is added and a reader of
+/// the newer format still reads the older, so only the major is checked.
+pub const format_major_read = "2"
+
+/// The nine sections, in the order the format gives them, which is not the
+/// order a sorted key list or a `group by` would give.
+pub const section_names = [
+  "head", "toc", "body", "endnotes", "glossary", "bibliography", "bookindex",
+  "blurb", "tail",
+]
+
+/// The four blocks that are document preamble rather than sections of
+/// objects.
+pub const preamble_blocks = ["source", "meta", "make", "doc_has"]
+
+/// A repeatable `.image` record: the file as the abstraction found it.
+pub type Image {
+  Image(
+    /// filename as the object refers to it
+    name: String,
+    /// hex, uppercase
+    sha256: String,
+    bytes: Int,
+    /// the file's own pixel dimensions, not any display width
+    width: Int,
+    height: Int,
+    /// absent or unreadable, so no digest and no size
+    missing: Bool,
+    /// any further key:value fields, kept rather than dropped
+    fields: Dict(String, String),
+  )
+}
+
+pub fn new_image() -> Image {
+  Image(
+    name: "",
+    sha256: "",
+    bytes: 0,
+    width: 0,
+    height: 0,
+    missing: False,
+    fields: dict.new(),
+  )
+}
+
+/// One object of the abstraction.
+///
+/// `text` is one string with the lines joined by newline, as in the D
+/// library and for the same reason: an object whose only text line is
+/// blank is then indistinguishable from an object with no text, in both
+/// libraries, in the same place. A difference that exists in one language
+/// and not the other would be worse than a loss both share.
+pub type Object {
+  Object(
+    ocn: Int,
+    /// one of the sixteen kinds
+    is_a: String,
+    /// the section this object was found in
+    section: String,
+    /// headings: A B C D 1 2 3 4
+    heading_level: String,
+    identifier: String,
+    part: String,
+    /// how the text behaves: para, block, comment
+    is_of_type: String,
+    /// the object's own section, when it differs from the block it sits in
+    own_section: String,
+    segment_html_is: String,
+    table_aligns: String,
+    stow_link: List(String),
+    parent_ocn: Int,
+    last_descendant_ocn: Int,
+    children: List(Int),
+    ancestors: List(Int),
+    dummy_heading: Bool,
+    ocn_off: Bool,
+    indent_base: Int,
+    indent_hang: Int,
+    bullet: Bool,
+    lang: String,
+    has_links: Bool,
+    has_notes_reg: Bool,
+    has_notes_star: Bool,
+    has_images: Bool,
+    segment: String,
+    segment_prev: String,
+    segment_next: String,
+    anchor: String,
+    table_cols: Int,
+    table_widths: String,
+    table_header: Bool,
+    code_syntax: String,
+    code_linenumbers: Bool,
+    heading_ancestors_text: String,
+    sha256: String,
+    images: List(Image),
+    text: String,
+  )
+}
+
+pub fn new_object() -> Object {
+  Object(
+    ocn: 0,
+    is_a: "",
+    section: "",
+    heading_level: "",
+    identifier: "",
+    part: "",
+    is_of_type: "",
+    own_section: "",
+    segment_html_is: "",
+    table_aligns: "",
+    stow_link: [],
+    parent_ocn: 0,
+    last_descendant_ocn: 0,
+    children: [],
+    ancestors: [],
+    dummy_heading: False,
+    ocn_off: False,
+    indent_base: 0,
+    indent_hang: 0,
+    bullet: False,
+    lang: "",
+    has_links: False,
+    has_notes_reg: False,
+    has_notes_star: False,
+    has_images: False,
+    segment: "",
+    segment_prev: "",
+    segment_next: "",
+    anchor: "",
+    table_cols: 0,
+    table_widths: "",
+    table_header: False,
+    code_syntax: "",
+    code_linenumbers: False,
+    heading_ancestors_text: "",
+    sha256: "",
+    images: [],
+    text: "",
+  )
+}
+
+/// Whether this object is rendered at all.
+///
+/// `is_of_type` "comment" is markup commentary: carried by the abstraction
+/// so that nothing is lost, and not put into output. An output that
+/// renders it leaks the author's notes into the document, which is what
+/// spine itself does not do.
+pub fn renders(object: Object) -> Bool {
+  object.is_of_type != "comment"
+}
+
+/// The section this object belongs to: the block it sits in, unless it
+/// says otherwise.
+pub fn section_of(object: Object) -> String {
+  case object.own_section {
+    "" -> object.section
+    own -> own
+  }
+}
+
+/// What the loader met and did not understand.
+///
+/// A loader that silently drops what it does not know cannot say whether
+/// it understood the document, and so cannot be used to check a
+/// specification against an implementation: silence and understanding look
+/// alike.
+pub type Unknown {
+  Unknown(
+    /// `.name` -> times seen
+    property: Dict(String, Int),
+    /// `@name` -> times seen
+    block: Dict(String, Int),
+    /// first character -> times seen, for lines of no known shape
+    line: Dict(String, Int),
+  )
+}
+
+pub fn new_unknown() -> Unknown {
+  Unknown(property: dict.new(), block: dict.new(), line: dict.new())
+}
+
+pub fn any_unknown(unknown: Unknown) -> Bool {
+  dict.size(unknown.property) > 0
+  || dict.size(unknown.block) > 0
+  || dict.size(unknown.line) > 0
+}
+
+pub type Document {
+  Document(
+    /// as declared on line 1 of a `.ssp` or in `schema.version`, "" when
+    /// absent
+    format_version: String,
+    source_filename: String,
+    /// language, languages, digest
+    source: Dict(String, String),
+    unknown: Unknown,
+    metadata: Dict(String, String),
+    make: Dict(String, String),
+    doc_has: Dict(String, String),
+    /// section name -> objects, in document order
+    sections: Dict(String, List(Object)),
+  )
+}
+
+pub fn new_document() -> Document {
+  Document(
+    format_version: "",
+    source_filename: "",
+    source: dict.new(),
+    unknown: new_unknown(),
+    metadata: dict.new(),
+    make: dict.new(),
+    doc_has: dict.new(),
+    sections: dict.new(),
+  )
+}
+
+/// The major part of the declared version, "" when none was declared.
+pub fn version_major(document: Document) -> String {
+  case string.split_once(document.format_version, ".") {
+    Ok(#(major, _)) -> major
+    Error(_) -> document.format_version
+  }
+}
+
+pub fn meta(document: Document, key: String) -> String {
+  case dict.get(document.metadata, key) {
+    Ok(value) -> value
+    Error(_) -> ""
+  }
+}
+
+/// The objects of a named section, empty when the document has none.
+pub fn section(document: Document, name: String) -> List(Object) {
+  case dict.get(document.sections, name) {
+    Ok(objects) -> objects
+    Error(_) -> []
+  }
+}
+
+/// Is this one of the nine sections of the format?
+pub fn is_section(name: String) -> Bool {
+  list.contains(section_names, name)
+}
+
+/// Is this one of the four preamble blocks?
+pub fn is_preamble_block(name: String) -> Bool {
+  list.contains(preamble_blocks, name)
+}