format 2.1: ocda-read, the readers folded in

ocda-read, one reader where there were two. The .ssp reader and the
.ocda.db reader each had their own parser; this one is built on the
library, so every mode reads a .ssp, a .ocda.db and a pod, and says
the same thing of each. Its output modules are the .ssp reader's,
copied (the reader repositories are kept, frozen) and moved onto the
library's model; --languages and --files come from the database
reader.

  ocda-read [--tree|--html|--llm-text|--llm-jsonl|--images|
             --verify --image-dir=<d>|--languages|--files|
             --report-unknown|--validate] <file> [lang]

In D every mode of both readers; in Gleam the modes the Gleam readers
had (the summary, --tree, --languages, --files, --report-unknown),
whose output is the D reader's, byte for byte. In each language it
lives in src/read/ and is not part of the library: dub configurations
keep it out of a dependant's build in D, and Gleam compiles all of
src/ but a dependant need not use it.

Checked over the markup document samples: against the .ssp reader on
a .ssp, byte for byte after the first line in every D mode but the
summary, whose closing hint differs; against the database reader on a
.ocda.db, --files and --languages (its id column left out); the same
output from the .ssp, the .ocda.db and the .sisupod; and D == Gleam.
The database reader's --outline is not reproduced: it ordered
headings within their own section, so that other sections' headings
fell among the body's. --outline here is --tree.

What writing it found in the library: the database loader could not
say what it had not understood. It asked for columns by name and so
never saw the rest, and reported that it had understood everything.
It now counts them on the conditions under which the .ssp writes the
same property, and --report-unknown is the same from every form.

ocda.inventory answers two questions about the file rather than the
document: its languages, in the manifest's order, each with the
markup it was made from and the sha256 of its .ssp, for every form
alike; and what a .ocda.db carries besides the abstraction. In Gleam
the sha256 comes from OTP's crypto through a four line Erlang FFI.

The flake's default is all three D programs in one result: built
separately into the same link, each replaced the one before.

(assisted by Claude-Code)

1 files changed, 107 insertions, 1 deletions

diff --git a/gleam/src/ocda/load_db.gleam b/gleam/src/ocda/load_db.gleam
index 641923e..3ae4f0b 100644
--- a/gleam/src/ocda/load_db.gleam
+++ b/gleam/src/ocda/load_db.gleam
@@ -5,7 +5,7 @@
 //// then a function from the model to bytes and cannot tell which
 //// serialisation it was given, exactly as spine's own writers cannot.
 ////
-//// Written against `ocda-db-format-specification.md`, whose section 4
+//// Written against `ocda-db-format-specification.org`, whose section 4
 //// gives the column by column correspondence this module implements.
 
 import gleam/dict
@@ -112,6 +112,7 @@ pub fn load_file(
     )
   let document = metadata(conn, chosen.doc_id, document)
   let document = objects(conn, chosen.doc_id, document)
+  let document = count_unread(conn, chosen.doc_id, document)
   Ok(document)
 }
 
@@ -561,3 +562,108 @@ fn json_joined(value: String) -> String {
     parts -> string.join(parts, "|")
   }
 }
+
+/// What the database holds and the model does not.
+///
+/// The text loader meets every property line and counts those it does not
+/// keep, so a reader can say what was read and not understood. This loader
+/// asks for columns by name, and would never see the ones it did not ask
+/// for: it would report that it had understood everything, which is the
+/// silence the count exists to prevent. So it counts them, on the
+/// conditions under which the `.ssp` writes the same property
+/// (specification section 6.3), and the report is the same from either
+/// form, as the D loader's is.
+fn count_unread(
+  conn: sqlight.Connection,
+  doc_id: Int,
+  document: Document,
+) -> Document {
+  let id = int.to_string(doc_id)
+  let counts = [
+    #("ancestors_collapsed", "coalesce(ancestors_collapsed,'') != ''"),
+    #("dom_status", "coalesce(dom_status,'') != ''"),
+    #("dom_status_collapsed", "coalesce(dom_status_collapsed,'') != ''"),
+    #("heading_lev_collapsed", "heading_lev_collapsed is not null"),
+    #("parent_lev", "coalesce(parent_lev,0) != 0"),
+    #("segment_lv4_is", "coalesce(segment_lv4_is,'') != ''"),
+    #("attrib", "coalesce(attrib,'') != ''"),
+  ]
+  let property =
+    list.fold(counts, document.unknown.property, fn(acc, c) {
+      let n =
+        count(
+          conn,
+          "select count(*) from objects where doc_id = " <> id <> " and " <> c.1,
+        )
+      case n > 0 {
+        True -> dict.insert(acc, c.0, n)
+        False -> acc
+      }
+    })
+  let property = case has(conn, "object_subtoc") {
+    False -> property
+    True -> {
+      let n =
+        count(
+          conn,
+          "select count(*) from object_subtoc s join objects o
+             on o.id = s.object_id where o.doc_id = " <> id,
+        )
+      case n > 0 {
+        True -> dict.insert(property, "lev4_subtoc", n)
+        False -> property
+      }
+    }
+  }
+  model.Document(
+    ..document,
+    unknown: model.Unknown(..document.unknown, property: property),
+  )
+}
+
+fn count(conn: sqlight.Connection, sql: String) -> Int {
+  let d = decode.field(0, decode.int, decode.success)
+  case sqlight.query(sql, on: conn, with: [], expecting: d) {
+    Ok([n, ..]) -> n
+    _ -> 0
+  }
+}
+
+/// Every language the database at `path` holds, in manifest order.
+pub fn languages_of_file(path: String) -> Result(List(Language), String) {
+  use conn <- with_connection("file:" <> path <> "?mode=ro", path)
+  Ok(languages(conn))
+}
+
+/// What the database at `path` carries besides the abstraction, by role
+/// and storage: role, compression, count, original bytes, stored bytes.
+pub fn carried_of_file(path: String) -> Result(List(List(String)), String) {
+  use conn <- with_connection("file:" <> path <> "?mode=ro", path)
+  case has(conn, "files") {
+    False -> Ok([])
+    True -> {
+      let d = {
+        use role <- decode.field(0, decode.string)
+        use compression <- decode.field(1, decode.string)
+        use n <- decode.field(2, decode.int)
+        use bytes <- decode.field(3, decode.int)
+        use stored <- decode.field(4, decode.int)
+        decode.success([
+          role,
+          compression,
+          int.to_string(n),
+          int.to_string(bytes),
+          int.to_string(stored),
+        ])
+      }
+      sqlight.query(
+        "select role, compression, count(*), sum(bytes), sum(length(data))
+           from files group by role, compression order by role",
+        on: conn,
+        with: [],
+        expecting: d,
+      )
+      |> result.map_error(fn(e) { path <> ": " <> e.message })
+    }
+  }
+}