summaryrefslogtreecommitdiffhomepage
path: root/gleam
diff options
context:
space:
mode:
authorRalph Amissah <ralph.amissah@gmail.com>2026-10-12 20:24:11 -0400
committerRalph Amissah <ralph.amissah@gmail.com>2026-10-12 20:24:11 -0400
commit4e4f2f8c2b2c043077390cb4ccf7ce97ffe4ac6b (patch)
treebfafd6fb9cfa975db49fbcbcf7e3b23a4f13e0bc /gleam
parentmodel 2.1: ocn_range, the verses a poem holds (diff)
format 2.1: ocda-read, the readers folded in
ocda-read, one reader where there were two. The .ssp reader and the .ocda.db reader each had their own parser; this one is built on the library, so every mode reads a .ssp, a .ocda.db and a pod, and says the same thing of each. Its output modules are the .ssp reader's, copied (the reader repositories are kept, frozen) and moved onto the library's model; --languages and --files come from the database reader. ocda-read [--tree|--html|--llm-text|--llm-jsonl|--images| --verify --image-dir=<d>|--languages|--files| --report-unknown|--validate] <file> [lang] In D every mode of both readers; in Gleam the modes the Gleam readers had (the summary, --tree, --languages, --files, --report-unknown), whose output is the D reader's, byte for byte. In each language it lives in src/read/ and is not part of the library: dub configurations keep it out of a dependant's build in D, and Gleam compiles all of src/ but a dependant need not use it. Checked over the markup document samples: against the .ssp reader on a .ssp, byte for byte after the first line in every D mode but the summary, whose closing hint differs; against the database reader on a .ocda.db, --files and --languages (its id column left out); the same output from the .ssp, the .ocda.db and the .sisupod; and D == Gleam. The database reader's --outline is not reproduced: it ordered headings within their own section, so that other sections' headings fell among the body's. --outline here is --tree. What writing it found in the library: the database loader could not say what it had not understood. It asked for columns by name and so never saw the rest, and reported that it had understood everything. It now counts them on the conditions under which the .ssp writes the same property, and --report-unknown is the same from every form. ocda.inventory answers two questions about the file rather than the document: its languages, in the manifest's order, each with the markup it was made from and the sha256 of its .ssp, for every form alike; and what a .ocda.db carries besides the abstraction. In Gleam the sha256 comes from OTP's crypto through a four line Erlang FFI. The flake's default is all three D programs in one result: built separately into the same link, each replaced the one before. (assisted by Claude-Code)
Diffstat (limited to 'gleam')
-rw-r--r--gleam/README.md44
-rw-r--r--gleam/flake.lock17
-rw-r--r--gleam/flake.nix8
-rw-r--r--gleam/src/ocda/inventory.gleam138
-rw-r--r--gleam/src/ocda/load_db.gleam108
-rw-r--r--gleam/src/ocda/load_pod.gleam6
-rw-r--r--gleam/src/ocda/load_ssp.gleam2
-rw-r--r--gleam/src/ocda_ffi.erl7
-rw-r--r--gleam/src/ocda_lib.gleam14
-rw-r--r--gleam/src/read/main.gleam459
-rw-r--r--gleam/test/ocda_lib_test.gleam2
11 files changed, 772 insertions, 33 deletions
diff --git a/gleam/README.md b/gleam/README.md
index d586bd3..482db38 100644
--- a/gleam/README.md
+++ b/gleam/README.md
@@ -1,15 +1,18 @@
# ocda-lib, in Gleam
-The abstraction's model and both its loaders, as a Gleam library, and the
-two programs that put it to the question.
-
-| | |
-|-----------------|----------------------------------------------------------------------------------|
-| `ocda/model` | the document, the object, the image. Depends on nothing but the standard library |
-| `ocda/load_ssp` | text -> model |
-| `ocda/load_db` | sqlite -> model |
-| `ocda/load` | what a path is, and loading it |
-| `ocda/dump` | the model written out canonically, as an oracle |
+The abstraction's model and both its loaders, as a Gleam library, with its
+reader and the two programs that put it to the question.
+
+| | |
+|------------------|-----------------------------------------------------------------------------------------|
+| `ocda/model` | the document, the object, the image. Depends on nothing but the standard library |
+| `ocda/load_ssp` | text -> model |
+| `ocda/load_db` | sqlite -> model |
+| `ocda/load` | what a path is, and loading it |
+| `ocda/inventory` | what a file holds beyond the one document: its languages, and what a `.ocda.db` carries |
+| `ocda/dump` | the model written out canonically, as an oracle |
+| `read/main` | `ocda-read`, the reader |
+| `ocda_ffi.erl` | a sha256, from OTP's crypto, which the standard library lacks |
## Use
@@ -27,6 +30,23 @@ case load.load(path, "en", False) {
}
```
+## The reader
+
+```sh
+nix build
+./result/bin/ocda-read [--tree|--outline|--languages|--files|--report-unknown]
+ <file>.ssp|<file>.ocda.db|<file>.zip [lang]
+```
+
+`ocda-lib read`, installed also as `ocda-read`. The modes the Gleam readers
+had, over the library, so each reads every form this library reads; the
+others are the D reader's. **Its output is the D `ocda-read`'s, byte for
+byte**, in every mode both have. In `src/read/`, because Gleam compiles
+nothing outside `src/`; a program depending on the library compiles it too,
+and does not use it.
+
+A `.sisupod` it refuses, saying why: the zstd frame, below.
+
## The two programs
```sh
@@ -97,6 +117,6 @@ one and the other must change with it.
## Format
-Specified in **sisudoc-abstraction-format**: `ocda-model.md` for the model,
-`ssp-format-specification.md` and `ocda-db-format-specification.md` for the
+Specified in **sisudoc-abstraction-format**: `ocda-model.org` for the model,
+`ssp-format-specification.org` and `ocda-db-format-specification.org` for the
two serialisations. At format **2.x**.
diff --git a/gleam/flake.lock b/gleam/flake.lock
index 5f236ab..12c4c3b 100644
--- a/gleam/flake.lock
+++ b/gleam/flake.lock
@@ -2,18 +2,15 @@
"nodes": {
"nixpkgs": {
"locked": {
- "lastModified": 1790185690,
- "narHash": "sha256-xJ+X4hBtOcAFGBOe5nAMyMUeF9foJBmIOu3NjBqBycU=",
- "owner": "NixOS",
- "repo": "nixpkgs",
- "rev": "4975466d324710c576dc11ad614684e6bd8cad8e",
- "type": "github"
+ "lastModified": 1790652569,
+ "narHash": "sha256-k8Fu4c9Z+4Nh7mUr0cfw++ITQiyEhlWxoJBOkI3tOcQ=",
+ "rev": "b6c8664de9b6cc07fe5666a29f91884ba81197c4",
+ "type": "tarball",
+ "url": "https://releases.nixos.org/nixpkgs/nixpkgs-26.11pre1082290.b6c8664de9b6/nixexprs.tar.zst"
},
"original": {
- "owner": "NixOS",
- "ref": "nixos-unstable",
- "repo": "nixpkgs",
- "type": "github"
+ "type": "tarball",
+ "url": "https://channels.nixos.org/nixpkgs-unstable/nixexprs.tar.zst"
}
},
"root": {
diff --git a/gleam/flake.nix b/gleam/flake.nix
index a5a3407..132e3fc 100644
--- a/gleam/flake.nix
+++ b/gleam/flake.nix
@@ -1,6 +1,6 @@
{
description = "ocda-lib: the object-centric document abstraction as a Gleam library";
- inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
+ inputs.nixpkgs.url = "https://channels.nixos.org/nixpkgs-unstable/nixexprs.tar.zst";
outputs = { self, nixpkgs }:
let
system = "x86_64-linux"; # adjust for aarch64-linux / darwin
@@ -51,10 +51,14 @@
makeWrapper $out/lib/${name}/entrypoint.sh $out/bin/ocda-lib \
--add-flags run \
--prefix PATH : ${pkgs.lib.makeBinPath [ pkgs.beamPackages.erlang pkgs.coreutils ]}
+ # the reader under its own name, as the D flake installs it
+ makeWrapper $out/lib/${name}/entrypoint.sh $out/bin/ocda-read \
+ --add-flags "run read" \
+ --prefix PATH : ${pkgs.lib.makeBinPath [ pkgs.beamPackages.erlang pkgs.coreutils ]}
runHook postInstall
'';
meta = {
- description = "the abstraction as a Gleam library";
+ description = "the abstraction as a Gleam library, with its reader, dump and agreement check";
mainProgram = "ocda-lib";
};
};
diff --git a/gleam/src/ocda/inventory.gleam b/gleam/src/ocda/inventory.gleam
new file mode 100644
index 0000000..6047313
--- /dev/null
+++ b/gleam/src/ocda/inventory.gleam
@@ -0,0 +1,138 @@
+//// What a source holds, beyond the one document the model is loaded from.
+////
+//// The model is one language of one document. Two questions a reader asks
+//// are about the file instead: which languages it holds, and, for a
+//// database, what else it carries. They are here rather than in a reader
+//// because the answer depends on the input form, which is the library's to
+//// know, and so that a reader asking them does not learn which form it was
+//// handed. A mirror of the D library's `ocda.inventory`.
+////
+//// `languages` answers for every form, and answers the same: the languages
+//// in the order the manifest gave them (which every `.ssp` records, in
+//// `@source languages`), each with the markup file it was made from and the
+//// sha256 of its `.ssp`. The database stores that digest; for a `.ssp` and a
+//// pod it is taken from the bytes, which are the same bytes.
+////
+//// `carried` is the database's: a `.ocda.db` holds the markup, images,
+//// configuration and catalogues it was built from, a `.ssp` holds none of
+//// them, and a pod's are its archive entries rather than anything the
+//// abstraction says.
+
+import gleam/bit_array
+import gleam/dict
+import gleam/list
+import gleam/result
+import gleam/string
+import ocda/load
+import ocda/load_db
+import ocda/load_pod
+import ocda/load_ssp
+import ocda/zip
+import simplifile
+
+pub type Language {
+ Language(lang: String, source_filename: String, ssp_digest: String)
+}
+
+pub type Carried {
+ Carried(
+ role: String,
+ compression: String,
+ count: String,
+ bytes: String,
+ stored: String,
+ )
+}
+
+pub fn languages(path: String) -> Result(List(Language), String) {
+ case load.source_of(path) {
+ load.OcdaDb -> {
+ use held <- result.try(load_db.languages_of_file(path))
+ Ok(
+ list.map(held, fn(l) {
+ Language(l.lang, l.source_filename, l.ssp_digest)
+ }),
+ )
+ }
+ load.Ssp -> {
+ use bytes <- result.try(
+ simplifile.read_bits(path)
+ |> result.map_error(fn(e) { path <> ": " <> simplifile.describe_error(e) }),
+ )
+ use text <- result.try(
+ bit_array.to_string(bytes)
+ |> result.replace_error(path <> ": not valid UTF-8"),
+ )
+ let doc = load_ssp.parse(text)
+ Ok([
+ Language(
+ source_value(doc.source, "language"),
+ doc.source_filename,
+ sha256_hex(bytes),
+ ),
+ ])
+ }
+ load.Pod -> {
+ use archive <- result.try(load_pod.archive_bytes(path))
+ use entries <- result.try(zip.entries(archive))
+ use found <- result.try(
+ zip.find_suffix(entries, ".ssp")
+ |> list.try_map(fn(e) {
+ use bytes <- result.try(zip.read(archive, e))
+ use text <- result.try(
+ bit_array.to_string(bytes)
+ |> result.replace_error(e.name <> ": not valid UTF-8"),
+ )
+ let doc = load_ssp.parse(text)
+ let lang = result.unwrap(load_pod.lang_of(e.name), "")
+ Ok(#(
+ Language(lang, doc.source_filename, sha256_hex(bytes)),
+ string.split(source_value(doc.source, "languages"), " "),
+ ))
+ }),
+ )
+ let order = case found {
+ [#(_, o), ..] -> list.filter(o, fn(l) { l != "" })
+ [] -> []
+ }
+ let by_lang = list.map(found, fn(f) { #({ f.0 }.lang, f.0) }) |> dict.from_list
+ // the manifest's order, and anything it did not name after it
+ let named = list.filter_map(order, fn(l) { dict.get(by_lang, l) })
+ let rest =
+ list.filter(found, fn(f) { !list.contains(order, { f.0 }.lang) })
+ |> list.map(fn(f) { f.0 })
+ Ok(list.append(named, rest))
+ }
+ load.Unknown -> Error(path <> ": " <> load.source_name(load.Unknown))
+ }
+}
+
+/// Only a `.ocda.db` carries files.
+pub fn is_carrier(path: String) -> Bool {
+ load.source_of(path) == load.OcdaDb
+}
+
+pub fn carried(path: String) -> Result(List(Carried), String) {
+ case is_carrier(path) {
+ False -> Ok([])
+ True -> {
+ use rows <- result.try(load_db.carried_of_file(path))
+ Ok(
+ list.filter_map(rows, fn(r) {
+ case r {
+ [role, compression, n, bytes, stored] ->
+ Ok(Carried(role, compression, n, bytes, stored))
+ _ -> Error(Nil)
+ }
+ }),
+ )
+ }
+ }
+}
+
+fn source_value(source: dict.Dict(String, String), key: String) -> String {
+ result.unwrap(dict.get(source, key), "")
+}
+
+@external(erlang, "ocda_ffi", "sha256_hex")
+fn sha256_hex(bytes: BitArray) -> String
diff --git a/gleam/src/ocda/load_db.gleam b/gleam/src/ocda/load_db.gleam
index 641923e..3ae4f0b 100644
--- a/gleam/src/ocda/load_db.gleam
+++ b/gleam/src/ocda/load_db.gleam
@@ -5,7 +5,7 @@
//// then a function from the model to bytes and cannot tell which
//// serialisation it was given, exactly as spine's own writers cannot.
////
-//// Written against `ocda-db-format-specification.md`, whose section 4
+//// Written against `ocda-db-format-specification.org`, whose section 4
//// gives the column by column correspondence this module implements.
import gleam/dict
@@ -112,6 +112,7 @@ pub fn load_file(
)
let document = metadata(conn, chosen.doc_id, document)
let document = objects(conn, chosen.doc_id, document)
+ let document = count_unread(conn, chosen.doc_id, document)
Ok(document)
}
@@ -561,3 +562,108 @@ fn json_joined(value: String) -> String {
parts -> string.join(parts, "|")
}
}
+
+/// What the database holds and the model does not.
+///
+/// The text loader meets every property line and counts those it does not
+/// keep, so a reader can say what was read and not understood. This loader
+/// asks for columns by name, and would never see the ones it did not ask
+/// for: it would report that it had understood everything, which is the
+/// silence the count exists to prevent. So it counts them, on the
+/// conditions under which the `.ssp` writes the same property
+/// (specification section 6.3), and the report is the same from either
+/// form, as the D loader's is.
+fn count_unread(
+ conn: sqlight.Connection,
+ doc_id: Int,
+ document: Document,
+) -> Document {
+ let id = int.to_string(doc_id)
+ let counts = [
+ #("ancestors_collapsed", "coalesce(ancestors_collapsed,'') != ''"),
+ #("dom_status", "coalesce(dom_status,'') != ''"),
+ #("dom_status_collapsed", "coalesce(dom_status_collapsed,'') != ''"),
+ #("heading_lev_collapsed", "heading_lev_collapsed is not null"),
+ #("parent_lev", "coalesce(parent_lev,0) != 0"),
+ #("segment_lv4_is", "coalesce(segment_lv4_is,'') != ''"),
+ #("attrib", "coalesce(attrib,'') != ''"),
+ ]
+ let property =
+ list.fold(counts, document.unknown.property, fn(acc, c) {
+ let n =
+ count(
+ conn,
+ "select count(*) from objects where doc_id = " <> id <> " and " <> c.1,
+ )
+ case n > 0 {
+ True -> dict.insert(acc, c.0, n)
+ False -> acc
+ }
+ })
+ let property = case has(conn, "object_subtoc") {
+ False -> property
+ True -> {
+ let n =
+ count(
+ conn,
+ "select count(*) from object_subtoc s join objects o
+ on o.id = s.object_id where o.doc_id = " <> id,
+ )
+ case n > 0 {
+ True -> dict.insert(property, "lev4_subtoc", n)
+ False -> property
+ }
+ }
+ }
+ model.Document(
+ ..document,
+ unknown: model.Unknown(..document.unknown, property: property),
+ )
+}
+
+fn count(conn: sqlight.Connection, sql: String) -> Int {
+ let d = decode.field(0, decode.int, decode.success)
+ case sqlight.query(sql, on: conn, with: [], expecting: d) {
+ Ok([n, ..]) -> n
+ _ -> 0
+ }
+}
+
+/// Every language the database at `path` holds, in manifest order.
+pub fn languages_of_file(path: String) -> Result(List(Language), String) {
+ use conn <- with_connection("file:" <> path <> "?mode=ro", path)
+ Ok(languages(conn))
+}
+
+/// What the database at `path` carries besides the abstraction, by role
+/// and storage: role, compression, count, original bytes, stored bytes.
+pub fn carried_of_file(path: String) -> Result(List(List(String)), String) {
+ use conn <- with_connection("file:" <> path <> "?mode=ro", path)
+ case has(conn, "files") {
+ False -> Ok([])
+ True -> {
+ let d = {
+ use role <- decode.field(0, decode.string)
+ use compression <- decode.field(1, decode.string)
+ use n <- decode.field(2, decode.int)
+ use bytes <- decode.field(3, decode.int)
+ use stored <- decode.field(4, decode.int)
+ decode.success([
+ role,
+ compression,
+ int.to_string(n),
+ int.to_string(bytes),
+ int.to_string(stored),
+ ])
+ }
+ sqlight.query(
+ "select role, compression, count(*), sum(bytes), sum(length(data))
+ from files group by role, compression order by role",
+ on: conn,
+ with: [],
+ expecting: d,
+ )
+ |> result.map_error(fn(e) { path <> ": " <> e.message })
+ }
+ }
+}
diff --git a/gleam/src/ocda/load_pod.gleam b/gleam/src/ocda/load_pod.gleam
index 35483c6..05f7add 100644
--- a/gleam/src/ocda/load_pod.gleam
+++ b/gleam/src/ocda/load_pod.gleam
@@ -89,7 +89,7 @@ fn langs(entries: List(zip.Entry)) -> List(String) {
}
/// `<uid>.<lang>.ssp` -> `<lang>`
-fn lang_of(name: String) -> Result(String, Nil) {
+pub fn lang_of(name: String) -> Result(String, Nil) {
case string.ends_with(name, ".ssp") {
False -> Error(Nil)
True -> {
@@ -104,7 +104,7 @@ fn lang_of(name: String) -> Result(String, Nil) {
/// The bytes decide, not the name: a file misnamed either way is treated by
/// what it holds.
-fn archive_bytes(path: String) -> Result(BitArray, String) {
+pub fn archive_bytes(path: String) -> Result(BitArray, String) {
case simplifile.read_bits(path) {
Error(e) -> Error(path <> ": " <> simplifile.describe_error(e))
Ok(bytes) ->
@@ -133,6 +133,6 @@ fn is_zstd_frame(bytes: BitArray) -> Bool {
@external(erlang, "unicode", "characters_to_binary")
fn bit_array_to_string_ffi(b: BitArray) -> String
-fn bit_array_to_string(b: BitArray) -> Result(String, Nil) {
+pub fn bit_array_to_string(b: BitArray) -> Result(String, Nil) {
Ok(bit_array_to_string_ffi(b))
}
diff --git a/gleam/src/ocda/load_ssp.gleam b/gleam/src/ocda/load_ssp.gleam
index ec93ef5..324319e 100644
--- a/gleam/src/ocda/load_ssp.gleam
+++ b/gleam/src/ocda/load_ssp.gleam
@@ -4,7 +4,7 @@
//// apart by its first character, so the loader is a fold over the lines
//// carrying the state the shapes need.
////
-//// Written against `ssp-format-specification.md`. Where it and the D
+//// Written against `ssp-format-specification.org`. Where it and the D
//// library differ the D library is the bug, or this is: they are meant to
//// give the same model, and `ocda-dump` says whether they do.
diff --git a/gleam/src/ocda_ffi.erl b/gleam/src/ocda_ffi.erl
new file mode 100644
index 0000000..2e395cd
--- /dev/null
+++ b/gleam/src/ocda_ffi.erl
@@ -0,0 +1,7 @@
+%% The one thing ocda-lib asks of Erlang directly: a sha256, which OTP's
+%% crypto has and the Gleam standard library does not. Upper case hex, as
+%% the database stores it and the D library writes it.
+-module(ocda_ffi).
+-export([sha256_hex/1]).
+
+sha256_hex(Bin) -> binary:encode_hex(crypto:hash(sha256, Bin)).
diff --git a/gleam/src/ocda_lib.gleam b/gleam/src/ocda_lib.gleam
index 1e8a5ae..6eccaff 100644
--- a/gleam/src/ocda_lib.gleam
+++ b/gleam/src/ocda_lib.gleam
@@ -1,10 +1,11 @@
//// ocda-lib, in Gleam: the abstraction as a library, put to the question.
////
-//// A library has no binary, so this is the two programs that test its
-//// claims.
+//// A library has no binary, so this is the programs over it: the reader,
+//// and the two that test its claims.
////
//// ocda-lib dump [--unknown] <file>.ssp|<file>.ocda.db [lang]
//// ocda-lib agree <doc>.ssp <doc>.ocda.db [lang]
+//// ocda-lib read [options] <file> [lang] (also installed as ocda-read)
////
//// `dump` writes the whole loaded model out canonically, byte for byte as
//// the D library's `ocda-dump` writes it, so that the two implementations
@@ -20,6 +21,7 @@ import gleam/string
import ocda/dump
import ocda/load
import ocda/model.{type Document}
+import read/main as reader
pub fn main() {
case argv.load().arguments {
@@ -29,6 +31,11 @@ pub fn main() {
["dump", path, lang] -> dump_file(path, lang, False)
["agree", ssp, db] -> agree(ssp, db, "")
["agree", ssp, db, lang] -> agree(ssp, db, lang)
+ ["read", ..rest] ->
+ case reader.run(rest) {
+ 0 -> Nil
+ code -> halt(code)
+ }
_ -> usage()
}
}
@@ -36,7 +43,8 @@ pub fn main() {
fn usage() -> Nil {
io.println_error(
"usage: ocda-lib dump [--unknown] <file>.ssp|<file>.ocda.db [lang]
- ocda-lib agree <doc>.ssp <doc>.ocda.db [lang]",
+ ocda-lib agree <doc>.ssp <doc>.ocda.db [lang]
+ ocda-lib read [--help] [options] <file> [lang]",
)
halt(2)
}
diff --git a/gleam/src/read/main.gleam b/gleam/src/read/main.gleam
new file mode 100644
index 0000000..307b8d8
--- /dev/null
+++ b/gleam/src/read/main.gleam
@@ -0,0 +1,459 @@
+//// ocda-read, in Gleam: reads the abstraction, in any of the forms the
+//// library reads, and says what is in it.
+////
+//// ocda-read [--tree|--outline|--languages|--files|--report-unknown]
+//// <file>.ssp | <file>.ocda.db | <file>.zip [lang]
+////
+//// The modes the Gleam readers had (the summary, the outline, the unknown
+//// report, and the database's languages and files), over the library, so
+//// that each reads every form. Their output is the D ocda-read's, byte for
+//// byte: two readers in unrelated languages, neither written from the other
+//// line by line, agreeing is what says each has read the model and not its
+//// own idea of it. A `.sisupod` is the one form the Gleam library cannot
+//// open (its zstd frame; see `ocda/load_pod`), and it says so.
+
+import gleam/dict
+import gleam/int
+import gleam/io
+import gleam/list
+import gleam/result
+import gleam/string
+import ocda/inventory.{type Language}
+import ocda/load
+import ocda/model.{type Document, type Object}
+
+/// The format major this reader knows.
+const major_read = "2"
+
+type Options {
+ Options(
+ tree: Bool,
+ languages: Bool,
+ files: Bool,
+ report_unknown: Bool,
+ any_version: Bool,
+ lang: String,
+ positional: List(String),
+ )
+}
+
+/// Run with the arguments after the program name; returns the exit status.
+pub fn run(args: List(String)) -> Int {
+ case parse(args, Options(False, False, False, False, False, "", [])) {
+ Error(message) -> {
+ io.println_error("ocda-read: " <> message <> "; --help lists the options")
+ 2
+ }
+ Ok(Options(positional: [], ..)) -> {
+ help()
+ 0
+ }
+ Ok(options) -> read(options)
+ }
+}
+
+fn parse(args: List(String), o: Options) -> Result(Options, String) {
+ case args {
+ [] -> Ok(Options(..o, positional: list.reverse(o.positional)))
+ ["--help", ..] | ["-h", ..] -> Ok(Options(..o, positional: []))
+ ["--tree", ..rest] | ["--outline", ..rest] ->
+ parse(rest, Options(..o, tree: True))
+ ["--languages", ..rest] -> parse(rest, Options(..o, languages: True))
+ ["--files", ..rest] -> parse(rest, Options(..o, files: True))
+ ["--report-unknown", ..rest] ->
+ parse(rest, Options(..o, report_unknown: True))
+ ["--any-version", ..rest] -> parse(rest, Options(..o, any_version: True))
+ ["--lang", code, ..rest] -> parse(rest, Options(..o, lang: code))
+ [flag, ..rest] ->
+ case string.starts_with(flag, "--lang=") {
+ True -> parse(rest, Options(..o, lang: string.drop_start(flag, 7)))
+ False ->
+ case string.starts_with(flag, "-") {
+ True ->
+ case
+ list.contains(
+ [
+ "--html", "--llm-text", "--llm-jsonl", "--images", "--verify",
+ "--validate",
+ ],
+ flag,
+ )
+ {
+ True ->
+ Error(flag <> " is a mode of the D ocda-read, not of this one")
+ False -> Error("Unrecognized option " <> flag)
+ }
+ False ->
+ parse(rest, Options(..o, positional: [flag, ..o.positional]))
+ }
+ }
+ }
+}
+
+fn help() -> Nil {
+ io.println("ocda-read - reads the document abstraction, in any of its forms")
+ io.println("")
+ io.println(
+ "Usage: ocda-read [options] <file>.ssp|<file>.ocda.db|<file>.zip [lang]",
+ )
+ io.println("")
+ io.println(" --tree the headings, as a tree")
+ io.println(" --outline the same as --tree")
+ io.println(" --languages the languages the file holds")
+ io.println(" --files what a .ocda.db carries besides the abstraction")
+ io.println(" --report-unknown what was read and not understood")
+ io.println(" --lang=<code> the language to read (default: the first)")
+ io.println(" --any-version read a format major this reader does not know")
+}
+
+fn read(o: Options) -> Int {
+ let assert [path, ..more] = o.positional
+ let lang = case o.lang, more {
+ "", [given, ..] -> given
+ l, _ -> l
+ }
+ let lang_named = lang != ""
+ let loaded = {
+ use held <- result.try(inventory.languages(path))
+ let lang = case lang, held {
+ "", [first, ..] -> first.lang
+ l, _ -> l
+ }
+ use _ <- result.try(case
+ held != [] && !list.any(held, fn(l) { l.lang == lang })
+ {
+ True ->
+ Error(
+ "no language '"
+ <> lang
+ <> "' in "
+ <> path
+ <> "; it holds: "
+ <> string.join(list.map(held, fn(l) { l.lang }), " "),
+ )
+ False -> Ok(Nil)
+ })
+ use doc <- result.try(load.load(path, lang, o.any_version))
+ Ok(#(held, lang, doc))
+ }
+ case loaded {
+ Error(message) -> {
+ io.println_error("Error reading " <> path <> ": " <> message)
+ 2
+ }
+ Ok(#(held, lang, doc)) ->
+ case gate(path, doc, o.any_version) {
+ False -> 3
+ True -> {
+ summary(path, doc, held, lang, lang_named)
+ case o.languages {
+ True -> print_languages(held)
+ False -> Nil
+ }
+ case o.files {
+ True -> print_files(path)
+ False -> Nil
+ }
+ case o.tree {
+ True -> tree(doc)
+ False -> Nil
+ }
+ case o.report_unknown {
+ True -> report_unknown(doc)
+ False -> Nil
+ }
+ case o.tree || o.languages || o.files || o.report_unknown {
+ True -> Nil
+ False ->
+ io.println(
+ "\n(this is the summary: --help lists the modes, each adding its output)",
+ )
+ }
+ 0
+ }
+ }
+ }
+}
+
+/// The major part, and refuse what is not known.
+fn gate(path: String, doc: Document, any_version: Bool) -> Bool {
+ case doc.format_version, model.version_major(doc) == major_read, any_version {
+ "", _, _ -> {
+ io.println_error("warning: " <> path <> " declares no format version")
+ True
+ }
+ _, True, _ -> True
+ v, False, True -> {
+ io.println_error(
+ "warning: "
+ <> path
+ <> " is format v"
+ <> v
+ <> ", this reader knows v"
+ <> major_read
+ <> ".x; reading anyway",
+ )
+ True
+ }
+ v, False, False -> {
+ io.println_error(
+ "error: "
+ <> path
+ <> " is format v"
+ <> v
+ <> ", this reader knows v"
+ <> major_read
+ <> ".x",
+ )
+ io.println_error(
+ " a major version differs: a property may have changed meaning or gone away.",
+ )
+ io.println_error(" --any-version reads it regardless.")
+ False
+ }
+ }
+}
+
+const sections = [
+ "head", "toc", "body", "endnotes", "glossary", "bibliography", "bookindex",
+ "blurb", "tail",
+]
+
+/// The sections whose images are counted: every one but the tail, as the D
+/// reader counts them.
+const image_sections = [
+ "head", "toc", "body", "endnotes", "glossary", "bibliography", "bookindex",
+ "blurb",
+]
+
+fn summary(
+ path: String,
+ doc: Document,
+ held: List(Language),
+ lang: String,
+ lang_named: Bool,
+) -> Nil {
+ io.println("Read: " <> path)
+ case doc.format_version {
+ "" -> Nil
+ v -> io.println(" Format: v" <> v)
+ }
+ case dict.get(doc.source, "language") {
+ Error(_) -> Nil
+ Ok(language) -> {
+ let of = case dict.get(doc.source, "languages") {
+ Ok(ls) -> " of " <> ls
+ Error(_) -> ""
+ }
+ let digest = case dict.get(doc.source, "digest") {
+ Ok(d) -> ", markup digest " <> string.slice(d, 0, 16) <> "..."
+ Error(_) -> ""
+ }
+ io.println(" Source: " <> language <> of <> digest)
+ }
+ }
+ case list.length(held) > 1 && !lang_named {
+ True -> io.println(" (reading " <> lang <> "; --lang=<code> for another)")
+ False -> Nil
+ }
+ io.println(" Title: " <> model.meta(doc, "title.main"))
+ io.println(" Author: " <> model.meta(doc, "creator.author"))
+ io.println(
+ " Sections: "
+ <> string.join(
+ list.map(sections, fn(s) {
+ s <> "=" <> int.to_string(list.length(model.section(doc, s)))
+ }),
+ " ",
+ ),
+ )
+ let records =
+ list.fold(image_sections, 0, fn(n, s) {
+ list.fold(model.section(doc, s), n, fn(n, o) {
+ n + list.length(o.images)
+ })
+ })
+ case records {
+ 0 -> Nil
+ 1 -> io.println(" Images: 1 record")
+ n -> io.println(" Images: " <> int.to_string(n) <> " records")
+ }
+}
+
+fn print_languages(held: List(Language)) -> Nil {
+ io.println("")
+ io.println(" lang source ssp digest")
+ list.each(held, fn(l) {
+ let digest = case string.length(l.ssp_digest) > 16 {
+ True -> string.slice(l.ssp_digest, 0, 16) <> "..."
+ False -> l.ssp_digest
+ }
+ io.println(
+ " "
+ <> string.pad_end(l.lang, 7, " ")
+ <> " "
+ <> string.pad_end(l.source_filename, 31, " ")
+ <> " "
+ <> digest,
+ )
+ })
+}
+
+fn print_files(path: String) -> Nil {
+ io.println("")
+ case inventory.is_carrier(path) {
+ False ->
+ io.println(
+ " carried files: only a .ocda.db carries files; this is "
+ <> load.source_name(load.source_of(path)),
+ )
+ True ->
+ case inventory.carried(path) {
+ Error(message) -> io.println_error("error: " <> message)
+ Ok([]) -> io.println(" carries no files")
+ Ok(rows) -> {
+ io.println(" carried files:")
+ io.println(" role stored n original in the file")
+ list.each(rows, fn(r) {
+ io.println(
+ " "
+ <> string.pad_end(r.role, 9, " ")
+ <> " "
+ <> string.pad_end(r.compression, 7, " ")
+ <> " "
+ <> string.pad_end(r.count, 5, " ")
+ <> " "
+ <> string.pad_end(r.bytes, 13, " ")
+ <> " "
+ <> r.stored,
+ )
+ })
+ }
+ }
+ }
+}
+
+/// The headings, as a tree: from the document's root heading, down the
+/// children each heading lists, as the D reader walks it.
+fn tree(doc: Document) -> Nil {
+ io.println("Document: " <> model.meta(doc, "title.main"))
+ case model.meta(doc, "title.subtitle") {
+ "" -> Nil
+ sub -> io.println(" " <> sub)
+ }
+ io.println("Author: " <> model.meta(doc, "creator.author"))
+ io.println("")
+ let headings =
+ model.section(doc, "body")
+ |> list.filter(fn(o) { o.is_a == "heading" && o.ocn > 0 })
+ |> list.map(fn(o) { #(o.ocn, o) })
+ |> dict.from_list
+ case model.section(doc, "head") {
+ [root, ..] if root.is_a == "heading" -> {
+ io.println(heading_line(root, 0))
+ walk(root, headings, 1)
+ }
+ _ -> Nil
+ }
+}
+
+fn walk(parent: Object, headings: dict.Dict(Int, Object), depth: Int) -> Nil {
+ list.each(parent.children, fn(ocn) {
+ case dict.get(headings, ocn) {
+ Ok(child) -> {
+ io.println(heading_line(child, depth))
+ walk(child, headings, depth + 1)
+ }
+ Error(_) -> Nil
+ }
+ })
+}
+
+fn heading_line(o: Object, depth: Int) -> String {
+ let level = case o.heading_level {
+ "" -> "?"
+ l -> l
+ }
+ let ocn = case o.ocn > 0 {
+ True -> " [" <> int.to_string(o.ocn) <> "]"
+ False -> ""
+ }
+ string.repeat(" ", depth) <> level <> "~" <> ocn <> " " <> strip_markup(o.text)
+}
+
+/// The internal markers a heading's text may carry, removed for display, as
+/// the D reader removes them.
+fn strip_markup(text: String) -> String {
+ text
+ |> string.replace("⑆*", "")
+ |> string.replace("⑆!", "")
+ |> string.replace("⑆/", "")
+ |> string.replace("⑆_", "")
+ |> string.replace("⑆^", "")
+ |> string.replace("⑆,", "")
+ |> string.replace("┨", "")
+ |> string.replace("┣", "")
+ |> string.replace("┙", " ")
+ |> string.replace("┘", "\n")
+}
+
+/// Say what was read and not understood.
+fn report_unknown(doc: Document) -> Nil {
+ let u = doc.unknown
+ case model.any_unknown(u) {
+ False ->
+ io.println(
+ "\nread and understood: every property, block and line was known",
+ )
+ True -> {
+ io.println("\nread and NOT understood:")
+ section_of_unknown(u.property, " properties ignored (", fn(name, n) {
+ " ."
+ <> string.pad_end(name, 26, " ")
+ <> " "
+ <> string.pad_start(int.to_string(n), 8, " ")
+ })
+ section_of_unknown(u.block, " blocks ignored (", fn(name, n) {
+ " @"
+ <> string.pad_end(name, 26, " ")
+ <> " "
+ <> string.pad_start(int.to_string(n), 8, " ")
+ })
+ case dict.size(u.line) {
+ 0 -> Nil
+ _ -> {
+ io.println(" lines of no known shape, by first character:")
+ dict.to_list(u.line)
+ |> list.sort(fn(a, b) { string.compare(a.0, b.0) })
+ |> list.each(fn(e) {
+ io.println(
+ " "
+ <> string.pad_end("\"" <> e.0 <> "\"", 27, " ")
+ <> " "
+ <> string.pad_start(int.to_string(e.1), 8, " "),
+ )
+ })
+ }
+ }
+ io.println(
+ " (ignoring these is conforming within a major version; the count is so that it is not silent)",
+ )
+ }
+ }
+}
+
+fn section_of_unknown(
+ counts: dict.Dict(String, Int),
+ heading: String,
+ line: fn(String, Int) -> String,
+) -> Nil {
+ case dict.size(counts) {
+ 0 -> Nil
+ n -> {
+ io.println(heading <> int.to_string(n) <> " distinct):")
+ dict.to_list(counts)
+ |> list.sort(fn(a, b) { string.compare(a.0, b.0) })
+ |> list.each(fn(e) { io.println(line(e.0, e.1)) })
+ }
+ }
+}
diff --git a/gleam/test/ocda_lib_test.gleam b/gleam/test/ocda_lib_test.gleam
index 02fe463..c0e10e5 100644
--- a/gleam/test/ocda_lib_test.gleam
+++ b/gleam/test/ocda_lib_test.gleam
@@ -36,7 +36,7 @@ const minimal = "% SiSU Document Abstraction v2.0
.part: body
.parent: 1
| First line
-|
+|
| Third line, after a blank one
}