summaryrefslogtreecommitdiffhomepage
diff options
context:
space:
mode:
authorRalph Amissah <ralph.amissah@gmail.com>2026-10-12 20:24:11 -0400
committerRalph Amissah <ralph.amissah@gmail.com>2026-10-12 20:24:11 -0400
commit4e4f2f8c2b2c043077390cb4ccf7ce97ffe4ac6b (patch)
treebfafd6fb9cfa975db49fbcbcf7e3b23a4f13e0bc
parentmodel 2.1: ocn_range, the verses a poem holds (diff)
format 2.1: ocda-read, the readers folded in
ocda-read, one reader where there were two. The .ssp reader and the .ocda.db reader each had their own parser; this one is built on the library, so every mode reads a .ssp, a .ocda.db and a pod, and says the same thing of each. Its output modules are the .ssp reader's, copied (the reader repositories are kept, frozen) and moved onto the library's model; --languages and --files come from the database reader. ocda-read [--tree|--html|--llm-text|--llm-jsonl|--images| --verify --image-dir=<d>|--languages|--files| --report-unknown|--validate] <file> [lang] In D every mode of both readers; in Gleam the modes the Gleam readers had (the summary, --tree, --languages, --files, --report-unknown), whose output is the D reader's, byte for byte. In each language it lives in src/read/ and is not part of the library: dub configurations keep it out of a dependant's build in D, and Gleam compiles all of src/ but a dependant need not use it. Checked over the markup document samples: against the .ssp reader on a .ssp, byte for byte after the first line in every D mode but the summary, whose closing hint differs; against the database reader on a .ocda.db, --files and --languages (its id column left out); the same output from the .ssp, the .ocda.db and the .sisupod; and D == Gleam. The database reader's --outline is not reproduced: it ordered headings within their own section, so that other sections' headings fell among the body's. --outline here is --tree. What writing it found in the library: the database loader could not say what it had not understood. It asked for columns by name and so never saw the rest, and reported that it had understood everything. It now counts them on the conditions under which the .ssp writes the same property, and --report-unknown is the same from every form. ocda.inventory answers two questions about the file rather than the document: its languages, in the manifest's order, each with the markup it was made from and the sha256 of its .ssp, for every form alike; and what a .ocda.db carries besides the abstraction. In Gleam the sha256 comes from OTP's crypto through a four line Erlang FFI. The flake's default is all three D programs in one result: built separately into the same link, each replaced the one before. (assisted by Claude-Code)
-rw-r--r--README.md42
-rw-r--r--dlang/README.md45
-rw-r--r--dlang/dub.json16
-rw-r--r--dlang/flake.lock17
-rw-r--r--dlang/flake.nix56
-rw-r--r--dlang/src/ocda/inventory.d117
-rw-r--r--dlang/src/ocda/load_db.d33
-rw-r--r--dlang/src/ocda/load_pod.d4
-rw-r--r--dlang/src/read/html.d388
-rw-r--r--dlang/src/read/images.d114
-rw-r--r--dlang/src/read/llm_jsonl.d121
-rw-r--r--dlang/src/read/llm_text.d182
-rw-r--r--dlang/src/read/main.d228
-rw-r--r--dlang/src/read/report_unknown.d41
-rw-r--r--dlang/src/read/tree.d69
-rw-r--r--dlang/src/read/validate.d265
-rw-r--r--dlang/src/read/verify.d214
-rw-r--r--gleam/README.md44
-rw-r--r--gleam/flake.lock17
-rw-r--r--gleam/flake.nix8
-rw-r--r--gleam/src/ocda/inventory.gleam138
-rw-r--r--gleam/src/ocda/load_db.gleam108
-rw-r--r--gleam/src/ocda/load_pod.gleam6
-rw-r--r--gleam/src/ocda/load_ssp.gleam2
-rw-r--r--gleam/src/ocda_ffi.erl7
-rw-r--r--gleam/src/ocda_lib.gleam14
-rw-r--r--gleam/src/read/main.gleam459
-rw-r--r--gleam/test/ocda_lib_test.gleam2
28 files changed, 2674 insertions, 83 deletions
diff --git a/README.md b/README.md
index 5e328c0..ad3a9a9 100644
--- a/README.md
+++ b/README.md
@@ -16,8 +16,10 @@ at the direction of and reviewed by Ralph Amissah.
## What it is for
-The readers in `sisudoc-ssp-reader` and `sisudoc-ocda-db-reader` each read
-one serialisation and print. A **writer** needs neither: it needs the document.
+The first programs written from the specification were two readers,
+`sisudoc-ssp-reader` and `sisudoc-ocda-db-reader`, each reading one
+serialisation and printing. A **writer** needs neither: it needs the
+document.
```
.ssp ─┐
@@ -29,8 +31,38 @@ So this library holds the model and both loaders, and **a writer cannot
tell which serialisation it was handed** - exactly as spine's own writers
cannot, since spine 0.25.0.
-That is the whole of the idea. The readers become thin programs over this,
-and the writers are functions from the model to bytes.
+That is the whole of the idea. The writers are functions from the model to
+bytes, and the reader is a program over the model.
+
+## The reader
+
+```sh
+ocda-read [--tree|--html|--llm-text|--llm-jsonl|--images|--verify --image-dir=<d>|
+ --languages|--files|--report-unknown|--validate]
+ <file>.ssp | <file>.ocda.db | <file>.sisupod [lang]
+```
+
+One reader, in each language, where the two earlier readers had a parser
+each. Built on this library, **every mode reads every form, and every mode
+that describes the document says the same thing of its `.ssp`, `.ocda.db`
+and `.sisupod`**, 36 of 36 document-languages. Two modes describe the file
+instead, and differ as the files do: `--languages` lists the one language a
+`.ssp` holds and every language of the other two, and `--files` finds
+carried files only in a `.ocda.db`. The D reader has every mode of both
+old readers; the Gleam one the modes the Gleam readers had (the summary,
+`--tree`, `--languages`, `--files`, `--report-unknown`), and in those **its
+output is the D one's, byte for byte**, 36 of 36.
+
+It is in each language's `src/read/`, and not part of the library: a
+program using the library does not build it (in D, by dub configuration;
+in Gleam a dependant compiles it and does not use it, Gleam building
+everything in `src/`).
+
+Writing it found one defect in the library: **the database loader could
+not say what it had not understood**. It asks for columns by name, and so
+never saw the ones it did not ask for, and reported that it had understood
+everything. It now counts them on the conditions under which the `.ssp`
+writes the same property, and the report is the same from either form.
## The checks that matter
@@ -72,7 +104,7 @@ Nothing else had caught it. The two `.ssp` readers had been compared on
*section counts*, which this does not change. It took loading the same
document two ways and comparing the objects.
-Fixed here and in `sisudoc-ssp-reader/dlang`, which carried the same code.
+Fixed here and in the `.ssp` reader, which carried the same code.
### the whole-model dump: four more
diff --git a/dlang/README.md b/dlang/README.md
index 124e4d6..65d762e 100644
--- a/dlang/README.md
+++ b/dlang/README.md
@@ -4,14 +4,16 @@ The abstraction's model and both its loaders, as a dub library.
## Modules
-| | |
-|-----------------|---------------------------------------------------------------|
-| `ocda.model` | the document, the object, the eight sections. Imports nothing |
-| `ocda.load_ssp` | text -> model |
-| `ocda.load_db` | sqlite -> model |
-| `ocda.load` | what a path is, and loading it |
-| `ocda.dump` | the model written out canonically, as an oracle |
-| `ocda.sqlite` | the sqlite3 C API, declared rather than depended upon |
+| | |
+|------------------|------------------------------------------------------------------------------------------------------------|
+| `ocda.model` | the document, the object, the eight sections. Imports nothing |
+| `ocda.load_ssp` | text -> model |
+| `ocda.load_db` | sqlite -> model |
+| `ocda.load` | what a path is, and loading it |
+| `ocda.inventory` | what a file holds beyond the one document: its languages (every form alike), and what a `.ocda.db` carries |
+| `ocda.dump` | the model written out canonically, as an oracle |
+| `ocda.sqlite` | the sqlite3 C API, declared rather than depended upon |
+| `read.*` | `ocda-read`, the reader: in `src/read/`, and *not* part of the library (below) |
`ocda.model` importing nothing is the property that matters: a writer
depends on the model and on neither loader.
@@ -31,6 +33,28 @@ foreach (obj; doc.body) { ... }
Link `sqlite3`; the library declares the dozen functions it needs rather
than taking a binding from the registry, so there is nothing to fetch.
+## The reader
+
+```sh
+nix build .#ocda-read && ./result/bin/ocda-read [--tree|--html|--llm-text|--llm-jsonl|
+ --images|--verify --image-dir=<d>|--languages|--files|--report-unknown|--validate]
+ <file>.ssp|<file>.ocda.db|<file>.sisupod [lang]
+```
+
+The library's reader, one program where `sisudoc-ssp-reader` and
+`sisudoc-ocda-db-reader` were two. Its output modules are the `.ssp`
+reader's, moved onto the library's model, so every mode reads every form
+and, where it describes the document, **says the same thing of its `.ssp`,
+`.ocda.db` and `.sisupod`**. `--languages` lists the one language a `.ssp`
+holds and every language of the other two; `--files` answers for a
+`.ocda.db`, the one form that carries files; `--validate` checks a `.ssp`
+as text and refuses anything else.
+
+It lives in `src/read/` beside the library, and dub keeps it out of the
+library: the configuration `library` excludes `src/read/*`, the
+configuration `ocda-read` builds it. A program using the library by dub, or
+by naming `src/ocda/*.d` as the writer's flake does, never compiles it.
+
## The two checks
```sh
@@ -55,8 +79,11 @@ repository README and in the history.
## Build
```sh
+nix build # all three programs, in one result/bin
+nix build .#ocda-read # the reader
nix build .#ocda-agree # the agreement check
nix build .#ocda-dump # the canonical dump
-nix develop # ldc, dub, dtools, sqlite
+nix develop # ldc, dub, dtools, sqlite, zstd
dub build # the library
+dub build --config=ocda-read # the reader
```
diff --git a/dlang/dub.json b/dlang/dub.json
index 4f59aec..861c2cc 100644
--- a/dlang/dub.json
+++ b/dlang/dub.json
@@ -6,8 +6,20 @@
"description": "the object-centric document abstraction as a library: the model, and loaders for its two serialisations",
"homepage": "https://sisudoc.org",
"license": "AGPL-3.0+",
- "targetType": "library",
"sourcePaths": [ "./src" ],
"importPaths": [ "./src" ],
- "libs": [ "sqlite3", "zstd" ]
+ "libs": [ "sqlite3", "zstd" ],
+ "configurations": [
+ {
+ "name": "library",
+ "targetType": "library",
+ "excludedSourceFiles": [ "src/read/*" ]
+ },
+ {
+ "name": "ocda-read",
+ "targetType": "executable",
+ "targetName": "ocda-read",
+ "mainSourceFile": "src/read/main.d"
+ }
+ ]
}
diff --git a/dlang/flake.lock b/dlang/flake.lock
index 5f236ab..12c4c3b 100644
--- a/dlang/flake.lock
+++ b/dlang/flake.lock
@@ -2,18 +2,15 @@
"nodes": {
"nixpkgs": {
"locked": {
- "lastModified": 1790185690,
- "narHash": "sha256-xJ+X4hBtOcAFGBOe5nAMyMUeF9foJBmIOu3NjBqBycU=",
- "owner": "NixOS",
- "repo": "nixpkgs",
- "rev": "4975466d324710c576dc11ad614684e6bd8cad8e",
- "type": "github"
+ "lastModified": 1790652569,
+ "narHash": "sha256-k8Fu4c9Z+4Nh7mUr0cfw++ITQiyEhlWxoJBOkI3tOcQ=",
+ "rev": "b6c8664de9b6cc07fe5666a29f91884ba81197c4",
+ "type": "tarball",
+ "url": "https://releases.nixos.org/nixpkgs/nixpkgs-26.11pre1082290.b6c8664de9b6/nixexprs.tar.zst"
},
"original": {
- "owner": "NixOS",
- "ref": "nixos-unstable",
- "repo": "nixpkgs",
- "type": "github"
+ "type": "tarball",
+ "url": "https://channels.nixos.org/nixpkgs-unstable/nixexprs.tar.zst"
}
},
"root": {
diff --git a/dlang/flake.nix b/dlang/flake.nix
index 3aa9a60..2241af6 100644
--- a/dlang/flake.nix
+++ b/dlang/flake.nix
@@ -1,21 +1,23 @@
{
description = "ocda-lib: the object-centric document abstraction as a D library";
-
- inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
-
+ inputs.nixpkgs.url = "https://channels.nixos.org/nixpkgs-unstable/nixexprs.tar.zst";
outputs = { self, nixpkgs }:
let
system = "x86_64-linux"; # adjust for aarch64-linux / darwin
pkgs = import nixpkgs { inherit system; };
-
- /* A library, so there is nothing to install as a binary. What is
- built here are the two programs that put the library to the
- question: the agreement check, which loads a document's .ssp and
- its .ocda.db and compares the two models, and the dump, which
+ /* A library, so the library itself is not a build product: a program
+ that uses it compiles its modules alongside its own, as the writer's
+ flake does, or takes it as a dub dependency. What is built here are
+ programs over it: the reader, and the two that put the library to
+ the question - the agreement check, which loads a document's .ssp
+ and its .ocda.db and compares the two models, and the dump, which
writes a loaded model out canonically so that an implementation in
another language can be diffed against this one.
+ .
+ `mains` is what is compiled besides the library: the checks are one
+ file each in test/, the reader is src/read/.
*/
- mkTool = { tool, binary, what, compiler, compilerBin }:
+ mkTool = { tool, binary, what, mains, compiler, compilerBin }:
pkgs.stdenv.mkDerivation {
pname = "ocda-lib-${tool}";
version = "2.0.0";
@@ -25,7 +27,7 @@
buildPhase = ''
runHook preBuild
export HOME=$TMPDIR
- ${compilerBin} -I=src -of=${tool} src/ocda/*.d test/${tool}.d -L-lsqlite3 -L-lzstd
+ ${compilerBin} -I=src -of=${tool} src/ocda/*.d ${mains} -L-lsqlite3 -L-lzstd
runHook postBuild
'';
installPhase = ''
@@ -39,21 +41,31 @@
mainProgram = binary;
};
};
-
ldc = { compiler = pkgs.ldc; compilerBin = "ldmd2"; };
+ ocda-read = mkTool (ldc // {
+ tool = "read"; binary = "ocda-read"; mains = "src/read/*.d";
+ what = "reads the abstraction in any of its forms and says what is in it";
+ });
+ ocda-agree = mkTool (ldc // {
+ tool = "agree"; binary = "ocda-agree"; mains = "test/agree.d";
+ what = "checks that the .ssp and .ocda.db loaders give one model";
+ });
+ ocda-dump = mkTool (ldc // {
+ tool = "dump"; binary = "ocda-dump"; mains = "test/dump.d";
+ what = "writes a loaded abstraction out canonically, as an oracle";
+ });
in {
packages.${system} = {
- default = self.packages.${system}.ocda-agree;
-
- ocda-agree = mkTool (ldc // {
- tool = "agree"; binary = "ocda-agree";
- what = "checks that the .ssp and .ocda.db loaders give one model";
- });
-
- ocda-dump = mkTool (ldc // {
- tool = "dump"; binary = "ocda-dump";
- what = "writes a loaded abstraction out canonically, as an oracle";
- });
+ /* All three in one bin/, so that one build leaves every program in
+ one result link. Built separately into the same link, each build
+ replaced the one before, and a script looking for ocda-dump beside
+ ocda-agree found nothing.
+ */
+ default = pkgs.symlinkJoin {
+ name = "ocda-lib-programs-2.0.0";
+ paths = [ ocda-read ocda-agree ocda-dump ];
+ };
+ inherit ocda-read ocda-agree ocda-dump;
};
devShells.${system}.default = pkgs.mkShell {
diff --git a/dlang/src/ocda/inventory.d b/dlang/src/ocda/inventory.d
new file mode 100644
index 0000000..dba2156
--- /dev/null
+++ b/dlang/src/ocda/inventory.d
@@ -0,0 +1,117 @@
+module ocda.inventory;
+
+/+ ↓ what a source holds, beyond the one document the model is loaded from.
+ .
+ The model is one language of one document. Two questions a reader asks
+ are about the file instead: which languages it holds, and, for a
+ database, what else it carries. They are here rather than in a reader
+ because the answer depends on the input form, which is the library's
+ to know, and so that a reader asking them does not learn which form it
+ was handed.
+ .
+ languages() answers for every form, and answers the same: the
+ languages in the order the manifest gave them (which every .ssp
+ records, in @source languages), each with the markup file it was made
+ from and the sha256 of its .ssp. The database stores that digest; for a
+ .ssp and a pod it is taken from the bytes, which are the same bytes.
+ .
+ carried() is the database's: a .ocda.db holds the markup, images,
+ configuration and catalogues it was built from, a .ssp holds none of
+ them, and a pod's are its archive entries rather than anything the
+ abstraction says. So it answers for a database and says so otherwise.
++/
+
+import std.algorithm : endsWith, countUntil;
+import std.array : array, split;
+import std.conv : to;
+import std.digest : toHexString, LetterCase;
+import std.digest.sha : sha256Of;
+import std.file : read;
+import std.string : strip;
+import ocda.model;
+import ocda.load;
+import ocda.load_ssp;
+import ocda.load_db;
+import ocda.load_pod;
+import ocda.sqlite;
+import ocda.zip;
+
+struct OcdaLanguage {
+ string lang;
+ string source_filename; // the markup file the abstraction was made from
+ string ssp_digest; // sha256 of that language's .ssp, upper case hex
+}
+
+struct OcdaCarried {
+ string role; // source, image, conf, manifest, tools
+ string compression; // how it is stored: none, zstd
+ string count;
+ string bytes; // the files' own size, whatever the storage
+ string stored; // what they take in the database
+}
+
+OcdaLanguage[] languages(string path) {
+ final switch (sourceOf(path)) {
+ case Source.ocda_db:
+ auto db = Db.open(path);
+ scope(exit) { db.close(); }
+ OcdaLanguage[] out_;
+ foreach (l; dbLanguages(db)) {
+ out_ ~= OcdaLanguage(l.lang, l.source_filename, l.ssp_digest);
+ }
+ return out_;
+ case Source.ssp:
+ auto bytes = cast(ubyte[]) read(path);
+ auto doc = parseSsp(cast(string) bytes.idup);
+ return [ OcdaLanguage(sourceValue(doc, "language"), doc.source_filename, hexOf(bytes)) ];
+ case Source.pod:
+ auto archive = archiveBytes(path);
+ OcdaLanguage[string] found;
+ string[] order;
+ foreach (e; zipFindSuffix(zipEntries(archive), ".ssp")) {
+ auto bytes = zipRead(archive, e);
+ auto doc = parseSsp(cast(string) bytes.idup);
+ string lang = langOfSspName(e.name);
+ found[lang] = OcdaLanguage(lang, doc.source_filename, hexOf(bytes));
+ if (order.length == 0) {
+ order = sourceValue(doc, "languages").split;
+ }
+ }
+ /+ ↓ the manifest's order, and anything it did not name after it +/
+ OcdaLanguage[] out_;
+ foreach (l; order) { if (auto p = l in found) { out_ ~= *p; } }
+ foreach (l, v; found) { if (order.countUntil(l) < 0) { out_ ~= v; } }
+ return out_;
+ case Source.unknown:
+ throw new OcdaLoadError(path ~ ": " ~ sourceName(Source.unknown));
+ }
+}
+
+/+ ↓ what a database carries, by role and storage; empty for any other form,
+ which isCarrier() says in advance +/
+bool isCarrier(string path) {
+ return sourceOf(path) == Source.ocda_db;
+}
+
+OcdaCarried[] carried(string path) {
+ if (!isCarrier(path)) { return []; }
+ auto db = Db.open(path);
+ scope(exit) { db.close(); }
+ if (!db.has("files")) { return []; }
+ OcdaCarried[] out_;
+ foreach (r; db.rows(
+ "SELECT role, compression, count(*), sum(bytes), sum(length(data))
+ FROM files GROUP BY role, compression ORDER BY role")) {
+ out_ ~= OcdaCarried(r[0], r[1], r[2], r[3], r[4]);
+ }
+ return out_;
+}
+
+private string sourceValue(ref OcdaDocument doc, string key) {
+ if (auto p = key in doc.source) { return *p; }
+ return "";
+}
+
+private string hexOf(const(ubyte)[] bytes) {
+ return toHexString!(LetterCase.upper)(sha256Of(bytes)).idup;
+}
diff --git a/dlang/src/ocda/load_db.d b/dlang/src/ocda/load_db.d
index eb3d71b..2f59437 100644
--- a/dlang/src/ocda/load_db.d
+++ b/dlang/src/ocda/load_db.d
@@ -242,9 +242,42 @@ OcdaDocument loadDbFile(string path, string lang = "", bool any_version = false)
auto section = doc.sectionByName(r[0]);
if (section !is null) { *section ~= obj; }
}
+ countUnread(db, chosen.doc_id, doc);
return doc;
}
+/+ ↓ what the database holds and the model does not.
+ .
+ The text loader meets every property line and counts those it does not
+ keep, so a reader can say what was read and not understood. This
+ loader asks for columns by name, and would never see the ones it did
+ not ask for: it would report that it had understood everything, which
+ is the silence the count exists to prevent. So it counts them, on the
+ conditions under which the .ssp writes the same property (specification
+ section 6.3), and the report is the same from either form: checked,
+ 36 of 36.
++/
+private void countUnread(ref Db db, int doc_id, ref OcdaDocument doc) {
+ string id = doc_id.to!string;
+ void count(string property, string where) {
+ auto n = db.one("SELECT count(*) FROM objects WHERE doc_id = " ~ id
+ ~ " AND " ~ where).to!size_t;
+ if (n > 0) { doc.unknown.property[property] = n; }
+ }
+ count("ancestors_collapsed", "coalesce(ancestors_collapsed,'') != ''");
+ count("dom_status", "coalesce(dom_status,'') != ''");
+ count("dom_status_collapsed", "coalesce(dom_status_collapsed,'') != ''");
+ count("heading_lev_collapsed", "heading_lev_collapsed IS NOT NULL");
+ count("parent_lev", "coalesce(parent_lev,0) != 0");
+ count("segment_lv4_is", "coalesce(segment_lv4_is,'') != ''");
+ count("attrib", "coalesce(attrib,'') != ''");
+ if (db.has("object_subtoc")) {
+ auto n = db.one("SELECT count(*) FROM object_subtoc s JOIN objects o"
+ ~ " ON o.id = s.object_id WHERE o.doc_id = " ~ id).to!size_t;
+ if (n > 0) { doc.unknown.property["lev4_subtoc"] = n; }
+ }
+}
+
/+ ↓ model section 4: the eight levels and their indices +/
private string levelName(int index) {
switch (index) {
diff --git a/dlang/src/ocda/load_pod.d b/dlang/src/ocda/load_pod.d
index 75993be..8f59fe3 100644
--- a/dlang/src/ocda/load_pod.d
+++ b/dlang/src/ocda/load_pod.d
@@ -92,7 +92,7 @@ OcdaDocument loadPodFile(string path, string lang = "") {
The bytes decide, not the name: a .sisupod is a zstd frame and a .zip
is not, and a file misnamed either way still reads.
+/
-private ubyte[] archiveBytes(string path) {
+package ubyte[] archiveBytes(string path) {
auto bytes = cast(ubyte[]) read(path);
return zstdIsFrame(bytes)
? zstdDecompress(bytes, max_pod_size)
@@ -100,7 +100,7 @@ private ubyte[] archiveBytes(string path) {
}
/+ ↓ <uid>.<lang>.ssp -> <lang> +/
-private string langOfSspName(string name) {
+package string langOfSspName(string name) {
if (!name.endsWith(".ssp")) { return ""; }
string stem = name[0 .. $ - 4];
auto dot = stem.lastIndexOf('.');
diff --git a/dlang/src/read/html.d b/dlang/src/read/html.d
new file mode 100644
index 0000000..cd679e4
--- /dev/null
+++ b/dlang/src/read/html.d
@@ -0,0 +1,388 @@
+module read.html;
+
+import std.array : replace, split, join, appender;
+import std.conv : to;
+import std.stdio : writeln, File;
+import std.string : strip, indexOf, startsWith;
+import ocda.model;
+
+void outputHtmlScroll(OcdaDocument doc, string output_file) {
+ auto buf = appender!(string[]);
+
+ buf ~= htmlHead(doc);
+
+ /+ ↓ title +/
+ if (doc.head.length > 0) {
+ auto title_obj = doc.head[0];
+ buf ~= `<div class="title">`;
+ buf ~= `<h1>` ~ htmlEscape(doc.meta("title.main")) ~ `</h1>`;
+ if (doc.meta("title.subtitle").length > 0)
+ buf ~= `<h2 class="subtitle">` ~ htmlEscape(doc.meta("title.subtitle")) ~ `</h2>`;
+ buf ~= `<p class="author">` ~ htmlEscape(doc.meta("creator.author")) ~ `</p>`;
+ buf ~= `</div>`;
+ }
+
+ /+ ↓ table of contents +/
+ if (doc.toc.length > 0) {
+ buf ~= `<div class="toc">`;
+ buf ~= `<h2>Table of Contents</h2>`;
+ foreach (obj; doc.toc) {
+ if (obj.is_a == "toc") {
+ string text = convertInlineMarkup(obj.text);
+ int indent = obj.indent_base;
+ string cls = (indent > 4) ? "toc-sub" : "toc-entry";
+ buf ~= `<p class="` ~ cls ~ `">` ~ text ~ `</p>`;
+ }
+ }
+ buf ~= `</div>`;
+ buf ~= `<hr/>`;
+ }
+
+ /+ ↓ body +/
+ buf ~= `<div class="body">`;
+ foreach (obj; doc.body) {
+ buf ~= renderObject(obj);
+ }
+ buf ~= `</div>`;
+
+ /+ ↓ endnotes +/
+ bool has_endnotes = false;
+ foreach (obj; doc.endnotes) {
+ if (obj.is_a == "endnote") { has_endnotes = true; break; }
+ }
+ if (has_endnotes) {
+ buf ~= `<div class="endnotes">`;
+ buf ~= `<hr/>`;
+ buf ~= `<h2>Endnotes</h2>`;
+ foreach (obj; doc.endnotes) {
+ if (obj.is_a == "endnote") {
+ string text = convertInlineMarkup(obj.text);
+ string parent_ref = (obj.parent_ocn > 0)
+ ? ` <a href="#o` ~ obj.parent_ocn.to!string ~ `">[` ~ obj.parent_ocn.to!string ~ `]</a>`
+ : "";
+ buf ~= `<p class="endnote">` ~ text ~ parent_ref ~ `</p>`;
+ }
+ }
+ buf ~= `</div>`;
+ }
+
+ /+ ↓ glossary +/
+ if (doc.glossary.length > 0) {
+ buf ~= `<div class="glossary">`;
+ buf ~= `<hr/>`;
+ buf ~= `<h2>Glossary</h2>`;
+ foreach (obj; doc.glossary) {
+ string text = convertInlineMarkup(obj.text);
+ buf ~= `<p class="glossary-entry">` ~ text ~ `</p>`;
+ }
+ buf ~= `</div>`;
+ }
+
+ /+ ↓ bookindex +/
+ bool has_bookindex = false;
+ foreach (obj; doc.bookindex) {
+ if (obj.is_a == "bookindex") { has_bookindex = true; break; }
+ }
+ if (has_bookindex) {
+ buf ~= `<div class="bookindex">`;
+ buf ~= `<hr/>`;
+ buf ~= `<h2>Book Index</h2>`;
+ foreach (obj; doc.bookindex) {
+ if (obj.is_a == "bookindex") {
+ string text = convertInlineMarkup(obj.text);
+ buf ~= `<p class="index-entry">` ~ text ~ `</p>`;
+ }
+ }
+ buf ~= `</div>`;
+ }
+
+ buf ~= htmlTail();
+
+ /+ ↓ write output +/
+ string[] output = buf.data;
+ if (output_file.length > 0) {
+ auto f = File(output_file, "w");
+ foreach (line; output) f.writeln(line);
+ } else {
+ foreach (line; output) writeln(line);
+ }
+}
+
+private string[] renderObject(OcdaObject obj) {
+ /+ ↓ markup commentary is carried by the abstraction and is not output.
+ Rendering it would put the author notes into the document, which
+ spine itself does not do
+ +/
+ if (!obj.renders) { return []; }
+ string[] lines;
+ string text = convertInlineMarkup(obj.text);
+ string ocn_anchor = (obj.ocn > 0 && !obj.ocn_off)
+ ? ` id="o` ~ obj.ocn.to!string ~ `"`
+ : "";
+ string ocn_display = (obj.ocn > 0 && !obj.ocn_off)
+ ? `<span class="ocn">` ~ obj.ocn.to!string ~ `</span>`
+ : "";
+
+ switch (obj.is_a) {
+ case "heading":
+ string tag = headingTag(obj.heading_level);
+ string seg_anchor = (obj.segment.length > 0)
+ ? ` id="` ~ htmlEscape(obj.segment) ~ `"`
+ : ocn_anchor;
+ lines ~= `<` ~ tag ~ seg_anchor ~ `>` ~ text ~ `</` ~ tag ~ `>`;
+ break;
+ case "para":
+ string indent_cls = "";
+ if (obj.indent_base > 0) indent_cls = ` class="indent-` ~ obj.indent_base.to!string ~ `"`;
+ else if (obj.bullet) indent_cls = ` class="bullet"`;
+ lines ~= `<p` ~ ocn_anchor ~ indent_cls ~ `>` ~ text ~ ocn_display ~ `</p>`;
+ break;
+ case "code":
+ lines ~= `<pre` ~ ocn_anchor ~ `><code>` ~ htmlEscape(obj.text) ~ `</code></pre>`;
+ break;
+ case "table":
+ lines ~= renderTable(obj, ocn_anchor);
+ break;
+ case "group":
+ lines ~= `<div` ~ ocn_anchor ~ ` class="group">` ~ text ~ `</div>`;
+ break;
+ case "block":
+ lines ~= `<pre` ~ ocn_anchor ~ ` class="block">` ~ htmlEscape(obj.text) ~ `</pre>`;
+ break;
+ case "verse":
+ lines ~= `<p` ~ ocn_anchor ~ ` class="verse">` ~ text.replace("\n", "<br/>\n") ~ ocn_display ~ `</p>`;
+ break;
+ case "quote":
+ lines ~= `<blockquote` ~ ocn_anchor ~ `>` ~ text ~ ocn_display ~ `</blockquote>`;
+ break;
+ default:
+ if (text.length > 0)
+ lines ~= `<p` ~ ocn_anchor ~ `>` ~ text ~ ocn_display ~ `</p>`;
+ break;
+ }
+ return lines;
+}
+
+private string headingTag(string level) {
+ switch (level) {
+ case "A": return "h1";
+ case "B": return "h2";
+ case "C": return "h3";
+ case "D": return "h4";
+ case "1": return "h3";
+ case "2": return "h4";
+ case "3": return "h5";
+ case "4": return "h6";
+ default: return "h4";
+ }
+}
+
+private string renderTable(OcdaObject obj, string ocn_anchor) {
+ string[] lines;
+ lines ~= `<table` ~ ocn_anchor ~ `>`;
+ /+ ↓ table content uses ┊ as cell separator, rows separated by newlines +/
+ auto rows = obj.text.split("\n");
+ bool first_row = obj.table_header;
+ foreach (row; rows) {
+ if (row.strip.length == 0) continue;
+ string tag = first_row ? "th" : "td";
+ lines ~= `<tr>`;
+ foreach (cell; row.split("┊")) {
+ lines ~= `<` ~ tag ~ `>` ~ convertInlineMarkup(cell.strip) ~ `</` ~ tag ~ `>`;
+ }
+ lines ~= `</tr>`;
+ first_row = false;
+ }
+ lines ~= `</table>`;
+ return lines.join("\n");
+}
+
+/+ ↓ convert spine internal Unicode markup to HTML +/
+string convertInlineMarkup(string text) {
+ if (text.length == 0) return "";
+ string result = htmlEscape(text);
+
+ /+ ↓ font face markers: ⑆*bold⑆*, ⑆/italic⑆/, etc.
+ Pattern: ⑆X┨content┣X or ⑆Xcontent⑆X
+ The ┨┣ form is used within heading content +/
+
+ /+ ↓ bold +/
+ result = replaceInlineMarker(result, "⑆!", "!", "<b>", "</b>");
+ result = replaceInlineMarkerBracketed(result, "⑆!", "┨", "┣!", "<b>", "</b>");
+
+ /+ ↓ emphasis +/
+ result = replaceInlineMarker(result, "⑆*", "*", "<em>", "</em>");
+ result = replaceInlineMarkerBracketed(result, "⑆*", "┨", "┣*", "<em>", "</em>");
+
+ /+ ↓ italic +/
+ result = replaceInlineMarker(result, "⑆/", "/", "<i>", "</i>");
+ result = replaceInlineMarkerBracketed(result, "⑆/", "┨", "┣/", "<i>", "</i>");
+
+ /+ ↓ underscore +/
+ result = replaceInlineMarker(result, "⑆_", "_", "<u>", "</u>");
+ result = replaceInlineMarkerBracketed(result, "⑆_", "┨", "┣_", "<u>", "</u>");
+
+ /+ ↓ superscript +/
+ result = replaceInlineMarker(result, "⑆^", "^", "<sup>", "</sup>");
+ result = replaceInlineMarkerBracketed(result, "⑆^", "┨", "┣^", "<sup>", "</sup>");
+
+ /+ ↓ subscript +/
+ result = replaceInlineMarker(result, "⑆,", ",", "<sub>", "</sub>");
+ result = replaceInlineMarkerBracketed(result, "⑆,", "┨", "┣,", "<sub>", "</sub>");
+
+ /+ ↓ monospace +/
+ result = replaceInlineMarker(result, "⑆■", "■", "<code>", "</code>");
+
+ /+ ↓ links: ┥link text┝┤url├ +/
+ result = replaceLinks(result);
+
+ /+ ↓ footnote markers: 【N text】 +/
+ result = replaceFootnotes(result);
+
+ /+ ↓ line breaks +/
+ result = result.replace("┘", "<br/>");
+ result = result.replace("┙", "<br/>");
+
+ /+ ↓ non-breaking spaces +/
+ result = result.replace("░", "&nbsp;");
+
+ /+ ↓ bullet markers +/
+ result = result.replace("● ", "&#8226; ");
+
+ /+ ↓ page break markers +/
+ result = result.replace("┼", "").replace("┿", "").replace("╂", "");
+
+ return result;
+}
+
+private string replaceInlineMarker(string text, string open_mark, string close_suffix, string html_open, string html_close) {
+ /+ ↓ replace ⑆Xcontent⑆X pattern (no bracket form) +/
+ string result = text;
+ while (true) {
+ auto open_pos = result.indexOf(open_mark);
+ if (open_pos < 0) break;
+ auto after_open = open_pos + open_mark.length;
+ /+ ↓ look for closing ⑆X +/
+ string close_mark = "⑆" ~ close_suffix;
+ auto close_pos = indexOf(result[after_open .. $], close_mark);
+ if (close_pos < 0) break;
+ close_pos += after_open;
+ string content = result[after_open .. close_pos];
+ result = result[0 .. open_pos] ~ html_open ~ content ~ html_close
+ ~ result[close_pos + close_mark.length .. $];
+ }
+ return result;
+}
+
+private string replaceInlineMarkerBracketed(string text, string open_mark, string bracket_open, string bracket_close, string html_open, string html_close) {
+ /+ ↓ replace ⑆X┨content┣X pattern (bracketed form) +/
+ string result = text;
+ string full_open = open_mark ~ bracket_open;
+ while (true) {
+ auto open_pos = result.indexOf(full_open);
+ if (open_pos < 0) break;
+ auto after_open = open_pos + full_open.length;
+ auto close_pos = indexOf(result[after_open .. $], bracket_close);
+ if (close_pos < 0) break;
+ close_pos += after_open;
+ string content = result[after_open .. close_pos];
+ result = result[0 .. open_pos] ~ html_open ~ content ~ html_close
+ ~ result[close_pos + bracket_close.length .. $];
+ }
+ return result;
+}
+
+private string replaceLinks(string text) {
+ /+ ↓ replace ┥link text┝┤url├ with <a href="url">link text</a> +/
+ string result = text;
+ while (true) {
+ auto link_open = result.indexOf("┥");
+ if (link_open < 0) break;
+ auto link_mid = indexOf(result[link_open .. $], "┝┤");
+ if (link_mid < 0) break;
+ link_mid += link_open;
+ auto link_close = indexOf(result[link_mid .. $], "├");
+ if (link_close < 0) break;
+ link_close += link_mid;
+ string link_text = result[link_open + "┥".length .. link_mid];
+ string url = result[link_mid + "┝┤".length .. link_close];
+ /+ ↓ handle internal links (¤ prefix) +/
+ if (url.length > 0 && url[0] == '¤') {
+ url = url[("¤").length .. $];
+ }
+ /+ ↓ handle OCN-only links (#N) +/
+ if (url.length > 0 && url[0] == '#') {
+ url = "#o" ~ url[1 .. $];
+ }
+ result = result[0 .. link_open]
+ ~ `<a href="` ~ url ~ `">` ~ link_text ~ `</a>`
+ ~ result[link_close + "├".length .. $];
+ }
+ return result;
+}
+
+private string replaceFootnotes(string text) {
+ /+ ↓ replace 【N content】 with superscript note number +/
+ string result = text;
+ while (true) {
+ auto fn_open = result.indexOf("【");
+ if (fn_open < 0) break;
+ auto fn_close = indexOf(result[fn_open .. $], "】");
+ if (fn_close < 0) break;
+ fn_close += fn_open;
+ string fn_content = result[fn_open + "【".length .. fn_close];
+ /+ ↓ extract note number (first token before space) +/
+ auto space_pos = fn_content.indexOf(' ');
+ string note_num = (space_pos > 0) ? fn_content[0 .. space_pos] : fn_content;
+ result = result[0 .. fn_open]
+ ~ `<sup class="fn">` ~ note_num ~ `</sup>`
+ ~ result[fn_close + "】".length .. $];
+ }
+ return result;
+}
+
+string htmlEscape(string text) {
+ return text.replace("&", "&amp;").replace("<", "&lt;").replace(">", "&gt;")
+ .replace("\"", "&quot;");
+}
+
+private string[] htmlHead(OcdaDocument doc) {
+ string[] lines;
+ lines ~= `<!DOCTYPE html>`;
+ lines ~= `<html>`;
+ lines ~= `<head>`;
+ lines ~= `<meta charset="utf-8"/>`;
+ lines ~= `<title>` ~ htmlEscape(doc.meta("title.main")) ~ `</title>`;
+ lines ~= `<style>`;
+ lines ~= ` body { max-width: 48em; margin: 2em auto; padding: 0 1em; font-family: serif; line-height: 1.5; }`;
+ lines ~= ` .title { text-align: center; margin-bottom: 2em; }`;
+ lines ~= ` .subtitle { font-weight: normal; font-style: italic; }`;
+ lines ~= ` .ocn { float: right; font-size: 0.7em; color: #999; }`;
+ lines ~= ` .toc { margin-bottom: 2em; }`;
+ lines ~= ` .toc-entry { margin: 0.2em 0; }`;
+ lines ~= ` .toc-sub { margin: 0.2em 0 0.2em 2em; }`;
+ lines ~= ` .indent-1 { margin-left: 2em; }`;
+ lines ~= ` .indent-2 { margin-left: 4em; }`;
+ lines ~= ` .indent-3 { margin-left: 6em; }`;
+ lines ~= ` .bullet { margin-left: 2em; }`;
+ lines ~= ` .bullet:before { content: "\2022 "; }`;
+ lines ~= ` .verse { white-space: pre-wrap; font-style: italic; }`;
+ lines ~= ` .group { margin: 1em 2em; }`;
+ lines ~= ` .block { margin: 1em 2em; background: #f8f8f8; padding: 1em; }`;
+ lines ~= ` pre code { display: block; background: #f4f4f4; padding: 1em; overflow-x: auto; }`;
+ lines ~= ` table { border-collapse: collapse; margin: 1em 0; }`;
+ lines ~= ` th, td { border: 1px solid #ccc; padding: 0.4em 0.8em; }`;
+ lines ~= ` th { background: #f0f0f0; }`;
+ lines ~= ` .endnote { font-size: 0.9em; }`;
+ lines ~= ` .fn { color: #666; }`;
+ lines ~= ` .index-entry { font-size: 0.9em; }`;
+ lines ~= ` blockquote { margin: 1em 2em; padding-left: 1em; border-left: 3px solid #ccc; }`;
+ lines ~= `</style>`;
+ lines ~= `</head>`;
+ lines ~= `<body>`;
+ return lines;
+}
+
+private string[] htmlTail() {
+ return [`</body>`, `</html>`];
+}
diff --git a/dlang/src/read/images.d b/dlang/src/read/images.d
new file mode 100644
index 0000000..afddc74
--- /dev/null
+++ b/dlang/src/read/images.d
@@ -0,0 +1,114 @@
+module read.images;
+
+import std.array : join;
+import std.conv : to;
+import std.stdio : writeln, File;
+import ocda.model;
+
+/+ ↓ the .image records of a document, gathered per distinct file, with the
+ objects referencing each file, in order of first appearance +/
+struct ImageEntry {
+ string name;
+ string sha256;
+ ulong bytes;
+ int width;
+ int height;
+ bool missing;
+ string[] refs;
+ string[] other_sha256; // same filename, differing digest
+}
+
+struct ImageRecords {
+ int records; // .image lines, an image referenced twice counts twice
+ ImageEntry[] entries; // distinct files
+}
+
+ImageRecords gatherImageRecords(OcdaDocument doc) {
+ ImageRecords ret;
+ size_t[string] seen;
+ foreach (section; sectionNames) {
+ auto objs = doc.sectionByName(section);
+ if (objs is null) continue;
+ foreach (obj; *objs) {
+ foreach (img; obj.images) {
+ ret.records += 1;
+ string _ref = section ~ "[" ~ obj.ocn.to!string ~ "]";
+ if (auto at = img.name in seen) {
+ if (ret.entries[*at].sha256 != img.sha256
+ && img.sha256.length > 0
+ && ret.entries[*at].sha256.length > 0) {
+ ret.entries[*at].other_sha256 ~= img.sha256;
+ }
+ ret.entries[*at].refs ~= _ref;
+ } else {
+ ImageEntry e_;
+ e_.name = img.name;
+ e_.sha256 = img.sha256;
+ e_.bytes = img.bytes;
+ e_.width = img.width;
+ e_.height = img.height;
+ e_.missing = img.missing;
+ e_.refs = [_ref];
+ seen[img.name] = ret.entries.length;
+ ret.entries ~= e_;
+ }
+ }
+ }
+ }
+ return ret;
+}
+
+/+ ↓ image manifest, line oriented and greppable:
+
+ <sha256> <bytes> <px> <filename> refs: body[97] body[412]
+ MISSING <filename> refs: body[840]
+
+ The point of the record in the .ssp is verification (is the image
+ shipped in this pod the image the abstraction was built from), so this
+ output is the shape a verifier wants: one line per distinct file.
++/
+void outputImages(OcdaDocument doc, string output_file) {
+ auto gathered = gatherImageRecords(doc);
+ string[] lines;
+ int missing = 0;
+ foreach (e; gathered.entries) {
+ if (e.missing) missing += 1;
+ }
+ lines ~= "Images: " ~ gathered.records.to!string ~ " record"
+ ~ ((gathered.records == 1) ? "" : "s")
+ ~ ", " ~ gathered.entries.length.to!string ~ " distinct file"
+ ~ ((gathered.entries.length == 1) ? "" : "s")
+ ~ ", " ~ missing.to!string ~ " missing";
+ lines ~= "";
+ foreach (e; gathered.entries) {
+ string px = (e.width > 0 && e.height > 0)
+ ? e.width.to!string ~ "x" ~ e.height.to!string : "-";
+ lines ~= (e.missing)
+ ? "MISSING" ~ " " ~ e.name ~ " refs: " ~ e.refs.join(" ")
+ : e.sha256 ~ " " ~ e.bytes.to!string ~ " " ~ px
+ ~ " " ~ e.name ~ " refs: " ~ e.refs.join(" ");
+ /+ ↓ one filename, more than one digest: the .ssp is inconsistent,
+ or two distinct files share a name +/
+ foreach (other; e.other_sha256) {
+ lines ~= " WARNING differing digest for same filename: " ~ other;
+ }
+ }
+
+ /+ ↓ write output +/
+ if (output_file.length > 0) {
+ auto f = File(output_file, "w");
+ foreach (line; lines) f.writeln(line);
+ } else {
+ foreach (line; lines) writeln(line);
+ }
+}
+
+/+ ↓ count of .image records in a document, for the parse summary +/
+int imageRecordCount(OcdaDocument doc) {
+ return gatherImageRecords(doc).records;
+}
+
+package immutable string[] sectionNames = [
+ "head", "toc", "body", "endnotes",
+ "glossary", "bibliography", "bookindex", "blurb",
+];
diff --git a/dlang/src/read/llm_jsonl.d b/dlang/src/read/llm_jsonl.d
new file mode 100644
index 0000000..5b81e68
--- /dev/null
+++ b/dlang/src/read/llm_jsonl.d
@@ -0,0 +1,121 @@
+module read.llm_jsonl;
+
+import std.array : split, join, replace;
+import std.conv : to;
+import std.stdio : writeln, File;
+import std.string : strip, indexOf, stripRight;
+import ocda.model;
+import read.llm_text : stripMarkup;
+
+/+ ↓ JSONL output with embeddings-ready chunks for RAG pipelines
+ one JSON object per line, each containing:
+ doc, ocn, path (breadcrumb), type, text (clean),
+ images (only where the object embeds one, so that a chunk whose
+ text reads around a figure carries the figure's identity),
+ index_terms (from bookindex entries associated with this object)
++/
+void outputLlmJsonl(OcdaDocument doc, string output_file) {
+ string[] lines;
+ string doc_id = doc.meta("title.main");
+ string doc_author = doc.meta("creator.author");
+
+ /+ ↓ build bookindex -> OCN mapping for index_terms
+ bookindex objects in the .ssp have parent_ocn linking to body objects
+ but the index terms are in the text of bookindex objects, not
+ directly attached to body objects. We use a simpler approach:
+ headings carry heading_ancestors_text which gives context. +/
+
+ /+ ↓ track current heading path +/
+ string current_path = "";
+
+ foreach (obj; doc.body) {
+ if (obj.is_a == "heading") {
+ current_path = buildPath(doc_id, obj.heading_ancestors_text, obj);
+ /+ ↓ emit heading as a chunk too +/
+ string heading_text = stripMarkup(obj.text);
+ if (heading_text.strip.length > 0) {
+ lines ~= jsonLine(doc_id, doc_author, obj.ocn, current_path, "heading", heading_text, obj.images);
+ }
+ continue;
+ }
+
+ /+ ↓ markup commentary is not content, and least of all is it a chunk
+ to embed: it would be indexed as though the author had written it
+ into the document
+ +/
+ if (!obj.renders) { continue; }
+
+ string text = stripMarkup(obj.text);
+ if (text.strip.length == 0) continue;
+
+ lines ~= jsonLine(doc_id, doc_author, obj.ocn, current_path, obj.is_a, text, obj.images);
+ }
+
+ /+ ↓ write output +/
+ if (output_file.length > 0) {
+ auto f = File(output_file, "w");
+ foreach (line; lines) f.writeln(line);
+ } else {
+ foreach (line; lines) writeln(line);
+ }
+}
+
+private string jsonLine(string doc_id, string doc_author, int ocn, string path, string type, string text, OcdaImage[] images) {
+ return `{"doc":` ~ jsonStr(doc_id)
+ ~ `,"author":` ~ jsonStr(doc_author)
+ ~ `,"ocn":` ~ ocn.to!string
+ ~ `,"path":` ~ jsonStr(path)
+ ~ `,"type":` ~ jsonStr(type)
+ ~ `,"text":` ~ jsonStr(text)
+ ~ jsonImages(images)
+ ~ `}`;
+}
+
+private string jsonImages(OcdaImage[] images) {
+ /+ ↓ omitted entirely where the object embeds no image +/
+ if (images.length == 0) return "";
+ string[] items;
+ foreach (img; images) {
+ string item = `{"name":` ~ jsonStr(img.name);
+ if (img.missing) {
+ item ~= `,"missing":true`;
+ } else {
+ item ~= `,"sha256":` ~ jsonStr(img.sha256)
+ ~ `,"bytes":` ~ img.bytes.to!string;
+ if (img.width > 0 && img.height > 0) {
+ item ~= `,"width":` ~ img.width.to!string
+ ~ `,"height":` ~ img.height.to!string;
+ }
+ }
+ items ~= item ~ `}`;
+ }
+ return `,"images":[` ~ items.join(",") ~ `]`;
+}
+
+private string buildPath(string doc_title, string ancestors_text, OcdaObject obj) {
+ string[] parts;
+ if (ancestors_text.length > 0) {
+ foreach (part; ancestors_text.split("|")) {
+ string clean = stripMarkup(part).strip;
+ if (clean.length > 0 && clean != doc_title) {
+ parts ~= clean;
+ }
+ }
+ }
+ string heading_text = stripMarkup(obj.text).strip;
+ if (parts.length > 0)
+ return parts.join(" > ") ~ " > " ~ heading_text;
+ else
+ return heading_text;
+}
+
+private string jsonStr(string s) {
+ /+ ↓ JSON string escaping +/
+ string r = s;
+ r = r.replace(`\`, `\\`);
+ r = r.replace(`"`, `\"`);
+ r = r.replace("\n", `\n`);
+ r = r.replace("\r", `\r`);
+ r = r.replace("\t", `\t`);
+ return `"` ~ r ~ `"`;
+}
diff --git a/dlang/src/read/llm_text.d b/dlang/src/read/llm_text.d
new file mode 100644
index 0000000..e61f2b7
--- /dev/null
+++ b/dlang/src/read/llm_text.d
@@ -0,0 +1,182 @@
+module read.llm_text;
+
+import std.array : replace, split, join;
+import std.conv : to;
+import std.stdio : writeln, File;
+import std.string : strip, indexOf, stripRight;
+import ocda.model;
+
+/+ ↓ plain structured text with heading breadcrumbs for direct LLM consumption
+ format:
+ ## Document Title > Chapter > Section > [OCN]
+
+ Clean paragraph text with inline markup stripped.
++/
+void outputLlmBreadcrumbs(OcdaDocument doc, string output_file) {
+ string[] lines;
+ string doc_title = doc.meta("title.main");
+
+ /+ ↓ document header +/
+ lines ~= "# " ~ doc_title;
+ if (doc.meta("title.subtitle").length > 0)
+ lines ~= "# " ~ doc.meta("title.subtitle");
+ lines ~= "# " ~ doc.meta("creator.author");
+ lines ~= "";
+
+ /+ ↓ build heading text lookup from ancestors +/
+ string[int] heading_text;
+ foreach (obj; doc.head) {
+ if (obj.is_a == "heading" && obj.ocn > 0)
+ heading_text[obj.ocn] = stripMarkup(obj.text);
+ }
+ foreach (obj; doc.body) {
+ if (obj.is_a == "heading")
+ heading_text[obj.ocn] = stripMarkup(obj.text);
+ }
+
+ /+ ↓ body objects +/
+ string current_breadcrumb = "";
+ foreach (obj; doc.body) {
+ if (obj.is_a == "heading") {
+ current_breadcrumb = buildBreadcrumb(doc_title, obj.heading_ancestors_text, obj);
+ lines ~= "";
+ lines ~= "## " ~ current_breadcrumb;
+ lines ~= "";
+ continue;
+ }
+
+ /+ ↓ markup commentary is not content +/
+ if (!obj.renders) { continue; }
+
+ string text = stripMarkup(obj.text);
+ if (text.strip.length == 0) continue;
+
+ string ocn_ref = (obj.ocn > 0 && !obj.ocn_off)
+ ? " [" ~ obj.ocn.to!string ~ "]"
+ : "";
+
+ switch (obj.is_a) {
+ case "para":
+ lines ~= text ~ ocn_ref;
+ lines ~= "";
+ break;
+ case "code":
+ lines ~= "```";
+ lines ~= obj.text; /+ preserve raw text for code +/
+ lines ~= "```";
+ lines ~= "";
+ break;
+ case "table":
+ lines ~= text ~ ocn_ref;
+ lines ~= "";
+ break;
+ case "verse":
+ lines ~= text ~ ocn_ref;
+ lines ~= "";
+ break;
+ case "quote":
+ lines ~= "> " ~ text ~ ocn_ref;
+ lines ~= "";
+ break;
+ case "group":
+ case "block":
+ lines ~= text ~ ocn_ref;
+ lines ~= "";
+ break;
+ default:
+ if (text.length > 0) {
+ lines ~= text ~ ocn_ref;
+ lines ~= "";
+ }
+ break;
+ }
+ }
+
+ /+ ↓ write output +/
+ if (output_file.length > 0) {
+ auto f = File(output_file, "w");
+ foreach (line; lines) f.writeln(line);
+ } else {
+ foreach (line; lines) writeln(line);
+ }
+}
+
+private string buildBreadcrumb(string doc_title, string ancestors_text, OcdaObject obj) {
+ /+ ↓ ancestors_text is pipe-separated: "Title|Part||Chapter|||" +/
+ string[] parts;
+ if (ancestors_text.length > 0) {
+ foreach (part; ancestors_text.split("|")) {
+ string clean = stripMarkup(part).strip;
+ if (clean.length > 0 && clean != doc_title) {
+ /+ ↓ truncate very long heading text for breadcrumb +/
+ if (clean.length > 60)
+ clean = clean[0 .. 57] ~ "...";
+ parts ~= clean;
+ }
+ }
+ }
+ /+ ↓ add the current heading itself +/
+ string heading_text = stripMarkup(obj.text).strip;
+ if (heading_text.length > 60)
+ heading_text = heading_text[0 .. 57] ~ "...";
+
+ string ocn_ref = (obj.ocn > 0) ? " [" ~ obj.ocn.to!string ~ "]" : "";
+
+ if (parts.length > 0)
+ return doc_title ~ " > " ~ parts.join(" > ") ~ " > " ~ heading_text ~ ocn_ref;
+ else
+ return doc_title ~ " > " ~ heading_text ~ ocn_ref;
+}
+
+/+ ↓ strip all internal Unicode markup to produce clean plain text +/
+string stripMarkup(string text) {
+ if (text.length == 0) return "";
+ string r = text;
+
+ /+ ↓ links: ┥link text┝┤url├ -> link text +/
+ while (true) {
+ auto lo = r.indexOf("┥");
+ if (lo < 0) break;
+ auto lm = indexOf(r[lo .. $], "┝┤");
+ if (lm < 0) break;
+ lm += lo;
+ auto lc = indexOf(r[lm .. $], "├");
+ if (lc < 0) break;
+ lc += lm;
+ string link_text = r[lo + "┥".length .. lm];
+ r = r[0 .. lo] ~ link_text ~ r[lc + "├".length .. $];
+ }
+
+ /+ ↓ footnotes: 【N content】 -> (remove entirely for clean text) +/
+ while (true) {
+ auto fo = r.indexOf("【");
+ if (fo < 0) break;
+ auto fc = indexOf(r[fo .. $], "】");
+ if (fc < 0) break;
+ fc += fo;
+ r = r[0 .. fo] ~ r[fc + "】".length .. $];
+ }
+
+ /+ ↓ font face markers +/
+ r = r.replace("⑆*", "").replace("⑆!", "").replace("⑆/", "")
+ .replace("⑆_", "").replace("⑆^", "").replace("⑆,", "")
+ .replace("⑆■", "")
+ .replace("┨", "").replace("┣*", "").replace("┣!", "")
+ .replace("┣/", "").replace("┣_", "").replace("┣^", "")
+ .replace("┣,", "").replace("┣", "");
+
+ /+ ↓ line breaks and special chars +/
+ r = r.replace("┘", "\n").replace("┙", " ").replace("┚", "\n");
+ r = r.replace("░", " ");
+ r = r.replace("● ", "- ");
+ r = r.replace("┼", "").replace("┿", "").replace("╂", "");
+
+ /+ ↓ table cell separators +/
+ r = r.replace("┊", " | ");
+
+ /+ ↓ clean up multiple spaces +/
+ while (r.indexOf(" ") >= 0)
+ r = r.replace(" ", " ");
+
+ return r.strip;
+}
diff --git a/dlang/src/read/main.d b/dlang/src/read/main.d
new file mode 100644
index 0000000..ae0d318
--- /dev/null
+++ b/dlang/src/read/main.d
@@ -0,0 +1,228 @@
+/++
+ ocda-read - reads the object-centric document abstraction, in any of the
+ forms it is written in, and says what is in it.
+
+ ocda-read [options] <file>.ssp | <file>.ocda.db | <file>.sisupod [lang]
+
+ One reader where there were two: sisudoc-ssp-reader read a .ssp and
+ sisudoc-ocda-db-reader a .ocda.db, each with its own parser. This one is
+ built on the library, so every mode reads every form, and a mode given the
+ .ssp, the .ocda.db and the .sisupod of a document says the same thing of
+ each: it is told what the document is, not which file it came from.
+
+ Two things are about the file rather than the document, and say so:
+ --files, what a .ocda.db carries besides the abstraction, and --validate,
+ which checks a .ssp as text. Everything else is the model's.
+
+ The output modules are the .ssp reader's, moved onto the library's model.
++/
+module read.main;
+
+import std.algorithm : canFind, map;
+import std.array : array, join;
+import std.getopt;
+import std.stdio : writeln, writefln, stderr;
+import ocda.model;
+import ocda.load;
+import ocda.inventory;
+import read.tree;
+import read.html;
+import read.llm_text;
+import read.llm_jsonl;
+import read.images;
+import read.verify;
+import read.report_unknown;
+import read.validate;
+
+int main(string[] args) {
+ bool do_tree = false;
+ bool do_html = false;
+ bool do_llm_text = false;
+ bool do_llm_jsonl = false;
+ bool do_images = false;
+ bool do_verify = false;
+ bool do_report_unknown = false;
+ bool do_validate = false;
+ bool do_languages = false;
+ bool do_files = false;
+ bool any_version = false;
+ string image_dir = "";
+ string output_file = "";
+ string lang = "";
+
+ GetoptResult helpInfo;
+ try {
+ helpInfo = getopt(args,
+ "tree", "the headings, as a tree", &do_tree,
+ "outline", "the same as --tree", &do_tree,
+ "html", "the document as one html page", &do_html,
+ "llm-text", "plain text with heading breadcrumbs", &do_llm_text,
+ "llm-jsonl", "JSONL, chunks ready for embedding", &do_llm_jsonl,
+ "images", "the image records: digest, size, pixels", &do_images,
+ "verify", "check the image records against --image-dir", &do_verify,
+ "image-dir", "the images to verify against", &image_dir,
+ "languages", "the languages the file holds", &do_languages,
+ "files", "what a .ocda.db carries besides the abstraction", &do_files,
+ "report-unknown", "what was read and not understood", &do_report_unknown,
+ "validate", "check a .ssp against its specification, as text", &do_validate,
+ "lang", "=<code> the language to read (default: the first)", &lang,
+ "output", "write the mode's output to a file", &output_file,
+ "any-version", "read a format major this reader does not know", &any_version,
+ );
+ } catch (GetOptException e) {
+ stderr.writeln("ocda-read: ", e.msg, "; --help lists the options");
+ return 2;
+ }
+
+ if (helpInfo.helpWanted || args.length < 2) {
+ defaultGetoptPrinter(
+ "ocda-read - reads the document abstraction, in any of its forms\n\n"
+ ~ "Usage: ocda-read [options] <file>.ssp|<file>.ocda.db|<file>.sisupod [lang]\n",
+ helpInfo.options,
+ );
+ return 0;
+ }
+
+ string input_file = args[1];
+ if (lang.length == 0 && args.length > 2) { lang = args[2]; }
+ bool lang_named = (lang.length > 0);
+
+ /+ ↓ --validate is about the .ssp as text, so it is asked of a .ssp or not
+ at all: there is no text form of a database to check, and a pod's
+ .ssp is checked by taking it out +/
+ if (do_validate && sourceOf(input_file) != Source.ssp) {
+ stderr.writeln("--validate checks a .ssp as text, against the .ssp",
+ " specification; ", input_file, " is ", sourceName(sourceOf(input_file)));
+ return 2;
+ }
+
+ /+ ↓ which language. A .ocda.db and a pod may hold several; where none is
+ named, every form reads the first in the manifest's order and says
+ so, rather than the database taking one and the pod refusing +/
+ OcdaLanguage[] held;
+ OcdaDocument doc;
+ try {
+ held = languages(input_file);
+ if (lang.length == 0 && held.length > 0) { lang = held[0].lang; }
+ if (held.length > 0 && !held.map!(l => l.lang).canFind(lang)) {
+ stderr.writeln("error: no language '", lang, "' in ", input_file,
+ "; it holds: ", held.map!(l => l.lang).array.join(" "));
+ return 2;
+ }
+ doc = load(input_file, lang, any_version);
+ } catch (Exception e) {
+ stderr.writeln("Error reading ", input_file, ": ", e.msg);
+ return 2;
+ }
+
+ /+ ↓ the version gate: the major part, and refuse what is not known +/
+ if (doc.format_version.length == 0) {
+ stderr.writeln("warning: ", input_file, " declares no format version");
+ } else if (doc.formatVersionMajor != ssp_format_major_read) {
+ if (any_version) {
+ stderr.writeln("warning: ", input_file, " is format v", doc.format_version,
+ ", this reader knows v", ssp_format_major_read, ".x; reading anyway");
+ } else {
+ stderr.writeln("error: ", input_file, " is format v", doc.format_version,
+ ", this reader knows v", ssp_format_major_read, ".x");
+ stderr.writeln(" a major version differs: a property may have",
+ " changed meaning or gone away.");
+ stderr.writeln(" --any-version reads it regardless.");
+ return 3;
+ }
+ }
+
+ writeln("Read: ", input_file);
+ if (doc.format_version.length > 0) {
+ writeln(" Format: v", doc.format_version);
+ }
+ if (auto _lang = "language" in doc.source) {
+ writeln(" Source: ", *_lang,
+ ("languages" in doc.source) ? " of " ~ doc.source["languages"] : "",
+ ("digest" in doc.source) ? ", markup digest " ~ doc.source["digest"][0 .. 16] ~ "..." : "");
+ }
+ if (held.length > 1 && !lang_named) {
+ writeln(" (reading ", lang, "; --lang=<code> for another)");
+ }
+ writeln(" Title: ", doc.meta("title.main"));
+ writeln(" Author: ", doc.meta("creator.author"));
+ writeln(" Sections: head=", doc.head.length,
+ " toc=", doc.toc.length,
+ " body=", doc.body.length,
+ " endnotes=", doc.endnotes.length,
+ " glossary=", doc.glossary.length,
+ " bibliography=", doc.bibliography.length,
+ " bookindex=", doc.bookindex.length,
+ " blurb=", doc.blurb.length,
+ " tail=", doc.tail.length);
+ auto image_records = imageRecordCount(doc);
+ if (image_records > 0) {
+ writeln(" Images: ", image_records, " record",
+ (image_records == 1) ? "" : "s");
+ }
+
+ if (do_languages) { printLanguages(held); }
+ if (do_files) { printFiles(input_file); }
+ if (do_tree) { outputTree(doc, output_file); }
+ if (do_html) { outputHtmlScroll(doc, output_file); }
+ if (do_llm_text) { outputLlmBreadcrumbs(doc, output_file); }
+ if (do_llm_jsonl) { outputLlmJsonl(doc, output_file); }
+ if (do_images) { outputImages(doc, output_file); }
+
+ int _verify_failures = 0;
+ if (do_verify) {
+ if (image_dir.length == 0) {
+ stderr.writeln("--verify needs --image-dir=<path to the pod's media/image>");
+ return 2;
+ }
+ _verify_failures = verifyImages(doc, image_dir, output_file);
+ }
+
+ if (do_report_unknown) { reportUnknown(doc); }
+
+ int _validation_findings = 0;
+ if (do_validate) {
+ auto _found = validateSspFile(input_file, ssp_format_major_read);
+ reportValidation(input_file, _found);
+ _validation_findings = cast(int) _found.length;
+ }
+
+ if (!do_tree && !do_html && !do_llm_text && !do_llm_jsonl && !do_images
+ && !do_verify && !do_report_unknown && !do_validate
+ && !do_languages && !do_files) {
+ writeln("\n(this is the summary: --help lists the modes, each adding its output)");
+ }
+
+ return (_verify_failures > 0 || _validation_findings > 0) ? 1 : 0;
+}
+
+/+ ↓ the languages, each with the markup it was made from and its .ssp's
+ digest: the same from every form, the database storing the digest and
+ the others taking it from the bytes +/
+void printLanguages(OcdaLanguage[] held) {
+ writeln();
+ writeln(" lang source ssp digest");
+ foreach (l; held) {
+ writefln(" %-7s %-31s %s", l.lang, l.source_filename,
+ (l.ssp_digest.length > 16) ? l.ssp_digest[0 .. 16] ~ "..." : l.ssp_digest);
+ }
+}
+
+/+ ↓ what a .ocda.db carries besides the abstraction. A .ssp carries
+ nothing, and a pod's files are its archive entries rather than anything
+ the abstraction records, so for those it says so +/
+void printFiles(string path) {
+ writeln();
+ if (!isCarrier(path)) {
+ writeln(" carried files: only a .ocda.db carries files; this is ",
+ sourceName(sourceOf(path)));
+ return;
+ }
+ auto rows = carried(path);
+ if (rows.length == 0) { writeln(" carries no files"); return; }
+ writeln(" carried files:");
+ writeln(" role stored n original in the file");
+ foreach (r; rows) {
+ writefln(" %-9s %-7s %-5s %-13s %s", r.role, r.compression, r.count, r.bytes, r.stored);
+ }
+}
diff --git a/dlang/src/read/report_unknown.d b/dlang/src/read/report_unknown.d
new file mode 100644
index 0000000..2f81702
--- /dev/null
+++ b/dlang/src/read/report_unknown.d
@@ -0,0 +1,41 @@
+module read.report_unknown;
+
+import std.algorithm : sort;
+import std.array : array;
+import std.stdio : writeln, writefln;
+import ocda.model;
+
+/+ ↓ say what was read and not understood.
+ This is what makes the reader usable as a check on the format
+ specification: a reader that ignores what it does not know is
+ conforming, but one that cannot say what it ignored cannot tell you
+ whether the specification it was written from was complete. Silence
+ and understanding look the same from outside.
++/
+void reportUnknown(ref OcdaDocument doc) {
+ if (!doc.unknown.any) {
+ writeln("\nread and understood: every property, block and line was known");
+ return;
+ }
+ writeln("\nread and NOT understood:");
+ if (doc.unknown.property.length > 0) {
+ writeln(" properties ignored (", doc.unknown.property.length, " distinct):");
+ foreach (name; doc.unknown.property.keys.sort) {
+ writefln(" .%-26s %8d", name, doc.unknown.property[name]);
+ }
+ }
+ if (doc.unknown.block.length > 0) {
+ writeln(" blocks ignored (", doc.unknown.block.length, " distinct):");
+ foreach (name; doc.unknown.block.keys.sort) {
+ writefln(" @%-26s %8d", name, doc.unknown.block[name]);
+ }
+ }
+ if (doc.unknown.line.length > 0) {
+ writeln(" lines of no known shape, by first character:");
+ foreach (c; doc.unknown.line.keys.sort) {
+ writefln(" %-27s %8d", "\"" ~ c ~ "\"", doc.unknown.line[c]);
+ }
+ }
+ writeln(" (ignoring these is conforming within a major version;",
+ " the count is so that it is not silent)");
+}
diff --git a/dlang/src/read/tree.d b/dlang/src/read/tree.d
new file mode 100644
index 0000000..9a45b02
--- /dev/null
+++ b/dlang/src/read/tree.d
@@ -0,0 +1,69 @@
+module read.tree;
+
+import std.conv : to;
+import std.stdio : writeln, writef, writefln, File;
+import std.array : replicate;
+import ocda.model;
+
+void outputTree(OcdaDocument doc, string output_file) {
+ string[] lines;
+
+ lines ~= "Document: " ~ doc.meta("title.main");
+ if (doc.meta("title.subtitle").length > 0)
+ lines ~= " " ~ doc.meta("title.subtitle");
+ lines ~= "Author: " ~ doc.meta("creator.author");
+ lines ~= "";
+
+ /+ ↓ build OCN -> object lookup for body headings +/
+ OcdaObject[int] headings;
+ foreach (obj; doc.body) {
+ if (obj.is_a == "heading" && obj.ocn > 0) {
+ headings[obj.ocn] = obj;
+ }
+ }
+
+ /+ ↓ find root heading (from head section, or first body heading) +/
+ if (doc.head.length > 0 && doc.head[0].is_a == "heading") {
+ auto root = doc.head[0];
+ lines ~= formatHeading(root, 0);
+ walkChildren(root, headings, lines, 1);
+ }
+
+ /+ ↓ write output +/
+ if (output_file.length > 0) {
+ auto f = File(output_file, "w");
+ foreach (line; lines) f.writeln(line);
+ } else {
+ foreach (line; lines) writeln(line);
+ }
+}
+
+private void walkChildren(OcdaObject parent, ref OcdaObject[int] headings, ref string[] lines, int depth) {
+ foreach (child_ocn; parent.children) {
+ if (auto p = child_ocn in headings) {
+ lines ~= formatHeading(*p, depth);
+ walkChildren(*p, headings, lines, depth + 1);
+ }
+ }
+}
+
+private string formatHeading(OcdaObject obj, int depth) {
+ string indent = replicate(" ", depth);
+ string level_tag = (obj.heading_level.length > 0) ? obj.heading_level : "?";
+ string ocn_str = (obj.ocn > 0) ? " [" ~ obj.ocn.to!string ~ "]" : "";
+ /+ ↓ strip internal markup from heading text for clean display +/
+ string text = stripInternalMarkup(obj.text);
+ return indent ~ level_tag ~ "~" ~ ocn_str ~ " " ~ text;
+}
+
+private string stripInternalMarkup(string text) {
+ /+ ↓ simple removal of common internal markers for display +/
+ import std.regex : replaceAll, regex;
+ import std.array : replace;
+ string result = text;
+ result = result.replace("⑆*", "").replace("⑆!", "").replace("⑆/", "")
+ .replace("⑆_", "").replace("⑆^", "").replace("⑆,", "")
+ .replace("┨", "").replace("┣", "")
+ .replace("┙", " ").replace("┘", "\n");
+ return result;
+}
diff --git a/dlang/src/read/validate.d b/dlang/src/read/validate.d
new file mode 100644
index 0000000..4a77f8b
--- /dev/null
+++ b/dlang/src/read/validate.d
@@ -0,0 +1,265 @@
+module read.validate;
+
+/+ ↓ check a .ssp against the format specification.
+ .
+ Written from specs/sisudoc-abstraction-format/ssp-format-specification.md
+ and from nothing else: the point of it is to be what the specification
+ says, executed, so that a file can be held against the document rather
+ than against an implementation. Each check names the section it comes
+ from, so a disagreement can be taken back to the text.
++/
+
+import std.algorithm : canFind, startsWith;
+import std.array : array, split;
+import std.conv : to;
+import std.file : readText;
+import std.stdio : writeln, writefln;
+import std.string : indexOf, strip;
+
+struct SspFinding {
+ size_t line; // 1-based
+ string section; // the specification section the rule is from
+ string detail;
+}
+
+/+ ↓ specification section 6.3: the properties, in the order they are
+ emitted. A reader may rely on the order, so a validator can check it +/
+private immutable string[] property_order = [
+ "identifier", "part", "section", "parent", "last_descendant", "ocn_range",
+ "children",
+ "ancestors", "ancestors_collapsed", "dom_status", "dom_status_collapsed",
+ "heading_lev_collapsed", "parent_lev", "dummy", "ocn_off", "is_of_type",
+ "attrib", "meta_lang", "meta_syntax", "sha256", "image", "indent",
+ "bullet", "lang", "has", "table_cols", "table_widths", "table_aligns",
+ "table_header", "code_linenumbers", "stow_link", "segment", "anchor",
+ "segment_prev", "segment_next", "heading_lev_anchor", "segment_epub",
+ "segment_html_is", "segment_epub_is", "segment_lv4_is",
+ "heading_ancestors_text", "lev4_subtoc", "anchor_tag",
+];
+
+/+ ↓ section 6.1: the sixteen kinds +/
+private immutable string[] kinds = [
+ "para", "heading", "toc", "bookindex", "endnote", "code", "bibliography",
+ "comment", "verse", "glossary", "quote", "poem", "block", "table",
+ "blurb", "group",
+];
+
+/+ ↓ section 4: the nine sections, in the order they appear +/
+private immutable string[] section_order = [
+ "head", "toc", "body", "endnotes", "glossary", "bibliography",
+ "bookindex", "blurb", "tail",
+];
+
+/+ ↓ section 4: the four preamble blocks, in the order they appear +/
+private immutable string[] preamble_order = ["source", "meta", "make", "doc_has"];
+
+/+ ↓ section 6.1: the heading levels +/
+private immutable string[] levels = ["A", "B", "C", "D", "1", "2", "3", "4"];
+
+private int propertyRank(string name) {
+ foreach (i, p; property_order) { if (p == name) { return cast(int) i; } }
+ return -1;
+}
+
+SspFinding[] validateSspFile(string filename, string major_known) {
+ SspFinding[] found;
+ auto lines = readText(filename).split("\n");
+
+ bool seen_banner = false;
+ bool seen_source_line = false;
+ string block = ""; // the block currently open, "" when none
+ int preamble_at = -1; // how far through the preamble order
+ int section_at = -1; // how far through the section order
+ bool in_object = false;
+ int prev_rank = -1;
+ size_t doc_has_keys = 0;
+ bool in_text = false; // text lines come after properties
+
+ foreach (i, line; lines) {
+ size_t no = i + 1;
+
+ /+ ↓ section 2: the version, on the first line +/
+ if (no == 1) {
+ enum banner = "% SiSU Document Abstraction v";
+ if (!line.startsWith(banner)) {
+ found ~= SspFinding(no, "2", "first line is not the version banner");
+ } else {
+ seen_banner = true;
+ string v = line[banner.length .. $].strip;
+ auto dot = v.indexOf('.');
+ string major = (dot > 0) ? v[0 .. dot] : v;
+ if (major != major_known) {
+ found ~= SspFinding(no, "2.1",
+ "format major v" ~ major ~ ", this validator knows v" ~ major_known);
+ }
+ }
+ continue;
+ }
+ if (no == 2) {
+ if (!line.startsWith("% Source: ")) {
+ found ~= SspFinding(no, "4", "second line is not \"% Source: \"");
+ } else { seen_source_line = true; }
+ continue;
+ }
+
+ if (line.length == 0) { in_object = false; in_text = false; prev_rank = -1; continue; }
+
+ /+ ↓ section 3: a comment is the file header only +/
+ if (line[0] == '%') {
+ found ~= SspFinding(no, "3", "a comment outside the two header lines");
+ continue;
+ }
+
+ /+ ↓ block open +/
+ if (line[0] == '@') {
+ auto brace = line.indexOf(" {");
+ if (brace < 1 || line[$ - 1] != '{') {
+ found ~= SspFinding(no, "3", "block open is not \"@name {\"");
+ continue;
+ }
+ string name = line[1 .. brace];
+ if (preamble_order.canFind(name)) {
+ int at = 0;
+ foreach (k, p; preamble_order) { if (p == name) { at = cast(int) k; } }
+ if (at <= preamble_at) {
+ found ~= SspFinding(no, "4", "preamble block @" ~ name ~ " out of order");
+ }
+ preamble_at = at;
+ if (section_at >= 0) {
+ found ~= SspFinding(no, "4", "preamble block @" ~ name ~ " after a section");
+ }
+ } else if (section_order.canFind(name)) {
+ int at = 0;
+ foreach (k, sname; section_order) { if (sname == name) { at = cast(int) k; } }
+ if (at <= section_at) {
+ found ~= SspFinding(no, "4", "section @" ~ name ~ " out of order");
+ }
+ section_at = at;
+ } else {
+ found ~= SspFinding(no, "4", "@" ~ name ~ " is neither a preamble block nor a section");
+ }
+ block = name;
+ doc_has_keys = 0;
+ continue;
+ }
+
+ if (line[0] == '}') {
+ if (line.length != 1) {
+ found ~= SspFinding(no, "3", "block close carries more than \"}\"");
+ }
+ /+ ↓ section 5.4: ten keys, always +/
+ if (block == "doc_has" && doc_has_keys != 10) {
+ found ~= SspFinding(no, "5.4",
+ "@doc_has has " ~ doc_has_keys.to!string ~ " keys, the specification says ten");
+ }
+ block = "";
+ in_object = false;
+ prev_rank = -1;
+ continue;
+ }
+
+ /+ ↓ section 5: inside a preamble block, " key: value" +/
+ if (preamble_order.canFind(block)) {
+ if (!line.startsWith(" ") || line.indexOf(": ") < 3) {
+ found ~= SspFinding(no, "5", "not \" key: value\" inside @" ~ block);
+ } else if (block == "doc_has") {
+ doc_has_keys++;
+ }
+ continue;
+ }
+
+ /+ ↓ section 6.1: the object declaration line +/
+ if (line[0] == '[') {
+ auto close = line.indexOf("] ");
+ if (close < 2) {
+ found ~= SspFinding(no, "6.1", "declaration is not \"[ocn] ...\"");
+ continue;
+ }
+ string ocn = line[1 .. close];
+ foreach (c; ocn) {
+ if (c < '0' || c > '9') {
+ found ~= SspFinding(no, "6.1", "ocn \"" ~ ocn ~ "\" is not a number");
+ break;
+ }
+ }
+ auto rest = line[close + 2 .. $].split(" ");
+ if (rest.length == 0 || !kinds.canFind(rest[0])) {
+ found ~= SspFinding(no, "6.1",
+ "\"" ~ ((rest.length > 0) ? rest[0] : "") ~ "\" is not one of the sixteen kinds");
+ } else if (rest[0] == "heading") {
+ if (rest.length < 2 || rest[1].length < 2 || rest[1][0] != ':') {
+ found ~= SspFinding(no, "6.1", "a heading declaration carries no \":level\"");
+ } else if (!levels.canFind(rest[1][1 .. $])) {
+ found ~= SspFinding(no, "6.1", "level \"" ~ rest[1][1 .. $] ~ "\" is not one of the eight");
+ }
+ }
+ in_object = true;
+ in_text = false;
+ prev_rank = -1;
+ continue;
+ }
+
+ /+ ↓ section 6.3: properties, known and in order +/
+ if (line[0] == '.') {
+ if (!in_object) {
+ found ~= SspFinding(no, "6", "a property outside an object record");
+ continue;
+ }
+ if (in_text) {
+ found ~= SspFinding(no, "6", "a property after the object text began");
+ }
+ auto colon = line.indexOf(": ");
+ if (colon < 2) {
+ found ~= SspFinding(no, "3", "property is not \".name: value\"");
+ continue;
+ }
+ string name = line[1 .. colon];
+ int rank = propertyRank(name);
+ if (rank < 0) {
+ found ~= SspFinding(no, "6.3", "." ~ name ~ " is not one of the " ~ property_order.length.to!string ~ " properties");
+ } else {
+ /+ ↓ the four repeatable properties may recur at the same rank +/
+ if (rank < prev_rank) {
+ found ~= SspFinding(no, "6.3", "." ~ name ~ " is out of the emitted order");
+ }
+ prev_rank = rank;
+ }
+ continue;
+ }
+
+ /+ ↓ section 6.2: text lines +/
+ if (line[0] == '|') {
+ if (!in_object) {
+ found ~= SspFinding(no, "6.2", "a text line outside an object record");
+ }
+ if (line.length < 2 || line[1] != ' ') {
+ found ~= SspFinding(no, "6.2", "a text line is not \"| \" then the text");
+ }
+ in_text = true;
+ continue;
+ }
+
+ found ~= SspFinding(no, "3", "line of no shape the specification defines");
+ }
+
+ if (!seen_banner) { found ~= SspFinding(0, "2", "no version banner"); }
+ if (!seen_source_line) { found ~= SspFinding(0, "4", "no \"% Source:\" line"); }
+ return found;
+}
+
+void reportValidation(string filename, SspFinding[] found) {
+ if (found.length == 0) {
+ writeln("\nvalid: ", filename,
+ " conforms to the .ssp specification as this validator reads it");
+ return;
+ }
+ writefln("\n%d finding%s against the specification:",
+ found.length, (found.length == 1) ? "" : "s");
+ foreach (f; found) {
+ if (f.line > 0) {
+ writefln(" line %-7d [%-4s] %s", f.line, f.section, f.detail);
+ } else {
+ writefln(" %-13s [%-4s] %s", "(file)", f.section, f.detail);
+ }
+ }
+}
diff --git a/dlang/src/read/verify.d b/dlang/src/read/verify.d
new file mode 100644
index 0000000..8c2cc65
--- /dev/null
+++ b/dlang/src/read/verify.d
@@ -0,0 +1,214 @@
+module read.verify;
+
+import std.algorithm : sort;
+import std.array : array, join;
+import std.conv : to;
+import std.digest : toHexString;
+import std.digest.sha : SHA256, digest;
+import std.file : dirEntries, exists, isDir, SpanMode, getSize;
+import std.path : baseName, buildPath, extension;
+import std.stdio : writeln, File;
+import std.string : toLower;
+import ocda.model;
+import read.images : gatherImageRecords;
+
+/+ ↓ verification: recompute what the .image records claim, against the
+ images as they stand in a directory (a pod's media/image).
+
+ ssp-reader --verify --image-dir=<pod>/media/image doc.ssp
+
+ Reports one line per distinct file:
+
+ OK won_benkler_2_1.png
+ MISMATCH won_benkler_2_1.png sha256 recorded 5BA4... file 1C3D...
+ ABSENT won_benkler_2_1.png (recorded sha256 5BA4..., not in directory)
+ PRESENT ffa.png (recorded missing, but the file is there)
+ EXTRA unused.png (in directory, no record in this .ssp)
+
+ Returns the number of failures. EXTRA is reported but is not a failure:
+ a pod carries the images of every language version of a document, while
+ one .ssp covers one language.
++/
+int verifyImages(OcdaDocument doc, string image_dir, string output_file) {
+ string[] lines;
+ int failures = 0;
+ int ok = 0;
+ int mismatch = 0;
+ int absent = 0;
+ int extra = 0;
+
+ if (!(image_dir.exists) || !(image_dir.isDir)) {
+ writeln("ERROR: image directory not found: ", image_dir);
+ return 1;
+ }
+ auto gathered = gatherImageRecords(doc);
+ bool[string] recorded;
+
+ foreach (e; gathered.entries) {
+ recorded[e.name] = true;
+ string pth = buildPath(image_dir, e.name);
+ if (!(pth.exists)) {
+ if (e.missing) {
+ /+ recorded missing, and missing here too: consistent +/
+ ok += 1;
+ lines ~= "OK " ~ e.name ~ " (recorded missing, absent here)";
+ } else {
+ absent += 1;
+ failures += 1;
+ lines ~= "ABSENT " ~ e.name
+ ~ " (recorded sha256 " ~ e.sha256 ~ ", not in directory)";
+ }
+ continue;
+ }
+ if (e.missing) {
+ mismatch += 1;
+ failures += 1;
+ lines ~= "PRESENT " ~ e.name ~ " (recorded missing, but the file is there)";
+ continue;
+ }
+ auto file_ = readFileStat(pth);
+ string[] differs;
+ if (file_.sha256 != e.sha256) {
+ differs ~= "sha256 recorded " ~ e.sha256 ~ " file " ~ file_.sha256;
+ }
+ if (file_.bytes != e.bytes) {
+ differs ~= "bytes recorded " ~ e.bytes.to!string
+ ~ " file " ~ file_.bytes.to!string;
+ }
+ /+ px is written only where the dimensions could be read, and read here
+ only for the formats below, so compare only when both are known +/
+ if (e.width > 0 && e.height > 0 && file_.width > 0 && file_.height > 0
+ && (file_.width != e.width || file_.height != e.height)) {
+ differs ~= "px recorded " ~ e.width.to!string ~ "x" ~ e.height.to!string
+ ~ " file " ~ file_.width.to!string ~ "x" ~ file_.height.to!string;
+ }
+ if (differs.length > 0) {
+ mismatch += 1;
+ failures += 1;
+ lines ~= "MISMATCH " ~ e.name ~ " " ~ differs.join("; ");
+ } else {
+ ok += 1;
+ lines ~= "OK " ~ e.name;
+ }
+ }
+
+ /+ ↓ files in the directory that no record names +/
+ string[] in_dir;
+ foreach (entry; dirEntries(image_dir, SpanMode.shallow)) {
+ if (!(entry.isFile)) continue;
+ string nm = entry.name.baseName;
+ switch (nm.extension.toLower) {
+ case ".png": case ".jpg": case ".jpeg": case ".gif":
+ in_dir ~= nm;
+ break;
+ default: break;
+ }
+ }
+ in_dir.sort;
+ foreach (nm; in_dir) {
+ if (nm !in recorded) {
+ extra += 1;
+ lines ~= "EXTRA " ~ nm ~ " (in directory, no record in this .ssp)";
+ }
+ }
+
+ string head = "Verify: " ~ gathered.entries.length.to!string ~ " distinct file"
+ ~ ((gathered.entries.length == 1) ? "" : "s")
+ ~ " in " ~ gathered.records.to!string ~ " record"
+ ~ ((gathered.records == 1) ? "" : "s")
+ ~ ", against " ~ image_dir;
+ lines = [head, ""] ~ lines;
+ lines ~= "";
+ lines ~= "verify: " ~ ok.to!string ~ " ok, "
+ ~ mismatch.to!string ~ " mismatch, "
+ ~ absent.to!string ~ " absent, "
+ ~ extra.to!string ~ " extra"
+ ~ ((failures == 0) ? " PASS" : " FAIL");
+
+ /+ ↓ write output +/
+ if (output_file.length > 0) {
+ auto f = File(output_file, "w");
+ foreach (line; lines) f.writeln(line);
+ } else {
+ foreach (line; lines) writeln(line);
+ }
+ return failures;
+}
+
+private struct FileStat_ {
+ string sha256;
+ ulong bytes;
+ int width;
+ int height;
+}
+
+private FileStat_ readFileStat(string pth) {
+ FileStat_ ret;
+ auto f = File(pth, "rb");
+ ubyte[32] sha = digest!SHA256(f.byChunk(4096 * 1024));
+ ret.sha256 = sha.toHexString.to!string;
+ ret.bytes = pth.getSize;
+ auto dim = readImageDimensions(pth);
+ ret.width = dim[0];
+ ret.height = dim[1];
+ return ret;
+}
+
+/+ ↓ pixel dimensions from the file header, for the formats spine's markup
+ admits (png, gif, jpg). Header read only, no decoding, and no dependency
+ beyond phobos. Returns [0, 0] where the header cannot be read
++/
+private int[2] readImageDimensions(string pth) {
+ int[2] ret = [0, 0];
+ ubyte[] head;
+ try {
+ auto f = File(pth, "rb");
+ head = new ubyte[](64 * 1024);
+ head = f.rawRead(head);
+ } catch (Exception ex) {
+ return ret;
+ }
+ if (head.length >= 24
+ && head[0 .. 8] == cast(ubyte[])[0x89, 'P', 'N', 'G', 0x0d, 0x0a, 0x1a, 0x0a]) {
+ /+ png: IHDR width & height, big endian, at 16 and 20 +/
+ ret[0] = beInt(head[16 .. 20]);
+ ret[1] = beInt(head[20 .. 24]);
+ return ret;
+ }
+ if (head.length >= 10 && (head[0 .. 3] == cast(ubyte[])['G', 'I', 'F'])) {
+ /+ gif: logical screen width & height, little endian, at 6 and 8 +/
+ ret[0] = head[6] | (head[7] << 8);
+ ret[1] = head[8] | (head[9] << 8);
+ return ret;
+ }
+ if (head.length >= 4 && head[0] == 0xff && head[1] == 0xd8) {
+ /+ jpeg: walk the segments to a start of frame, height then width,
+ big endian, at offsets 5 and 7 of the segment
+ +/
+ size_t i = 2;
+ while (i + 9 < head.length) {
+ if (head[i] != 0xff) { i += 1; continue; }
+ ubyte marker = head[i + 1];
+ if (marker == 0xd8 || marker == 0x01
+ || (marker >= 0xd0 && marker <= 0xd7)) { i += 2; continue; }
+ if (marker == 0xff) { i += 1; continue; }
+ size_t seg_len = (head[i + 2] << 8) | head[i + 3];
+ bool is_sof = ((marker >= 0xc0 && marker <= 0xc3)
+ || (marker >= 0xc5 && marker <= 0xc7)
+ || (marker >= 0xc9 && marker <= 0xcb)
+ || (marker >= 0xcd && marker <= 0xcf));
+ if (is_sof) {
+ ret[1] = (head[i + 5] << 8) | head[i + 6];
+ ret[0] = (head[i + 7] << 8) | head[i + 8];
+ return ret;
+ }
+ if (seg_len < 2) break;
+ i += 2 + seg_len;
+ }
+ }
+ return ret;
+}
+
+private int beInt(const(ubyte)[] b) {
+ return cast(int)((b[0] << 24) | (b[1] << 16) | (b[2] << 8) | b[3]);
+}
diff --git a/gleam/README.md b/gleam/README.md
index d586bd3..482db38 100644
--- a/gleam/README.md
+++ b/gleam/README.md
@@ -1,15 +1,18 @@
# ocda-lib, in Gleam
-The abstraction's model and both its loaders, as a Gleam library, and the
-two programs that put it to the question.
-
-| | |
-|-----------------|----------------------------------------------------------------------------------|
-| `ocda/model` | the document, the object, the image. Depends on nothing but the standard library |
-| `ocda/load_ssp` | text -> model |
-| `ocda/load_db` | sqlite -> model |
-| `ocda/load` | what a path is, and loading it |
-| `ocda/dump` | the model written out canonically, as an oracle |
+The abstraction's model and both its loaders, as a Gleam library, with its
+reader and the two programs that put it to the question.
+
+| | |
+|------------------|-----------------------------------------------------------------------------------------|
+| `ocda/model` | the document, the object, the image. Depends on nothing but the standard library |
+| `ocda/load_ssp` | text -> model |
+| `ocda/load_db` | sqlite -> model |
+| `ocda/load` | what a path is, and loading it |
+| `ocda/inventory` | what a file holds beyond the one document: its languages, and what a `.ocda.db` carries |
+| `ocda/dump` | the model written out canonically, as an oracle |
+| `read/main` | `ocda-read`, the reader |
+| `ocda_ffi.erl` | a sha256, from OTP's crypto, which the standard library lacks |
## Use
@@ -27,6 +30,23 @@ case load.load(path, "en", False) {
}
```
+## The reader
+
+```sh
+nix build
+./result/bin/ocda-read [--tree|--outline|--languages|--files|--report-unknown]
+ <file>.ssp|<file>.ocda.db|<file>.zip [lang]
+```
+
+`ocda-lib read`, installed also as `ocda-read`. The modes the Gleam readers
+had, over the library, so each reads every form this library reads; the
+others are the D reader's. **Its output is the D `ocda-read`'s, byte for
+byte**, in every mode both have. In `src/read/`, because Gleam compiles
+nothing outside `src/`; a program depending on the library compiles it too,
+and does not use it.
+
+A `.sisupod` it refuses, saying why: the zstd frame, below.
+
## The two programs
```sh
@@ -97,6 +117,6 @@ one and the other must change with it.
## Format
-Specified in **sisudoc-abstraction-format**: `ocda-model.md` for the model,
-`ssp-format-specification.md` and `ocda-db-format-specification.md` for the
+Specified in **sisudoc-abstraction-format**: `ocda-model.org` for the model,
+`ssp-format-specification.org` and `ocda-db-format-specification.org` for the
two serialisations. At format **2.x**.
diff --git a/gleam/flake.lock b/gleam/flake.lock
index 5f236ab..12c4c3b 100644
--- a/gleam/flake.lock
+++ b/gleam/flake.lock
@@ -2,18 +2,15 @@
"nodes": {
"nixpkgs": {
"locked": {
- "lastModified": 1790185690,
- "narHash": "sha256-xJ+X4hBtOcAFGBOe5nAMyMUeF9foJBmIOu3NjBqBycU=",
- "owner": "NixOS",
- "repo": "nixpkgs",
- "rev": "4975466d324710c576dc11ad614684e6bd8cad8e",
- "type": "github"
+ "lastModified": 1790652569,
+ "narHash": "sha256-k8Fu4c9Z+4Nh7mUr0cfw++ITQiyEhlWxoJBOkI3tOcQ=",
+ "rev": "b6c8664de9b6cc07fe5666a29f91884ba81197c4",
+ "type": "tarball",
+ "url": "https://releases.nixos.org/nixpkgs/nixpkgs-26.11pre1082290.b6c8664de9b6/nixexprs.tar.zst"
},
"original": {
- "owner": "NixOS",
- "ref": "nixos-unstable",
- "repo": "nixpkgs",
- "type": "github"
+ "type": "tarball",
+ "url": "https://channels.nixos.org/nixpkgs-unstable/nixexprs.tar.zst"
}
},
"root": {
diff --git a/gleam/flake.nix b/gleam/flake.nix
index a5a3407..132e3fc 100644
--- a/gleam/flake.nix
+++ b/gleam/flake.nix
@@ -1,6 +1,6 @@
{
description = "ocda-lib: the object-centric document abstraction as a Gleam library";
- inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
+ inputs.nixpkgs.url = "https://channels.nixos.org/nixpkgs-unstable/nixexprs.tar.zst";
outputs = { self, nixpkgs }:
let
system = "x86_64-linux"; # adjust for aarch64-linux / darwin
@@ -51,10 +51,14 @@
makeWrapper $out/lib/${name}/entrypoint.sh $out/bin/ocda-lib \
--add-flags run \
--prefix PATH : ${pkgs.lib.makeBinPath [ pkgs.beamPackages.erlang pkgs.coreutils ]}
+ # the reader under its own name, as the D flake installs it
+ makeWrapper $out/lib/${name}/entrypoint.sh $out/bin/ocda-read \
+ --add-flags "run read" \
+ --prefix PATH : ${pkgs.lib.makeBinPath [ pkgs.beamPackages.erlang pkgs.coreutils ]}
runHook postInstall
'';
meta = {
- description = "the abstraction as a Gleam library";
+ description = "the abstraction as a Gleam library, with its reader, dump and agreement check";
mainProgram = "ocda-lib";
};
};
diff --git a/gleam/src/ocda/inventory.gleam b/gleam/src/ocda/inventory.gleam
new file mode 100644
index 0000000..6047313
--- /dev/null
+++ b/gleam/src/ocda/inventory.gleam
@@ -0,0 +1,138 @@
+//// What a source holds, beyond the one document the model is loaded from.
+////
+//// The model is one language of one document. Two questions a reader asks
+//// are about the file instead: which languages it holds, and, for a
+//// database, what else it carries. They are here rather than in a reader
+//// because the answer depends on the input form, which is the library's to
+//// know, and so that a reader asking them does not learn which form it was
+//// handed. A mirror of the D library's `ocda.inventory`.
+////
+//// `languages` answers for every form, and answers the same: the languages
+//// in the order the manifest gave them (which every `.ssp` records, in
+//// `@source languages`), each with the markup file it was made from and the
+//// sha256 of its `.ssp`. The database stores that digest; for a `.ssp` and a
+//// pod it is taken from the bytes, which are the same bytes.
+////
+//// `carried` is the database's: a `.ocda.db` holds the markup, images,
+//// configuration and catalogues it was built from, a `.ssp` holds none of
+//// them, and a pod's are its archive entries rather than anything the
+//// abstraction says.
+
+import gleam/bit_array
+import gleam/dict
+import gleam/list
+import gleam/result
+import gleam/string
+import ocda/load
+import ocda/load_db
+import ocda/load_pod
+import ocda/load_ssp
+import ocda/zip
+import simplifile
+
+pub type Language {
+ Language(lang: String, source_filename: String, ssp_digest: String)
+}
+
+pub type Carried {
+ Carried(
+ role: String,
+ compression: String,
+ count: String,
+ bytes: String,
+ stored: String,
+ )
+}
+
+pub fn languages(path: String) -> Result(List(Language), String) {
+ case load.source_of(path) {
+ load.OcdaDb -> {
+ use held <- result.try(load_db.languages_of_file(path))
+ Ok(
+ list.map(held, fn(l) {
+ Language(l.lang, l.source_filename, l.ssp_digest)
+ }),
+ )
+ }
+ load.Ssp -> {
+ use bytes <- result.try(
+ simplifile.read_bits(path)
+ |> result.map_error(fn(e) { path <> ": " <> simplifile.describe_error(e) }),
+ )
+ use text <- result.try(
+ bit_array.to_string(bytes)
+ |> result.replace_error(path <> ": not valid UTF-8"),
+ )
+ let doc = load_ssp.parse(text)
+ Ok([
+ Language(
+ source_value(doc.source, "language"),
+ doc.source_filename,
+ sha256_hex(bytes),
+ ),
+ ])
+ }
+ load.Pod -> {
+ use archive <- result.try(load_pod.archive_bytes(path))
+ use entries <- result.try(zip.entries(archive))
+ use found <- result.try(
+ zip.find_suffix(entries, ".ssp")
+ |> list.try_map(fn(e) {
+ use bytes <- result.try(zip.read(archive, e))
+ use text <- result.try(
+ bit_array.to_string(bytes)
+ |> result.replace_error(e.name <> ": not valid UTF-8"),
+ )
+ let doc = load_ssp.parse(text)
+ let lang = result.unwrap(load_pod.lang_of(e.name), "")
+ Ok(#(
+ Language(lang, doc.source_filename, sha256_hex(bytes)),
+ string.split(source_value(doc.source, "languages"), " "),
+ ))
+ }),
+ )
+ let order = case found {
+ [#(_, o), ..] -> list.filter(o, fn(l) { l != "" })
+ [] -> []
+ }
+ let by_lang = list.map(found, fn(f) { #({ f.0 }.lang, f.0) }) |> dict.from_list
+ // the manifest's order, and anything it did not name after it
+ let named = list.filter_map(order, fn(l) { dict.get(by_lang, l) })
+ let rest =
+ list.filter(found, fn(f) { !list.contains(order, { f.0 }.lang) })
+ |> list.map(fn(f) { f.0 })
+ Ok(list.append(named, rest))
+ }
+ load.Unknown -> Error(path <> ": " <> load.source_name(load.Unknown))
+ }
+}
+
+/// Only a `.ocda.db` carries files.
+pub fn is_carrier(path: String) -> Bool {
+ load.source_of(path) == load.OcdaDb
+}
+
+pub fn carried(path: String) -> Result(List(Carried), String) {
+ case is_carrier(path) {
+ False -> Ok([])
+ True -> {
+ use rows <- result.try(load_db.carried_of_file(path))
+ Ok(
+ list.filter_map(rows, fn(r) {
+ case r {
+ [role, compression, n, bytes, stored] ->
+ Ok(Carried(role, compression, n, bytes, stored))
+ _ -> Error(Nil)
+ }
+ }),
+ )
+ }
+ }
+}
+
+fn source_value(source: dict.Dict(String, String), key: String) -> String {
+ result.unwrap(dict.get(source, key), "")
+}
+
+@external(erlang, "ocda_ffi", "sha256_hex")
+fn sha256_hex(bytes: BitArray) -> String
diff --git a/gleam/src/ocda/load_db.gleam b/gleam/src/ocda/load_db.gleam
index 641923e..3ae4f0b 100644
--- a/gleam/src/ocda/load_db.gleam
+++ b/gleam/src/ocda/load_db.gleam
@@ -5,7 +5,7 @@
//// then a function from the model to bytes and cannot tell which
//// serialisation it was given, exactly as spine's own writers cannot.
////
-//// Written against `ocda-db-format-specification.md`, whose section 4
+//// Written against `ocda-db-format-specification.org`, whose section 4
//// gives the column by column correspondence this module implements.
import gleam/dict
@@ -112,6 +112,7 @@ pub fn load_file(
)
let document = metadata(conn, chosen.doc_id, document)
let document = objects(conn, chosen.doc_id, document)
+ let document = count_unread(conn, chosen.doc_id, document)
Ok(document)
}
@@ -561,3 +562,108 @@ fn json_joined(value: String) -> String {
parts -> string.join(parts, "|")
}
}
+
+/// What the database holds and the model does not.
+///
+/// The text loader meets every property line and counts those it does not
+/// keep, so a reader can say what was read and not understood. This loader
+/// asks for columns by name, and would never see the ones it did not ask
+/// for: it would report that it had understood everything, which is the
+/// silence the count exists to prevent. So it counts them, on the
+/// conditions under which the `.ssp` writes the same property
+/// (specification section 6.3), and the report is the same from either
+/// form, as the D loader's is.
+fn count_unread(
+ conn: sqlight.Connection,
+ doc_id: Int,
+ document: Document,
+) -> Document {
+ let id = int.to_string(doc_id)
+ let counts = [
+ #("ancestors_collapsed", "coalesce(ancestors_collapsed,'') != ''"),
+ #("dom_status", "coalesce(dom_status,'') != ''"),
+ #("dom_status_collapsed", "coalesce(dom_status_collapsed,'') != ''"),
+ #("heading_lev_collapsed", "heading_lev_collapsed is not null"),
+ #("parent_lev", "coalesce(parent_lev,0) != 0"),
+ #("segment_lv4_is", "coalesce(segment_lv4_is,'') != ''"),
+ #("attrib", "coalesce(attrib,'') != ''"),
+ ]
+ let property =
+ list.fold(counts, document.unknown.property, fn(acc, c) {
+ let n =
+ count(
+ conn,
+ "select count(*) from objects where doc_id = " <> id <> " and " <> c.1,
+ )
+ case n > 0 {
+ True -> dict.insert(acc, c.0, n)
+ False -> acc
+ }
+ })
+ let property = case has(conn, "object_subtoc") {
+ False -> property
+ True -> {
+ let n =
+ count(
+ conn,
+ "select count(*) from object_subtoc s join objects o
+ on o.id = s.object_id where o.doc_id = " <> id,
+ )
+ case n > 0 {
+ True -> dict.insert(property, "lev4_subtoc", n)
+ False -> property
+ }
+ }
+ }
+ model.Document(
+ ..document,
+ unknown: model.Unknown(..document.unknown, property: property),
+ )
+}
+
+fn count(conn: sqlight.Connection, sql: String) -> Int {
+ let d = decode.field(0, decode.int, decode.success)
+ case sqlight.query(sql, on: conn, with: [], expecting: d) {
+ Ok([n, ..]) -> n
+ _ -> 0
+ }
+}
+
+/// Every language the database at `path` holds, in manifest order.
+pub fn languages_of_file(path: String) -> Result(List(Language), String) {
+ use conn <- with_connection("file:" <> path <> "?mode=ro", path)
+ Ok(languages(conn))
+}
+
+/// What the database at `path` carries besides the abstraction, by role
+/// and storage: role, compression, count, original bytes, stored bytes.
+pub fn carried_of_file(path: String) -> Result(List(List(String)), String) {
+ use conn <- with_connection("file:" <> path <> "?mode=ro", path)
+ case has(conn, "files") {
+ False -> Ok([])
+ True -> {
+ let d = {
+ use role <- decode.field(0, decode.string)
+ use compression <- decode.field(1, decode.string)
+ use n <- decode.field(2, decode.int)
+ use bytes <- decode.field(3, decode.int)
+ use stored <- decode.field(4, decode.int)
+ decode.success([
+ role,
+ compression,
+ int.to_string(n),
+ int.to_string(bytes),
+ int.to_string(stored),
+ ])
+ }
+ sqlight.query(
+ "select role, compression, count(*), sum(bytes), sum(length(data))
+ from files group by role, compression order by role",
+ on: conn,
+ with: [],
+ expecting: d,
+ )
+ |> result.map_error(fn(e) { path <> ": " <> e.message })
+ }
+ }
+}
diff --git a/gleam/src/ocda/load_pod.gleam b/gleam/src/ocda/load_pod.gleam
index 35483c6..05f7add 100644
--- a/gleam/src/ocda/load_pod.gleam
+++ b/gleam/src/ocda/load_pod.gleam
@@ -89,7 +89,7 @@ fn langs(entries: List(zip.Entry)) -> List(String) {
}
/// `<uid>.<lang>.ssp` -> `<lang>`
-fn lang_of(name: String) -> Result(String, Nil) {
+pub fn lang_of(name: String) -> Result(String, Nil) {
case string.ends_with(name, ".ssp") {
False -> Error(Nil)
True -> {
@@ -104,7 +104,7 @@ fn lang_of(name: String) -> Result(String, Nil) {
/// The bytes decide, not the name: a file misnamed either way is treated by
/// what it holds.
-fn archive_bytes(path: String) -> Result(BitArray, String) {
+pub fn archive_bytes(path: String) -> Result(BitArray, String) {
case simplifile.read_bits(path) {
Error(e) -> Error(path <> ": " <> simplifile.describe_error(e))
Ok(bytes) ->
@@ -133,6 +133,6 @@ fn is_zstd_frame(bytes: BitArray) -> Bool {
@external(erlang, "unicode", "characters_to_binary")
fn bit_array_to_string_ffi(b: BitArray) -> String
-fn bit_array_to_string(b: BitArray) -> Result(String, Nil) {
+pub fn bit_array_to_string(b: BitArray) -> Result(String, Nil) {
Ok(bit_array_to_string_ffi(b))
}
diff --git a/gleam/src/ocda/load_ssp.gleam b/gleam/src/ocda/load_ssp.gleam
index ec93ef5..324319e 100644
--- a/gleam/src/ocda/load_ssp.gleam
+++ b/gleam/src/ocda/load_ssp.gleam
@@ -4,7 +4,7 @@
//// apart by its first character, so the loader is a fold over the lines
//// carrying the state the shapes need.
////
-//// Written against `ssp-format-specification.md`. Where it and the D
+//// Written against `ssp-format-specification.org`. Where it and the D
//// library differ the D library is the bug, or this is: they are meant to
//// give the same model, and `ocda-dump` says whether they do.
diff --git a/gleam/src/ocda_ffi.erl b/gleam/src/ocda_ffi.erl
new file mode 100644
index 0000000..2e395cd
--- /dev/null
+++ b/gleam/src/ocda_ffi.erl
@@ -0,0 +1,7 @@
+%% The one thing ocda-lib asks of Erlang directly: a sha256, which OTP's
+%% crypto has and the Gleam standard library does not. Upper case hex, as
+%% the database stores it and the D library writes it.
+-module(ocda_ffi).
+-export([sha256_hex/1]).
+
+sha256_hex(Bin) -> binary:encode_hex(crypto:hash(sha256, Bin)).
diff --git a/gleam/src/ocda_lib.gleam b/gleam/src/ocda_lib.gleam
index 1e8a5ae..6eccaff 100644
--- a/gleam/src/ocda_lib.gleam
+++ b/gleam/src/ocda_lib.gleam
@@ -1,10 +1,11 @@
//// ocda-lib, in Gleam: the abstraction as a library, put to the question.
////
-//// A library has no binary, so this is the two programs that test its
-//// claims.
+//// A library has no binary, so this is the programs over it: the reader,
+//// and the two that test its claims.
////
//// ocda-lib dump [--unknown] <file>.ssp|<file>.ocda.db [lang]
//// ocda-lib agree <doc>.ssp <doc>.ocda.db [lang]
+//// ocda-lib read [options] <file> [lang] (also installed as ocda-read)
////
//// `dump` writes the whole loaded model out canonically, byte for byte as
//// the D library's `ocda-dump` writes it, so that the two implementations
@@ -20,6 +21,7 @@ import gleam/string
import ocda/dump
import ocda/load
import ocda/model.{type Document}
+import read/main as reader
pub fn main() {
case argv.load().arguments {
@@ -29,6 +31,11 @@ pub fn main() {
["dump", path, lang] -> dump_file(path, lang, False)
["agree", ssp, db] -> agree(ssp, db, "")
["agree", ssp, db, lang] -> agree(ssp, db, lang)
+ ["read", ..rest] ->
+ case reader.run(rest) {
+ 0 -> Nil
+ code -> halt(code)
+ }
_ -> usage()
}
}
@@ -36,7 +43,8 @@ pub fn main() {
fn usage() -> Nil {
io.println_error(
"usage: ocda-lib dump [--unknown] <file>.ssp|<file>.ocda.db [lang]
- ocda-lib agree <doc>.ssp <doc>.ocda.db [lang]",
+ ocda-lib agree <doc>.ssp <doc>.ocda.db [lang]
+ ocda-lib read [--help] [options] <file> [lang]",
)
halt(2)
}
diff --git a/gleam/src/read/main.gleam b/gleam/src/read/main.gleam
new file mode 100644
index 0000000..307b8d8
--- /dev/null
+++ b/gleam/src/read/main.gleam
@@ -0,0 +1,459 @@
+//// ocda-read, in Gleam: reads the abstraction, in any of the forms the
+//// library reads, and says what is in it.
+////
+//// ocda-read [--tree|--outline|--languages|--files|--report-unknown]
+//// <file>.ssp | <file>.ocda.db | <file>.zip [lang]
+////
+//// The modes the Gleam readers had (the summary, the outline, the unknown
+//// report, and the database's languages and files), over the library, so
+//// that each reads every form. Their output is the D ocda-read's, byte for
+//// byte: two readers in unrelated languages, neither written from the other
+//// line by line, agreeing is what says each has read the model and not its
+//// own idea of it. A `.sisupod` is the one form the Gleam library cannot
+//// open (its zstd frame; see `ocda/load_pod`), and it says so.
+
+import gleam/dict
+import gleam/int
+import gleam/io
+import gleam/list
+import gleam/result
+import gleam/string
+import ocda/inventory.{type Language}
+import ocda/load
+import ocda/model.{type Document, type Object}
+
+/// The format major this reader knows.
+const major_read = "2"
+
+type Options {
+ Options(
+ tree: Bool,
+ languages: Bool,
+ files: Bool,
+ report_unknown: Bool,
+ any_version: Bool,
+ lang: String,
+ positional: List(String),
+ )
+}
+
+/// Run with the arguments after the program name; returns the exit status.
+pub fn run(args: List(String)) -> Int {
+ case parse(args, Options(False, False, False, False, False, "", [])) {
+ Error(message) -> {
+ io.println_error("ocda-read: " <> message <> "; --help lists the options")
+ 2
+ }
+ Ok(Options(positional: [], ..)) -> {
+ help()
+ 0
+ }
+ Ok(options) -> read(options)
+ }
+}
+
+fn parse(args: List(String), o: Options) -> Result(Options, String) {
+ case args {
+ [] -> Ok(Options(..o, positional: list.reverse(o.positional)))
+ ["--help", ..] | ["-h", ..] -> Ok(Options(..o, positional: []))
+ ["--tree", ..rest] | ["--outline", ..rest] ->
+ parse(rest, Options(..o, tree: True))
+ ["--languages", ..rest] -> parse(rest, Options(..o, languages: True))
+ ["--files", ..rest] -> parse(rest, Options(..o, files: True))
+ ["--report-unknown", ..rest] ->
+ parse(rest, Options(..o, report_unknown: True))
+ ["--any-version", ..rest] -> parse(rest, Options(..o, any_version: True))
+ ["--lang", code, ..rest] -> parse(rest, Options(..o, lang: code))
+ [flag, ..rest] ->
+ case string.starts_with(flag, "--lang=") {
+ True -> parse(rest, Options(..o, lang: string.drop_start(flag, 7)))
+ False ->
+ case string.starts_with(flag, "-") {
+ True ->
+ case
+ list.contains(
+ [
+ "--html", "--llm-text", "--llm-jsonl", "--images", "--verify",
+ "--validate",
+ ],
+ flag,
+ )
+ {
+ True ->
+ Error(flag <> " is a mode of the D ocda-read, not of this one")
+ False -> Error("Unrecognized option " <> flag)
+ }
+ False ->
+ parse(rest, Options(..o, positional: [flag, ..o.positional]))
+ }
+ }
+ }
+}
+
+fn help() -> Nil {
+ io.println("ocda-read - reads the document abstraction, in any of its forms")
+ io.println("")
+ io.println(
+ "Usage: ocda-read [options] <file>.ssp|<file>.ocda.db|<file>.zip [lang]",
+ )
+ io.println("")
+ io.println(" --tree the headings, as a tree")
+ io.println(" --outline the same as --tree")
+ io.println(" --languages the languages the file holds")
+ io.println(" --files what a .ocda.db carries besides the abstraction")
+ io.println(" --report-unknown what was read and not understood")
+ io.println(" --lang=<code> the language to read (default: the first)")
+ io.println(" --any-version read a format major this reader does not know")
+}
+
+fn read(o: Options) -> Int {
+ let assert [path, ..more] = o.positional
+ let lang = case o.lang, more {
+ "", [given, ..] -> given
+ l, _ -> l
+ }
+ let lang_named = lang != ""
+ let loaded = {
+ use held <- result.try(inventory.languages(path))
+ let lang = case lang, held {
+ "", [first, ..] -> first.lang
+ l, _ -> l
+ }
+ use _ <- result.try(case
+ held != [] && !list.any(held, fn(l) { l.lang == lang })
+ {
+ True ->
+ Error(
+ "no language '"
+ <> lang
+ <> "' in "
+ <> path
+ <> "; it holds: "
+ <> string.join(list.map(held, fn(l) { l.lang }), " "),
+ )
+ False -> Ok(Nil)
+ })
+ use doc <- result.try(load.load(path, lang, o.any_version))
+ Ok(#(held, lang, doc))
+ }
+ case loaded {
+ Error(message) -> {
+ io.println_error("Error reading " <> path <> ": " <> message)
+ 2
+ }
+ Ok(#(held, lang, doc)) ->
+ case gate(path, doc, o.any_version) {
+ False -> 3
+ True -> {
+ summary(path, doc, held, lang, lang_named)
+ case o.languages {
+ True -> print_languages(held)
+ False -> Nil
+ }
+ case o.files {
+ True -> print_files(path)
+ False -> Nil
+ }
+ case o.tree {
+ True -> tree(doc)
+ False -> Nil
+ }
+ case o.report_unknown {
+ True -> report_unknown(doc)
+ False -> Nil
+ }
+ case o.tree || o.languages || o.files || o.report_unknown {
+ True -> Nil
+ False ->
+ io.println(
+ "\n(this is the summary: --help lists the modes, each adding its output)",
+ )
+ }
+ 0
+ }
+ }
+ }
+}
+
+/// The major part, and refuse what is not known.
+fn gate(path: String, doc: Document, any_version: Bool) -> Bool {
+ case doc.format_version, model.version_major(doc) == major_read, any_version {
+ "", _, _ -> {
+ io.println_error("warning: " <> path <> " declares no format version")
+ True
+ }
+ _, True, _ -> True
+ v, False, True -> {
+ io.println_error(
+ "warning: "
+ <> path
+ <> " is format v"
+ <> v
+ <> ", this reader knows v"
+ <> major_read
+ <> ".x; reading anyway",
+ )
+ True
+ }
+ v, False, False -> {
+ io.println_error(
+ "error: "
+ <> path
+ <> " is format v"
+ <> v
+ <> ", this reader knows v"
+ <> major_read
+ <> ".x",
+ )
+ io.println_error(
+ " a major version differs: a property may have changed meaning or gone away.",
+ )
+ io.println_error(" --any-version reads it regardless.")
+ False
+ }
+ }
+}
+
+const sections = [
+ "head", "toc", "body", "endnotes", "glossary", "bibliography", "bookindex",
+ "blurb", "tail",
+]
+
+/// The sections whose images are counted: every one but the tail, as the D
+/// reader counts them.
+const image_sections = [
+ "head", "toc", "body", "endnotes", "glossary", "bibliography", "bookindex",
+ "blurb",
+]
+
+fn summary(
+ path: String,
+ doc: Document,
+ held: List(Language),
+ lang: String,
+ lang_named: Bool,
+) -> Nil {
+ io.println("Read: " <> path)
+ case doc.format_version {
+ "" -> Nil
+ v -> io.println(" Format: v" <> v)
+ }
+ case dict.get(doc.source, "language") {
+ Error(_) -> Nil
+ Ok(language) -> {
+ let of = case dict.get(doc.source, "languages") {
+ Ok(ls) -> " of " <> ls
+ Error(_) -> ""
+ }
+ let digest = case dict.get(doc.source, "digest") {
+ Ok(d) -> ", markup digest " <> string.slice(d, 0, 16) <> "..."
+ Error(_) -> ""
+ }
+ io.println(" Source: " <> language <> of <> digest)
+ }
+ }
+ case list.length(held) > 1 && !lang_named {
+ True -> io.println(" (reading " <> lang <> "; --lang=<code> for another)")
+ False -> Nil
+ }
+ io.println(" Title: " <> model.meta(doc, "title.main"))
+ io.println(" Author: " <> model.meta(doc, "creator.author"))
+ io.println(
+ " Sections: "
+ <> string.join(
+ list.map(sections, fn(s) {
+ s <> "=" <> int.to_string(list.length(model.section(doc, s)))
+ }),
+ " ",
+ ),
+ )
+ let records =
+ list.fold(image_sections, 0, fn(n, s) {
+ list.fold(model.section(doc, s), n, fn(n, o) {
+ n + list.length(o.images)
+ })
+ })
+ case records {
+ 0 -> Nil
+ 1 -> io.println(" Images: 1 record")
+ n -> io.println(" Images: " <> int.to_string(n) <> " records")
+ }
+}
+
+fn print_languages(held: List(Language)) -> Nil {
+ io.println("")
+ io.println(" lang source ssp digest")
+ list.each(held, fn(l) {
+ let digest = case string.length(l.ssp_digest) > 16 {
+ True -> string.slice(l.ssp_digest, 0, 16) <> "..."
+ False -> l.ssp_digest
+ }
+ io.println(
+ " "
+ <> string.pad_end(l.lang, 7, " ")
+ <> " "
+ <> string.pad_end(l.source_filename, 31, " ")
+ <> " "
+ <> digest,
+ )
+ })
+}
+
+fn print_files(path: String) -> Nil {
+ io.println("")
+ case inventory.is_carrier(path) {
+ False ->
+ io.println(
+ " carried files: only a .ocda.db carries files; this is "
+ <> load.source_name(load.source_of(path)),
+ )
+ True ->
+ case inventory.carried(path) {
+ Error(message) -> io.println_error("error: " <> message)
+ Ok([]) -> io.println(" carries no files")
+ Ok(rows) -> {
+ io.println(" carried files:")
+ io.println(" role stored n original in the file")
+ list.each(rows, fn(r) {
+ io.println(
+ " "
+ <> string.pad_end(r.role, 9, " ")
+ <> " "
+ <> string.pad_end(r.compression, 7, " ")
+ <> " "
+ <> string.pad_end(r.count, 5, " ")
+ <> " "
+ <> string.pad_end(r.bytes, 13, " ")
+ <> " "
+ <> r.stored,
+ )
+ })
+ }
+ }
+ }
+}
+
+/// The headings, as a tree: from the document's root heading, down the
+/// children each heading lists, as the D reader walks it.
+fn tree(doc: Document) -> Nil {
+ io.println("Document: " <> model.meta(doc, "title.main"))
+ case model.meta(doc, "title.subtitle") {
+ "" -> Nil
+ sub -> io.println(" " <> sub)
+ }
+ io.println("Author: " <> model.meta(doc, "creator.author"))
+ io.println("")
+ let headings =
+ model.section(doc, "body")
+ |> list.filter(fn(o) { o.is_a == "heading" && o.ocn > 0 })
+ |> list.map(fn(o) { #(o.ocn, o) })
+ |> dict.from_list
+ case model.section(doc, "head") {
+ [root, ..] if root.is_a == "heading" -> {
+ io.println(heading_line(root, 0))
+ walk(root, headings, 1)
+ }
+ _ -> Nil
+ }
+}
+
+fn walk(parent: Object, headings: dict.Dict(Int, Object), depth: Int) -> Nil {
+ list.each(parent.children, fn(ocn) {
+ case dict.get(headings, ocn) {
+ Ok(child) -> {
+ io.println(heading_line(child, depth))
+ walk(child, headings, depth + 1)
+ }
+ Error(_) -> Nil
+ }
+ })
+}
+
+fn heading_line(o: Object, depth: Int) -> String {
+ let level = case o.heading_level {
+ "" -> "?"
+ l -> l
+ }
+ let ocn = case o.ocn > 0 {
+ True -> " [" <> int.to_string(o.ocn) <> "]"
+ False -> ""
+ }
+ string.repeat(" ", depth) <> level <> "~" <> ocn <> " " <> strip_markup(o.text)
+}
+
+/// The internal markers a heading's text may carry, removed for display, as
+/// the D reader removes them.
+fn strip_markup(text: String) -> String {
+ text
+ |> string.replace("⑆*", "")
+ |> string.replace("⑆!", "")
+ |> string.replace("⑆/", "")
+ |> string.replace("⑆_", "")
+ |> string.replace("⑆^", "")
+ |> string.replace("⑆,", "")
+ |> string.replace("┨", "")
+ |> string.replace("┣", "")
+ |> string.replace("┙", " ")
+ |> string.replace("┘", "\n")
+}
+
+/// Say what was read and not understood.
+fn report_unknown(doc: Document) -> Nil {
+ let u = doc.unknown
+ case model.any_unknown(u) {
+ False ->
+ io.println(
+ "\nread and understood: every property, block and line was known",
+ )
+ True -> {
+ io.println("\nread and NOT understood:")
+ section_of_unknown(u.property, " properties ignored (", fn(name, n) {
+ " ."
+ <> string.pad_end(name, 26, " ")
+ <> " "
+ <> string.pad_start(int.to_string(n), 8, " ")
+ })
+ section_of_unknown(u.block, " blocks ignored (", fn(name, n) {
+ " @"
+ <> string.pad_end(name, 26, " ")
+ <> " "
+ <> string.pad_start(int.to_string(n), 8, " ")
+ })
+ case dict.size(u.line) {
+ 0 -> Nil
+ _ -> {
+ io.println(" lines of no known shape, by first character:")
+ dict.to_list(u.line)
+ |> list.sort(fn(a, b) { string.compare(a.0, b.0) })
+ |> list.each(fn(e) {
+ io.println(
+ " "
+ <> string.pad_end("\"" <> e.0 <> "\"", 27, " ")
+ <> " "
+ <> string.pad_start(int.to_string(e.1), 8, " "),
+ )
+ })
+ }
+ }
+ io.println(
+ " (ignoring these is conforming within a major version; the count is so that it is not silent)",
+ )
+ }
+ }
+}
+
+fn section_of_unknown(
+ counts: dict.Dict(String, Int),
+ heading: String,
+ line: fn(String, Int) -> String,
+) -> Nil {
+ case dict.size(counts) {
+ 0 -> Nil
+ n -> {
+ io.println(heading <> int.to_string(n) <> " distinct):")
+ dict.to_list(counts)
+ |> list.sort(fn(a, b) { string.compare(a.0, b.0) })
+ |> list.each(fn(e) { io.println(line(e.0, e.1)) })
+ }
+ }
+}
diff --git a/gleam/test/ocda_lib_test.gleam b/gleam/test/ocda_lib_test.gleam
index 02fe463..c0e10e5 100644
--- a/gleam/test/ocda_lib_test.gleam
+++ b/gleam/test/ocda_lib_test.gleam
@@ -36,7 +36,7 @@ const minimal = "% SiSU Document Abstraction v2.0
.part: body
.parent: 1
| First line
-|
+|
| Third line, after a blank one
}