summaryrefslogtreecommitdiffhomepage
diff options
context:
space:
mode:
Diffstat
-rw-r--r--README.md42
-rw-r--r--dlang/README.md45
-rw-r--r--dlang/dub.json16
-rw-r--r--dlang/flake.lock17
-rw-r--r--dlang/flake.nix56
-rw-r--r--dlang/src/ocda/inventory.d117
-rw-r--r--dlang/src/ocda/load_db.d33
-rw-r--r--dlang/src/ocda/load_pod.d4
-rw-r--r--dlang/src/read/html.d388
-rw-r--r--dlang/src/read/images.d114
-rw-r--r--dlang/src/read/llm_jsonl.d121
-rw-r--r--dlang/src/read/llm_text.d182
-rw-r--r--dlang/src/read/main.d228
-rw-r--r--dlang/src/read/report_unknown.d41
-rw-r--r--dlang/src/read/tree.d69
-rw-r--r--dlang/src/read/validate.d265
-rw-r--r--dlang/src/read/verify.d214
-rw-r--r--gleam/README.md44
-rw-r--r--gleam/flake.lock17
-rw-r--r--gleam/flake.nix8
-rw-r--r--gleam/src/ocda/inventory.gleam138
-rw-r--r--gleam/src/ocda/load_db.gleam108
-rw-r--r--gleam/src/ocda/load_pod.gleam6
-rw-r--r--gleam/src/ocda/load_ssp.gleam2
-rw-r--r--gleam/src/ocda_ffi.erl7
-rw-r--r--gleam/src/ocda_lib.gleam14
-rw-r--r--gleam/src/read/main.gleam459
-rw-r--r--gleam/test/ocda_lib_test.gleam2
28 files changed, 2674 insertions, 83 deletions
diff --git a/README.md b/README.md
index 5e328c0..ad3a9a9 100644
--- a/README.md
+++ b/README.md
@@ -16,8 +16,10 @@ at the direction of and reviewed by Ralph Amissah.
## What it is for
-The readers in `sisudoc-ssp-reader` and `sisudoc-ocda-db-reader` each read
-one serialisation and print. A **writer** needs neither: it needs the document.
+The first programs written from the specification were two readers,
+`sisudoc-ssp-reader` and `sisudoc-ocda-db-reader`, each reading one
+serialisation and printing. A **writer** needs neither: it needs the
+document.
```
.ssp ─┐
@@ -29,8 +31,38 @@ So this library holds the model and both loaders, and **a writer cannot
tell which serialisation it was handed** - exactly as spine's own writers
cannot, since spine 0.25.0.
-That is the whole of the idea. The readers become thin programs over this,
-and the writers are functions from the model to bytes.
+That is the whole of the idea. The writers are functions from the model to
+bytes, and the reader is a program over the model.
+
+## The reader
+
+```sh
+ocda-read [--tree|--html|--llm-text|--llm-jsonl|--images|--verify --image-dir=<d>|
+ --languages|--files|--report-unknown|--validate]
+ <file>.ssp | <file>.ocda.db | <file>.sisupod [lang]
+```
+
+One reader, in each language, where the two earlier readers had a parser
+each. Built on this library, **every mode reads every form, and every mode
+that describes the document says the same thing of its `.ssp`, `.ocda.db`
+and `.sisupod`**, 36 of 36 document-languages. Two modes describe the file
+instead, and differ as the files do: `--languages` lists the one language a
+`.ssp` holds and every language of the other two, and `--files` finds
+carried files only in a `.ocda.db`. The D reader has every mode of both
+old readers; the Gleam one the modes the Gleam readers had (the summary,
+`--tree`, `--languages`, `--files`, `--report-unknown`), and in those **its
+output is the D one's, byte for byte**, 36 of 36.
+
+It is in each language's `src/read/`, and not part of the library: a
+program using the library does not build it (in D, by dub configuration;
+in Gleam a dependant compiles it and does not use it, Gleam building
+everything in `src/`).
+
+Writing it found one defect in the library: **the database loader could
+not say what it had not understood**. It asks for columns by name, and so
+never saw the ones it did not ask for, and reported that it had understood
+everything. It now counts them on the conditions under which the `.ssp`
+writes the same property, and the report is the same from either form.
## The checks that matter
@@ -72,7 +104,7 @@ Nothing else had caught it. The two `.ssp` readers had been compared on
*section counts*, which this does not change. It took loading the same
document two ways and comparing the objects.
-Fixed here and in `sisudoc-ssp-reader/dlang`, which carried the same code.
+Fixed here and in the `.ssp` reader, which carried the same code.
### the whole-model dump: four more
diff --git a/dlang/README.md b/dlang/README.md
index 124e4d6..65d762e 100644
--- a/dlang/README.md
+++ b/dlang/README.md
@@ -4,14 +4,16 @@ The abstraction's model and both its loaders, as a dub library.
## Modules
-| | |
-|-----------------|---------------------------------------------------------------|
-| `ocda.model` | the document, the object, the eight sections. Imports nothing |
-| `ocda.load_ssp` | text -> model |
-| `ocda.load_db` | sqlite -> model |
-| `ocda.load` | what a path is, and loading it |
-| `ocda.dump` | the model written out canonically, as an oracle |
-| `ocda.sqlite` | the sqlite3 C API, declared rather than depended upon |
+| | |
+|------------------|------------------------------------------------------------------------------------------------------------|
+| `ocda.model` | the document, the object, the eight sections. Imports nothing |
+| `ocda.load_ssp` | text -> model |
+| `ocda.load_db` | sqlite -> model |
+| `ocda.load` | what a path is, and loading it |
+| `ocda.inventory` | what a file holds beyond the one document: its languages (every form alike), and what a `.ocda.db` carries |
+| `ocda.dump` | the model written out canonically, as an oracle |
+| `ocda.sqlite` | the sqlite3 C API, declared rather than depended upon |
+| `read.*` | `ocda-read`, the reader: in `src/read/`, and *not* part of the library (below) |
`ocda.model` importing nothing is the property that matters: a writer
depends on the model and on neither loader.
@@ -31,6 +33,28 @@ foreach (obj; doc.body) { ... }
Link `sqlite3`; the library declares the dozen functions it needs rather
than taking a binding from the registry, so there is nothing to fetch.
+## The reader
+
+```sh
+nix build .#ocda-read && ./result/bin/ocda-read [--tree|--html|--llm-text|--llm-jsonl|
+ --images|--verify --image-dir=<d>|--languages|--files|--report-unknown|--validate]
+ <file>.ssp|<file>.ocda.db|<file>.sisupod [lang]
+```
+
+The library's reader, one program where `sisudoc-ssp-reader` and
+`sisudoc-ocda-db-reader` were two. Its output modules are the `.ssp`
+reader's, moved onto the library's model, so every mode reads every form
+and, where it describes the document, **says the same thing of its `.ssp`,
+`.ocda.db` and `.sisupod`**. `--languages` lists the one language a `.ssp`
+holds and every language of the other two; `--files` answers for a
+`.ocda.db`, the one form that carries files; `--validate` checks a `.ssp`
+as text and refuses anything else.
+
+It lives in `src/read/` beside the library, and dub keeps it out of the
+library: the configuration `library` excludes `src/read/*`, the
+configuration `ocda-read` builds it. A program using the library by dub, or
+by naming `src/ocda/*.d` as the writer's flake does, never compiles it.
+
## The two checks
```sh
@@ -55,8 +79,11 @@ repository README and in the history.
## Build
```sh
+nix build # all three programs, in one result/bin
+nix build .#ocda-read # the reader
nix build .#ocda-agree # the agreement check
nix build .#ocda-dump # the canonical dump
-nix develop # ldc, dub, dtools, sqlite
+nix develop # ldc, dub, dtools, sqlite, zstd
dub build # the library
+dub build --config=ocda-read # the reader
```
diff --git a/dlang/dub.json b/dlang/dub.json
index 4f59aec..861c2cc 100644
--- a/dlang/dub.json
+++ b/dlang/dub.json
@@ -6,8 +6,20 @@
"description": "the object-centric document abstraction as a library: the model, and loaders for its two serialisations",
"homepage": "https://sisudoc.org",
"license": "AGPL-3.0+",
- "targetType": "library",
"sourcePaths": [ "./src" ],
"importPaths": [ "./src" ],
- "libs": [ "sqlite3", "zstd" ]
+ "libs": [ "sqlite3", "zstd" ],
+ "configurations": [
+ {
+ "name": "library",
+ "targetType": "library",
+ "excludedSourceFiles": [ "src/read/*" ]
+ },
+ {
+ "name": "ocda-read",
+ "targetType": "executable",
+ "targetName": "ocda-read",
+ "mainSourceFile": "src/read/main.d"
+ }
+ ]
}
diff --git a/dlang/flake.lock b/dlang/flake.lock
index 5f236ab..12c4c3b 100644
--- a/dlang/flake.lock
+++ b/dlang/flake.lock
@@ -2,18 +2,15 @@
"nodes": {
"nixpkgs": {
"locked": {
- "lastModified": 1790185690,
- "narHash": "sha256-xJ+X4hBtOcAFGBOe5nAMyMUeF9foJBmIOu3NjBqBycU=",
- "owner": "NixOS",
- "repo": "nixpkgs",
- "rev": "4975466d324710c576dc11ad614684e6bd8cad8e",
- "type": "github"
+ "lastModified": 1790652569,
+ "narHash": "sha256-k8Fu4c9Z+4Nh7mUr0cfw++ITQiyEhlWxoJBOkI3tOcQ=",
+ "rev": "b6c8664de9b6cc07fe5666a29f91884ba81197c4",
+ "type": "tarball",
+ "url": "https://releases.nixos.org/nixpkgs/nixpkgs-26.11pre1082290.b6c8664de9b6/nixexprs.tar.zst"
},
"original": {
- "owner": "NixOS",
- "ref": "nixos-unstable",
- "repo": "nixpkgs",
- "type": "github"
+ "type": "tarball",
+ "url": "https://channels.nixos.org/nixpkgs-unstable/nixexprs.tar.zst"
}
},
"root": {
diff --git a/dlang/flake.nix b/dlang/flake.nix
index 3aa9a60..2241af6 100644
--- a/dlang/flake.nix
+++ b/dlang/flake.nix
@@ -1,21 +1,23 @@
{
description = "ocda-lib: the object-centric document abstraction as a D library";
-
- inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
-
+ inputs.nixpkgs.url = "https://channels.nixos.org/nixpkgs-unstable/nixexprs.tar.zst";
outputs = { self, nixpkgs }:
let
system = "x86_64-linux"; # adjust for aarch64-linux / darwin
pkgs = import nixpkgs { inherit system; };
-
- /* A library, so there is nothing to install as a binary. What is
- built here are the two programs that put the library to the
- question: the agreement check, which loads a document's .ssp and
- its .ocda.db and compares the two models, and the dump, which
+ /* A library, so the library itself is not a build product: a program
+ that uses it compiles its modules alongside its own, as the writer's
+ flake does, or takes it as a dub dependency. What is built here are
+ programs over it: the reader, and the two that put the library to
+ the question - the agreement check, which loads a document's .ssp
+ and its .ocda.db and compares the two models, and the dump, which
writes a loaded model out canonically so that an implementation in
another language can be diffed against this one.
+ .
+ `mains` is what is compiled besides the library: the checks are one
+ file each in test/, the reader is src/read/.
*/
- mkTool = { tool, binary, what, compiler, compilerBin }:
+ mkTool = { tool, binary, what, mains, compiler, compilerBin }:
pkgs.stdenv.mkDerivation {
pname = "ocda-lib-${tool}";
version = "2.0.0";
@@ -25,7 +27,7 @@
buildPhase = ''
runHook preBuild
export HOME=$TMPDIR
- ${compilerBin} -I=src -of=${tool} src/ocda/*.d test/${tool}.d -L-lsqlite3 -L-lzstd
+ ${compilerBin} -I=src -of=${tool} src/ocda/*.d ${mains} -L-lsqlite3 -L-lzstd
runHook postBuild
'';
installPhase = ''
@@ -39,21 +41,31 @@
mainProgram = binary;
};
};
-
ldc = { compiler = pkgs.ldc; compilerBin = "ldmd2"; };
+ ocda-read = mkTool (ldc // {
+ tool = "read"; binary = "ocda-read"; mains = "src/read/*.d";
+ what = "reads the abstraction in any of its forms and says what is in it";
+ });
+ ocda-agree = mkTool (ldc // {
+ tool = "agree"; binary = "ocda-agree"; mains = "test/agree.d";
+ what = "checks that the .ssp and .ocda.db loaders give one model";
+ });
+ ocda-dump = mkTool (ldc // {
+ tool = "dump"; binary = "ocda-dump"; mains = "test/dump.d";
+ what = "writes a loaded abstraction out canonically, as an oracle";
+ });
in {
packages.${system} = {
- default = self.packages.${system}.ocda-agree;
-
- ocda-agree = mkTool (ldc // {
- tool = "agree"; binary = "ocda-agree";
- what = "checks that the .ssp and .ocda.db loaders give one model";
- });
-
- ocda-dump = mkTool (ldc // {
- tool = "dump"; binary = "ocda-dump";
- what = "writes a loaded abstraction out canonically, as an oracle";
- });
+ /* All three in one bin/, so that one build leaves every program in
+ one result link. Built separately into the same link, each build
+ replaced the one before, and a script looking for ocda-dump beside
+ ocda-agree found nothing.
+ */
+ default = pkgs.symlinkJoin {
+ name = "ocda-lib-programs-2.0.0";
+ paths = [ ocda-read ocda-agree ocda-dump ];
+ };
+ inherit ocda-read ocda-agree ocda-dump;
};
devShells.${system}.default = pkgs.mkShell {
diff --git a/dlang/src/ocda/inventory.d b/dlang/src/ocda/inventory.d
new file mode 100644
index 0000000..dba2156
--- /dev/null
+++ b/dlang/src/ocda/inventory.d
@@ -0,0 +1,117 @@
+module ocda.inventory;
+
+/+ ↓ what a source holds, beyond the one document the model is loaded from.
+ .
+ The model is one language of one document. Two questions a reader asks
+ are about the file instead: which languages it holds, and, for a
+ database, what else it carries. They are here rather than in a reader
+ because the answer depends on the input form, which is the library's
+ to know, and so that a reader asking them does not learn which form it
+ was handed.
+ .
+ languages() answers for every form, and answers the same: the
+ languages in the order the manifest gave them (which every .ssp
+ records, in @source languages), each with the markup file it was made
+ from and the sha256 of its .ssp. The database stores that digest; for a
+ .ssp and a pod it is taken from the bytes, which are the same bytes.
+ .
+ carried() is the database's: a .ocda.db holds the markup, images,
+ configuration and catalogues it was built from, a .ssp holds none of
+ them, and a pod's are its archive entries rather than anything the
+ abstraction says. So it answers for a database and says so otherwise.
++/
+
+import std.algorithm : endsWith, countUntil;
+import std.array : array, split;
+import std.conv : to;
+import std.digest : toHexString, LetterCase;
+import std.digest.sha : sha256Of;
+import std.file : read;
+import std.string : strip;
+import ocda.model;
+import ocda.load;
+import ocda.load_ssp;
+import ocda.load_db;
+import ocda.load_pod;
+import ocda.sqlite;
+import ocda.zip;
+
+struct OcdaLanguage {
+ string lang;
+ string source_filename; // the markup file the abstraction was made from
+ string ssp_digest; // sha256 of that language's .ssp, upper case hex
+}
+
+struct OcdaCarried {
+ string role; // source, image, conf, manifest, tools
+ string compression; // how it is stored: none, zstd
+ string count;
+ string bytes; // the files' own size, whatever the storage
+ string stored; // what they take in the database
+}
+
+OcdaLanguage[] languages(string path) {
+ final switch (sourceOf(path)) {
+ case Source.ocda_db:
+ auto db = Db.open(path);
+ scope(exit) { db.close(); }
+ OcdaLanguage[] out_;
+ foreach (l; dbLanguages(db)) {
+ out_ ~= OcdaLanguage(l.lang, l.source_filename, l.ssp_digest);
+ }
+ return out_;
+ case Source.ssp:
+ auto bytes = cast(ubyte[]) read(path);
+ auto doc = parseSsp(cast(string) bytes.idup);
+ return [ OcdaLanguage(sourceValue(doc, "language"), doc.source_filename, hexOf(bytes)) ];
+ case Source.pod:
+ auto archive = archiveBytes(path);
+ OcdaLanguage[string] found;
+ string[] order;
+ foreach (e; zipFindSuffix(zipEntries(archive), ".ssp")) {
+ auto bytes = zipRead(archive, e);
+ auto doc = parseSsp(cast(string) bytes.idup);
+ string lang = langOfSspName(e.name);
+ found[lang] = OcdaLanguage(lang, doc.source_filename, hexOf(bytes));
+ if (order.length == 0) {
+ order = sourceValue(doc, "languages").split;
+ }
+ }
+ /+ ↓ the manifest's order, and anything it did not name after it +/
+ OcdaLanguage[] out_;
+ foreach (l; order) { if (auto p = l in found) { out_ ~= *p; } }
+ foreach (l, v; found) { if (order.countUntil(l) < 0) { out_ ~= v; } }
+ return out_;
+ case Source.unknown:
+ throw new OcdaLoadError(path ~ ": " ~ sourceName(Source.unknown));
+ }
+}
+
+/+ ↓ what a database carries, by role and storage; empty for any other form,
+ which isCarrier() says in advance +/
+bool isCarrier(string path) {
+ return sourceOf(path) == Source.ocda_db;
+}
+
+OcdaCarried[] carried(string path) {
+ if (!isCarrier(path)) { return []; }
+ auto db = Db.open(path);
+ scope(exit) { db.close(); }
+ if (!db.has("files")) { return []; }
+ OcdaCarried[] out_;
+ foreach (r; db.rows(
+ "SELECT role, compression, count(*), sum(bytes), sum(length(data))
+ FROM files GROUP BY role, compression ORDER BY role")) {
+ out_ ~= OcdaCarried(r[0], r[1], r[2], r[3], r[4]);
+ }
+ return out_;
+}
+
+private string sourceValue(ref OcdaDocument doc, string key) {
+ if (auto p = key in doc.source) { return *p; }
+ return "";
+}
+
+private string hexOf(const(ubyte)[] bytes) {
+ return toHexString!(LetterCase.upper)(sha256Of(bytes)).idup;
+}
diff --git a/dlang/src/ocda/load_db.d b/dlang/src/ocda/load_db.d
index eb3d71b..2f59437 100644
--- a/dlang/src/ocda/load_db.d
+++ b/dlang/src/ocda/load_db.d
@@ -242,9 +242,42 @@ OcdaDocument loadDbFile(string path, string lang = "", bool any_version = false)
auto section = doc.sectionByName(r[0]);
if (section !is null) { *section ~= obj; }
}
+ countUnread(db, chosen.doc_id, doc);
return doc;
}
+/+ ↓ what the database holds and the model does not.
+ .
+ The text loader meets every property line and counts those it does not
+ keep, so a reader can say what was read and not understood. This
+ loader asks for columns by name, and would never see the ones it did
+ not ask for: it would report that it had understood everything, which
+ is the silence the count exists to prevent. So it counts them, on the
+ conditions under which the .ssp writes the same property (specification
+ section 6.3), and the report is the same from either form: checked,
+ 36 of 36.
++/
+private void countUnread(ref Db db, int doc_id, ref OcdaDocument doc) {
+ string id = doc_id.to!string;
+ void count(string property, string where) {
+ auto n = db.one("SELECT count(*) FROM objects WHERE doc_id = " ~ id
+ ~ " AND " ~ where).to!size_t;
+ if (n > 0) { doc.unknown.property[property] = n; }
+ }
+ count("ancestors_collapsed", "coalesce(ancestors_collapsed,'') != ''");
+ count("dom_status", "coalesce(dom_status,'') != ''");
+ count("dom_status_collapsed", "coalesce(dom_status_collapsed,'') != ''");
+ count("heading_lev_collapsed", "heading_lev_collapsed IS NOT NULL");
+ count("parent_lev", "coalesce(parent_lev,0) != 0");
+ count("segment_lv4_is", "coalesce(segment_lv4_is,'') != ''");
+ count("attrib", "coalesce(attrib,'') != ''");
+ if (db.has("object_subtoc")) {
+ auto n = db.one("SELECT count(*) FROM object_subtoc s JOIN objects o"
+ ~ " ON o.id = s.object_id WHERE o.doc_id = " ~ id).to!size_t;
+ if (n > 0) { doc.unknown.property["lev4_subtoc"] = n; }
+ }
+}
+
/+ ↓ model section 4: the eight levels and their indices +/
private string levelName(int index) {
switch (index) {
diff --git a/dlang/src/ocda/load_pod.d b/dlang/src/ocda/load_pod.d
index 75993be..8f59fe3 100644
--- a/dlang/src/ocda/load_pod.d
+++ b/dlang/src/ocda/load_pod.d
@@ -92,7 +92,7 @@ OcdaDocument loadPodFile(string path, string lang = "") {
The bytes decide, not the name: a .sisupod is a zstd frame and a .zip
is not, and a file misnamed either way still reads.
+/
-private ubyte[] archiveBytes(string path) {
+package ubyte[] archiveBytes(string path) {
auto bytes = cast(ubyte[]) read(path);
return zstdIsFrame(bytes)
? zstdDecompress(bytes, max_pod_size)
@@ -100,7 +100,7 @@ private ubyte[] archiveBytes(string path) {
}
/+ ↓ <uid>.<lang>.ssp -> <lang> +/
-private string langOfSspName(string name) {
+package string langOfSspName(string name) {
if (!name.endsWith(".ssp")) { return ""; }
string stem = name[0 .. $ - 4];
auto dot = stem.lastIndexOf('.');
diff --git a/dlang/src/read/html.d b/dlang/src/read/html.d
new file mode 100644
index 0000000..cd679e4
--- /dev/null
+++ b/dlang/src/read/html.d
@@ -0,0 +1,388 @@
+module read.html;
+
+import std.array : replace, split, join, appender;
+import std.conv : to;
+import std.stdio : writeln, File;
+import std.string : strip, indexOf, startsWith;
+import ocda.model;
+
+void outputHtmlScroll(OcdaDocument doc, string output_file) {
+ auto buf = appender!(string[]);
+
+ buf ~= htmlHead(doc);
+
+ /+ ↓ title +/
+ if (doc.head.length > 0) {
+ auto title_obj = doc.head[0];
+ buf ~= `<div class="title">`;
+ buf ~= `<h1>` ~ htmlEscape(doc.meta("title.main")) ~ `</h1>`;
+ if (doc.meta("title.subtitle").length > 0)
+ buf ~= `<h2 class="subtitle">` ~ htmlEscape(doc.meta("title.subtitle")) ~ `</h2>`;
+ buf ~= `<p class="author">` ~ htmlEscape(doc.meta("creator.author")) ~ `</p>`;
+ buf ~= `</div>`;
+ }
+
+ /+ ↓ table of contents +/
+ if (doc.toc.length > 0) {
+ buf ~= `<div class="toc">`;
+ buf ~= `<h2>Table of Contents</h2>`;
+ foreach (obj; doc.toc) {
+ if (obj.is_a == "toc") {
+ string text = convertInlineMarkup(obj.text);
+ int indent = obj.indent_base;
+ string cls = (indent > 4) ? "toc-sub" : "toc-entry";
+ buf ~= `<p class="` ~ cls ~ `">` ~ text ~ `</p>`;
+ }
+ }
+ buf ~= `</div>`;
+ buf ~= `<hr/>`;
+ }
+
+ /+ ↓ body +/
+ buf ~= `<div class="body">`;
+ foreach (obj; doc.body) {
+ buf ~= renderObject(obj);
+ }
+ buf ~= `</div>`;
+
+ /+ ↓ endnotes +/
+ bool has_endnotes = false;
+ foreach (obj; doc.endnotes) {
+ if (obj.is_a == "endnote") { has_endnotes = true; break; }
+ }
+ if (has_endnotes) {
+ buf ~= `<div class="endnotes">`;
+ buf ~= `<hr/>`;
+ buf ~= `<h2>Endnotes</h2>`;
+ foreach (obj; doc.endnotes) {
+ if (obj.is_a == "endnote") {
+ string text = convertInlineMarkup(obj.text);
+ string parent_ref = (obj.parent_ocn > 0)
+ ? ` <a href="#o` ~ obj.parent_ocn.to!string ~ `">[` ~ obj.parent_ocn.to!string ~ `]</a>`
+ : "";
+ buf ~= `<p class="endnote">` ~ text ~ parent_ref ~ `</p>`;
+ }
+ }
+ buf ~= `</div>`;
+ }
+
+ /+ ↓ glossary +/
+ if (doc.glossary.length > 0) {
+ buf ~= `<div class="glossary">`;
+ buf ~= `<hr/>`;
+ buf ~= `<h2>Glossary</h2>`;
+ foreach (obj; doc.glossary) {
+ string text = convertInlineMarkup(obj.text);
+ buf ~= `<p class="glossary-entry">` ~ text ~ `</p>`;
+ }
+ buf ~= `</div>`;
+ }
+
+ /+ ↓ bookindex +/
+ bool has_bookindex = false;
+ foreach (obj; doc.bookindex) {
+ if (obj.is_a == "bookindex") { has_bookindex = true; break; }
+ }
+ if (has_bookindex) {
+ buf ~= `<div class="bookindex">`;
+ buf ~= `<hr/>`;
+ buf ~= `<h2>Book Index</h2>`;
+ foreach (obj; doc.bookindex) {
+ if (obj.is_a == "bookindex") {
+ string text = convertInlineMarkup(obj.text);
+ buf ~= `<p class="index-entry">` ~ text ~ `</p>`;
+ }
+ }
+ buf ~= `</div>`;
+ }
+
+ buf ~= htmlTail();
+
+ /+ ↓ write output +/
+ string[] output = buf.data;
+ if (output_file.length > 0) {
+ auto f = File(output_file, "w");
+ foreach (line; output) f.writeln(line);
+ } else {
+ foreach (line; output) writeln(line);
+ }
+}
+
+private string[] renderObject(OcdaObject obj) {
+ /+ ↓ markup commentary is carried by the abstraction and is not output.
+ Rendering it would put the author notes into the document, which
+ spine itself does not do
+ +/
+ if (!obj.renders) { return []; }
+ string[] lines;
+ string text = convertInlineMarkup(obj.text);
+ string ocn_anchor = (obj.ocn > 0 && !obj.ocn_off)
+ ? ` id="o` ~ obj.ocn.to!string ~ `"`
+ : "";
+ string ocn_display = (obj.ocn > 0 && !obj.ocn_off)
+ ? `<span class="ocn">` ~ obj.ocn.to!string ~ `</span>`
+ : "";
+
+ switch (obj.is_a) {
+ case "heading":
+ string tag = headingTag(obj.heading_level);
+ string seg_anchor = (obj.segment.length > 0)
+ ? ` id="` ~ htmlEscape(obj.segment) ~ `"`
+ : ocn_anchor;
+ lines ~= `<` ~ tag ~ seg_anchor ~ `>` ~ text ~ `</` ~ tag ~ `>`;
+ break;
+ case "para":
+ string indent_cls = "";
+ if (obj.indent_base > 0) indent_cls = ` class="indent-` ~ obj.indent_base.to!string ~ `"`;
+ else if (obj.bullet) indent_cls = ` class="bullet"`;
+ lines ~= `<p` ~ ocn_anchor ~ indent_cls ~ `>` ~ text ~ ocn_display ~ `</p>`;
+ break;
+ case "code":
+ lines ~= `<pre` ~ ocn_anchor ~ `><code>` ~ htmlEscape(obj.text) ~ `</code></pre>`;
+ break;
+ case "table":
+ lines ~= renderTable(obj, ocn_anchor);
+ break;
+ case "group":
+ lines ~= `<div` ~ ocn_anchor ~ ` class="group">` ~ text ~ `</div>`;
+ break;
+ case "block":
+ lines ~= `<pre` ~ ocn_anchor ~ ` class="block">` ~ htmlEscape(obj.text) ~ `</pre>`;
+ break;
+ case "verse":
+ lines ~= `<p` ~ ocn_anchor ~ ` class="verse">` ~ text.replace("\n", "<br/>\n") ~ ocn_display ~ `</p>`;
+ break;
+ case "quote":
+ lines ~= `<blockquote` ~ ocn_anchor ~ `>` ~ text ~ ocn_display ~ `</blockquote>`;
+ break;
+ default:
+ if (text.length > 0)
+ lines ~= `<p` ~ ocn_anchor ~ `>` ~ text ~ ocn_display ~ `</p>`;
+ break;
+ }
+ return lines;
+}
+
+private string headingTag(string level) {
+ switch (level) {
+ case "A": return "h1";
+ case "B": return "h2";
+ case "C": return "h3";
+ case "D": return "h4";
+ case "1": return "h3";
+ case "2": return "h4";
+ case "3": return "h5";
+ case "4": return "h6";
+ default: return "h4";
+ }
+}
+
+private string renderTable(OcdaObject obj, string ocn_anchor) {
+ string[] lines;
+ lines ~= `<table` ~ ocn_anchor ~ `>`;
+ /+ ↓ table content uses ┊ as cell separator, rows separated by newlines +/
+ auto rows = obj.text.split("\n");
+ bool first_row = obj.table_header;
+ foreach (row; rows) {
+ if (row.strip.length == 0) continue;
+ string tag = first_row ? "th" : "td";
+ lines ~= `<tr>`;
+ foreach (cell; row.split("┊")) {
+ lines ~= `<` ~ tag ~ `>` ~ convertInlineMarkup(cell.strip) ~ `</` ~ tag ~ `>`;
+ }
+ lines ~= `</tr>`;
+ first_row = false;
+ }
+ lines ~= `</table>`;
+ return lines.join("\n");
+}
+
+/+ ↓ convert spine internal Unicode markup to HTML +/
+string convertInlineMarkup(string text) {
+ if (text.length == 0) return "";
+ string result = htmlEscape(text);
+
+ /+ ↓ font face markers: ⑆*bold⑆*, ⑆/italic⑆/, etc.
+ Pattern: ⑆X┨content┣X or ⑆Xcontent⑆X
+ The ┨┣ form is used within heading content +/
+
+ /+ ↓ bold +/
+ result = replaceInlineMarker(result, "⑆!", "!", "<b>", "</b>");
+ result = replaceInlineMarkerBracketed(result, "⑆!", "┨", "┣!", "<b>", "</b>");
+
+ /+ ↓ emphasis +/
+ result = replaceInlineMarker(result, "⑆*", "*", "<em>", "</em>");
+ result = replaceInlineMarkerBracketed(result, "⑆*", "┨", "┣*", "<em>", "</em>");
+
+ /+ ↓ italic +/
+ result = replaceInlineMarker(result, "⑆/", "/", "<i>", "</i>");
+ result = replaceInlineMarkerBracketed(result, "⑆/", "┨", "┣/", "<i>", "</i>");
+
+ /+ ↓ underscore +/
+ result = replaceInlineMarker(result, "⑆_", "_", "<u>", "</u>");
+ result = replaceInlineMarkerBracketed(result, "⑆_", "┨", "┣_", "<u>", "</u>");
+
+ /+ ↓ superscript +/
+ result = replaceInlineMarker(result, "⑆^", "^", "<sup>", "</sup>");
+ result = replaceInlineMarkerBracketed(result, "⑆^", "┨", "┣^", "<sup>", "</sup>");
+
+ /+ ↓ subscript +/
+ result = replaceInlineMarker(result, "⑆,", ",", "<sub>", "</sub>");
+ result = replaceInlineMarkerBracketed(result, "⑆,", "┨", "┣,", "<sub>", "</sub>");
+
+ /+ ↓ monospace +/
+ result = replaceInlineMarker(result, "⑆■", "■", "<code>", "</code>");
+
+ /+ ↓ links: ┥link text┝┤url├ +/
+ result = replaceLinks(result);
+
+ /+ ↓ footnote markers: 【N text】 +/
+ result = replaceFootnotes(result);
+
+ /+ ↓ line breaks +/
+ result = result.replace("┘", "<br/>");
+ result = result.replace("┙", "<br/>");
+
+ /+ ↓ non-breaking spaces +/
+ result = result.replace("░", "&nbsp;");
+
+ /+ ↓ bullet markers +/
+ result = result.replace("● ", "&#8226; ");
+
+ /+ ↓ page break markers +/
+ result = result.replace("┼", "").replace("┿", "").replace("╂", "");
+
+ return result;
+}
+
+private string replaceInlineMarker(string text, string open_mark, string close_suffix, string html_open, string html_close) {
+ /+ ↓ replace ⑆Xcontent⑆X pattern (no bracket form) +/
+ string result = text;
+ while (true) {
+ auto open_pos = result.indexOf(open_mark);
+ if (open_pos < 0) break;
+ auto after_open = open_pos + open_mark.length;
+ /+ ↓ look for closing ⑆X +/
+ string close_mark = "⑆" ~ close_suffix;
+ auto close_pos = indexOf(result[after_open .. $], close_mark);
+ if (close_pos < 0) break;
+ close_pos += after_open;
+ string content = result[after_open .. close_pos];
+ result = result[0 .. open_pos] ~ html_open ~ content ~ html_close
+ ~ result[close_pos + close_mark.length .. $];
+ }
+ return result;
+}
+
+private string replaceInlineMarkerBracketed(string text, string open_mark, string bracket_open, string bracket_close, string html_open, string html_close) {
+ /+ ↓ replace ⑆X┨content┣X pattern (bracketed form) +/
+ string result = text;
+ string full_open = open_mark ~ bracket_open;
+ while (true) {
+ auto open_pos = result.indexOf(full_open);
+ if (open_pos < 0) break;
+ auto after_open = open_pos + full_open.length;
+ auto close_pos = indexOf(result[after_open .. $], bracket_close);
+ if (close_pos < 0) break;
+ close_pos += after_open;
+ string content = result[after_open .. close_pos];
+ result = result[0 .. open_pos] ~ html_open ~ content ~ html_close
+ ~ result[close_pos + bracket_close.length .. $];
+ }
+ return result;
+}
+
+private string replaceLinks(string text) {
+ /+ ↓ replace ┥link text┝┤url├ with <a href="url">link text</a> +/
+ string result = text;
+ while (true) {
+ auto link_open = result.indexOf("┥");
+ if (link_open < 0) break;
+ auto link_mid = indexOf(result[link_open .. $], "┝┤");
+ if (link_mid < 0) break;
+ link_mid += link_open;
+ auto link_close = indexOf(result[link_mid .. $], "├");
+ if (link_close < 0) break;
+ link_close += link_mid;
+ string link_text = result[link_open + "┥".length .. link_mid];
+ string url = result[link_mid + "┝┤".length .. link_close];
+ /+ ↓ handle internal links (¤ prefix) +/
+ if (url.length > 0 && url[0] == '¤') {
+ url = url[("¤").length .. $];
+ }
+ /+ ↓ handle OCN-only links (#N) +/
+ if (url.length > 0 && url[0] == '#') {
+ url = "#o" ~ url[1 .. $];
+ }
+ result = result[0 .. link_open]
+ ~ `<a href="` ~ url ~ `">` ~ link_text ~ `</a>`
+ ~ result[link_close + "├".length .. $];
+ }
+ return result;
+}
+
+private string replaceFootnotes(string text) {
+ /+ ↓ replace 【N content】 with superscript note number +/
+ string result = text;
+ while (true) {
+ auto fn_open = result.indexOf("【");
+ if (fn_open < 0) break;
+ auto fn_close = indexOf(result[fn_open .. $], "】");
+ if (fn_close < 0) break;
+ fn_close += fn_open;
+ string fn_content = result[fn_open + "【".length .. fn_close];
+ /+ ↓ extract note number (first token before space) +/
+ auto space_pos = fn_content.indexOf(' ');
+ string note_num = (space_pos > 0) ? fn_content[0 .. space_pos] : fn_content;
+ result = result[0 .. fn_open]
+ ~ `<sup class="fn">` ~ note_num ~ `</sup>`
+ ~ result[fn_close + "】".length .. $];
+ }
+ return result;
+}
+
+string htmlEscape(string text) {
+ return text.replace("&", "&amp;").replace("<", "&lt;").replace(">", "&gt;")
+ .replace("\"", "&quot;");
+}
+
+private string[] htmlHead(OcdaDocument doc) {
+ string[] lines;
+ lines ~= `<!DOCTYPE html>`;
+ lines ~= `<html>`;
+ lines ~= `<head>`;
+ lines ~= `<meta charset="utf-8"/>`;
+ lines ~= `<title>` ~ htmlEscape(doc.meta("title.main")) ~ `</title>`;
+ lines ~= `<style>`;
+ lines ~= ` body { max-width: 48em; margin: 2em auto; padding: 0 1em; font-family: serif; line-height: 1.5; }`;
+ lines ~= ` .title { text-align: center; margin-bottom: 2em; }`;
+ lines ~= ` .subtitle { font-weight: normal; font-style: italic; }`;
+ lines ~= ` .ocn { float: right; font-size: 0.7em; color: #999; }`;
+ lines ~= ` .toc { margin-bottom: 2em; }`;
+ lines ~= ` .toc-entry { margin: 0.2em 0; }`;
+ lines ~= ` .toc-sub { margin: 0.2em 0 0.2em 2em; }`;
+ lines ~= ` .indent-1 { margin-left: 2em; }`;
+ lines ~= ` .indent-2 { margin-left: 4em; }`;
+ lines ~= ` .indent-3 { margin-left: 6em; }`;
+ lines ~= ` .bullet { margin-left: 2em; }`;
+ lines ~= ` .bullet:before { content: "\2022 "; }`;
+ lines ~= ` .verse { white-space: pre-wrap; font-style: italic; }`;
+ lines ~= ` .group { margin: 1em 2em; }`;
+ lines ~= ` .block { margin: 1em 2em; background: #f8f8f8; padding: 1em; }`;
+ lines ~= ` pre code { display: block; background: #f4f4f4; padding: 1em; overflow-x: auto; }`;
+ lines ~= ` table { border-collapse: collapse; margin: 1em 0; }`;
+ lines ~= ` th, td { border: 1px solid #ccc; padding: 0.4em 0.8em; }`;
+ lines ~= ` th { background: #f0f0f0; }`;
+ lines ~= ` .endnote { font-size: 0.9em; }`;
+ lines ~= ` .fn { color: #666; }`;
+ lines ~= ` .index-entry { font-size: 0.9em; }`;
+ lines ~= ` blockquote { margin: 1em 2em; padding-left: 1em; border-left: 3px solid #ccc; }`;
+ lines ~= `</style>`;
+ lines ~= `</head>`;
+ lines ~= `<body>`;
+ return lines;
+}
+
+private string[] htmlTail() {
+ return [`</body>`, `</html>`];
+}
diff --git a/dlang/src/read/images.d b/dlang/src/read/images.d
new file mode 100644
index 0000000..afddc74
--- /dev/null
+++ b/dlang/src/read/images.d
@@ -0,0 +1,114 @@
+module read.images;
+
+import std.array : join;
+import std.conv : to;
+import std.stdio : writeln, File;
+import ocda.model;
+
+/+ ↓ the .image records of a document, gathered per distinct file, with the
+ objects referencing each file, in order of first appearance +/
+struct ImageEntry {
+ string name;
+ string sha256;
+ ulong bytes;
+ int width;
+ int height;
+ bool missing;
+ string[] refs;
+ string[] other_sha256; // same filename, differing digest
+}
+
+struct ImageRecords {
+ int records; // .image lines, an image referenced twice counts twice
+ ImageEntry[] entries; // distinct files
+}
+
+ImageRecords gatherImageRecords(OcdaDocument doc) {
+ ImageRecords ret;
+ size_t[string] seen;
+ foreach (section; sectionNames) {
+ auto objs = doc.sectionByName(section);
+ if (objs is null) continue;
+ foreach (obj; *objs) {
+ foreach (img; obj.images) {
+ ret.records += 1;
+ string _ref = section ~ "[" ~ obj.ocn.to!string ~ "]";
+ if (auto at = img.name in seen) {
+ if (ret.entries[*at].sha256 != img.sha256
+ && img.sha256.length > 0
+ && ret.entries[*at].sha256.length > 0) {
+ ret.entries[*at].other_sha256 ~= img.sha256;
+ }
+ ret.entries[*at].refs ~= _ref;
+ } else {
+ ImageEntry e_;
+ e_.name = img.name;
+ e_.sha256 = img.sha256;
+ e_.bytes = img.bytes;
+ e_.width = img.width;
+ e_.height = img.height;
+ e_.missing = img.missing;
+ e_.refs = [_ref];
+ seen[img.name] = ret.entries.length;
+ ret.entries ~= e_;
+ }
+ }
+ }
+ }
+ return ret;
+}
+
+/+ ↓ image manifest, line oriented and greppable:
+
+ <sha256> <bytes> <px> <filename> refs: body[97] body[412]
+ MISSING <filename> refs: body[840]
+
+ The point of the record in the .ssp is verification (is the image
+ shipped in this pod the image the abstraction was built from), so this
+ output is the shape a verifier wants: one line per distinct file.
++/
+void outputImages(OcdaDocument doc, string output_file) {
+ auto gathered = gatherImageRecords(doc);
+ string[] lines;
+ int missing = 0;
+ foreach (e; gathered.entries) {
+ if (e.missing) missing += 1;
+ }
+ lines ~= "Images: " ~ gathered.records.to!string ~ " record"
+ ~ ((gathered.records == 1) ? "" : "s")
+ ~ ", " ~ gathered.entries.length.to!string ~ " distinct file"
+ ~ ((gathered.entries.length == 1) ? "" : "s")
+ ~ ", " ~ missing.to!string ~ " missing";
+ lines ~= "";
+ foreach (e; gathered.entries) {
+ string px = (e.width > 0 && e.height > 0)
+ ? e.width.to!string ~ "x" ~ e.height.to!string : "-";
+ lines ~= (e.missing)
+ ? "MISSING" ~ " " ~ e.name ~ " refs: " ~ e.refs.join(" ")
+ : e.sha256 ~ " " ~ e.bytes.to!string ~ " " ~ px
+ ~ " " ~ e.name ~ " refs: " ~ e.refs.join(" ");
+ /+ ↓ one filename, more than one digest: the .ssp is inconsistent,
+ or two distinct files share a name +/
+ foreach (other; e.other_sha256) {
+ lines ~= " WARNING differing digest for same filename: " ~ other;
+ }
+ }
+
+ /+ ↓ write output +/
+ if (output_file.length > 0) {
+ auto f = File(output_file, "w");
+ foreach (line; lines) f.writeln(line);
+ } else {
+ foreach (line; lines) writeln(line);
+ }
+}
+
+/+ ↓ count of .image records in a document, for the parse summary +/
+int imageRecordCount(OcdaDocument doc) {
+ return gatherImageRecords(doc).records;
+}
+
+package immutable string[] sectionNames = [
+ "head", "toc", "body", "endnotes",
+ "glossary", "bibliography", "bookindex", "blurb",
+];
diff --git a/dlang/src/read/llm_jsonl.d b/dlang/src/read/llm_jsonl.d
new file mode 100644
index 0000000..5b81e68
--- /dev/null
+++ b/dlang/src/read/llm_jsonl.d
@@ -0,0 +1,121 @@
+module read.llm_jsonl;
+
+import std.array : split, join, replace;
+import std.conv : to;
+import std.stdio : writeln, File;
+import std.string : strip, indexOf, stripRight;
+import ocda.model;
+import read.llm_text : stripMarkup;
+
+/+ ↓ JSONL output with embeddings-ready chunks for RAG pipelines
+ one JSON object per line, each containing:
+ doc, ocn, path (breadcrumb), type, text (clean),
+ images (only where the object embeds one, so that a chunk whose
+ text reads around a figure carries the figure's identity),
+ index_terms (from bookindex entries associated with this object)
++/
+void outputLlmJsonl(OcdaDocument doc, string output_file) {
+ string[] lines;
+ string doc_id = doc.meta("title.main");
+ string doc_author = doc.meta("creator.author");
+
+ /+ ↓ build bookindex -> OCN mapping for index_terms
+ bookindex objects in the .ssp have parent_ocn linking to body objects
+ but the index terms are in the text of bookindex objects, not
+ directly attached to body objects. We use a simpler approach:
+ headings carry heading_ancestors_text which gives context. +/
+
+ /+ ↓ track current heading path +/
+ string current_path = "";
+
+ foreach (obj; doc.body) {
+ if (obj.is_a == "heading") {
+ current_path = buildPath(doc_id, obj.heading_ancestors_text, obj);
+ /+ ↓ emit heading as a chunk too +/
+ string heading_text = stripMarkup(obj.text);
+ if (heading_text.strip.length > 0) {
+ lines ~= jsonLine(doc_id, doc_author, obj.ocn, current_path, "heading", heading_text, obj.images);
+ }
+ continue;
+ }
+
+ /+ ↓ markup commentary is not content, and least of all is it a chunk
+ to embed: it would be indexed as though the author had written it
+ into the document
+ +/
+ if (!obj.renders) { continue; }
+
+ string text = stripMarkup(obj.text);
+ if (text.strip.length == 0) continue;
+
+ lines ~= jsonLine(doc_id, doc_author, obj.ocn, current_path, obj.is_a, text, obj.images);
+ }
+
+ /+ ↓ write output +/
+ if (output_file.length > 0) {
+ auto f = File(output_file, "w");
+ foreach (line; lines) f.writeln(line);
+ } else {
+ foreach (line; lines) writeln(line);
+ }
+}
+
+private string jsonLine(string doc_id, string doc_author, int ocn, string path, string type, string text, OcdaImage[] images) {
+ return `{"doc":` ~ jsonStr(doc_id)
+ ~ `,"author":` ~ jsonStr(doc_author)
+ ~ `,"ocn":` ~ ocn.to!string
+ ~ `,"path":` ~ jsonStr(path)
+ ~ `,"type":` ~ jsonStr(type)
+ ~ `,"text":` ~ jsonStr(text)
+ ~ jsonImages(images)
+ ~ `}`;
+}
+
+private string jsonImages(OcdaImage[] images) {
+ /+ ↓ omitted entirely where the object embeds no image +/
+ if (images.length == 0) return "";
+ string[] items;
+ foreach (img; images) {
+ string item = `{"name":` ~ jsonStr(img.name);
+ if (img.missing) {
+ item ~= `,"missing":true`;
+ } else {
+ item ~= `,"sha256":` ~ jsonStr(img.sha256)
+ ~ `,"bytes":` ~ img.bytes.to!string;
+ if (img.width > 0 && img.height > 0) {
+ item ~= `,"width":` ~ img.width.to!string
+ ~ `,"height":` ~ img.height.to!string;
+ }
+ }
+ items ~= item ~ `}`;
+ }
+ return `,"images":[` ~ items.join(",") ~ `]`;
+}
+
+private string buildPath(string doc_title, string ancestors_text, OcdaObject obj) {
+ string[] parts;
+ if (ancestors_text.length > 0) {
+ foreach (part; ancestors_text.split("|")) {
+ string clean = stripMarkup(part).strip;
+ if (clean.length > 0 && clean != doc_title) {
+ parts ~= clean;
+ }
+ }
+ }
+ string heading_text = stripMarkup(obj.text).strip;
+ if (parts.length > 0)
+ return parts.join(" > ") ~ " > " ~ heading_text;
+ else
+ return heading_text;
+}
+
+private string jsonStr(string s) {
+ /+ ↓ JSON string escaping +/
+ string r = s;
+ r = r.replace(`\`, `\\`);
+ r = r.replace(`"`, `\"`);
+ r = r.replace("\n", `\n`);
+ r = r.replace("\r", `\r`);
+ r = r.replace("\t", `\t`);
+ return `"` ~ r ~ `"`;
+}
diff --git a/dlang/src/read/llm_text.d b/dlang/src/read/llm_text.d
new file mode 100644
index 0000000..e61f2b7
--- /dev/null
+++ b/dlang/src/read/llm_text.d
@@ -0,0 +1,182 @@
+module read.llm_text;
+
+import std.array : replace, split, join;
+import std.conv : to;
+import std.stdio : writeln, File;
+import std.string : strip, indexOf, stripRight;
+import ocda.model;
+
+/+ ↓ plain structured text with heading breadcrumbs for direct LLM consumption
+ format:
+ ## Document Title > Chapter > Section > [OCN]
+
+ Clean paragraph text with inline markup stripped.
++/
+void outputLlmBreadcrumbs(OcdaDocument doc, string output_file) {
+ string[] lines;
+ string doc_title = doc.meta("title.main");
+
+ /+ ↓ document header +/
+ lines ~= "# " ~ doc_title;
+ if (doc.meta("title.subtitle").length > 0)
+ lines ~= "# " ~ doc.meta("title.subtitle");
+ lines ~= "# " ~ doc.meta("creator.author");
+ lines ~= "";
+
+ /+ ↓ build heading text lookup from ancestors +/
+ string[int] heading_text;
+ foreach (obj; doc.head) {
+ if (obj.is_a == "heading" && obj.ocn > 0)
+ heading_text[obj.ocn] = stripMarkup(obj.text);
+ }
+ foreach (obj; doc.body) {
+ if (obj.is_a == "heading")
+ heading_text[obj.ocn] = stripMarkup(obj.text);
+ }
+
+ /+ ↓ body objects +/
+ string current_breadcrumb = "";
+ foreach (obj; doc.body) {
+ if (obj.is_a == "heading") {
+ current_breadcrumb = buildBreadcrumb(doc_title, obj.heading_ancestors_text, obj);
+ lines ~= "";
+ lines ~= "## " ~ current_breadcrumb;
+ lines ~= "";
+ continue;
+ }
+
+ /+ ↓ markup commentary is not content +/
+ if (!obj.renders) { continue; }
+
+ string text = stripMarkup(obj.text);
+ if (text.strip.length == 0) continue;
+
+ string ocn_ref = (obj.ocn > 0 && !obj.ocn_off)
+ ? " [" ~ obj.ocn.to!string ~ "]"
+ : "";
+
+ switch (obj.is_a) {
+ case "para":
+ lines ~= text ~ ocn_ref;
+ lines ~= "";
+ break;
+ case "code":
+ lines ~= "```";
+ lines ~= obj.text; /+ preserve raw text for code +/
+ lines ~= "```";
+ lines ~= "";
+ break;
+ case "table":
+ lines ~= text ~ ocn_ref;
+ lines ~= "";
+ break;
+ case "verse":
+ lines ~= text ~ ocn_ref;
+ lines ~= "";
+ break;
+ case "quote":
+ lines ~= "> " ~ text ~ ocn_ref;
+ lines ~= "";
+ break;
+ case "group":
+ case "block":
+ lines ~= text ~ ocn_ref;
+ lines ~= "";
+ break;
+ default:
+ if (text.length > 0) {
+ lines ~= text ~ ocn_ref;
+ lines ~= "";
+ }
+ break;
+ }
+ }
+
+ /+ ↓ write output +/
+ if (output_file.length > 0) {
+ auto f = File(output_file, "w");
+ foreach (line; lines) f.writeln(line);
+ } else {
+ foreach (line; lines) writeln(line);
+ }
+}
+
+private string buildBreadcrumb(string doc_title, string ancestors_text, OcdaObject obj) {
+ /+ ↓ ancestors_text is pipe-separated: "Title|Part||Chapter|||" +/
+ string[] parts;
+ if (ancestors_text.length > 0) {
+ foreach (part; ancestors_text.split("|")) {
+ string clean = stripMarkup(part).strip;
+ if (clean.length > 0 && clean != doc_title) {
+ /+ ↓ truncate very long heading text for breadcrumb +/
+ if (clean.length > 60)
+ clean = clean[0 .. 57] ~ "...";
+ parts ~= clean;
+ }
+ }
+ }
+ /+ ↓ add the current heading itself +/
+ string heading_text = stripMarkup(obj.text).strip;
+ if (heading_text.length > 60)
+ heading_text = heading_text[0 .. 57] ~ "...";
+
+ string ocn_ref = (obj.ocn > 0) ? " [" ~ obj.ocn.to!string ~ "]" : "";
+
+ if (parts.length > 0)
+ return doc_title ~ " > " ~ parts.join(" > ") ~ " > " ~ heading_text ~ ocn_ref;
+ else
+ return doc_title ~ " > " ~ heading_text ~ ocn_ref;
+}
+
+/+ ↓ strip all internal Unicode markup to produce clean plain text +/
+string stripMarkup(string text) {
+ if (text.length == 0) return "";
+ string r = text;
+
+ /+ ↓ links: ┥link text┝┤url├ -> link text +/
+ while (true) {
+ auto lo = r.indexOf("┥");
+ if (lo < 0) break;
+ auto lm = indexOf(r[lo .. $], "┝┤");
+ if (lm < 0) break;
+ lm += lo;
+ auto lc = indexOf(r[lm .. $], "├");
+ if (lc < 0) break;
+ lc += lm;
+ string link_text = r[lo + "┥".length .. lm];
+ r = r[0 .. lo] ~ link_text ~ r[lc + "├".length .. $];
+ }
+
+ /+ ↓ footnotes: 【N content】 -> (remove entirely for clean text) +/
+ while (true) {
+ auto fo = r.indexOf("【");
+ if (fo < 0) break;
+ auto fc = indexOf(r[fo .. $], "】");
+ if (fc < 0) break;
+ fc += fo;
+ r = r[0 .. fo] ~ r[fc + "】".length .. $];
+ }
+
+ /+ ↓ font face markers +/
+ r = r.replace("⑆*", "").replace("⑆!", "").replace("⑆/", "")
+ .replace("⑆_", "").replace("⑆^", "").replace("⑆,", "")
+ .replace("⑆■", "")
+ .replace("┨", "").replace("┣*", "").replace("┣!", "")
+ .replace("┣/", "").replace("┣_", "").replace("┣^", "")
+ .replace("┣,", "").replace("┣", "");
+
+ /+ ↓ line breaks and special chars +/
+ r = r.replace("┘", "\n").replace("┙", " ").replace("┚", "\n");
+ r = r.replace("░", " ");
+ r = r.replace("● ", "- ");
+ r = r.replace("┼", "").replace("┿", "").replace("╂", "");
+
+ /+ ↓ table cell separators +/
+ r = r.replace("┊", " | ");
+
+ /+ ↓ clean up multiple spaces +/
+ while (r.indexOf(" ") >= 0)
+ r = r.replace(" ", " ");
+
+ return r.strip;
+}
diff --git a/dlang/src/read/main.d b/dlang/src/read/main.d
new file mode 100644
index 0000000..ae0d318
--- /dev/null
+++ b/dlang/src/read/main.d
@@ -0,0 +1,228 @@
+/++
+ ocda-read - reads the object-centric document abstraction, in any of the
+ forms it is written in, and says what is in it.
+
+ ocda-read [options] <file>.ssp | <file>.ocda.db | <file>.sisupod [lang]
+
+ One reader where there were two: sisudoc-ssp-reader read a .ssp and
+ sisudoc-ocda-db-reader a .ocda.db, each with its own parser. This one is
+ built on the library, so every mode reads every form, and a mode given the
+ .ssp, the .ocda.db and the .sisupod of a document says the same thing of
+ each: it is told what the document is, not which file it came from.
+
+ Two things are about the file rather than the document, and say so:
+ --files, what a .ocda.db carries besides the abstraction, and --validate,
+ which checks a .ssp as text. Everything else is the model's.
+
+ The output modules are the .ssp reader's, moved onto the library's model.
++/
+module read.main;
+
+import std.algorithm : canFind, map;
+import std.array : array, join;
+import std.getopt;
+import std.stdio : writeln, writefln, stderr;
+import ocda.model;
+import ocda.load;
+import ocda.inventory;
+import read.tree;
+import read.html;
+import read.llm_text;
+import read.llm_jsonl;
+import read.images;
+import read.verify;
+import read.report_unknown;
+import read.validate;
+
+int main(string[] args) {
+ bool do_tree = false;
+ bool do_html = false;
+ bool do_llm_text = false;
+ bool do_llm_jsonl = false;
+ bool do_images = false;
+ bool do_verify = false;
+ bool do_report_unknown = false;
+ bool do_validate = false;
+ bool do_languages = false;
+ bool do_files = false;
+ bool any_version = false;
+ string image_dir = "";
+ string output_file = "";
+ string lang = "";
+
+ GetoptResult helpInfo;
+ try {
+ helpInfo = getopt(args,
+ "tree", "the headings, as a tree", &do_tree,
+ "outline", "the same as --tree", &do_tree,
+ "html", "the document as one html page", &do_html,
+ "llm-text", "plain text with heading breadcrumbs", &do_llm_text,
+ "llm-jsonl", "JSONL, chunks ready for embedding", &do_llm_jsonl,
+ "images", "the image records: digest, size, pixels", &do_images,
+ "verify", "check the image records against --image-dir", &do_verify,
+ "image-dir", "the images to verify against", &image_dir,
+ "languages", "the languages the file holds", &do_languages,
+ "files", "what a .ocda.db carries besides the abstraction", &do_files,
+ "report-unknown", "what was read and not understood", &do_report_unknown,
+ "validate", "check a .ssp against its specification, as text", &do_validate,
+ "lang", "=<code> the language to read (default: the first)", &lang,
+ "output", "write the mode's output to a file", &output_file,
+ "any-version", "read a format major this reader does not know", &any_version,
+ );
+ } catch (GetOptException e) {
+ stderr.writeln("ocda-read: ", e.msg, "; --help lists the options");
+ return 2;
+ }
+
+ if (helpInfo.helpWanted || args.length < 2) {
+ defaultGetoptPrinter(
+ "ocda-read - reads the document abstraction, in any of its forms\n\n"
+ ~ "Usage: ocda-read [options] <file>.ssp|<file>.ocda.db|<file>.sisupod [lang]\n",
+ helpInfo.options,
+ );
+ return 0;
+ }
+
+ string input_file = args[1];
+ if (lang.length == 0 && args.length > 2) { lang = args[2]; }
+ bool lang_named = (lang.length > 0);
+
+ /+ ↓ --validate is about the .ssp as text, so it is asked of a .ssp or not
+ at all: there is no text form of a database to check, and a pod's
+ .ssp is checked by taking it out +/
+ if (do_validate && sourceOf(input_file) != Source.ssp) {
+ stderr.writeln("--validate checks a .ssp as text, against the .ssp",
+ " specification; ", input_file, " is ", sourceName(sourceOf(input_file)));
+ return 2;
+ }
+
+ /+ ↓ which language. A .ocda.db and a pod may hold several; where none is
+ named, every form reads the first in the manifest's order and says
+ so, rather than the database taking one and the pod refusing +/
+ OcdaLanguage[] held;
+ OcdaDocument doc;
+ try {
+ held = languages(input_file);
+ if (lang.length == 0 && held.length > 0) { lang = held[0].lang; }
+ if (held.length > 0 && !held.map!(l => l.lang).canFind(lang)) {
+ stderr.writeln("error: no language '", lang, "' in ", input_file,
+ "; it holds: ", held.map!(l => l.lang).array.join(" "));
+ return 2;
+ }
+ doc = load(input_file, lang, any_version);
+ } catch (Exception e) {
+ stderr.writeln("Error reading ", input_file, ": ", e.msg);
+ return 2;
+ }
+
+ /+ ↓ the version gate: the major part, and refuse what is not known +/
+ if (doc.format_version.length == 0) {
+ stderr.writeln("warning: ", input_file, " declares no format version");
+ } else if (doc.formatVersionMajor != ssp_format_major_read) {
+ if (any_version) {
+ stderr.writeln("warning: ", input_file, " is format v", doc.format_version,
+ ", this reader knows v", ssp_format_major_read, ".x; reading anyway");
+ } else {
+ stderr.writeln("error: ", input_file, " is format v", doc.format_version,
+ ", this reader knows v", ssp_format_major_read, ".x");
+ stderr.writeln(" a major version differs: a property may have",
+ " changed meaning or gone away.");
+ stderr.writeln(" --any-version reads it regardless.");
+ return 3;
+ }
+ }
+
+ writeln("Read: ", input_file);
+ if (doc.format_version.length > 0) {
+ writeln(" Format: v", doc.format_version);
+ }
+ if (auto _lang = "language" in doc.source) {
+ writeln(" Source: ", *_lang,
+ ("languages" in doc.source) ? " of " ~ doc.source["languages"] : "",
+ ("digest" in doc.source) ? ", markup digest " ~ doc.source["digest"][0 .. 16] ~ "..." : "");
+ }
+ if (held.length > 1 && !lang_named) {
+ writeln(" (reading ", lang, "; --lang=<code> for another)");
+ }
+ writeln(" Title: ", doc.meta("title.main"));
+ writeln(" Author: ", doc.meta("creator.author"));
+ writeln(" Sections: head=", doc.head.length,
+ " toc=", doc.toc.length,
+ " body=", doc.body.length,
+ " endnotes=", doc.endnotes.length,
+ " glossary=", doc.glossary.length,
+ " bibliography=", doc.bibliography.length,
+ " bookindex=", doc.bookindex.length,
+ " blurb=", doc.blurb.length,
+ " tail=", doc.tail.length);
+ auto image_records = imageRecordCount(doc);
+ if (image_records > 0) {
+ writeln(" Images: ", image_records, " record",
+ (image_records == 1) ? "" : "s");
+ }
+
+ if (do_languages) { printLanguages(held); }
+ if (do_files) { printFiles(input_file); }
+ if (do_tree) { outputTree(doc, output_file); }
+ if (do_html) { outputHtmlScroll(doc, output_file); }
+ if (do_llm_text) { outputLlmBreadcrumbs(doc, output_file); }
+ if (do_llm_jsonl) { outputLlmJsonl(doc, output_file); }
+ if (do_images) { outputImages(doc, output_file); }
+
+ int _verify_failures = 0;
+ if (do_verify) {
+ if (image_dir.length == 0) {
+ stderr.writeln("--verify needs --image-dir=<path to the pod's media/image>");
+ return 2;
+ }
+ _verify_failures = verifyImages(doc, image_dir, output_file);
+ }
+
+ if (do_report_unknown) { reportUnknown(doc); }
+
+ int _validation_findings = 0;
+ if (do_validate) {
+ auto _found = validateSspFile(input_file, ssp_format_major_read);
+ reportValidation(input_file, _found);
+ _validation_findings = cast(int) _found.length;
+ }
+
+ if (!do_tree && !do_html && !do_llm_text && !do_llm_jsonl && !do_images
+ && !do_verify && !do_report_unknown && !do_validate
+ && !do_languages && !do_files) {
+ writeln("\n(this is the summary: --help lists the modes, each adding its output)");
+ }
+
+ return (_verify_failures > 0 || _validation_findings > 0) ? 1 : 0;
+}
+
+/+ ↓ the languages, each with the markup it was made from and its .ssp's
+ digest: the same from every form, the database storing the digest and
+ the others taking it from the bytes +/
+void printLanguages(OcdaLanguage[] held) {
+ writeln();
+ writeln(" lang source ssp digest");
+ foreach (l; held) {
+ writefln(" %-7s %-31s %s", l.lang, l.source_filename,
+ (l.ssp_digest.length > 16) ? l.ssp_digest[0 .. 16] ~ "..." : l.ssp_digest);
+ }
+}
+
+/+ ↓ what a .ocda.db carries besides the abstraction. A .ssp carries
+ nothing, and a pod's files are its archive entries rather than anything
+ the abstraction records, so for those it says so +/
+void printFiles(string path) {
+ writeln();
+ if (!isCarrier(path)) {
+ writeln(" carried files: only a .ocda.db carries files; this is ",
+ sourceName(sourceOf(path)));
+ return;
+ }
+ auto rows = carried(path);
+ if (rows.length == 0) { writeln(" carries no files"); return; }
+ writeln(" carried files:");
+ writeln(" role stored n original in the file");
+ foreach (r; rows) {
+ writefln(" %-9s %-7s %-5s %-13s %s", r.role, r.compression, r.count, r.bytes, r.stored);
+ }
+}
diff --git a/dlang/src/read/report_unknown.d b/dlang/src/read/report_unknown.d
new file mode 100644
index 0000000..2f81702
--- /dev/null
+++ b/dlang/src/read/report_unknown.d
@@ -0,0 +1,41 @@
+module read.report_unknown;
+
+import std.algorithm : sort;
+import std.array : array;
+import std.stdio : writeln, writefln;
+import ocda.model;
+
+/+ ↓ say what was read and not understood.
+ This is what makes the reader usable as a check on the format
+ specification: a reader that ignores what it does not know is
+ conforming, but one that cannot say what it ignored cannot tell you
+ whether the specification it was written from was complete. Silence
+ and understanding look the same from outside.
++/
+void reportUnknown(ref OcdaDocument doc) {
+ if (!doc.unknown.any) {
+ writeln("\nread and understood: every property, block and line was known");
+ return;
+ }
+ writeln("\nread and NOT understood:");
+ if (doc.unknown.property.length > 0) {
+ writeln(" properties ignored (", doc.unknown.property.length, " distinct):");
+ foreach (name; doc.unknown.property.keys.sort) {
+ writefln(" .%-26s %8d", name, doc.unknown.property[name]);
+ }
+ }
+ if (doc.unknown.block.length > 0) {
+ writeln(" blocks ignored (", doc.unknown.block.length, " distinct):");
+ foreach (name; doc.unknown.block.keys.sort) {
+ writefln(" @%-26s %8d", name, doc.unknown.block[name]);
+ }
+ }
+ if (doc.unknown.line.length > 0) {
+ writeln(" lines of no known shape, by first character:");
+ foreach (c; doc.unknown.line.keys.sort) {
+ writefln(" %-27s %8d", "\"" ~ c ~ "\"", doc.unknown.line[c]);
+ }
+ }
+ writeln(" (ignoring these is conforming within a major version;",
+ " the count is so that it is not silent)");
+}
diff --git a/dlang/src/read/tree.d b/dlang/src/read/tree.d
new file mode 100644
index 0000000..9a45b02
--- /dev/null
+++ b/dlang/src/read/tree.d
@@ -0,0 +1,69 @@
+module read.tree;
+
+import std.conv : to;
+import std.stdio : writeln, writef, writefln, File;
+import std.array : replicate;
+import ocda.model;
+
+void outputTree(OcdaDocument doc, string output_file) {
+ string[] lines;
+
+ lines ~= "Document: " ~ doc.meta("title.main");
+ if (doc.meta("title.subtitle").length > 0)
+ lines ~= " " ~ doc.meta("title.subtitle");
+ lines ~= "Author: " ~ doc.meta("creator.author");
+ lines ~= "";
+
+ /+ ↓ build OCN -> object lookup for body headings +/
+ OcdaObject[int] headings;
+ foreach (obj; doc.body) {
+ if (obj.is_a == "heading" && obj.ocn > 0) {
+ headings[obj.ocn] = obj;
+ }
+ }
+
+ /+ ↓ find root heading (from head section, or first body heading) +/
+ if (doc.head.length > 0 && doc.head[0].is_a == "heading") {
+ auto root = doc.head[0];
+ lines ~= formatHeading(root, 0);
+ walkChildren(root, headings, lines, 1);
+ }
+
+ /+ ↓ write output +/
+ if (output_file.length > 0) {
+ auto f = File(output_file, "w");
+ foreach (line; lines) f.writeln(line);
+ } else {
+ foreach (line; lines) writeln(line);
+ }
+}
+
+private void walkChildren(OcdaObject parent, ref OcdaObject[int] headings, ref string[] lines, int depth) {
+ foreach (child_ocn; parent.children) {
+ if (auto p = child_ocn in headings) {
+ lines ~= formatHeading(*p, depth);
+ walkChildren(*p, headings, lines, depth + 1);
+ }
+ }
+}
+
+private string formatHeading(OcdaObject obj, int depth) {
+ string indent = replicate(" ", depth);
+ string level_tag = (obj.heading_level.length > 0) ? obj.heading_level : "?";
+ string ocn_str = (obj.ocn > 0) ? " [" ~ obj.ocn.to!string ~ "]" : "";
+ /+ ↓ strip internal markup from heading text for clean display +/
+ string text = stripInternalMarkup(obj.text);
+ return indent ~ level_tag ~ "~" ~ ocn_str ~ " " ~ text;
+}
+
+private string stripInternalMarkup(string text) {
+ /+ ↓ simple removal of common internal markers for display +/
+ import std.regex : replaceAll, regex;
+ import std.array : replace;
+ string result = text;
+ result = result.replace("⑆*", "").replace("⑆!", "").replace("⑆/", "")
+ .replace("⑆_", "").replace("⑆^", "").replace("⑆,", "")
+ .replace("┨", "").replace("┣", "")
+ .replace("┙", " ").replace("┘", "\n");
+ return result;
+}
diff --git a/dlang/src/read/validate.d b/dlang/src/read/validate.d
new file mode 100644
index 0000000..4a77f8b
--- /dev/null
+++ b/dlang/src/read/validate.d
@@ -0,0 +1,265 @@
+module read.validate;
+
+/+ ↓ check a .ssp against the format specification.
+ .
+ Written from specs/sisudoc-abstraction-format/ssp-format-specification.md
+ and from nothing else: the point of it is to be what the specification
+ says, executed, so that a file can be held against the document rather
+ than against an implementation. Each check names the section it comes
+ from, so a disagreement can be taken back to the text.
++/
+
+import std.algorithm : canFind, startsWith;
+import std.array : array, split;
+import std.conv : to;
+import std.file : readText;
+import std.stdio : writeln, writefln;
+import std.string : indexOf, strip;
+
+struct SspFinding {
+ size_t line; // 1-based
+ string section; // the specification section the rule is from
+ string detail;
+}
+
+/+ ↓ specification section 6.3: the properties, in the order they are
+ emitted. A reader may rely on the order, so a validator can check it +/
+private immutable string[] property_order = [
+ "identifier", "part", "section", "parent", "last_descendant", "ocn_range",
+ "children",
+ "ancestors", "ancestors_collapsed", "dom_status", "dom_status_collapsed",
+ "heading_lev_collapsed", "parent_lev", "dummy", "ocn_off", "is_of_type",
+ "attrib", "meta_lang", "meta_syntax", "sha256", "image", "indent",
+ "bullet", "lang", "has", "table_cols", "table_widths", "table_aligns",
+ "table_header", "code_linenumbers", "stow_link", "segment", "anchor",
+ "segment_prev", "segment_next", "heading_lev_anchor", "segment_epub",
+ "segment_html_is", "segment_epub_is", "segment_lv4_is",
+ "heading_ancestors_text", "lev4_subtoc", "anchor_tag",
+];
+
+/+ ↓ section 6.1: the sixteen kinds +/
+private immutable string[] kinds = [
+ "para", "heading", "toc", "bookindex", "endnote", "code", "bibliography",
+ "comment", "verse", "glossary", "quote", "poem", "block", "table",
+ "blurb", "group",
+];
+
+/+ ↓ section 4: the nine sections, in the order they appear +/
+private immutable string[] section_order = [
+ "head", "toc", "body", "endnotes", "glossary", "bibliography",
+ "bookindex", "blurb", "tail",
+];
+
+/+ ↓ section 4: the four preamble blocks, in the order they appear +/
+private immutable string[] preamble_order = ["source", "meta", "make", "doc_has"];
+
+/+ ↓ section 6.1: the heading levels +/
+private immutable string[] levels = ["A", "B", "C", "D", "1", "2", "3", "4"];
+
+private int propertyRank(string name) {
+ foreach (i, p; property_order) { if (p == name) { return cast(int) i; } }
+ return -1;
+}
+
+SspFinding[] validateSspFile(string filename, string major_known) {
+ SspFinding[] found;
+ auto lines = readText(filename).split("\n");
+
+ bool seen_banner = false;
+ bool seen_source_line = false;
+ string block = ""; // the block currently open, "" when none
+ int preamble_at = -1; // how far through the preamble order
+ int section_at = -1; // how far through the section order
+ bool in_object = false;
+ int prev_rank = -1;
+ size_t doc_has_keys = 0;
+ bool in_text = false; // text lines come after properties
+
+ foreach (i, line; lines) {
+ size_t no = i + 1;
+
+ /+ ↓ section 2: the version, on the first line +/
+ if (no == 1) {
+ enum banner = "% SiSU Document Abstraction v";
+ if (!line.startsWith(banner)) {
+ found ~= SspFinding(no, "2", "first line is not the version banner");
+ } else {
+ seen_banner = true;
+ string v = line[banner.length .. $].strip;
+ auto dot = v.indexOf('.');
+ string major = (dot > 0) ? v[0 .. dot] : v;
+ if (major != major_known) {
+ found ~= SspFinding(no, "2.1",
+ "format major v" ~ major ~ ", this validator knows v" ~ major_known);
+ }
+ }
+ continue;
+ }
+ if (no == 2) {
+ if (!line.startsWith("% Source: ")) {
+ found ~= SspFinding(no, "4", "second line is not \"% Source: \"");
+ } else { seen_source_line = true; }
+ continue;
+ }
+
+ if (line.length == 0) { in_object = false; in_text = false; prev_rank = -1; continue; }
+
+ /+ ↓ section 3: a comment is the file header only +/
+ if (line[0] == '%') {
+ found ~= SspFinding(no, "3", "a comment outside the two header lines");
+ continue;
+ }
+
+ /+ ↓ block open +/
+ if (line[0] == '@') {
+ auto brace = line.indexOf(" {");
+ if (brace < 1 || line[$ - 1] != '{') {
+ found ~= SspFinding(no, "3", "block open is not \"@name {\"");
+ continue;
+ }
+ string name = line[1 .. brace];
+ if (preamble_order.canFind(name)) {
+ int at = 0;
+ foreach (k, p; preamble_order) { if (p == name) { at = cast(int) k; } }
+ if (at <= preamble_at) {
+ found ~= SspFinding(no, "4", "preamble block @" ~ name ~ " out of order");
+ }
+ preamble_at = at;
+ if (section_at >= 0) {
+ found ~= SspFinding(no, "4", "preamble block @" ~ name ~ " after a section");
+ }
+ } else if (section_order.canFind(name)) {
+ int at = 0;
+ foreach (k, sname; section_order) { if (sname == name) { at = cast(int) k; } }
+ if (at <= section_at) {
+ found ~= SspFinding(no, "4", "section @" ~ name ~ " out of order");
+ }
+ section_at = at;
+ } else {
+ found ~= SspFinding(no, "4", "@" ~ name ~ " is neither a preamble block nor a section");
+ }
+ block = name;
+ doc_has_keys = 0;
+ continue;
+ }
+
+ if (line[0] == '}') {
+ if (line.length != 1) {
+ found ~= SspFinding(no, "3", "block close carries more than \"}\"");
+ }
+ /+ ↓ section 5.4: ten keys, always +/
+ if (block == "doc_has" && doc_has_keys != 10) {
+ found ~= SspFinding(no, "5.4",
+ "@doc_has has " ~ doc_has_keys.to!string ~ " keys, the specification says ten");
+ }
+ block = "";
+ in_object = false;
+ prev_rank = -1;
+ continue;
+ }
+
+ /+ ↓ section 5: inside a preamble block, " key: value" +/
+ if (preamble_order.canFind(block)) {
+ if (!line.startsWith(" ") || line.indexOf(": ") < 3) {
+ found ~= SspFinding(no, "5", "not \" key: value\" inside @" ~ block);
+ } else if (block == "doc_has") {
+ doc_has_keys++;
+ }
+ continue;
+ }
+
+ /+ ↓ section 6.1: the object declaration line +/
+ if (line[0] == '[') {
+ auto close = line.indexOf("] ");
+ if (close < 2) {
+ found ~= SspFinding(no, "6.1", "declaration is not \"[ocn] ...\"");
+ continue;
+ }
+ string ocn = line[1 .. close];
+ foreach (c; ocn) {
+ if (c < '0' || c > '9') {
+ found ~= SspFinding(no, "6.1", "ocn \"" ~ ocn ~ "\" is not a number");
+ break;
+ }
+ }
+ auto rest = line[close + 2 .. $].split(" ");
+ if (rest.length == 0 || !kinds.canFind(rest[0])) {
+ found ~= SspFinding(no, "6.1",
+ "\"" ~ ((rest.length > 0) ? rest[0] : "") ~ "\" is not one of the sixteen kinds");
+ } else if (rest[0] == "heading") {
+ if (rest.length < 2 || rest[1].length < 2 || rest[1][0] != ':') {
+ found ~= SspFinding(no, "6.1", "a heading declaration carries no \":level\"");
+ } else if (!levels.canFind(rest[1][1 .. $])) {
+ found ~= SspFinding(no, "6.1", "level \"" ~ rest[1][1 .. $] ~ "\" is not one of the eight");
+ }
+ }
+ in_object = true;
+ in_text = false;
+ prev_rank = -1;
+ continue;
+ }
+
+ /+ ↓ section 6.3: properties, known and in order +/
+ if (line[0] == '.') {
+ if (!in_object) {
+ found ~= SspFinding(no, "6", "a property outside an object record");
+ continue;
+ }
+ if (in_text) {
+ found ~= SspFinding(no, "6", "a property after the object text began");
+ }
+ auto colon = line.indexOf(": ");
+ if (colon < 2) {
+ found ~= SspFinding(no, "3", "property is not \".name: value\"");
+ continue;
+ }
+ string name = line[1 .. colon];
+ int rank = propertyRank(name);
+ if (rank < 0) {
+ found ~= SspFinding(no, "6.3", "." ~ name ~ " is not one of the " ~ property_order.length.to!string ~ " properties");
+ } else {
+ /+ ↓ the four repeatable properties may recur at the same rank +/
+ if (rank < prev_rank) {
+ found ~= SspFinding(no, "6.3", "." ~ name ~ " is out of the emitted order");
+ }
+ prev_rank = rank;
+ }
+ continue;
+ }
+
+ /+ ↓ section 6.2: text lines +/
+ if (line[0] == '|') {
+ if (!in_object) {
+ found ~= SspFinding(no, "6.2", "a text line outside an object record");
+ }
+ if (line.length < 2 || line[1] != ' ') {
+ found ~= SspFinding(no, "6.2", "a text line is not \"| \" then the text");
+ }
+ in_text = true;
+ continue;
+ }
+
+ found ~= SspFinding(no, "3", "line of no shape the specification defines");
+ }
+
+ if (!seen_banner) { found ~= SspFinding(0, "2", "no version banner"); }
+ if (!seen_source_line) { found ~= SspFinding(0, "4", "no \"% Source:\" line"); }
+ return found;
+}
+
+void reportValidation(string filename, SspFinding[] found) {
+ if (found.length == 0) {
+ writeln("\nvalid: ", filename,
+ " conforms to the .ssp specification as this validator reads it");
+ return;
+ }
+ writefln("\n%d finding%s against the specification:",
+ found.length, (found.length == 1) ? "" : "s");
+ foreach (f; found) {
+ if (f.line > 0) {
+ writefln(" line %-7d [%-4s] %s", f.line, f.section, f.detail);
+ } else {
+ writefln(" %-13s [%-4s] %s", "(file)", f.section, f.detail);
+ }
+ }
+}
diff --git a/dlang/src/read/verify.d b/dlang/src/read/verify.d
new file mode 100644
index 0000000..8c2cc65
--- /dev/null
+++ b/dlang/src/read/verify.d
@@ -0,0 +1,214 @@
+module read.verify;
+
+import std.algorithm : sort;
+import std.array : array, join;
+import std.conv : to;
+import std.digest : toHexString;
+import std.digest.sha : SHA256, digest;
+import std.file : dirEntries, exists, isDir, SpanMode, getSize;
+import std.path : baseName, buildPath, extension;
+import std.stdio : writeln, File;
+import std.string : toLower;
+import ocda.model;
+import read.images : gatherImageRecords;
+
+/+ ↓ verification: recompute what the .image records claim, against the
+ images as they stand in a directory (a pod's media/image).
+
+ ssp-reader --verify --image-dir=<pod>/media/image doc.ssp
+
+ Reports one line per distinct file:
+
+ OK won_benkler_2_1.png
+ MISMATCH won_benkler_2_1.png sha256 recorded 5BA4... file 1C3D...
+ ABSENT won_benkler_2_1.png (recorded sha256 5BA4..., not in directory)
+ PRESENT ffa.png (recorded missing, but the file is there)
+ EXTRA unused.png (in directory, no record in this .ssp)
+
+ Returns the number of failures. EXTRA is reported but is not a failure:
+ a pod carries the images of every language version of a document, while
+ one .ssp covers one language.
++/
+int verifyImages(OcdaDocument doc, string image_dir, string output_file) {
+ string[] lines;
+ int failures = 0;
+ int ok = 0;
+ int mismatch = 0;
+ int absent = 0;
+ int extra = 0;
+
+ if (!(image_dir.exists) || !(image_dir.isDir)) {
+ writeln("ERROR: image directory not found: ", image_dir);
+ return 1;
+ }
+ auto gathered = gatherImageRecords(doc);
+ bool[string] recorded;
+
+ foreach (e; gathered.entries) {
+ recorded[e.name] = true;
+ string pth = buildPath(image_dir, e.name);
+ if (!(pth.exists)) {
+ if (e.missing) {
+ /+ recorded missing, and missing here too: consistent +/
+ ok += 1;
+ lines ~= "OK " ~ e.name ~ " (recorded missing, absent here)";
+ } else {
+ absent += 1;
+ failures += 1;
+ lines ~= "ABSENT " ~ e.name
+ ~ " (recorded sha256 " ~ e.sha256 ~ ", not in directory)";
+ }
+ continue;
+ }
+ if (e.missing) {
+ mismatch += 1;
+ failures += 1;
+ lines ~= "PRESENT " ~ e.name ~ " (recorded missing, but the file is there)";
+ continue;
+ }
+ auto file_ = readFileStat(pth);
+ string[] differs;
+ if (file_.sha256 != e.sha256) {
+ differs ~= "sha256 recorded " ~ e.sha256 ~ " file " ~ file_.sha256;
+ }
+ if (file_.bytes != e.bytes) {
+ differs ~= "bytes recorded " ~ e.bytes.to!string
+ ~ " file " ~ file_.bytes.to!string;
+ }
+ /+ px is written only where the dimensions could be read, and read here
+ only for the formats below, so compare only when both are known +/
+ if (e.width > 0 && e.height > 0 && file_.width > 0 && file_.height > 0
+ && (file_.width != e.width || file_.height != e.height)) {
+ differs ~= "px recorded " ~ e.width.to!string ~ "x" ~ e.height.to!string
+ ~ " file " ~ file_.width.to!string ~ "x" ~ file_.height.to!string;
+ }
+ if (differs.length > 0) {
+ mismatch += 1;
+ failures += 1;
+ lines ~= "MISMATCH " ~ e.name ~ " " ~ differs.join("; ");
+ } else {
+ ok += 1;
+ lines ~= "OK " ~ e.name;
+ }
+ }
+
+ /+ ↓ files in the directory that no record names +/
+ string[] in_dir;
+ foreach (entry; dirEntries(image_dir, SpanMode.shallow)) {
+ if (!(entry.isFile)) continue;
+ string nm = entry.name.baseName;
+ switch (nm.extension.toLower) {
+ case ".png": case ".jpg": case ".jpeg": case ".gif":
+ in_dir ~= nm;
+ break;
+ default: break;
+ }
+ }
+ in_dir.sort;
+ foreach (nm; in_dir) {
+ if (nm !in recorded) {
+ extra += 1;
+ lines ~= "EXTRA " ~ nm ~ " (in directory, no record in this .ssp)";
+ }
+ }
+
+ string head = "Verify: " ~ gathered.entries.length.to!string ~ " distinct file"
+ ~ ((gathered.entries.length == 1) ? "" : "s")
+ ~ " in " ~ gathered.records.to!string ~ " record"
+ ~ ((gathered.records == 1) ? "" : "s")
+ ~ ", against " ~ image_dir;
+ lines = [head, ""] ~ lines;
+ lines ~= "";
+ lines ~= "verify: " ~ ok.to!string ~ " ok, "
+ ~ mismatch.to!string ~ " mismatch, "
+ ~ absent.to!string ~ " absent, "
+ ~ extra.to!string ~ " extra"
+ ~ ((failures == 0) ? " PASS" : " FAIL");
+
+ /+ ↓ write output +/
+ if (output_file.length > 0) {
+ auto f = File(output_file, "w");
+ foreach (line; lines) f.writeln(line);
+ } else {
+ foreach (line; lines) writeln(line);
+ }
+ return failures;
+}
+
+private struct FileStat_ {
+ string sha256;
+ ulong bytes;
+ int width;
+ int height;
+}
+
+private FileStat_ readFileStat(string pth) {
+ FileStat_ ret;
+ auto f = File(pth, "rb");
+ ubyte[32] sha = digest!SHA256(f.byChunk(4096 * 1024));
+ ret.sha256 = sha.toHexString.to!string;
+ ret.bytes = pth.getSize;
+ auto dim = readImageDimensions(pth);
+ ret.width = dim[0];
+ ret.height = dim[1];
+ return ret;
+}
+
+/+ ↓ pixel dimensions from the file header, for the formats spine's markup
+ admits (png, gif, jpg). Header read only, no decoding, and no dependency
+ beyond phobos. Returns [0, 0] where the header cannot be read
++/
+private int[2] readImageDimensions(string pth) {
+ int[2] ret = [0, 0];
+ ubyte[] head;
+ try {
+ auto f = File(pth, "rb");
+ head = new ubyte[](64 * 1024);
+ head = f.rawRead(head);
+ } catch (Exception ex) {
+ return ret;
+ }
+ if (head.length >= 24
+ && head[0 .. 8] == cast(ubyte[])[0x89, 'P', 'N', 'G', 0x0d, 0x0a, 0x1a, 0x0a]) {
+ /+ png: IHDR width & height, big endian, at 16 and 20 +/
+ ret[0] = beInt(head[16 .. 20]);
+ ret[1] = beInt(head[20 .. 24]);
+ return ret;
+ }
+ if (head.length >= 10 && (head[0 .. 3] == cast(ubyte[])['G', 'I', 'F'])) {
+ /+ gif: logical screen width & height, little endian, at 6 and 8 +/
+ ret[0] = head[6] | (head[7] << 8);
+ ret[1] = head[8] | (head[9] << 8);
+ return ret;
+ }
+ if (head.length >= 4 && head[0] == 0xff && head[1] == 0xd8) {
+ /+ jpeg: walk the segments to a start of frame, height then width,
+ big endian, at offsets 5 and 7 of the segment
+ +/
+ size_t i = 2;
+ while (i + 9 < head.length) {
+ if (head[i] != 0xff) { i += 1; continue; }
+ ubyte marker = head[i + 1];
+ if (marker == 0xd8 || marker == 0x01
+ || (marker >= 0xd0 && marker <= 0xd7)) { i += 2; continue; }
+ if (marker == 0xff) { i += 1; continue; }
+ size_t seg_len = (head[i + 2] << 8) | head[i + 3];
+ bool is_sof = ((marker >= 0xc0 && marker <= 0xc3)
+ || (marker >= 0xc5 && marker <= 0xc7)
+ || (marker >= 0xc9 && marker <= 0xcb)
+ || (marker >= 0xcd && marker <= 0xcf));
+ if (is_sof) {
+ ret[1] = (head[i + 5] << 8) | head[i + 6];
+ ret[0] = (head[i + 7] << 8) | head[i + 8];
+ return ret;
+ }
+ if (seg_len < 2) break;
+ i += 2 + seg_len;
+ }
+ }
+ return ret;
+}
+
+private int beInt(const(ubyte)[] b) {
+ return cast(int)((b[0] << 24) | (b[1] << 16) | (b[2] << 8) | b[3]);
+}
diff --git a/gleam/README.md b/gleam/README.md
index d586bd3..482db38 100644
--- a/gleam/README.md
+++ b/gleam/README.md
@@ -1,15 +1,18 @@
# ocda-lib, in Gleam
-The abstraction's model and both its loaders, as a Gleam library, and the
-two programs that put it to the question.
-
-| | |
-|-----------------|----------------------------------------------------------------------------------|
-| `ocda/model` | the document, the object, the image. Depends on nothing but the standard library |
-| `ocda/load_ssp` | text -> model |
-| `ocda/load_db` | sqlite -> model |
-| `ocda/load` | what a path is, and loading it |
-| `ocda/dump` | the model written out canonically, as an oracle |
+The abstraction's model and both its loaders, as a Gleam library, with its
+reader and the two programs that put it to the question.
+
+| | |
+|------------------|-----------------------------------------------------------------------------------------|
+| `ocda/model` | the document, the object, the image. Depends on nothing but the standard library |
+| `ocda/load_ssp` | text -> model |
+| `ocda/load_db` | sqlite -> model |
+| `ocda/load` | what a path is, and loading it |
+| `ocda/inventory` | what a file holds beyond the one document: its languages, and what a `.ocda.db` carries |
+| `ocda/dump` | the model written out canonically, as an oracle |
+| `read/main` | `ocda-read`, the reader |
+| `ocda_ffi.erl` | a sha256, from OTP's crypto, which the standard library lacks |
## Use
@@ -27,6 +30,23 @@ case load.load(path, "en", False) {
}
```
+## The reader
+
+```sh
+nix build
+./result/bin/ocda-read [--tree|--outline|--languages|--files|--report-unknown]
+ <file>.ssp|<file>.ocda.db|<file>.zip [lang]
+```
+
+`ocda-lib read`, installed also as `ocda-read`. The modes the Gleam readers
+had, over the library, so each reads every form this library reads; the
+others are the D reader's. **Its output is the D `ocda-read`'s, byte for
+byte**, in every mode both have. In `src/read/`, because Gleam compiles
+nothing outside `src/`; a program depending on the library compiles it too,
+and does not use it.
+
+A `.sisupod` it refuses, saying why: the zstd frame, below.
+
## The two programs
```sh
@@ -97,6 +117,6 @@ one and the other must change with it.
## Format
-Specified in **sisudoc-abstraction-format**: `ocda-model.md` for the model,
-`ssp-format-specification.md` and `ocda-db-format-specification.md` for the
+Specified in **sisudoc-abstraction-format**: `ocda-model.org` for the model,
+`ssp-format-specification.org` and `ocda-db-format-specification.org` for the
two serialisations. At format **2.x**.
diff --git a/gleam/flake.lock b/gleam/flake.lock
index 5f236ab..12c4c3b 100644
--- a/gleam/flake.lock
+++ b/gleam/flake.lock
@@ -2,18 +2,15 @@
"nodes": {
"nixpkgs": {
"locked": {
- "lastModified": 1790185690,
- "narHash": "sha256-xJ+X4hBtOcAFGBOe5nAMyMUeF9foJBmIOu3NjBqBycU=",
- "owner": "NixOS",
- "repo": "nixpkgs",
- "rev": "4975466d324710c576dc11ad614684e6bd8cad8e",
- "type": "github"
+ "lastModified": 1790652569,
+ "narHash": "sha256-k8Fu4c9Z+4Nh7mUr0cfw++ITQiyEhlWxoJBOkI3tOcQ=",
+ "rev": "b6c8664de9b6cc07fe5666a29f91884ba81197c4",
+ "type": "tarball",
+ "url": "https://releases.nixos.org/nixpkgs/nixpkgs-26.11pre1082290.b6c8664de9b6/nixexprs.tar.zst"
},
"original": {
- "owner": "NixOS",
- "ref": "nixos-unstable",
- "repo": "nixpkgs",
- "type": "github"
+ "type": "tarball",
+ "url": "https://channels.nixos.org/nixpkgs-unstable/nixexprs.tar.zst"
}
},
"root": {
diff --git a/gleam/flake.nix b/gleam/flake.nix
index a5a3407..132e3fc 100644
--- a/gleam/flake.nix
+++ b/gleam/flake.nix
@@ -1,6 +1,6 @@
{
description = "ocda-lib: the object-centric document abstraction as a Gleam library";
- inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
+ inputs.nixpkgs.url = "https://channels.nixos.org/nixpkgs-unstable/nixexprs.tar.zst";
outputs = { self, nixpkgs }:
let
system = "x86_64-linux"; # adjust for aarch64-linux / darwin
@@ -51,10 +51,14 @@
makeWrapper $out/lib/${name}/entrypoint.sh $out/bin/ocda-lib \
--add-flags run \
--prefix PATH : ${pkgs.lib.makeBinPath [ pkgs.beamPackages.erlang pkgs.coreutils ]}
+ # the reader under its own name, as the D flake installs it
+ makeWrapper $out/lib/${name}/entrypoint.sh $out/bin/ocda-read \
+ --add-flags "run read" \
+ --prefix PATH : ${pkgs.lib.makeBinPath [ pkgs.beamPackages.erlang pkgs.coreutils ]}
runHook postInstall
'';
meta = {
- description = "the abstraction as a Gleam library";
+ description = "the abstraction as a Gleam library, with its reader, dump and agreement check";
mainProgram = "ocda-lib";
};
};
diff --git a/gleam/src/ocda/inventory.gleam b/gleam/src/ocda/inventory.gleam
new file mode 100644
index 0000000..6047313
--- /dev/null
+++ b/gleam/src/ocda/inventory.gleam
@@ -0,0 +1,138 @@
+//// What a source holds, beyond the one document the model is loaded from.
+////
+//// The model is one language of one document. Two questions a reader asks
+//// are about the file instead: which languages it holds, and, for a
+//// database, what else it carries. They are here rather than in a reader
+//// because the answer depends on the input form, which is the library's to
+//// know, and so that a reader asking them does not learn which form it was
+//// handed. A mirror of the D library's `ocda.inventory`.
+////
+//// `languages` answers for every form, and answers the same: the languages
+//// in the order the manifest gave them (which every `.ssp` records, in
+//// `@source languages`), each with the markup file it was made from and the
+//// sha256 of its `.ssp`. The database stores that digest; for a `.ssp` and a
+//// pod it is taken from the bytes, which are the same bytes.
+////
+//// `carried` is the database's: a `.ocda.db` holds the markup, images,
+//// configuration and catalogues it was built from, a `.ssp` holds none of
+//// them, and a pod's are its archive entries rather than anything the
+//// abstraction says.
+
+import gleam/bit_array
+import gleam/dict
+import gleam/list
+import gleam/result
+import gleam/string
+import ocda/load
+import ocda/load_db
+import ocda/load_pod
+import ocda/load_ssp
+import ocda/zip
+import simplifile
+
+pub type Language {
+ Language(lang: String, source_filename: String, ssp_digest: String)
+}
+
+pub type Carried {
+ Carried(
+ role: String,
+ compression: String,
+ count: String,
+ bytes: String,
+ stored: String,
+ )
+}
+
+pub fn languages(path: String) -> Result(List(Language), String) {
+ case load.source_of(path) {
+ load.OcdaDb -> {
+ use held <- result.try(load_db.languages_of_file(path))
+ Ok(
+ list.map(held, fn(l) {
+ Language(l.lang, l.source_filename, l.ssp_digest)
+ }),
+ )
+ }
+ load.Ssp -> {
+ use bytes <- result.try(
+ simplifile.read_bits(path)
+ |> result.map_error(fn(e) { path <> ": " <> simplifile.describe_error(e) }),
+ )
+ use text <- result.try(
+ bit_array.to_string(bytes)
+ |> result.replace_error(path <> ": not valid UTF-8"),
+ )
+ let doc = load_ssp.parse(text)
+ Ok([
+ Language(
+ source_value(doc.source, "language"),
+ doc.source_filename,
+ sha256_hex(bytes),
+ ),
+ ])
+ }
+ load.Pod -> {
+ use archive <- result.try(load_pod.archive_bytes(path))
+ use entries <- result.try(zip.entries(archive))
+ use found <- result.try(
+ zip.find_suffix(entries, ".ssp")
+ |> list.try_map(fn(e) {
+ use bytes <- result.try(zip.read(archive, e))
+ use text <- result.try(
+ bit_array.to_string(bytes)
+ |> result.replace_error(e.name <> ": not valid UTF-8"),
+ )
+ let doc = load_ssp.parse(text)
+ let lang = result.unwrap(load_pod.lang_of(e.name), "")
+ Ok(#(
+ Language(lang, doc.source_filename, sha256_hex(bytes)),
+ string.split(source_value(doc.source, "languages"), " "),
+ ))
+ }),
+ )
+ let order = case found {
+ [#(_, o), ..] -> list.filter(o, fn(l) { l != "" })
+ [] -> []
+ }
+ let by_lang = list.map(found, fn(f) { #({ f.0 }.lang, f.0) }) |> dict.from_list
+ // the manifest's order, and anything it did not name after it
+ let named = list.filter_map(order, fn(l) { dict.get(by_lang, l) })
+ let rest =
+ list.filter(found, fn(f) { !list.contains(order, { f.0 }.lang) })
+ |> list.map(fn(f) { f.0 })
+ Ok(list.append(named, rest))
+ }
+ load.Unknown -> Error(path <> ": " <> load.source_name(load.Unknown))
+ }
+}
+
+/// Only a `.ocda.db` carries files.
+pub fn is_carrier(path: String) -> Bool {
+ load.source_of(path) == load.OcdaDb
+}
+
+pub fn carried(path: String) -> Result(List(Carried), String) {
+ case is_carrier(path) {
+ False -> Ok([])
+ True -> {
+ use rows <- result.try(load_db.carried_of_file(path))
+ Ok(
+ list.filter_map(rows, fn(r) {
+ case r {
+ [role, compression, n, bytes, stored] ->
+ Ok(Carried(role, compression, n, bytes, stored))
+ _ -> Error(Nil)
+ }
+ }),
+ )
+ }
+ }
+}
+
+fn source_value(source: dict.Dict(String, String), key: String) -> String {
+ result.unwrap(dict.get(source, key), "")
+}
+
+@external(erlang, "ocda_ffi", "sha256_hex")
+fn sha256_hex(bytes: BitArray) -> String
diff --git a/gleam/src/ocda/load_db.gleam b/gleam/src/ocda/load_db.gleam
index 641923e..3ae4f0b 100644
--- a/gleam/src/ocda/load_db.gleam
+++ b/gleam/src/ocda/load_db.gleam
@@ -5,7 +5,7 @@
//// then a function from the model to bytes and cannot tell which
//// serialisation it was given, exactly as spine's own writers cannot.
////
-//// Written against `ocda-db-format-specification.md`, whose section 4
+//// Written against `ocda-db-format-specification.org`, whose section 4
//// gives the column by column correspondence this module implements.
import gleam/dict
@@ -112,6 +112,7 @@ pub fn load_file(
)
let document = metadata(conn, chosen.doc_id, document)
let document = objects(conn, chosen.doc_id, document)
+ let document = count_unread(conn, chosen.doc_id, document)
Ok(document)
}
@@ -561,3 +562,108 @@ fn json_joined(value: String) -> String {
parts -> string.join(parts, "|")
}
}
+
+/// What the database holds and the model does not.
+///
+/// The text loader meets every property line and counts those it does not
+/// keep, so a reader can say what was read and not understood. This loader
+/// asks for columns by name, and would never see the ones it did not ask
+/// for: it would report that it had understood everything, which is the
+/// silence the count exists to prevent. So it counts them, on the
+/// conditions under which the `.ssp` writes the same property
+/// (specification section 6.3), and the report is the same from either
+/// form, as the D loader's is.
+fn count_unread(
+ conn: sqlight.Connection,
+ doc_id: Int,
+ document: Document,
+) -> Document {
+ let id = int.to_string(doc_id)
+ let counts = [
+ #("ancestors_collapsed", "coalesce(ancestors_collapsed,'') != ''"),
+ #("dom_status", "coalesce(dom_status,'') != ''"),
+ #("dom_status_collapsed", "coalesce(dom_status_collapsed,'') != ''"),
+ #("heading_lev_collapsed", "heading_lev_collapsed is not null"),
+ #("parent_lev", "coalesce(parent_lev,0) != 0"),
+ #("segment_lv4_is", "coalesce(segment_lv4_is,'') != ''"),
+ #("attrib", "coalesce(attrib,'') != ''"),
+ ]
+ let property =
+ list.fold(counts, document.unknown.property, fn(acc, c) {
+ let n =
+ count(
+ conn,
+ "select count(*) from objects where doc_id = " <> id <> " and " <> c.1,
+ )
+ case n > 0 {
+ True -> dict.insert(acc, c.0, n)
+ False -> acc
+ }
+ })
+ let property = case has(conn, "object_subtoc") {
+ False -> property
+ True -> {
+ let n =
+ count(
+ conn,
+ "select count(*) from object_subtoc s join objects o
+ on o.id = s.object_id where o.doc_id = " <> id,
+ )
+ case n > 0 {
+ True -> dict.insert(property, "lev4_subtoc", n)
+ False -> property
+ }
+ }
+ }
+ model.Document(
+ ..document,
+ unknown: model.Unknown(..document.unknown, property: property),
+ )
+}
+
+fn count(conn: sqlight.Connection, sql: String) -> Int {
+ let d = decode.field(0, decode.int, decode.success)
+ case sqlight.query(sql, on: conn, with: [], expecting: d) {
+ Ok([n, ..]) -> n
+ _ -> 0
+ }
+}
+
+/// Every language the database at `path` holds, in manifest order.
+pub fn languages_of_file(path: String) -> Result(List(Language), String) {
+ use conn <- with_connection("file:" <> path <> "?mode=ro", path)
+ Ok(languages(conn))
+}
+
+/// What the database at `path` carries besides the abstraction, by role
+/// and storage: role, compression, count, original bytes, stored bytes.
+pub fn carried_of_file(path: String) -> Result(List(List(String)), String) {
+ use conn <- with_connection("file:" <> path <> "?mode=ro", path)
+ case has(conn, "files") {
+ False -> Ok([])
+ True -> {
+ let d = {
+ use role <- decode.field(0, decode.string)
+ use compression <- decode.field(1, decode.string)
+ use n <- decode.field(2, decode.int)
+ use bytes <- decode.field(3, decode.int)
+ use stored <- decode.field(4, decode.int)
+ decode.success([
+ role,
+ compression,
+ int.to_string(n),
+ int.to_string(bytes),
+ int.to_string(stored),
+ ])
+ }
+ sqlight.query(
+ "select role, compression, count(*), sum(bytes), sum(length(data))
+ from files group by role, compression order by role",
+ on: conn,
+ with: [],
+ expecting: d,
+ )
+ |> result.map_error(fn(e) { path <> ": " <> e.message })
+ }
+ }
+}
diff --git a/gleam/src/ocda/load_pod.gleam b/gleam/src/ocda/load_pod.gleam
index 35483c6..05f7add 100644
--- a/gleam/src/ocda/load_pod.gleam
+++ b/gleam/src/ocda/load_pod.gleam
@@ -89,7 +89,7 @@ fn langs(entries: List(zip.Entry)) -> List(String) {
}
/// `<uid>.<lang>.ssp` -> `<lang>`
-fn lang_of(name: String) -> Result(String, Nil) {
+pub fn lang_of(name: String) -> Result(String, Nil) {
case string.ends_with(name, ".ssp") {
False -> Error(Nil)
True -> {
@@ -104,7 +104,7 @@ fn lang_of(name: String) -> Result(String, Nil) {
/// The bytes decide, not the name: a file misnamed either way is treated by
/// what it holds.
-fn archive_bytes(path: String) -> Result(BitArray, String) {
+pub fn archive_bytes(path: String) -> Result(BitArray, String) {
case simplifile.read_bits(path) {
Error(e) -> Error(path <> ": " <> simplifile.describe_error(e))
Ok(bytes) ->
@@ -133,6 +133,6 @@ fn is_zstd_frame(bytes: BitArray) -> Bool {
@external(erlang, "unicode", "characters_to_binary")
fn bit_array_to_string_ffi(b: BitArray) -> String
-fn bit_array_to_string(b: BitArray) -> Result(String, Nil) {
+pub fn bit_array_to_string(b: BitArray) -> Result(String, Nil) {
Ok(bit_array_to_string_ffi(b))
}
diff --git a/gleam/src/ocda/load_ssp.gleam b/gleam/src/ocda/load_ssp.gleam
index ec93ef5..324319e 100644
--- a/gleam/src/ocda/load_ssp.gleam
+++ b/gleam/src/ocda/load_ssp.gleam
@@ -4,7 +4,7 @@
//// apart by its first character, so the loader is a fold over the lines
//// carrying the state the shapes need.
////
-//// Written against `ssp-format-specification.md`. Where it and the D
+//// Written against `ssp-format-specification.org`. Where it and the D
//// library differ the D library is the bug, or this is: they are meant to
//// give the same model, and `ocda-dump` says whether they do.
diff --git a/gleam/src/ocda_ffi.erl b/gleam/src/ocda_ffi.erl
new file mode 100644
index 0000000..2e395cd
--- /dev/null
+++ b/gleam/src/ocda_ffi.erl
@@ -0,0 +1,7 @@
+%% The one thing ocda-lib asks of Erlang directly: a sha256, which OTP's
+%% crypto has and the Gleam standard library does not. Upper case hex, as
+%% the database stores it and the D library writes it.
+-module(ocda_ffi).
+-export([sha256_hex/1]).
+
+sha256_hex(Bin) -> binary:encode_hex(crypto:hash(sha256, Bin)).
diff --git a/gleam/src/ocda_lib.gleam b/gleam/src/ocda_lib.gleam
index 1e8a5ae..6eccaff 100644
--- a/gleam/src/ocda_lib.gleam
+++ b/gleam/src/ocda_lib.gleam
@@ -1,10 +1,11 @@
//// ocda-lib, in Gleam: the abstraction as a library, put to the question.
////
-//// A library has no binary, so this is the two programs that test its
-//// claims.
+//// A library has no binary, so this is the programs over it: the reader,
+//// and the two that test its claims.
////
//// ocda-lib dump [--unknown] <file>.ssp|<file>.ocda.db [lang]
//// ocda-lib agree <doc>.ssp <doc>.ocda.db [lang]
+//// ocda-lib read [options] <file> [lang] (also installed as ocda-read)
////
//// `dump` writes the whole loaded model out canonically, byte for byte as
//// the D library's `ocda-dump` writes it, so that the two implementations
@@ -20,6 +21,7 @@ import gleam/string
import ocda/dump
import ocda/load
import ocda/model.{type Document}
+import read/main as reader
pub fn main() {
case argv.load().arguments {
@@ -29,6 +31,11 @@ pub fn main() {
["dump", path, lang] -> dump_file(path, lang, False)
["agree", ssp, db] -> agree(ssp, db, "")
["agree", ssp, db, lang] -> agree(ssp, db, lang)
+ ["read", ..rest] ->
+ case reader.run(rest) {
+ 0 -> Nil
+ code -> halt(code)
+ }
_ -> usage()
}
}
@@ -36,7 +43,8 @@ pub fn main() {
fn usage() -> Nil {
io.println_error(
"usage: ocda-lib dump [--unknown] <file>.ssp|<file>.ocda.db [lang]
- ocda-lib agree <doc>.ssp <doc>.ocda.db [lang]",
+ ocda-lib agree <doc>.ssp <doc>.ocda.db [lang]
+ ocda-lib read [--help] [options] <file> [lang]",
)
halt(2)
}
diff --git a/gleam/src/read/main.gleam b/gleam/src/read/main.gleam
new file mode 100644
index 0000000..307b8d8
--- /dev/null
+++ b/gleam/src/read/main.gleam
@@ -0,0 +1,459 @@
+//// ocda-read, in Gleam: reads the abstraction, in any of the forms the
+//// library reads, and says what is in it.
+////
+//// ocda-read [--tree|--outline|--languages|--files|--report-unknown]
+//// <file>.ssp | <file>.ocda.db | <file>.zip [lang]
+////
+//// The modes the Gleam readers had (the summary, the outline, the unknown
+//// report, and the database's languages and files), over the library, so
+//// that each reads every form. Their output is the D ocda-read's, byte for
+//// byte: two readers in unrelated languages, neither written from the other
+//// line by line, agreeing is what says each has read the model and not its
+//// own idea of it. A `.sisupod` is the one form the Gleam library cannot
+//// open (its zstd frame; see `ocda/load_pod`), and it says so.
+
+import gleam/dict
+import gleam/int
+import gleam/io
+import gleam/list
+import gleam/result
+import gleam/string
+import ocda/inventory.{type Language}
+import ocda/load
+import ocda/model.{type Document, type Object}
+
+/// The format major this reader knows.
+const major_read = "2"
+
+type Options {
+ Options(
+ tree: Bool,
+ languages: Bool,
+ files: Bool,
+ report_unknown: Bool,
+ any_version: Bool,
+ lang: String,
+ positional: List(String),
+ )
+}
+
+/// Run with the arguments after the program name; returns the exit status.
+pub fn run(args: List(String)) -> Int {
+ case parse(args, Options(False, False, False, False, False, "", [])) {
+ Error(message) -> {
+ io.println_error("ocda-read: " <> message <> "; --help lists the options")
+ 2
+ }
+ Ok(Options(positional: [], ..)) -> {
+ help()
+ 0
+ }
+ Ok(options) -> read(options)
+ }
+}
+
+fn parse(args: List(String), o: Options) -> Result(Options, String) {
+ case args {
+ [] -> Ok(Options(..o, positional: list.reverse(o.positional)))
+ ["--help", ..] | ["-h", ..] -> Ok(Options(..o, positional: []))
+ ["--tree", ..rest] | ["--outline", ..rest] ->
+ parse(rest, Options(..o, tree: True))
+ ["--languages", ..rest] -> parse(rest, Options(..o, languages: True))
+ ["--files", ..rest] -> parse(rest, Options(..o, files: True))
+ ["--report-unknown", ..rest] ->
+ parse(rest, Options(..o, report_unknown: True))
+ ["--any-version", ..rest] -> parse(rest, Options(..o, any_version: True))
+ ["--lang", code, ..rest] -> parse(rest, Options(..o, lang: code))
+ [flag, ..rest] ->
+ case string.starts_with(flag, "--lang=") {
+ True -> parse(rest, Options(..o, lang: string.drop_start(flag, 7)))
+ False ->
+ case string.starts_with(flag, "-") {
+ True ->
+ case
+ list.contains(
+ [
+ "--html", "--llm-text", "--llm-jsonl", "--images", "--verify",
+ "--validate",
+ ],
+ flag,
+ )
+ {
+ True ->
+ Error(flag <> " is a mode of the D ocda-read, not of this one")
+ False -> Error("Unrecognized option " <> flag)
+ }
+ False ->
+ parse(rest, Options(..o, positional: [flag, ..o.positional]))
+ }
+ }
+ }
+}
+
+fn help() -> Nil {
+ io.println("ocda-read - reads the document abstraction, in any of its forms")
+ io.println("")
+ io.println(
+ "Usage: ocda-read [options] <file>.ssp|<file>.ocda.db|<file>.zip [lang]",
+ )
+ io.println("")
+ io.println(" --tree the headings, as a tree")
+ io.println(" --outline the same as --tree")
+ io.println(" --languages the languages the file holds")
+ io.println(" --files what a .ocda.db carries besides the abstraction")
+ io.println(" --report-unknown what was read and not understood")
+ io.println(" --lang=<code> the language to read (default: the first)")
+ io.println(" --any-version read a format major this reader does not know")
+}
+
+fn read(o: Options) -> Int {
+ let assert [path, ..more] = o.positional
+ let lang = case o.lang, more {
+ "", [given, ..] -> given
+ l, _ -> l
+ }
+ let lang_named = lang != ""
+ let loaded = {
+ use held <- result.try(inventory.languages(path))
+ let lang = case lang, held {
+ "", [first, ..] -> first.lang
+ l, _ -> l
+ }
+ use _ <- result.try(case
+ held != [] && !list.any(held, fn(l) { l.lang == lang })
+ {
+ True ->
+ Error(
+ "no language '"
+ <> lang
+ <> "' in "
+ <> path
+ <> "; it holds: "
+ <> string.join(list.map(held, fn(l) { l.lang }), " "),
+ )
+ False -> Ok(Nil)
+ })
+ use doc <- result.try(load.load(path, lang, o.any_version))
+ Ok(#(held, lang, doc))
+ }
+ case loaded {
+ Error(message) -> {
+ io.println_error("Error reading " <> path <> ": " <> message)
+ 2
+ }
+ Ok(#(held, lang, doc)) ->
+ case gate(path, doc, o.any_version) {
+ False -> 3
+ True -> {
+ summary(path, doc, held, lang, lang_named)
+ case o.languages {
+ True -> print_languages(held)
+ False -> Nil
+ }
+ case o.files {
+ True -> print_files(path)
+ False -> Nil
+ }
+ case o.tree {
+ True -> tree(doc)
+ False -> Nil
+ }
+ case o.report_unknown {
+ True -> report_unknown(doc)
+ False -> Nil
+ }
+ case o.tree || o.languages || o.files || o.report_unknown {
+ True -> Nil
+ False ->
+ io.println(
+ "\n(this is the summary: --help lists the modes, each adding its output)",
+ )
+ }
+ 0
+ }
+ }
+ }
+}
+
+/// The major part, and refuse what is not known.
+fn gate(path: String, doc: Document, any_version: Bool) -> Bool {
+ case doc.format_version, model.version_major(doc) == major_read, any_version {
+ "", _, _ -> {
+ io.println_error("warning: " <> path <> " declares no format version")
+ True
+ }
+ _, True, _ -> True
+ v, False, True -> {
+ io.println_error(
+ "warning: "
+ <> path
+ <> " is format v"
+ <> v
+ <> ", this reader knows v"
+ <> major_read
+ <> ".x; reading anyway",
+ )
+ True
+ }
+ v, False, False -> {
+ io.println_error(
+ "error: "
+ <> path
+ <> " is format v"
+ <> v
+ <> ", this reader knows v"
+ <> major_read
+ <> ".x",
+ )
+ io.println_error(
+ " a major version differs: a property may have changed meaning or gone away.",
+ )
+ io.println_error(" --any-version reads it regardless.")
+ False
+ }
+ }
+}
+
+const sections = [
+ "head", "toc", "body", "endnotes", "glossary", "bibliography", "bookindex",
+ "blurb", "tail",
+]
+
+/// The sections whose images are counted: every one but the tail, as the D
+/// reader counts them.
+const image_sections = [
+ "head", "toc", "body", "endnotes", "glossary", "bibliography", "bookindex",
+ "blurb",
+]
+
+fn summary(
+ path: String,
+ doc: Document,
+ held: List(Language),
+ lang: String,
+ lang_named: Bool,
+) -> Nil {
+ io.println("Read: " <> path)
+ case doc.format_version {
+ "" -> Nil
+ v -> io.println(" Format: v" <> v)
+ }
+ case dict.get(doc.source, "language") {
+ Error(_) -> Nil
+ Ok(language) -> {
+ let of = case dict.get(doc.source, "languages") {
+ Ok(ls) -> " of " <> ls
+ Error(_) -> ""
+ }
+ let digest = case dict.get(doc.source, "digest") {
+ Ok(d) -> ", markup digest " <> string.slice(d, 0, 16) <> "..."
+ Error(_) -> ""
+ }
+ io.println(" Source: " <> language <> of <> digest)
+ }
+ }
+ case list.length(held) > 1 && !lang_named {
+ True -> io.println(" (reading " <> lang <> "; --lang=<code> for another)")
+ False -> Nil
+ }
+ io.println(" Title: " <> model.meta(doc, "title.main"))
+ io.println(" Author: " <> model.meta(doc, "creator.author"))
+ io.println(
+ " Sections: "
+ <> string.join(
+ list.map(sections, fn(s) {
+ s <> "=" <> int.to_string(list.length(model.section(doc, s)))
+ }),
+ " ",
+ ),
+ )
+ let records =
+ list.fold(image_sections, 0, fn(n, s) {
+ list.fold(model.section(doc, s), n, fn(n, o) {
+ n + list.length(o.images)
+ })
+ })
+ case records {
+ 0 -> Nil
+ 1 -> io.println(" Images: 1 record")
+ n -> io.println(" Images: " <> int.to_string(n) <> " records")
+ }
+}
+
+fn print_languages(held: List(Language)) -> Nil {
+ io.println("")
+ io.println(" lang source ssp digest")
+ list.each(held, fn(l) {
+ let digest = case string.length(l.ssp_digest) > 16 {
+ True -> string.slice(l.ssp_digest, 0, 16) <> "..."
+ False -> l.ssp_digest
+ }
+ io.println(
+ " "
+ <> string.pad_end(l.lang, 7, " ")
+ <> " "
+ <> string.pad_end(l.source_filename, 31, " ")
+ <> " "
+ <> digest,
+ )
+ })
+}
+
+fn print_files(path: String) -> Nil {
+ io.println("")
+ case inventory.is_carrier(path) {
+ False ->
+ io.println(
+ " carried files: only a .ocda.db carries files; this is "
+ <> load.source_name(load.source_of(path)),
+ )
+ True ->
+ case inventory.carried(path) {
+ Error(message) -> io.println_error("error: " <> message)
+ Ok([]) -> io.println(" carries no files")
+ Ok(rows) -> {
+ io.println(" carried files:")
+ io.println(" role stored n original in the file")
+ list.each(rows, fn(r) {
+ io.println(
+ " "
+ <> string.pad_end(r.role, 9, " ")
+ <> " "
+ <> string.pad_end(r.compression, 7, " ")
+ <> " "
+ <> string.pad_end(r.count, 5, " ")
+ <> " "
+ <> string.pad_end(r.bytes, 13, " ")
+ <> " "
+ <> r.stored,
+ )
+ })
+ }
+ }
+ }
+}
+
+/// The headings, as a tree: from the document's root heading, down the
+/// children each heading lists, as the D reader walks it.
+fn tree(doc: Document) -> Nil {
+ io.println("Document: " <> model.meta(doc, "title.main"))
+ case model.meta(doc, "title.subtitle") {
+ "" -> Nil
+ sub -> io.println(" " <> sub)
+ }
+ io.println("Author: " <> model.meta(doc, "creator.author"))
+ io.println("")
+ let headings =
+ model.section(doc, "body")
+ |> list.filter(fn(o) { o.is_a == "heading" && o.ocn > 0 })
+ |> list.map(fn(o) { #(o.ocn, o) })
+ |> dict.from_list
+ case model.section(doc, "head") {
+ [root, ..] if root.is_a == "heading" -> {
+ io.println(heading_line(root, 0))
+ walk(root, headings, 1)
+ }
+ _ -> Nil
+ }
+}
+
+fn walk(parent: Object, headings: dict.Dict(Int, Object), depth: Int) -> Nil {
+ list.each(parent.children, fn(ocn) {
+ case dict.get(headings, ocn) {
+ Ok(child) -> {
+ io.println(heading_line(child, depth))
+ walk(child, headings, depth + 1)
+ }
+ Error(_) -> Nil
+ }
+ })
+}
+
+fn heading_line(o: Object, depth: Int) -> String {
+ let level = case o.heading_level {
+ "" -> "?"
+ l -> l
+ }
+ let ocn = case o.ocn > 0 {
+ True -> " [" <> int.to_string(o.ocn) <> "]"
+ False -> ""
+ }
+ string.repeat(" ", depth) <> level <> "~" <> ocn <> " " <> strip_markup(o.text)
+}
+
+/// The internal markers a heading's text may carry, removed for display, as
+/// the D reader removes them.
+fn strip_markup(text: String) -> String {
+ text
+ |> string.replace("⑆*", "")
+ |> string.replace("⑆!", "")
+ |> string.replace("⑆/", "")
+ |> string.replace("⑆_", "")
+ |> string.replace("⑆^", "")
+ |> string.replace("⑆,", "")
+ |> string.replace("┨", "")
+ |> string.replace("┣", "")
+ |> string.replace("┙", " ")
+ |> string.replace("┘", "\n")
+}
+
+/// Say what was read and not understood.
+fn report_unknown(doc: Document) -> Nil {
+ let u = doc.unknown
+ case model.any_unknown(u) {
+ False ->
+ io.println(
+ "\nread and understood: every property, block and line was known",
+ )
+ True -> {
+ io.println("\nread and NOT understood:")
+ section_of_unknown(u.property, " properties ignored (", fn(name, n) {
+ " ."
+ <> string.pad_end(name, 26, " ")
+ <> " "
+ <> string.pad_start(int.to_string(n), 8, " ")
+ })
+ section_of_unknown(u.block, " blocks ignored (", fn(name, n) {
+ " @"
+ <> string.pad_end(name, 26, " ")
+ <> " "
+ <> string.pad_start(int.to_string(n), 8, " ")
+ })
+ case dict.size(u.line) {
+ 0 -> Nil
+ _ -> {
+ io.println(" lines of no known shape, by first character:")
+ dict.to_list(u.line)
+ |> list.sort(fn(a, b) { string.compare(a.0, b.0) })
+ |> list.each(fn(e) {
+ io.println(
+ " "
+ <> string.pad_end("\"" <> e.0 <> "\"", 27, " ")
+ <> " "
+ <> string.pad_start(int.to_string(e.1), 8, " "),
+ )
+ })
+ }
+ }
+ io.println(
+ " (ignoring these is conforming within a major version; the count is so that it is not silent)",
+ )
+ }
+ }
+}
+
+fn section_of_unknown(
+ counts: dict.Dict(String, Int),
+ heading: String,
+ line: fn(String, Int) -> String,
+) -> Nil {
+ case dict.size(counts) {
+ 0 -> Nil
+ n -> {
+ io.println(heading <> int.to_string(n) <> " distinct):")
+ dict.to_list(counts)
+ |> list.sort(fn(a, b) { string.compare(a.0, b.0) })
+ |> list.each(fn(e) { io.println(line(e.0, e.1)) })
+ }
+ }
+}
diff --git a/gleam/test/ocda_lib_test.gleam b/gleam/test/ocda_lib_test.gleam
index 02fe463..c0e10e5 100644
--- a/gleam/test/ocda_lib_test.gleam
+++ b/gleam/test/ocda_lib_test.gleam
@@ -36,7 +36,7 @@ const minimal = "% SiSU Document Abstraction v2.0
.part: body
.parent: 1
| First line
-|
+|
| Third line, after a blank one
}
generated by cgit v1.2.3 (git 2.25.1) at 2026-11-15 13:37:42 +0000