summaryrefslogtreecommitdiffhomepage
path: root/dlang/src
diff options
context:
space:
mode:
authorRalph Amissah <ralph.amissah@gmail.com>2026-09-26 09:47:07 -0400
committerRalph Amissah <ralph.amissah@gmail.com>2026-09-26 09:47:07 -0400
commitfda48e417435e31a3122bfa7ca563333628aed99 (patch)
treec91f50006de299a9db42247bc21df4f59f9c7a82 /dlang/src
format 2.0: abstraction as library, in D and Gleam
The object-centric document abstraction that sisudoc-spine builds from SiSU markup, read as a library: the model, and a loader for each form it is written in - the .ssp (text), the .ocda.db (sqlite), and a pod archive carrying a .ssp (.sisupod, .zip). Begun 2026-09-25; this is the library as it stood at format 2.0. The readers (sisudoc-ssp-reader, sisudoc-ocda-db-reader) each read one form and print. A writer needs neither: it needs the document. So the model and the loaders live here, ocda.model imports nothing, and a program built on the library cannot tell which form it was handed. ocda.load is the one place that decides what a path is, so a new input form is added once and every program gains it. The model and the .ssp loader began as the .ssp reader's code, moved and renamed to the level they belong to; the .ocda.db loader is new. The Gleam library is a deliberate field-for-field mirror of the D one, including where that is not the most idiomatic Gleam: a difference in one language and not the other would show in every comparison and mean nothing. The claim holds only if the loaders agree, so the library asks: ocda-agree <doc>.ssp <doc>.ocda.db [lang] ten fields, per object ocda-dump <file> [lang] the whole model, canonically The dump is an oracle, not an output format, and the D and Gleam dumps are byte identical, so four questions are a diff: D and Gleam, each from the .ssp and from the .ocda.db. Over the markup document samples all four hold, every document-language; a .sisupod gives the same model as the .ssp beside it. None of that held at first, and what the checks found is the record of how the specification became enough: - an object whose first text line is blank lost it (the loader asked whether the text was empty, not whether a line had been seen); seven sample documents open verse that way. Found by agree; fixed here and in the .ssp reader, which carried the same code. - found by the dump: the identifier and the anchor are omitted by the text form when they can be derived and stored by the database always, so the loaders derive them, and the rule went into the specification (derived values); make and doc_has share one table in the database under a key prefix, which the loaders strip; the JSON columns were scanned, not parsed, and Phobos writes "/" as "\/"; and translation.ocn_aligned is a fact about the file, not the document, so neither loader puts it in the model. - in Gleam, a JSON element decoder fell through to a fallback giving "", so a float column read as empty strings: a value that could not be read turned into one that looked read. - found by the writers: anchor_tags, heading_lev_anchor, segment_epub and segment_epub_is were in the specification and read by nothing. ocda.images gives an image's bytes from whichever form the document came from (the database's files table, the archive entry, or beside a .ssp at media/image/), and says beforehand whether it can. The Gleam library reads a .zip pod but not a .sisupod: the zstd frame around it has no implementation on the BEAM that builds sandboxed. It says so, and says zstd -d, rather than calling the file malformed. gleam/ builds with nix-gleam's builder, kept in gleam/nix-gleam/ (MIT) Use: dlang/ nix build ocda-dump, ocda-agree; or a dub path dependency on dlang/ gleam/ nix build ocda-lib dump | agree See README.md, and dlang/README.md and gleam/README.md. (assisted by Claude-Code)
Diffstat (limited to 'dlang/src')
-rw-r--r--dlang/src/COPYRIGHT45
-rw-r--r--dlang/src/ocda/dump.d164
-rw-r--r--dlang/src/ocda/images.d100
-rw-r--r--dlang/src/ocda/load.d69
-rw-r--r--dlang/src/ocda/load_db.d310
-rw-r--r--dlang/src/ocda/load_pod.d115
-rw-r--r--dlang/src/ocda/load_ssp.d337
-rw-r--r--dlang/src/ocda/model.d147
-rw-r--r--dlang/src/ocda/sqlite.d122
-rw-r--r--dlang/src/ocda/zip.d115
-rw-r--r--dlang/src/ocda/zstd.d79
11 files changed, 1603 insertions, 0 deletions
diff --git a/dlang/src/COPYRIGHT b/dlang/src/COPYRIGHT
new file mode 100644
index 0000000..7e7dd56
--- /dev/null
+++ b/dlang/src/COPYRIGHT
@@ -0,0 +1,45 @@
+- Name: sisudoc-ocda-lib
+ - Description: part of SiSU spine (Doc Reform): documents, structuring,
+ processing, publishing and search
+ - [https://www.sisudoc.org]
+
+ - Author: Ralph Amissah
+ [ralph.amissah@gmail.com]
+
+ - Copyright: (C) 2026 Ralph Amissah, All Rights Reserved.
+
+ - License: AGPL 3 or later, for everything in this repository unless a file or
+ a directory specifies otherwise.:
+
+ sisudoc-ocda-lib
+
+ part of SiSU spine (Doc Reform): documents, structuring,
+ processing, publishing and search
+
+ Copyright (C) Ralph Amissah
+
+ This program is free software: you can redistribute it and/or modify it
+ under the terms of the GNU Affero General Public License (AGPLv3) as
+ published by the Free Software Foundation, either version 3 of the License,
+ or (at your option) any later version.
+
+ This program is distributed in the hope that it will be useful, but
+ WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public
+ License for more details.
+
+ You should have received a copy of the GNU Affero General Public License
+ along with this program. If not, see [https://www.gnu.org/licenses/].
+
+ If you have Internet connection, the latest version of the AGPL should be
+ available at: [https://www.gnu.org/licenses/agpl-3.0.html]
+
+ - Spine, Doc Reform (related to SiSU) uses standard:
+ - docReform markup syntax (based on SiSU markup)
+ - standard SiSU markup syntax with modified headers and minor
+ modifications
+ - docReform object numbering (based on SiSU object citation numbering)
+ - standard SiSU document object numbering
+
+ - Homepages:
+ [https://www.sisudoc.org]
diff --git a/dlang/src/ocda/dump.d b/dlang/src/ocda/dump.d
new file mode 100644
index 0000000..7a4f11a
--- /dev/null
+++ b/dlang/src/ocda/dump.d
@@ -0,0 +1,164 @@
+module ocda.dump;
+
+/+ ↓ the model, written out canonically.
+ .
+ Not an output format: an *oracle*. Two libraries in two languages
+ implement the same model from the same specification, and the cheap
+ way to ask whether they agree is to have each write the whole loaded
+ document as text and diff the two. That is only useful if the text is
+ fully determined by the model, so:
+ .
+ - every line is "<key> <value>", the value running to end of line,
+ and only one free-form value per line;
+ - a field at its default is *omitted*, so the output says what the
+ document has rather than what the struct declares;
+ - dictionaries are emitted in sorted key order, because neither
+ language's hash map has an order worth relying on;
+ - sections come in the fixed order of the format, not
+ alphabetically.
+ .
+ Anything that reads this to reconstruct a document has misunderstood
+ it. Load the .ssp or the .ocda.db.
++/
+
+import std.algorithm : sort, splitter;
+import std.array : appender, array, join;
+import std.conv : to;
+import ocda.model;
+
+/+ ↓ the nine sections, in format order +/
+static immutable string[] dump_sections = ["head", "toc", "body", "endnotes",
+ "glossary", "bibliography", "bookindex", "blurb", "tail"];
+
+string dumpDocument(ref OcdaDocument doc) {
+ auto out_ = appender!string;
+
+ void kv(string key, string value) {
+ if (value.length == 0) { return; }
+ out_ ~= key; out_ ~= " "; out_ ~= value; out_ ~= "\n";
+ }
+ void kn(string key, long value) {
+ if (value == 0) { return; }
+ kv(key, value.to!string);
+ }
+ void kb(string key, bool value) {
+ if (!value) { return; }
+ kv(key, "1");
+ }
+ void dict(string prefix, string[string] d) {
+ foreach (key; d.keys.dup.sort) {
+ out_ ~= prefix; out_ ~= " "; out_ ~= key; out_ ~= " ";
+ out_ ~= d[key]; out_ ~= "\n";
+ }
+ }
+
+ kv("version", doc.format_version);
+ kv("source_filename", doc.source_filename);
+ dict("source", doc.source);
+ dict("meta", doc.metadata);
+ dict("make", doc.make);
+ dict("doc_has", doc.doc_has);
+
+ foreach (name; dump_sections) {
+ auto sec = doc.sectionByName(name);
+ if (sec is null || (*sec).length == 0) { continue; }
+ kv("section", name ~ " " ~ (*sec).length.to!string);
+ }
+
+ foreach (name; dump_sections) {
+ auto sec = doc.sectionByName(name);
+ if (sec is null) { continue; }
+ foreach (i, obj; *sec) {
+ out_ ~= "object "; out_ ~= name; out_ ~= " ";
+ out_ ~= i.to!string; out_ ~= "\n";
+ kn("ocn", obj.ocn);
+ kv("is_a", obj.is_a);
+ kv("own_section", obj.own_section);
+ kv("level", obj.heading_level);
+ kv("identifier", obj.identifier);
+ kv("part", obj.part);
+ kv("is_of_type", obj.is_of_type);
+ kv("segment_html_is", obj.segment_html_is);
+ kv("segment_epub", obj.segment_epub);
+ kv("segment_epub_is", obj.segment_epub_is);
+ kn("parent", obj.parent_ocn);
+ kn("last_descendant", obj.last_descendant_ocn);
+ kv("children", ints(obj.children));
+ kv("ancestors", ints(obj.ancestors));
+ kb("dummy", obj.dummy_heading);
+ kb("ocn_off", obj.ocn_off);
+ kn("indent_base", obj.indent_base);
+ kn("indent_hang", obj.indent_hang);
+ kb("bullet", obj.bullet);
+ kv("lang", obj.lang);
+ kb("has_links", obj.has_links);
+ kb("has_notes_reg", obj.has_notes_reg);
+ kb("has_notes_star", obj.has_notes_star);
+ kb("has_images", obj.has_images);
+ kv("segment", obj.segment);
+ kv("segment_prev", obj.segment_prev);
+ kv("segment_next", obj.segment_next);
+ kv("anchor", obj.anchor);
+ kv("heading_lev_anchor", obj.heading_lev_anchor);
+ kn("table_cols", obj.table_cols);
+ kv("table_widths", obj.table_widths);
+ kv("table_aligns", obj.table_aligns);
+ kb("table_header", obj.table_header);
+ kv("code_syntax", obj.code_syntax);
+ kb("code_linenumbers", obj.code_linenumbers);
+ kv("heading_ancestors_text", obj.heading_ancestors_text);
+ kv("sha256", obj.sha256);
+ foreach (url; obj.stow_link) { kv("stow_link", url); }
+ foreach (a; obj.anchor_tags) { kv("anchor_tag", a); }
+ foreach (img; obj.images) {
+ /+ ↓ the name last, because a filename may hold a space where
+ none of the other fields can +/
+ kv("image", img.sha256 ~ " " ~ img.bytes.to!string ~ " "
+ ~ img.width.to!string ~ "x" ~ img.height.to!string ~ " "
+ ~ (img.missing ? "1" : "0") ~ " " ~ img.name);
+ foreach (key; img.fields.keys.dup.sort) {
+ kv("image_field", key ~ " " ~ img.fields[key]);
+ }
+ }
+ /+ ↓ text, when there is any. An object whose only text line is
+ blank holds "" here, indistinguishable from no text at all,
+ because the model joins the lines into one string. Both
+ libraries lose the same distinction in the same place, which is
+ what makes them comparable; it is recorded rather than hidden.
+ +/
+ if (obj.text.length > 0) {
+ auto lines = obj.text.splitter("\n").array;
+ kv("text", lines.length.to!string);
+ foreach (line; lines) {
+ out_ ~= "| "; out_ ~= line; out_ ~= "\n";
+ }
+ }
+ }
+ }
+ return out_.data;
+}
+
+/+ ↓ what the reader met and did not understand, sorted. Separate from the
+ document dump: it describes the reader, not the document, and the
+ .ocda.db loader has nothing to report where the .ssp loader does.
++/
+string dumpUnknown(ref OcdaDocument doc) {
+ auto out_ = appender!string;
+ void counts(string prefix, size_t[string] d) {
+ foreach (key; d.keys.dup.sort) {
+ out_ ~= prefix; out_ ~= " "; out_ ~= key; out_ ~= " ";
+ out_ ~= d[key].to!string; out_ ~= "\n";
+ }
+ }
+ counts("property", doc.unknown.property);
+ counts("block", doc.unknown.block);
+ counts("line", doc.unknown.line);
+ return out_.data;
+}
+
+private string ints(int[] xs) {
+ if (xs.length == 0) { return ""; }
+ string[] parts;
+ foreach (x; xs) { parts ~= x.to!string; }
+ return parts.join(",");
+}
diff --git a/dlang/src/ocda/images.d b/dlang/src/ocda/images.d
new file mode 100644
index 0000000..b3a9652
--- /dev/null
+++ b/dlang/src/ocda/images.d
@@ -0,0 +1,100 @@
+module ocda.images;
+
+/+ ↓ an image's bytes, from whichever source the document came from.
+ .
+ The model *names* images, with digest, byte size and pixel size; it does
+ not hold them. That is right - an abstraction of a document is not a
+ bundle of its assets - but a writer of any format that shows images
+ needs the bytes, so something has to know where they are. This is that
+ something, and it is in the library rather than in a writer because the
+ answer depends on the *input form* and not on the output.
+ .
+ Three answers, and one refusal:
+ .
+ .ocda.db the files table, role 'image', by name
+ .sisupod the archive entry whose path ends in /image/<name>
+ .zip the same
+ .ssp beside it: a pod puts the abstraction in
+ media/abstraction/ and its images in media/image/, so the
+ path is documented rather than guessed
+ .
+ A .ssp that is not in a pod has nowhere to look, and that is reported as
+ absent rather than as an error: a reader that cannot find an image
+ should say so and carry on, because the document is still readable.
++/
+
+import std.algorithm : endsWith;
+import std.array : array;
+import std.file : exists, isFile, read;
+import std.path : buildPath, dirName;
+import ocda.load;
+import ocda.sqlite;
+import ocda.zip;
+import ocda.zstd;
+
+/+ ↓ the bytes, or an empty slice when the source has no such image +/
+ubyte[] imageBytes(string source, string name) {
+ final switch (sourceOf(source)) {
+ case Source.ocda_db: return fromDb(source, name);
+ case Source.pod: return fromArchive(source, name);
+ case Source.ssp: return fromBeside(source, name);
+ case Source.unknown: return [];
+ }
+}
+
+/+ ↓ can this source give images at all?
+ A writer asks before it decides whether to emit them, because an <img>
+ pointing at a file it did not include is worse than no image.
++/
+bool canGiveImages(string source) {
+ final switch (sourceOf(source)) {
+ case Source.ocda_db: return true;
+ case Source.pod: return true;
+ case Source.ssp: return imageDirBeside(source).length > 0;
+ case Source.unknown: return false;
+ }
+}
+
+private ubyte[] fromDb(string path, string name) {
+ auto db = Db.open(path);
+ scope(exit) { db.close(); }
+ if (!db.has("files")) { return []; }
+ /+ ↓ the name is from the abstraction, not from a caller, and the quote
+ doubling is what keeps it a value rather than syntax +/
+ return db.blob("SELECT data FROM files WHERE role = 'image' AND name = '"
+ ~ sqlQuote(name) ~ "'");
+}
+
+private ubyte[] fromArchive(string path, string name) {
+ auto bytes = cast(ubyte[]) read(path);
+ if (zstdIsFrame(bytes)) { bytes = zstdDecompress(bytes, 512 * 1024 * 1024); }
+ auto entries = zipEntries(bytes);
+ foreach (e; entries) {
+ if (e.name.endsWith("/image/" ~ name)) { return zipRead(bytes, e); }
+ }
+ return [];
+}
+
+private ubyte[] fromBeside(string path, string name) {
+ string dir = imageDirBeside(path);
+ if (dir.length == 0) { return []; }
+ string file = buildPath(dir, name);
+ return file.exists && file.isFile ? cast(ubyte[]) read(file) : [];
+}
+
+/+ ↓ media/abstraction/<x>.ssp -> media/image/ +/
+private string imageDirBeside(string path) {
+ string dir = path.dirName;
+ if (!dir.endsWith("abstraction")) { return ""; }
+ string image_dir = buildPath(dir.dirName, "image");
+ return image_dir.exists ? image_dir : "";
+}
+
+/+ ↓ a single quote doubled is how sql says a single quote +/
+private string sqlQuote(string s) {
+ string out_;
+ foreach (c; s) {
+ out_ ~= (c == '\'') ? "''" : [c];
+ }
+ return out_;
+}
diff --git a/dlang/src/ocda/load.d b/dlang/src/ocda/load.d
new file mode 100644
index 0000000..89c8dcc
--- /dev/null
+++ b/dlang/src/ocda/load.d
@@ -0,0 +1,69 @@
+module ocda.load;
+
+/+ ↓ what a path is, and loading it.
+ .
+ One place that decides, so that every reader and every writer accepts
+ the same inputs and a new input type is added once. This mirrors
+ spine's own ocda/abstraction/load.d, which names the sources it can
+ enter the pipeline from and tells them apart by suffix.
+ .
+ Deliberately *not* here: .sst and .ssm markup. Reading markup means
+ implementing the parser, which is spine's work and a far larger thing
+ than a reader or a sample writer needs. This library reads
+ abstractions.
++/
+
+import std.algorithm : endsWith;
+import ocda.model;
+import ocda.load_ssp;
+import ocda.load_db;
+import ocda.load_pod;
+
+enum Source {
+ ssp, // .ssp, the abstraction as text
+ ocda_db, // .ocda.db, the abstraction as sqlite
+ pod, // .sisupod or .zip, an archive carrying a .ssp
+ unknown,
+}
+
+Source sourceOf(string path) {
+ if (path.endsWith(".ocda.db")) { return Source.ocda_db; }
+ if (path.endsWith(".ssp")) { return Source.ssp; }
+ /+ ↓ .sisupod is the archive spine writes now; .zip is what it wrote
+ before, and every pod already published carries that name, so both
+ are read. What is inside decides how it is opened, not the name.
+ +/
+ if (path.endsWith(".sisupod")) { return Source.pod; }
+ if (path.endsWith(".zip")) { return Source.pod; }
+ if (path.endsWith(".db")) { return Source.ocda_db; }
+ return Source.unknown;
+}
+
+string sourceName(Source s) {
+ final switch (s) {
+ case Source.ssp: return ".ssp (the abstraction as text)";
+ case Source.ocda_db: return ".ocda.db (the abstraction as sqlite)";
+ case Source.pod: return "a pod archive carrying a .ssp";
+ case Source.unknown: return "not an abstraction this library reads";
+ }
+}
+
+class OcdaLoadError : Exception {
+ this(string msg) { super(msg); }
+}
+
+/+ ↓ load whatever the path is into the model.
+ lang names which language to take from a .ocda.db, which holds
+ every language of its document, and from a pod, which may carry more
+ than one .ssp. A .ssp is one language by construction and ignores it.
++/
+OcdaDocument load(string path, string lang = "", bool any_version = false) {
+ final switch (sourceOf(path)) {
+ case Source.ssp: return parseSspFile(path);
+ case Source.ocda_db: return loadDbFile(path, lang, any_version);
+ case Source.pod: return loadPodFile(path, lang);
+ case Source.unknown:
+ throw new OcdaLoadError(path ~ ": " ~ sourceName(Source.unknown)
+ ~ " (expected .ssp, .ocda.db, .sisupod or .zip)");
+ }
+}
diff --git a/dlang/src/ocda/load_db.d b/dlang/src/ocda/load_db.d
new file mode 100644
index 0000000..dec0626
--- /dev/null
+++ b/dlang/src/ocda/load_db.d
@@ -0,0 +1,310 @@
+module ocda.load_db;
+
+/+ ↓ build the model from a .ocda.db.
+ .
+ The point of this module is that it returns *the same value*
+ ocda.load_ssp returns from the same document's .ssp. A writer is then
+ a function from the model to bytes, and cannot tell which
+ serialisation it was given, exactly as spine's own writers cannot.
+ .
+ Written from the sisudoc-abstraction-format specification,
+ ocda-db-format-specification.md, whose section 4 gives the column by
+ column correspondence this module implements.
++/
+
+import std.array : join, split;
+import std.json : JSONType, parseJSON;
+import std.conv : to;
+import std.algorithm : startsWith;
+import std.string : strip;
+import ocda.model;
+import ocda.sqlite;
+
+/+ ↓ the schema major this loader knows (specification section 2) +/
+enum schema_major_read = "2";
+
+class OcdaDbError : Exception {
+ this(string msg) { super(msg); }
+}
+
+/+ ↓ every language the database holds, in manifest order +/
+struct DbLanguage {
+ int doc_id;
+ string lang;
+ string source_filename;
+ string ssp_digest;
+}
+
+DbLanguage[] dbLanguages(ref Db db) {
+ DbLanguage[] out_;
+ foreach (r; db.rows(
+ "SELECT id, lang, coalesce(source_filename,''), coalesce(ssp_digest,'')
+ FROM documents ORDER BY id")) {
+ out_ ~= DbLanguage(r[0].to!int, r[1], r[2], r[3]);
+ }
+ return out_;
+}
+
+/+ ↓ load one language of a database as the model.
+ lang == "" takes the first language the manifest gave.
++/
+OcdaDocument loadDbFile(string path, string lang = "", bool any_version = false) {
+ auto db = Db.open(path);
+ scope(exit) { db.close(); }
+
+ if (!db.has("documents") || !db.has("objects") || !db.has("metadata")) {
+ throw new OcdaDbError(path ~ ": no documents/objects/metadata tables;"
+ ~ " it does not look like a .ocda.db");
+ }
+
+ /+ ↓ specification sections 2 and 3.2: the rows describing the file rather
+ than a document in it are the only ones with doc_id null +/
+ string schema_version = db.one(
+ "SELECT value FROM metadata WHERE doc_id IS NULL AND key = 'schema.version'");
+ if (schema_version.length > 0 && !any_version) {
+ auto major = schema_version.split(".")[0];
+ if (major != schema_major_read) {
+ throw new OcdaDbError(path ~ " is schema v" ~ schema_version
+ ~ ", this loader knows v" ~ schema_major_read ~ ".x");
+ }
+ }
+
+ auto languages = dbLanguages(db);
+ if (languages.length == 0) {
+ throw new OcdaDbError(path ~ ": no documents rows");
+ }
+ DbLanguage chosen = languages[0];
+ if (lang.length > 0) {
+ bool found = false;
+ foreach (l; languages) { if (l.lang == lang) { chosen = l; found = true; break; } }
+ if (!found) {
+ string have;
+ foreach (i, l; languages) { have ~= (i > 0 ? " " : "") ~ l.lang; }
+ throw new OcdaDbError(path ~ ": no language '" ~ lang ~ "'; it holds: " ~ have);
+ }
+ }
+
+ OcdaDocument doc;
+ doc.format_version = schema_version;
+ doc.source_filename = chosen.source_filename;
+
+ /+ ↓ @source of the .ssp corresponds to the documents row and the
+ source.digest metadata (specification section 5.1 of the .ssp, and
+ section 7 here) +/
+ doc.source["language"] = chosen.lang;
+ string languages_joined;
+ foreach (i, l; languages) { languages_joined ~= (i > 0 ? " " : "") ~ l.lang; }
+ doc.source["languages"] = languages_joined;
+ auto digest = db.one("SELECT value FROM metadata WHERE doc_id = "
+ ~ chosen.doc_id.to!string ~ " AND key = 'source.digest'");
+ if (digest.length > 0) { doc.source["digest"] = digest; }
+
+ /+ ↓ @meta: every metadata row of this language that is not a source.*
+ property. source.* describes the artefact rather than the document
+ and lives in @source in the text form.
+ +/
+ foreach (r; db.rows("SELECT key, value FROM metadata WHERE doc_id = "
+ ~ chosen.doc_id.to!string ~ " ORDER BY key")) {
+ string key = r[0];
+ if (key.startsWith("source.")) { continue; }
+ /+ ↓ translation.ocn_aligned says whether the languages of this
+ document share one numbering. A .ocda.db can know it because it
+ holds every language; a .ssp is one language and cannot. So it is
+ a fact about the *file* rather than about the document, and putting
+ it in the document's metadata would let a writer tell which
+ serialisation it was handed, which is the one thing this library
+ exists to prevent. Left out until the model has a place for
+ collection-wide facts.
+ +/
+ if (key.startsWith("translation.")) { continue; }
+ /+ ↓ the database has one metadata table where the text form has three
+ blocks, and tells them apart by a key prefix. Unprefixing them
+ here is what makes the two loaders give one model: a writer asks
+ doc.make["footer"] whichever serialisation it was handed, and
+ never doc.metadata["make.footer"].
+ +/
+ if (key.startsWith("make.")) {
+ doc.make[key["make.".length .. $]] = r[1].strip;
+ } else if (key.startsWith("doc_has.")) {
+ doc.doc_has[key["doc_has.".length .. $]] = r[1].strip;
+ } else {
+ doc.metadata[key] = r[1].strip;
+ }
+ }
+
+ /+ ↓ the objects, in document order within each section +/
+ foreach (r; db.rows(
+ "SELECT section, seq, ocn, is_a, coalesce(is_of_part,''),
+ coalesce(is_of_section,''), coalesce(is_of_type,''),
+ coalesce(heading_level,-1), coalesce(identifier,''),
+ coalesce(parent_ocn,0), coalesce(last_descendant_ocn,0),
+ coalesce(children,''), coalesce(ancestors,''),
+ coalesce(dummy_heading,0), coalesce(object_number_off,0),
+ coalesce(indent_base,0), coalesce(indent_hang,0),
+ coalesce(bullet,0), coalesce(lang,''),
+ coalesce(has_links,0), coalesce(has_notes_reg,0),
+ coalesce(has_notes_star,0), coalesce(has_images,0),
+ coalesce(segment,''), coalesce(segment_prev,''),
+ coalesce(segment_next,''), coalesce(anchor,''),
+ coalesce(segment_html_is,''),
+ coalesce(table_cols,0), coalesce(table_widths,''),
+ coalesce(table_aligns,''), coalesce(table_header,0),
+ coalesce(code_linenumbers,0),
+ coalesce(heading_ancestors_text,''), coalesce(sha256,''),
+ coalesce(text,''), id, coalesce(heading_lev_anchor,''),
+ coalesce(segment_epub,''), coalesce(segment_epub_is,'')
+ FROM objects WHERE doc_id = " ~ chosen.doc_id.to!string
+ ~ " ORDER BY CASE section
+ WHEN 'head' THEN 1 WHEN 'toc' THEN 2 WHEN 'body' THEN 3
+ WHEN 'endnotes' THEN 4 WHEN 'glossary' THEN 5
+ WHEN 'bibliography' THEN 6 WHEN 'bookindex' THEN 7
+ WHEN 'blurb' THEN 8 WHEN 'tail' THEN 9 ELSE 10 END, seq")) {
+ OcdaObject obj;
+ obj.section = r[0];
+ obj.ocn = r[2].to!int;
+ obj.is_a = r[3];
+ obj.part = r[4];
+ obj.own_section = (r[5] == r[0]) ? "" : r[5];
+ obj.is_of_type = r[6];
+ /+ ↓ the database holds the level as a number, the text form as the
+ letter or digit the markup wrote. Model section 4 is the mapping. +/
+ obj.heading_level = levelName(r[7].to!int);
+ obj.identifier = r[8];
+ obj.parent_ocn = r[9].to!int;
+ obj.last_descendant_ocn = r[10].to!int;
+ obj.children = jsonInts(r[11]);
+ obj.ancestors = jsonInts(r[12]);
+ obj.dummy_heading = (r[13] != "0");
+ obj.ocn_off = (r[14] != "0");
+ obj.indent_base = r[15].to!int;
+ obj.indent_hang = r[16].to!int;
+ obj.bullet = (r[17] != "0");
+ obj.lang = r[18];
+ obj.has_links = (r[19] != "0");
+ obj.has_notes_reg = (r[20] != "0");
+ obj.has_notes_star = (r[21] != "0");
+ obj.has_images = (r[22] != "0");
+ obj.segment = r[23];
+ obj.segment_prev = r[24];
+ obj.segment_next = r[25];
+ obj.anchor = r[26];
+ obj.segment_html_is = r[27];
+ obj.heading_lev_anchor = r[37];
+ obj.segment_epub = r[38];
+ obj.segment_epub_is = r[39];
+ obj.table_cols = r[28].to!int;
+ obj.table_widths = jsonWords(r[29]);
+ obj.table_aligns = jsonWords(r[30]);
+ obj.table_header = (r[31] != "0");
+ obj.code_linenumbers = (r[32] != "0");
+ obj.heading_ancestors_text = jsonJoined(r[33]);
+ obj.sha256 = r[34];
+ obj.text = r[35];
+
+ /+ ↓ the four repeatable properties of the text form are child tables
+ here (specification section 4) +/
+ string object_id = r[36];
+ foreach (i; db.rows("SELECT name, coalesce(sha256,''), coalesce(bytes,0),
+ coalesce(width,0), coalesce(height,0),
+ coalesce(missing,0)
+ FROM object_images WHERE object_id = " ~ object_id
+ ~ " ORDER BY seq")) {
+ OcdaImage img;
+ img.name = i[0];
+ img.sha256 = i[1];
+ img.bytes = i[2].to!ulong;
+ img.width = i[3].to!int;
+ img.height = i[4].to!int;
+ img.missing = (i[5] != "0");
+ obj.images ~= img;
+ }
+ foreach (l; db.rows("SELECT url FROM object_links WHERE object_id = "
+ ~ object_id ~ " ORDER BY seq")) {
+ obj.stow_link ~= l[0];
+ }
+ foreach (a; db.rows("SELECT anchor FROM object_anchors WHERE object_id = "
+ ~ object_id ~ " ORDER BY seq")) {
+ obj.anchor_tags ~= a[0];
+ }
+
+ auto section = doc.sectionByName(r[0]);
+ if (section !is null) { *section ~= obj; }
+ }
+ return doc;
+}
+
+/+ ↓ model section 4: the eight levels and their indices +/
+private string levelName(int index) {
+ switch (index) {
+ case 0: return "A";
+ case 1: return "B";
+ case 2: return "C";
+ case 3: return "D";
+ case 4: return "1";
+ case 5: return "2";
+ case 6: return "3";
+ case 7: return "4";
+ default: return "";
+ }
+}
+
+/+ ↓ the JSON columns, read as JSON.
+ .
+ The fixed width arrays are JSON in the database and space separated
+ (or "|" separated) in the text form. This was first written as a
+ character scan that threw away brackets and quotes, which is wrong the
+ moment a value holds an escape: Phobos' JSONValue.toString escapes
+ "/" as "\/", so every heading containing a slash came back with a
+ backslash in it. 36 of 36 documents differed on
+ heading_ancestors_text and nothing else had shown it.
+ .
+ So: parse. The values are small, there are few of them, and a real
+ parser is shorter than the scan it replaces.
++/
+private string[] jsonStrings(string v) {
+ if (v.length == 0) { return []; }
+ string[] out_;
+ try {
+ auto j = parseJSON(v);
+ if (j.type != JSONType.array) { return []; }
+ foreach (e; j.array) {
+ switch (e.type) {
+ case JSONType.string: out_ ~= e.str; break;
+ case JSONType.integer: out_ ~= e.integer.to!string; break;
+ case JSONType.uinteger: out_ ~= e.uinteger.to!string; break;
+ case JSONType.float_: out_ ~= e.floating.to!string; break;
+ case JSONType.true_: out_ ~= "true"; break;
+ case JSONType.false_: out_ ~= "false"; break;
+ default: out_ ~= ""; break;
+ }
+ }
+ } catch (Exception) {
+ return [];
+ }
+ return out_;
+}
+
+private int[] jsonInts(string v) {
+ int[] out_;
+ foreach (s; jsonStrings(v)) {
+ if (s.length == 0) { continue; }
+ try { out_ ~= s.to!int; } catch (Exception) {}
+ }
+ return out_;
+}
+
+/+ ↓ space separated in the text form: table_widths, table_aligns +/
+private string jsonWords(string v) {
+ return jsonStrings(v).join(" ");
+}
+
+/+ ↓ heading_ancestors_text is eight slots, joined with "|" in the text
+ form and a JSON array here. The empty slots are significant: they say
+ which levels the heading sits under, so the separators carry as much
+ as the text does.
++/
+private string jsonJoined(string v) {
+ auto parts = jsonStrings(v);
+ if (parts.length == 0) { return ""; }
+ return parts.join("|");
+}
diff --git a/dlang/src/ocda/load_pod.d b/dlang/src/ocda/load_pod.d
new file mode 100644
index 0000000..75993be
--- /dev/null
+++ b/dlang/src/ocda/load_pod.d
@@ -0,0 +1,115 @@
+module ocda.load_pod;
+
+/+ ↓ the abstraction out of a pod archive.
+ .
+ A .sisupod is one zstd frame wrapping a stored zip; a .zip is the same
+ archive unwrapped, which is what spine published before it wrapped them.
+ Either way, a pod built with --pod2 carries the abstraction inside it,
+ at media/abstraction/<uid>.<lang>.ssp, and that is what is read.
+ .
+ So this is a third input form and *not* a third model: the .ssp inside
+ is parsed by ocda.load_ssp and comes back as the same value. A writer
+ over the library gains .sisupod without being touched, which is the
+ whole reason the dispatch is in one place.
+ .
+ A pod built without --pod2 carries markup and no abstraction. That is
+ refused, and the refusal says which flag makes a pod this reader can
+ read, because "no abstraction in it" is a fact about the pod rather
+ than about the reader.
++/
+
+import std.algorithm : endsWith, sort;
+import std.array : array;
+import std.file : read;
+import std.string : indexOf;
+import ocda.model;
+import ocda.load_ssp;
+import ocda.zip;
+import ocda.zstd;
+
+/+ ↓ a limit on what a frame may expand to.
+ A reader of an artefact it did not make declines to be told how much
+ memory to allocate. 512 MiB is far above the largest pod in the
+ reference collection (4 MiB) and far below anything that matters.
++/
+enum size_t max_pod_size = 512 * 1024 * 1024;
+
+class OcdaPodError : Exception {
+ this(string msg) { super(msg); }
+}
+
+/+ ↓ which languages of the abstraction the pod carries, in sorted order +/
+string[] podLanguages(string path) {
+ auto bytes = archiveBytes(path);
+ string[] out_;
+ foreach (e; zipFindSuffix(zipEntries(bytes), ".ssp")) {
+ string lang = langOfSspName(e.name);
+ if (lang.length > 0) { out_ ~= lang; }
+ }
+ return out_.sort.array;
+}
+
+/+ ↓ load one language of the abstraction a pod carries.
+ lang == "" takes the only one there is, and where there is a choice it
+ says so rather than choosing, as the .ocda.db loader does.
++/
+OcdaDocument loadPodFile(string path, string lang = "") {
+ auto bytes = archiveBytes(path);
+ auto entries = zipEntries(bytes);
+ auto ssps = zipFindSuffix(entries, ".ssp");
+ if (ssps.length == 0) {
+ throw new OcdaPodError(path ~ ": carries no .ssp."
+ ~ " A pod carries the abstraction only when spine was asked for it:"
+ ~ " spine --pod2 puts it in, spine --source does not.");
+ }
+ ZipEntry chosen = ssps[0];
+ if (lang.length > 0) {
+ bool found = false;
+ foreach (e; ssps) {
+ if (langOfSspName(e.name) == lang) { chosen = e; found = true; break; }
+ }
+ if (!found) {
+ string have;
+ foreach (i, e; ssps) {
+ have ~= (i > 0 ? " " : "") ~ langOfSspName(e.name);
+ }
+ throw new OcdaPodError(path ~ ": no language '" ~ lang
+ ~ "'; it holds: " ~ have);
+ }
+ } else if (ssps.length > 1) {
+ string have;
+ foreach (i, e; ssps) {
+ have ~= (i > 0 ? " " : "") ~ langOfSspName(e.name);
+ }
+ throw new OcdaPodError(path ~ ": holds more than one language ("
+ ~ have ~ "); name the one wanted");
+ }
+ auto ssp = zipRead(bytes, chosen);
+ return parseSsp(cast(string) ssp.idup);
+}
+
+/+ ↓ the archive, unwrapped if it was wrapped.
+ The bytes decide, not the name: a .sisupod is a zstd frame and a .zip
+ is not, and a file misnamed either way still reads.
++/
+private ubyte[] archiveBytes(string path) {
+ auto bytes = cast(ubyte[]) read(path);
+ return zstdIsFrame(bytes)
+ ? zstdDecompress(bytes, max_pod_size)
+ : bytes;
+}
+
+/+ ↓ <uid>.<lang>.ssp -> <lang> +/
+private string langOfSspName(string name) {
+ if (!name.endsWith(".ssp")) { return ""; }
+ string stem = name[0 .. $ - 4];
+ auto dot = stem.lastIndexOf('.');
+ return (dot > 0) ? stem[dot + 1 .. $] : "";
+}
+
+private ptrdiff_t lastIndexOf(string s, char c) {
+ for (ptrdiff_t i = s.length - 1; i >= 0; i--) {
+ if (s[i] == c) { return i; }
+ }
+ return -1;
+}
diff --git a/dlang/src/ocda/load_ssp.d b/dlang/src/ocda/load_ssp.d
new file mode 100644
index 0000000..34d156c
--- /dev/null
+++ b/dlang/src/ocda/load_ssp.d
@@ -0,0 +1,337 @@
+module ocda.load_ssp;
+
+import std.algorithm : splitter, startsWith;
+import std.array : array, split;
+import std.conv : to, ConvException;
+import std.file : readText;
+import std.stdio : writeln;
+import std.string : strip, stripLeft, indexOf;
+import ocda.model;
+
+OcdaDocument parseSspFile(string filename) {
+ string content = readText(filename);
+ return parseSsp(content);
+}
+
+OcdaDocument parseSsp(string content) {
+ OcdaDocument doc;
+ OcdaObject current_obj;
+ bool in_object = false;
+ /+ ↓ whether a text line has been seen for the object being read.
+ Not "is the text non-empty": an object whose first text line is blank
+ (a verse opening on an empty line, which the sample set has seven of)
+ would then lose it, the next line overwriting rather than appending.
+ +/
+ bool seen_text = false;
+ string current_section = "";
+ string current_block = ""; // meta, make, doc_has, or section name
+
+ foreach (line; content.splitter("\n")) {
+ /+ ↓ comment. The first line of a .ssp declares the format version, and
+ it is the one comment that carries meaning: a reader checks the
+ major part and refuses what it does not know rather than loading
+ an unfamiliar artefact half populated in silence.
+ +/
+ if (line.length > 0 && line[0] == '%') {
+ enum _banner = "% SiSU Document Abstraction v";
+ enum _source = "% Source: ";
+ if (doc.format_version.length == 0 && line.startsWith(_banner)) {
+ doc.format_version = line[_banner.length .. $].strip;
+ } else if (doc.source_filename.length == 0 && line.startsWith(_source)) {
+ doc.source_filename = line[_source.length .. $].strip;
+ }
+ continue;
+ }
+
+ /+ ↓ blank line - finalize current object +/
+ if (line.length == 0 || line.strip.length == 0) {
+ if (in_object) {
+ commitObject(doc, current_section, current_obj);
+ current_obj = OcdaObject.init;
+ in_object = false;
+ seen_text = false;
+ }
+ continue;
+ }
+
+ /+ ↓ section/block open +/
+ if (line[0] == '@') {
+ auto brace_pos = line.indexOf('{');
+ if (brace_pos > 0) {
+ string name = line[1 .. brace_pos].strip;
+ current_block = name;
+ if (!isPreambleBlock(name)) {
+ current_section = name;
+ /+ ↓ a block that is neither preamble nor a section this reader
+ knows: counted, so that --report-unknown can say so +/
+ if (doc.sectionByName(name) is null) {
+ doc.unknown.block[name]++;
+ }
+ }
+ }
+ continue;
+ }
+
+ /+ ↓ section/block close +/
+ if (line[0] == '}') {
+ if (in_object) {
+ commitObject(doc, current_section, current_obj);
+ current_obj = OcdaObject.init;
+ in_object = false;
+ seen_text = false;
+ }
+ current_block = "";
+ continue;
+ }
+
+ /+ ↓ inside meta/make/doc_has blocks: key-value pairs +/
+ if (isPreambleBlock(current_block)) {
+ string stripped = line.stripLeft;
+ auto colon_pos = stripped.indexOf(':');
+ if (colon_pos > 0) {
+ string key = stripped[0 .. colon_pos].strip;
+ string value = (colon_pos + 1 < stripped.length)
+ ? stripped[colon_pos + 1 .. $].strip : "";
+ if (current_block == "meta") { doc.metadata[key] = value; }
+ else if (current_block == "make") { doc.make[key] = value; }
+ else if (current_block == "source") { doc.source[key] = value; }
+ else { doc.doc_has[key] = value; }
+ }
+ continue;
+ }
+
+ /+ ↓ object declaration: [N] type ... +/
+ if (line[0] == '[') {
+ if (in_object) {
+ commitObject(doc, current_section, current_obj);
+ }
+ current_obj = OcdaObject.init;
+ current_obj.section = current_section;
+ in_object = true;
+ seen_text = false;
+ parseObjectDeclaration(line, current_obj);
+ continue;
+ }
+
+ /+ ↓ property: .name: value +/
+ if (line[0] == '.') {
+ if (in_object) {
+ parseProperty(line, current_obj, doc.unknown);
+ }
+ continue;
+ }
+
+ /+ ↓ text content: | content +/
+ if (line.length >= 1 && line[0] == '|') {
+ if (in_object) {
+ string text_content = (line.length > 2) ? line[2 .. $] : "";
+ if (seen_text) {
+ current_obj.text ~= "\n" ~ text_content;
+ } else {
+ current_obj.text = text_content;
+ seen_text = true;
+ }
+ }
+ continue;
+ }
+
+ /+ ↓ no known shape. Counted by its first character so that a reader
+ can be told the file held something this one did not expect
+ +/
+ doc.unknown.line[line[0 .. 1].idup]++;
+ }
+
+ /+ ↓ finalize last object +/
+ if (in_object) {
+ commitObject(doc, current_section, current_obj);
+ }
+
+ return doc;
+}
+
+private void parseObjectDeclaration(string line, ref OcdaObject obj) {
+ /+ parse: [N] type :level identifier +/
+ auto bracket_close = line.indexOf(']');
+ if (bracket_close < 0) return;
+ try {
+ obj.ocn = line[1 .. bracket_close].strip.to!int;
+ } catch (ConvException) {
+ obj.ocn = 0;
+ }
+ string rest = (bracket_close + 1 < line.length)
+ ? line[bracket_close + 1 .. $].strip : "";
+ if (rest.length == 0) return;
+
+ auto parts = rest.split(" ");
+ if (parts.length == 0) return;
+ obj.is_a = parts[0];
+
+ if (obj.is_a == "heading" && parts.length >= 2) {
+ /+ parse :A, :B, :1, etc. +/
+ string lev = parts[1];
+ if (lev.length >= 2 && lev[0] == ':') {
+ obj.heading_level = lev[1 .. $];
+ }
+ /+ optional identifier after level +/
+ if (parts.length >= 3) {
+ obj.identifier = parts[2];
+ }
+ }
+}
+
+/+ ↓ the blocks that are document preamble rather than sections of objects +/
+private bool isPreambleBlock(string name) {
+ return name == "source" || name == "meta" || name == "make" || name == "doc_has";
+}
+
+private void parseProperty(string line, ref OcdaObject obj, ref OcdaUnknown unknown) {
+ /+ parse: .name: value +/
+ auto colon_pos = line.indexOf(':');
+ if (colon_pos < 1) return;
+ string name = line[1 .. colon_pos].strip;
+ string value = (colon_pos + 1 < line.length)
+ ? line[colon_pos + 1 .. $].strip : "";
+
+ switch (name) {
+ case "part": obj.part = value; break;
+ case "is_of_type": obj.is_of_type = value; break;
+ case "section": obj.own_section = value; break;
+ case "identifier": obj.identifier = value; break;
+ case "segment_html_is": obj.segment_html_is = value; break;
+ /+ ↓ the epub segments at heading levels 1 to 4 where the html
+ segments at level 4 alone, so the two are different files and
+ the abstraction records both +/
+ case "segment_epub": obj.segment_epub = value; break;
+ case "segment_epub_is": obj.segment_epub_is = value; break;
+ case "table_aligns": obj.table_aligns = value; break;
+ case "stow_link": obj.stow_link ~= value; break;
+ case "anchor_tag": obj.anchor_tags ~= value; break;
+ case "parent": obj.parent_ocn = toIntSafe(value); break;
+ case "last_descendant": obj.last_descendant_ocn = toIntSafe(value); break;
+ case "children": obj.children = parseIntArray(value); break;
+ case "ancestors": obj.ancestors = parseIntArray(value); break;
+ case "dummy": obj.dummy_heading = (value == "true"); break;
+ case "ocn_off": obj.ocn_off = (value == "true"); break;
+ case "indent":
+ auto parts = value.split(" ");
+ if (parts.length >= 1) obj.indent_base = toIntSafe(parts[0]);
+ if (parts.length >= 2) obj.indent_hang = toIntSafe(parts[1]);
+ break;
+ case "bullet": obj.bullet = (value == "true"); break;
+ case "lang": obj.lang = value; break;
+ case "has":
+ foreach (flag; value.split(" ")) {
+ switch (flag) {
+ case "links": obj.has_links = true; break;
+ case "notes_reg": obj.has_notes_reg = true; break;
+ case "notes_star": obj.has_notes_star = true; break;
+ case "images": obj.has_images = true; break;
+ default: break;
+ }
+ }
+ break;
+ case "segment": obj.segment = value; break;
+ case "segment_prev": obj.segment_prev = value; break;
+ case "segment_next": obj.segment_next = value; break;
+ case "anchor": obj.anchor = value; break;
+ case "heading_lev_anchor": obj.heading_lev_anchor = value; break;
+ case "table_cols": obj.table_cols = toIntSafe(value); break;
+ case "table_widths": obj.table_widths = value; break;
+ case "table_header": obj.table_header = (value == "true"); break;
+ case "code_syntax": obj.code_syntax = value; break;
+ case "code_linenumbers": obj.code_linenumbers = (value == "true"); break;
+ case "heading_ancestors_text": obj.heading_ancestors_text = value; break;
+ case "sha256": obj.sha256 = value; break;
+ case "image": obj.images ~= parseImageRecord(value); break;
+ /+ ↓ a property this reader has no use for, or has never heard of.
+ Ignored, which is the conforming behaviour within a major
+ version, but counted rather than passed over in silence
+ +/
+ default: unknown.property[name]++; break;
+ }
+}
+
+private OcdaImage parseImageRecord(string value) {
+ /+ parse: <filename> sha256:<HEX> bytes:<n> [key:value ...]
+ filename first, remaining fields order-independent; unrecognised
+ fields are kept rather than discarded +/
+ OcdaImage img;
+ bool first = true;
+ foreach (field; value.split(" ")) {
+ if (field.length == 0) continue;
+ if (first) {
+ img.name = field;
+ first = false;
+ continue;
+ }
+ auto colon_pos = field.indexOf(':');
+ if (colon_pos < 1) continue;
+ string key = field[0 .. colon_pos];
+ string val = (colon_pos + 1 < field.length) ? field[colon_pos + 1 .. $] : "";
+ switch (key) {
+ case "sha256": img.sha256 = val; break;
+ case "bytes": img.bytes = toUlongSafe(val); break;
+ case "px":
+ auto dims = val.split("x");
+ if (dims.length == 2) {
+ img.width = toIntSafe(dims[0]);
+ img.height = toIntSafe(dims[1]);
+ }
+ break;
+ case "missing": img.missing = (val == "true"); break;
+ default: img.fields[key] = val; break;
+ }
+ }
+ return img;
+}
+
+/+ ↓ two values the text form leaves out because they can be worked out.
+ .
+ The .ssp emits a property only when it carries information, so
+ .identifier is absent when it is simply the ocn, and .anchor is absent
+ when it is the segment name. The .ocda.db, being columns rather than
+ lines, stores both unconditionally. So a loader that takes the text
+ form literally gives a *different model* from the one the database
+ gives, and the difference is not small: in the reference collection
+ the identifier is omitted on all but a handful of objects.
+ .
+ The model is the thing both serialisations describe, so the loader
+ fills them in. Found by dumping both models and diffing them, which no
+ amount of comparing section counts would have shown.
++/
+private void deriveOmitted(ref OcdaObject obj) {
+ import std.conv : to;
+ if (obj.identifier.length == 0 && obj.ocn != 0) {
+ obj.identifier = obj.ocn.to!string;
+ }
+ if (obj.anchor.length == 0 && obj.segment.length > 0) {
+ obj.anchor = obj.segment;
+ }
+}
+
+private void commitObject(ref OcdaDocument doc, string section, ref OcdaObject obj) {
+ deriveOmitted(obj);
+ auto sec = doc.sectionByName(section);
+ if (sec !is null) {
+ *sec ~= obj;
+ }
+}
+
+private int toIntSafe(string s) {
+ try { return s.strip.to!int; }
+ catch (ConvException) { return 0; }
+}
+
+private ulong toUlongSafe(string s) {
+ try { return s.strip.to!ulong; }
+ catch (ConvException) { return 0; }
+}
+
+private int[] parseIntArray(string s) {
+ int[] result;
+ foreach (part; s.split(" ")) {
+ auto v = toIntSafe(part);
+ result ~= v;
+ }
+ return result;
+}
diff --git a/dlang/src/ocda/model.d b/dlang/src/ocda/model.d
new file mode 100644
index 0000000..403502a
--- /dev/null
+++ b/dlang/src/ocda/model.d
@@ -0,0 +1,147 @@
+module ocda.model;
+
+struct OcdaImage {
+ string name; // image filename as referenced by the object
+ string sha256; // hex-encoded (uppercase) digest of the file
+ ulong bytes; // file size
+ int width; // file's own pixel width (not display width)
+ int height; // file's own pixel height
+ bool missing; // file absent or unreadable, no sha256/bytes
+ string[string] fields; // any further/unrecognised key:value fields
+}
+
+struct OcdaObject {
+ int ocn;
+ string is_a; // heading, para, toc, table, code, group, block, verse, quote, endnote, glossary, bibliography, bookindex, blurb
+ string section; // head, toc, body, endnotes, glossary, bibliography, bookindex, blurb
+ string heading_level; // A, B, C, D, 1, 2, 3, 4 (for headings)
+ string identifier; // heading: from the declaration line; otherwise .identifier
+ string part; // frontmatter, body, backmatter
+ string is_of_type; // para, block, comment: how the text behaves
+ string own_section; // .section, when it differs from the block it sits in
+ string segment_html_is;// the html segment this object sits in
+ string segment_epub; // the epub segment this heading opens
+ string segment_epub_is;// the epub segment this object sits in
+ string table_aligns; // l c r per column
+ string[] stow_link; // links extracted from the object
+ string[] anchor_tags; // named targets the markup placed on this object
+ int parent_ocn;
+ int last_descendant_ocn;
+ int[] children; // direct child heading OCNs
+ int[] ancestors; // markedup_ancestors[8]
+ bool dummy_heading;
+ bool ocn_off;
+ int indent_base;
+ int indent_hang;
+ bool bullet;
+ string lang;
+ bool has_links;
+ bool has_notes_reg;
+ bool has_notes_star;
+ bool has_images;
+ string segment;
+ string segment_prev;
+ string segment_next;
+ string anchor;
+ string heading_lev_anchor; // a heading's own named target
+ int table_cols;
+ string table_widths;
+ bool table_header;
+ string code_syntax;
+ bool code_linenumbers;
+ string heading_ancestors_text;
+ string sha256;
+ OcdaImage[] images; // repeatable .image records, in order
+ string text; // content (lines joined with newline)
+
+ /+ ↓ whether this object is rendered at all.
+ is_of_type "comment" is markup commentary: carried by the
+ abstraction so that nothing is lost, and not put into output. An
+ output that renders it leaks the author notes into the document,
+ which is what spine itself does not do.
+ +/
+ bool renders() const {
+ return is_of_type != "comment";
+ }
+
+ /+ ↓ the section this object belongs to, which is the block it sits in
+ unless it says otherwise +/
+ string sectionOf() const {
+ return (own_section.length > 0) ? own_section : section;
+ }
+}
+
+/+ ↓ what the reader met and did not understand.
+ A reader that silently drops what it does not know cannot say whether
+ it understood the document, and cannot be used to check a
+ specification against an implementation: silence and understanding
+ look alike. So the parser counts what it ignored, and
+ --report-unknown prints it.
++/
+struct OcdaUnknown {
+ size_t[string] property; // .name -> times seen
+ size_t[string] block; // @name -> times seen
+ size_t[string] line; // first char -> times seen, lines of no known shape
+
+ bool any() const {
+ return property.length > 0 || block.length > 0 || line.length > 0;
+ }
+}
+
+struct OcdaDocument {
+ string format_version; // as declared on line 1, "" when absent
+ string source_filename; // from the "% Source:" line
+ string[string] source; // @source: language, languages, digest
+ OcdaUnknown unknown; // what was met and not understood
+ string[string] metadata; // key -> value
+ string[string] make; // key -> value
+ string[string] doc_has; // key -> value
+ OcdaObject[] head;
+ OcdaObject[] toc;
+ OcdaObject[] body;
+ OcdaObject[] endnotes;
+ OcdaObject[] glossary;
+ OcdaObject[] bibliography;
+ OcdaObject[] bookindex;
+ OcdaObject[] blurb;
+ OcdaObject[] tail;
+
+ /+ ↓ the major part of the declared version, "" when none was declared +/
+ string formatVersionMajor() const {
+ import std.string : indexOf;
+ if (format_version.length == 0) { return ""; }
+ auto dot = format_version.indexOf('.');
+ return (dot > 0) ? format_version[0 .. dot] : format_version;
+ }
+
+ string meta(string key) {
+ if (auto p = key in metadata) return *p;
+ return "";
+ }
+
+ OcdaObject[]* sectionByName(string name) {
+ switch (name) {
+ case "head": return &head;
+ case "toc": return &toc;
+ case "body": return &body;
+ case "endnotes": return &endnotes;
+ case "glossary": return &glossary;
+ case "bibliography": return &bibliography;
+ case "bookindex": return &bookindex;
+ case "blurb": return &blurb;
+ case "tail": return &tail;
+ default: return null;
+ }
+ }
+}
+
+/+ ↓ the format major version this reader knows.
+ The .ssp and the .ocda.db carry one version between them, declared in
+ band: the .ssp on its first line, the database in its schema.version
+ row. The minor part is raised when a property is added, and a reader of
+ the newer format still reads the older one, so only the major part is
+ checked here. Raise this when the reader has been brought to a new
+ major, not before: the point of the gate is to refuse what has not been
+ looked at.
++/
+enum ssp_format_major_read = "2";
diff --git a/dlang/src/ocda/sqlite.d b/dlang/src/ocda/sqlite.d
new file mode 100644
index 0000000..6f3753c
--- /dev/null
+++ b/dlang/src/ocda/sqlite.d
@@ -0,0 +1,122 @@
+module ocda.sqlite;
+
+/+ ↓ the sqlite3 C API, declared rather than depended upon.
+ .
+ A binding from the dub registry would be a dependency to fetch, and
+ the point of this reader is that it builds in a sandbox with no
+ network. The dozen functions a reader needs are declared here and the
+ library is linked directly, which costs a page of extern(C) and saves
+ a dependency.
++/
+
+import std.conv : to;
+import std.string : fromStringz, toStringz;
+
+extern (C) {
+ struct sqlite3;
+ struct sqlite3_stmt;
+
+ int sqlite3_open_v2(const(char)* filename, sqlite3** db, int flags, const(char)* vfs);
+ int sqlite3_close(sqlite3* db);
+ int sqlite3_prepare_v2(sqlite3* db, const(char)* sql, int nbyte,
+ sqlite3_stmt** stmt, const(char)** tail);
+ int sqlite3_step(sqlite3_stmt* stmt);
+ int sqlite3_finalize(sqlite3_stmt* stmt);
+ int sqlite3_column_count(sqlite3_stmt* stmt);
+ const(char)* sqlite3_column_name(sqlite3_stmt* stmt, int col);
+ const(ubyte)* sqlite3_column_text(sqlite3_stmt* stmt, int col);
+ int sqlite3_column_int(sqlite3_stmt* stmt, int col);
+ int sqlite3_column_type(sqlite3_stmt* stmt, int col);
+ const(char)* sqlite3_errmsg(sqlite3* db);
+ const(void)* sqlite3_column_blob(sqlite3_stmt* stmt, int col);
+ int sqlite3_column_bytes(sqlite3_stmt* stmt, int col);
+}
+
+enum SQLITE_OK = 0;
+enum SQLITE_ROW = 100;
+enum SQLITE_DONE = 101;
+enum SQLITE_NULL = 5;
+enum SQLITE_OPEN_READONLY = 0x00000001;
+
+class SqliteError : Exception {
+ this(string msg) { super(msg); }
+}
+
+/+ ↓ a database, opened read only.
+ Read only is deliberate: a reader has no business writing, and a
+ request that opens a database read-write can create an empty one where
+ a path is mistyped.
++/
+struct Db {
+ private sqlite3* _handle;
+
+ static Db open(string path) {
+ sqlite3* handle;
+ auto rc = sqlite3_open_v2(path.toStringz, &handle, SQLITE_OPEN_READONLY, null);
+ if (rc != SQLITE_OK) {
+ string msg = (handle is null)
+ ? "cannot open " ~ path
+ : "cannot open " ~ path ~ ": " ~ sqlite3_errmsg(handle).fromStringz.idup;
+ if (handle !is null) { sqlite3_close(handle); }
+ throw new SqliteError(msg);
+ }
+ Db db;
+ db._handle = handle;
+ return db;
+ }
+
+ void close() {
+ if (_handle !is null) { sqlite3_close(_handle); _handle = null; }
+ }
+
+ /+ ↓ every row of a query, each row as its columns in order +/
+ string[][] rows(string sql) {
+ sqlite3_stmt* stmt;
+ auto rc = sqlite3_prepare_v2(_handle, sql.toStringz, -1, &stmt, null);
+ if (rc != SQLITE_OK) {
+ throw new SqliteError(sqlite3_errmsg(_handle).fromStringz.idup ~ " in: " ~ sql);
+ }
+ scope(exit) { sqlite3_finalize(stmt); }
+ string[][] out_rows;
+ int cols = sqlite3_column_count(stmt);
+ while (sqlite3_step(stmt) == SQLITE_ROW) {
+ string[] row;
+ foreach (c; 0 .. cols) {
+ row ~= (sqlite3_column_type(stmt, c) == SQLITE_NULL)
+ ? "" : (cast(const(char)*) sqlite3_column_text(stmt, c)).fromStringz.idup;
+ }
+ out_rows ~= row;
+ }
+ return out_rows;
+ }
+
+ /+ ↓ the first column of the first row, "" when there is none +/
+ string one(string sql) {
+ auto r = rows(sql);
+ return (r.length > 0 && r[0].length > 0) ? r[0][0] : "";
+ }
+
+ /+ ↓ the first column of the first row as bytes, for a blob.
+ Separate from one(), because a blob is not text: an image read as text
+ stops at the first zero byte, and a png's ninth byte is zero.
+ +/
+ ubyte[] blob(string sql) {
+ sqlite3_stmt* stmt;
+ if (sqlite3_prepare_v2(_handle, sql.toStringz, -1, &stmt, null) != SQLITE_OK) {
+ throw new SqliteError("prepare: " ~ sqlite3_errmsg(_handle).fromStringz.idup
+ ~ " [" ~ sql ~ "]");
+ }
+ scope(exit) { sqlite3_finalize(stmt); }
+ if (sqlite3_step(stmt) != SQLITE_ROW) { return []; }
+ int n = sqlite3_column_bytes(stmt, 0);
+ if (n <= 0) { return []; }
+ auto p = cast(const(ubyte)*) sqlite3_column_blob(stmt, 0);
+ if (p is null) { return []; }
+ return p[0 .. n].dup;
+ }
+
+ /+ ↓ does a table or view of this name exist +/
+ bool has(string name) {
+ return one("SELECT count(*) FROM sqlite_master WHERE name = '" ~ name ~ "'") != "0";
+ }
+}
diff --git a/dlang/src/ocda/zip.d b/dlang/src/ocda/zip.d
new file mode 100644
index 0000000..9dff6f4
--- /dev/null
+++ b/dlang/src/ocda/zip.d
@@ -0,0 +1,115 @@
+module ocda.zip;
+
+/+ ↓ enough of the zip format to find one file in a pod and read it.
+ .
+ Not a zip library. A pod archive holds a handful of entries, spine
+ writes them **stored** (uncompressed) so that the outer zstd frame does
+ the compressing once over the whole thing, and what is wanted from it
+ here is one named entry. So: read the central directory, find the
+ entry, return its bytes.
+ .
+ Deflate is *refused rather than mishandled*. Phobos can inflate, but a
+ library that says it read an entry it in fact skipped is worse than one
+ that says it cannot, and spine's own archives do not need it.
++/
+
+import std.algorithm : endsWith;
+import std.conv : to;
+
+class OcdaZipError : Exception {
+ this(string msg) { super(msg); }
+}
+
+struct ZipEntry {
+ string name;
+ uint method; // 0 stored, 8 deflate
+ uint compressed;
+ uint uncompressed;
+ uint local_offset;
+}
+
+/+ ↓ the entries, from the central directory at the end of the file +/
+ZipEntry[] zipEntries(const(ubyte)[] bytes) {
+ enum uint eocd_sig = 0x06054b50;
+ enum uint cd_sig = 0x02014b50;
+ if (bytes.length < 22) {
+ throw new OcdaZipError("too short to be a zip archive");
+ }
+ /+ ↓ the end of central directory record is last, and may be followed by a
+ comment of up to 65535 bytes, so it is searched for backwards +/
+ ptrdiff_t eocd = -1;
+ ptrdiff_t lowest = bytes.length > (22 + 65535) ? bytes.length - (22 + 65535) : 0;
+ for (ptrdiff_t i = bytes.length - 22; i >= lowest; i--) {
+ if (u32(bytes, i) == eocd_sig) { eocd = i; break; }
+ }
+ if (eocd < 0) {
+ throw new OcdaZipError("no end of central directory record;"
+ ~ " it does not look like a zip archive");
+ }
+ uint count = u16(bytes, eocd + 10);
+ uint offset = u32(bytes, eocd + 16);
+ ZipEntry[] out_;
+ size_t p = offset;
+ foreach (_; 0 .. count) {
+ if (p + 46 > bytes.length || u32(bytes, p) != cd_sig) {
+ throw new OcdaZipError("central directory entry "
+ ~ out_.length.to!string ~ " is malformed");
+ }
+ ZipEntry e;
+ e.method = u16(bytes, p + 10);
+ e.compressed = u32(bytes, p + 20);
+ e.uncompressed = u32(bytes, p + 24);
+ uint name_len = u16(bytes, p + 28);
+ uint extra_len = u16(bytes, p + 30);
+ uint cmnt_len = u16(bytes, p + 32);
+ e.local_offset = u32(bytes, p + 42);
+ e.name = cast(string) bytes[p + 46 .. p + 46 + name_len].idup;
+ out_ ~= e;
+ p += 46 + name_len + extra_len + cmnt_len;
+ }
+ return out_;
+}
+
+/+ ↓ one entry's bytes, from its local header +/
+ubyte[] zipRead(const(ubyte)[] bytes, ZipEntry e) {
+ enum uint local_sig = 0x04034b50;
+ size_t p = e.local_offset;
+ if (p + 30 > bytes.length || u32(bytes, p) != local_sig) {
+ throw new OcdaZipError(e.name ~ ": local header is malformed");
+ }
+ uint name_len = u16(bytes, p + 26);
+ uint extra_len = u16(bytes, p + 28);
+ size_t start = p + 30 + name_len + extra_len;
+ size_t end = start + e.compressed;
+ if (end > bytes.length) {
+ throw new OcdaZipError(e.name ~ ": runs past the end of the archive");
+ }
+ if (e.method != 0) {
+ throw new OcdaZipError(e.name ~ ": compression method "
+ ~ e.method.to!string ~ "; this reader takes only stored entries,"
+ ~ " which is what spine writes");
+ }
+ return bytes[start .. end].dup;
+}
+
+/+ ↓ the first entry whose name ends this way, or none.
+ Used to find media/abstraction/<uid>.<lang>.ssp without knowing the uid.
++/
+ZipEntry[] zipFindSuffix(ZipEntry[] entries, string suffix) {
+ ZipEntry[] out_;
+ foreach (e; entries) {
+ if (e.name.endsWith(suffix)) { out_ ~= e; }
+ }
+ return out_;
+}
+
+private uint u16(const(ubyte)[] b, size_t i) {
+ return cast(uint) b[i] | (cast(uint) b[i + 1] << 8);
+}
+
+private uint u32(const(ubyte)[] b, size_t i) {
+ return cast(uint) b[i]
+ | (cast(uint) b[i + 1] << 8)
+ | (cast(uint) b[i + 2] << 16)
+ | (cast(uint) b[i + 3] << 24);
+}
diff --git a/dlang/src/ocda/zstd.d b/dlang/src/ocda/zstd.d
new file mode 100644
index 0000000..401f1a3
--- /dev/null
+++ b/dlang/src/ocda/zstd.d
@@ -0,0 +1,79 @@
+module ocda.zstd;
+
+/+ ↓ libzstd, declared rather than bound.
+ .
+ A .sisupod is one zstd frame wrapping a zip archive, so reading one
+ needs exactly two things from the library: is this a frame, and what is
+ in it. Both are declared here and the C library is linked, which is the
+ same arrangement ocda.sqlite uses: no vendored tree, no generated
+ header, and the prototypes say what is being called.
++/
+
+extern (C) @nogc nothrow private {
+ size_t ZSTD_decompress(void* dst, size_t dstCapacity,
+ const(void)* src, size_t compressedSize);
+ ulong ZSTD_getFrameContentSize(const(void)* src, size_t srcSize);
+ uint ZSTD_isError(size_t code);
+ const(char)* ZSTD_getErrorName(size_t code);
+}
+
+/+ ↓ the two values ZSTD_getFrameContentSize returns in place of a size +/
+private enum ulong contentSizeUnknown = ulong.max;
+private enum ulong contentSizeError = ulong.max - 1;
+
+/+ ↓ the magic number of a zstd frame, little endian 0xFD2FB528 +/
+private static immutable ubyte[4] zstd_magic = [0x28, 0xB5, 0x2F, 0xFD];
+
+class OcdaZstdError : Exception {
+ this(string msg) { super(msg); }
+}
+
+/+ ↓ does this begin with a zstd frame?
+ Asked of the bytes rather than of the filename, because what is inside
+ decides how a thing is opened. A .sisupod is a frame; a .zip published
+ before spine wrapped them is not, and both are read.
++/
+bool zstdIsFrame(const(ubyte)[] bytes) {
+ if (bytes.length < zstd_magic.length) { return false; }
+ foreach (i, b; zstd_magic) {
+ if (bytes[i] != b) { return false; }
+ }
+ return true;
+}
+
+/+ ↓ one frame in, its contents out.
+ max_size is a refusal rather than a buffer size: a frame declares how
+ much it expands to, and a frame that claims more than the caller is
+ willing to hold is not decompressed at all. A reader of an artefact it
+ did not make wants that limit.
++/
+ubyte[] zstdDecompress(const(ubyte)[] bytes, size_t max_size) {
+ if (!zstdIsFrame(bytes)) {
+ throw new OcdaZstdError("not a zstd frame");
+ }
+ ulong declared = ZSTD_getFrameContentSize(bytes.ptr, bytes.length);
+ if (declared == contentSizeError) {
+ throw new OcdaZstdError("zstd frame: could not read its content size");
+ }
+ if (declared == contentSizeUnknown) {
+ throw new OcdaZstdError("zstd frame declares no content size;"
+ ~ " this reader takes only frames that do");
+ }
+ if (declared > max_size) {
+ import std.conv : to;
+ throw new OcdaZstdError("zstd frame expands to " ~ declared.to!string
+ ~ " bytes, over the " ~ max_size.to!string ~ " byte limit");
+ }
+ auto out_ = new ubyte[](cast(size_t) declared);
+ size_t got = ZSTD_decompress(out_.ptr, out_.length, bytes.ptr, bytes.length);
+ if (ZSTD_isError(got)) {
+ import std.string : fromStringz;
+ throw new OcdaZstdError("zstd: "
+ ~ ZSTD_getErrorName(got).fromStringz.idup);
+ }
+ if (got != declared) {
+ throw new OcdaZstdError("zstd frame expanded to a different size"
+ ~ " than it declared");
+ }
+ return out_;
+}