diff options
| author | Ralph Amissah <ralph.amissah@gmail.com> | 2026-09-25 16:34:44 -0400 |
|---|---|---|
| committer | Ralph Amissah <ralph.amissah@gmail.com> | 2026-10-05 00:53:04 -0400 |
| commit | f7da955e7ddcdd0da83c3b318418ec86822ac731 (patch) | |
| tree | 16499028c3bb7e1e8b89a0b57c6b34a12fd71a00 /dlang/src/ocda | |
the abstraction as a library, and a defect it found at once
The readers each read one serialisation and print. A writer needs
neither: it needs the document. So the model and both loaders live
here, and a writer cannot tell which serialisation it was handed,
exactly as spine own writers cannot.
ocda.model imports nothing, which is the property that makes writers
possible: they depend on the model and on neither loader. The model
and the .ssp loader are the ssp-reader code, moved and renamed to the
level they belong to: the model is neither the .ssp nor the .ocda.db.
ocda.load_db is new, and returns the same value from sqlite that
load_ssp returns from text. ocda.load is the one place that decides
what a path is.
The claim is only true if the two loaders agree, so test/agree.d asks:
load a document both ways and compare the models object by object.
It found a real defect on its first run. Eight of thirty-six
disagreed, and the cause was the .ssp loader, not the formats: an
object whose first text line is blank lost it, the loader having
decided whether to start a new line by asking whether the accumulated
text was empty, which is a different question from whether a line has
been seen. Seven documents of the sample set have verse opening on a
blank line and every one of them was wrong.
Nothing else had caught it: the two .ssp readers were compared on
section counts, which the bug does not disturb. It took loading one
document two ways and comparing the objects. Fixed here, and in
ssp-reader/dlang which carried the same code. 36 of 36 now agree.
docs: the repository is sisudoc-ocda-lib
Prefix the repository name, as every other repository of this
project's own is prefixed. This one needed it most: "ocda-lib" is a
three letter acronym and the word "lib", and in a clone, away from
the parent directory that supplied the context, it reads as a generic
library belonging to no-one.
The dub path-dependency example and the note on where the blank-line
fix also went follow the new sibling name.
Binary, package and module names are unchanged: ocda-agree and the
ocda.* modules are read with the repository already in hand.
style: restore the comment arrows, double-encoded by an edit
The /+ ↓ +/ arrows had each of their three UTF-8 bytes re-encoded as
UTF-8 in turn, by an in-place edit that read the file as latin-1. 13
comments in load_ssp.d. Comment text only; the parser is unchanged and
the agreement check still passes.
ocda-dump, and the four disagreements it found
The agreement check compared ten fields of each object. That was enough
to find the blank-first-line defect and not enough for anything
subtler, so this adds ocda-dump: the whole loaded model written out
canonically, every field, sorted dictionaries, sections in format
order, defaults omitted. It is an oracle, not an output format. Two
implementations of this model can now be compared with diff.
Run against the reference collection it found four disagreements
between the two loaders, none of which the ten-field check could see.
1. .identifier and .anchor. The .ssp emits a property only when it
carries information, so the identifier is absent when it is simply
the ocn and the anchor is absent when it is the segment name; the
database stores both unconditionally. Over one document that is
1,314 identifiers and 69 anchors. The loader now derives them, which
is what the model means: an object's identifier is its ocn unless it
says otherwise.
2. make and doc_has. The database has one metadata table where the
text form has three blocks, and tells them apart by a key prefix.
The loader now unprefixes them, so a writer asks doc.make["footer"]
whichever serialisation it was handed and never
doc.metadata["make.footer"].
3. The JSON columns were read by a character scan that stripped
brackets and quotes, which is wrong the moment a value holds an
escape: JSONValue.toString escapes "/" as "\/", so every heading
containing a slash came back with a backslash in it. All 36
document-languages differed on heading_ancestors_text. Now parsed as
JSON.
4. translation.ocn_aligned is in the database and not in the .ssp,
legitimately: it says whether the languages share one numbering, and
a .ssp is one language and cannot know. It is a fact about the file
rather than the document, so the loader leaves it out rather than
letting a writer tell which serialisation it was given.
Metadata values are also stripped now, as the text loader strips them:
one topic_register value carried surrounding spaces in the database
that the text form cannot represent.
After all four: 36 of 36 document-languages give byte identical dumps
from their .ssp and from their .ocda.db.
ocda-lib in Gleam, and four agreements of 36 of 36
The model, both loaders, the dispatch and the canonical dump, in Gleam.
A deliberate field-for-field mirror of the D library, including where
that is not the most idiomatic Gleam: text is one string with the lines
joined by newline rather than a list of lines, and the properties are
typed fields rather than a list of pairs. A difference that existed in
one language and not the other would show in every dump and mean
nothing, so the two lose the same distinctions in the same places.
The dump is what makes the agreement testable. Both languages write it
byte for byte alike, so four questions are diff, and over the reference
collection all four hold:
D: .ssp == .ocda.db 36 of 36
Gleam: .ssp == .ocda.db 36 of 36
.ssp: D == Gleam 36 of 36
.ocda.db: D == Gleam 36 of 36
and a fifth, that the two report exactly the same ignored properties
with the same counts.
The Gleam .ssp loader agreed with the D one at the first attempt. The
Gleam .ocda.db loader had one defect of its own, found by the sweep and
worth recording because of its shape: a JSON element decoder handled
strings and integers and fell through to a fallback that yielded "".
table_widths is an integer array except where the columns divide
evenly, when it is [33.3333,...], and one table in the collection does
that. Its widths came back as three empty strings, so a value that
could not be read turned into one that looked read. That is the failure
the unknown counts exist to prevent, and a decoder fallback quietly
reintroduces it.
Shipped as an Erlang shipment rather than an escript, as the .ocda.db
reader is and for the same reason: sqlite on the BEAM is a NIF and a
NIF cannot load from inside an escript archive.
gleam.toml and the model came in with the previous commit, which was
about the dump; they belong here.
test: the fixture indented its property lines, so none were read
A `.ssp` property line is `.name: value` at the start of the line. The
Gleam test fixture indented them by two spaces, which makes each a line
of no known shape, so every property in the fixture was being counted as
malformed and ignored. The tests around it passed for the wrong reason.
Corrected, and a test added that reads a property, a digest and a
parent, and asserts that the fixture now holds no malformed line. Only
the fixture was wrong: the loaders agree with each other and with the D
library over the whole reference collection, which is real files.
a third input form: the abstraction out of a pod archive
A pod built with --pod2 carries the abstraction inside it, at
media/abstraction/<uid>.<lang>.ssp. So ocda.load gains .sisupod and .zip,
and because the dispatch is in one place a writer over the library gained
a third input form without being touched.
Over the reference collection a .sisupod gives the same model as the .ssp
beside it: 36 of 36 document-languages, byte identical dumps. And the
text writer, unchanged, renders a document from its .sisupod byte
identically to spine --text, 36 of 36.
Two small readers, neither of them a library: ocda.zstd declares the four
libzstd entry points needed to unwrap a frame, as ocda.sqlite declares
sqlite3; ocda.zip reads a central directory and one stored entry. Deflate
is refused rather than mishandled, since spine writes stored entries and
a library that says it read what it skipped is worse than one that says
it cannot. A frame that declares an expansion over 512 MiB is refused
before anything is allocated: a reader of an artefact it did not make
declines to be told how much memory to take.
The Gleam library reads the zip and not the frame. The only zstd on the
BEAM is a NIF that clones zstd from github in its own build, so it cannot
be built sandboxed without patching it. A .zip pod is read in both
languages, a .sisupod in D only, and the Gleam refusal says why and says
"zstd -d" rather than claiming the file is malformed. It is a packaging
limit, not anything about the format, and the only place the two
libraries differ in what they accept.
A pod without an abstraction is refused with a message naming the flag
that puts one in, because that is a fact about the pod.
two fields the model was missing, found by writing an html writer
anchor_tags and heading_lev_anchor. Both are in the specification's
property table and neither was read, so they went to the unknown counts
and a writer over the library had no way to emit a link target.
The cost was visible the moment an html writer existed: 154 of its
internal links pointed at nothing, because a note reference points at
the endnote's anchor and the endnote's anchor is an anchor_tag.
heading_lev_anchor is the name a document's own cross references use,
1,854 of them in the reference collection.
The .ssp carries them as repeated .anchor_tag and as
.heading_lev_anchor; the database as the object_anchors child table and
the heading_lev_anchor column. Both loaders read both now, the dump
reports them, and the four-way agreement still holds: 36 of 36 each way.
One thing to watch when adding a column to the database query: the read
is by position, so a column inserted in the middle shifts every one
after it. Adding heading_lev_anchor after segment_html_is broke all 36
documents until it moved to the end of the select.
images from the source, and the epub segment fields
Two additions the epub writer needed and could not have without.
ocda.images: an image's bytes, from whichever input form the document came
from. The model names images, with digest, byte size and pixel size, and
does not hold them - which is right, an abstraction is not a bundle of
assets - so something has to know where they are, and the answer depends
on the input form rather than on the output. Hence the library.
.ocda.db the files table, role 'image', by name
.sisupod the archive entry whose path ends in /image/<name>
.zip the same
.ssp beside it: a pod puts the abstraction in media/abstraction/
and its images in media/image/, so the path is documented
rather than guessed
canGiveImages says whether a source can answer at all, because a writer
has to know before it emits an <img>: a manifest item whose file is absent
and an <img> whose file is not in the manifest are both epubcheck errors.
A .ssp outside a pod has nowhere to look, and that is absence rather than
error - the document is still readable.
Reading a blob needed sqlite3_column_blob rather than _column_text: an
image read as text stops at the first zero byte, and a png's ninth byte
is zero.
And two more model fields: segment_epub and segment_epub_is. The epub
segments at heading levels 1 to 4 where the html segments at level 4
alone, so the two are different files and the abstraction records both.
Nine of the 36 documents have images; the four-way agreement still holds
at 36 of 36 each way.
model 2.1: ocn_range, the verses a poem holds
The format now states, for a container, the first and last ocn of
what it holds: .ocn_range in the .ssp, ocn_range_first and
ocn_range_last in the .ocda.db. A poem is the one container, and since
2.1 it carries ocn 0 and comes before its verses.
Both fields in the model, read by both loaders, in both languages, and
written by the dump. A 2.0 database has neither column, so each
loader asks sqlite whether it is there and reads a 2.0 file as having
no ranges, which is what 2.0 meant; checked on a published 2.0
database, D and Gleam giving the same dump.
Over the 36 document-languages written by spine at 2.1, the four
dumps (D and Gleam, .ssp and .ocda.db) are byte identical, 36 of 36.
D: what a source holds, and what a database holds unread
ocda.inventory answers two questions about the file rather than the
document. languages(path), for every form alike: the languages in the
manifest's order (which every .ssp records in @source languages), each
with the markup it was made from and the sha256 of its .ssp - stored
by the database, taken from the bytes for a .ssp or a pod, and the
same. carried(path), for a .ocda.db: what it holds besides the
abstraction, by role and storage.
And the database loader now counts what it holds and the model does
not. The text loader meets every property line and counts those it
does not keep; this one asks for columns by name and so never saw the
others, and reported that it had understood everything, which is the
silence the count exists to prevent. It counts them on the conditions
under which the .ssp writes the same property, and the report is now
the same from a .ssp, a .ocda.db and a .sisupod: 36 of 36.
The pod loader's archive and language-name helpers become package
visible for the inventory.
D: ocda-read, the reader, over the library, of every form
One reader where there were two. sisudoc-ssp-reader read a .ssp and
sisudoc-ocda-db-reader a .ocda.db, each with its own parser; this one
is built on the library, so every mode reads a .ssp, a .ocda.db and a
.sisupod, and says the same thing of each. Its output modules are the
.ssp reader's, copied (the reader repositories are unchanged) and
moved onto the library's model, which was the reader's types renamed
and added to. From the database reader, --languages and --files.
ocda-read [--tree|--outline|--html|--llm-text|--llm-jsonl|--images|
--verify --image-dir=<d>|--languages|--files|
--report-unknown|--validate] <file> [lang]
Two modes are about the file and say so: --files answers for a
.ocda.db, and --validate checks a .ssp as text and refuses anything
else. Where a file holds several languages and none is named, every
form reads the first and says so, where the database used to take one
and the pod refuse.
In src/read/ beside the library in src/ocda/; dub.json keeps it out of
the library (configurations "library" and "ocda-read"), and the flake
builds it as a package. The flake's default is now all three programs
in one bin/: built separately into the same result link, each replaced
the one before, and the pipeline's dump looked for ocda-dump where
only ocda-agree was left.
Checked over the 36 document-languages:
- against the .ssp reader, on a .ssp, byte for byte after the first
line (Parsed: is Read:): --tree, --html, --llm-text, --llm-jsonl,
--images, --verify, --validate, 36 of 36; the summary differs only
in the closing hint, which names the two new modes;
- against the database reader, on a .ocda.db: --files 27 of 27, and
--languages 27 of 27 on language, source and digest (its id column,
the database's own key, is left out);
- the same output from the .ssp, the .ocda.db and the .sisupod: the
summary and --tree, --html, --llm-text, --llm-jsonl, --images and
--report-unknown, 36 of 36; --languages, .ocda.db == .sisupod 27 of
27, and a single-language .ssp == its database, 26 of 26.
Not reproduced, deliberately: the database reader's --outline, which
ordered the headings by their position within their own section, so
that the toc and the "(skip)" headings of other sections fell among
the body's. --outline here is --tree.
Gleam: what a source holds, and what a database holds unread
The mirror of the D commit before it. ocda/inventory gives the
languages of any form, in the manifest's order, each with its markup
file and the sha256 of its .ssp (stored by the database, taken from
the bytes for a .ssp or a pod), and what a .ocda.db carries. The
sha256 comes from OTP's crypto through a four line Erlang FFI file,
since the standard library has none and a Hex package would be one
more fixed output hash; the whitelist admits src/*.erl for it.
The database loader counts the columns it does not read, on the
conditions the .ssp writer uses, so that it can say what it did not
understand; it could not before. The pod loader's archive, language
and text helpers become public for the inventory.
Checked: the four dumps, D and Gleam of .ssp and .ocda.db, identical
36 of 36, and with --unknown also 36 of 36; --languages of a database
== of its pod, 27 of 27.
Gleam: ocda-read, byte for byte the D reader's
ocda-lib read, installed also as ocda-read: the modes the Gleam readers
had (the summary, --tree/--outline, --languages, --files,
--report-unknown), over the library, in src/read/, which is where
Gleam must put it. A D-only mode asked of it says so.
Its output is the D ocda-read's. For that the D reader's closing hint
no longer lists its modes, which this one has fewer of: both say that
--help lists them.
Checked over the 36 document-languages, on the .ssp and the .ocda.db:
D == Gleam byte for byte in all five modes, 36 of 36. And the same
from the .ssp, the .ocda.db and the .zip (a .sisupod unwrapped, the
Gleam library having no zstd): the summary, --tree and
--report-unknown 36 of 36; --languages of a database == of its pod
27 of 27, and of a single-language .ssp == its database; --files
answers for a database and says why not for the others, in the same
words in both languages.
README: the reader, in each language, and what writing it found
The library's reader, ocda-read: where it lives and why (src/read/,
kept out of the library in D by dub configuration, in Gleam because
nothing outside src/ is compiled), its modes in each language, and
the two properties it holds, the same output from every form and D ==
Gleam byte for byte. And the one defect it found in the library, the
database loader unable to say what it had not read. The module tables
gain ocda.inventory and the reader, the build notes the three
programs in one result.
(assisted by Claude-Code)
Diffstat (limited to 'dlang/src/ocda')
| -rw-r--r-- | dlang/src/ocda/dump.d | 215 | ||||
| -rw-r--r-- | dlang/src/ocda/images.d | 149 | ||||
| -rw-r--r-- | dlang/src/ocda/inventory.d | 166 | ||||
| -rw-r--r-- | dlang/src/ocda/load.d | 118 | ||||
| -rw-r--r-- | dlang/src/ocda/load_db.d | 404 | ||||
| -rw-r--r-- | dlang/src/ocda/load_pod.d | 164 | ||||
| -rw-r--r-- | dlang/src/ocda/load_ssp.d | 391 | ||||
| -rw-r--r-- | dlang/src/ocda/model.d | 198 | ||||
| -rw-r--r-- | dlang/src/ocda/sqlite.d | 171 | ||||
| -rw-r--r-- | dlang/src/ocda/zip.d | 164 | ||||
| -rw-r--r-- | dlang/src/ocda/zstd.d | 128 |
11 files changed, 2268 insertions, 0 deletions
diff --git a/dlang/src/ocda/dump.d b/dlang/src/ocda/dump.d new file mode 100644 index 0000000..9b24cea --- /dev/null +++ b/dlang/src/ocda/dump.d @@ -0,0 +1,215 @@ +/+ +- Name: SisuDoc Spine, Doc Reform [a part of] + - Description: documents, structuring, processing, publishing, search + - static content generator + + - Author: Ralph Amissah + [ralph.amissah@gmail.com] + + - Copyright: (C) 2015 (continuously updated, current 2026) Ralph Amissah, All Rights Reserved. + + - License: AGPL 3 or later: + + Spine (SiSU), a framework for document structuring, publishing and + search + + Copyright (C) Ralph Amissah + + This program is free software: you can redistribute it and/or modify it + under the terms of the GNU AFERO General Public License as published by the + Free Software Foundation, either version 3 of the License, or (at your + option) any later version. + + This program is distributed in the hope that it will be useful, but WITHOUT + ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + more details. + + You should have received a copy of the GNU General Public License along with + this program. If not, see [https://www.gnu.org/licenses/]. + + If you have Internet connection, the latest version of the AGPL should be + available at these locations: + [https://www.fsf.org/licensing/licenses/agpl.html] + [https://www.gnu.org/licenses/agpl.html] + + - Spine (by Doc Reform, related to SiSU) uses standard: + - docReform markup syntax + - standard SiSU markup syntax with modified headers and minor modifications + - docReform object numbering + - standard SiSU object citation numbering & system + + - Homepages: + [https://www.sisudoc.org] + [https://www.doc-reform.org] + + - Git + [https://git.sisudoc.org/] + ++/ +module ocda.dump; + +/+ ↓ the model, written out canonically. + . + Not an output format: an *oracle*. Two libraries in two languages + implement the same model from the same specification, and the cheap + way to ask whether they agree is to have each write the whole loaded + document as text and diff the two. That is only useful if the text is + fully determined by the model, so: + . + - every line is "<key> <value>", the value running to end of line, + and only one free-form value per line; + - a field at its default is *omitted*, so the output says what the + document has rather than what the struct declares; + - dictionaries are emitted in sorted key order, because neither + language's hash map has an order worth relying on; + - sections come in the fixed order of the format, not + alphabetically. + . + Anything that reads this to reconstruct a document has misunderstood + it. Load the .ssp or the .ocda.db. ++/ + +import std.algorithm : sort, splitter; +import std.array : appender, array, join; +import std.conv : to; +import ocda.model; + +/+ ↓ the nine sections, in format order +/ +static immutable string[] dump_sections = ["head", "toc", "body", "endnotes", + "glossary", "bibliography", "bookindex", "blurb", "tail"]; + +string dumpDocument(ref OcdaDocument doc) { + auto out_ = appender!string; + + void kv(string key, string value) { + if (value.length == 0) { return; } + out_ ~= key; out_ ~= " "; out_ ~= value; out_ ~= "\n"; + } + void kn(string key, long value) { + if (value == 0) { return; } + kv(key, value.to!string); + } + void kb(string key, bool value) { + if (!value) { return; } + kv(key, "1"); + } + void dict(string prefix, string[string] d) { + foreach (key; d.keys.dup.sort) { + out_ ~= prefix; out_ ~= " "; out_ ~= key; out_ ~= " "; + out_ ~= d[key]; out_ ~= "\n"; + } + } + + kv("version", doc.format_version); + kv("source_filename", doc.source_filename); + dict("source", doc.source); + dict("meta", doc.metadata); + dict("make", doc.make); + dict("doc_has", doc.doc_has); + + foreach (name; dump_sections) { + auto sec = doc.sectionByName(name); + if (sec is null || (*sec).length == 0) { continue; } + kv("section", name ~ " " ~ (*sec).length.to!string); + } + + foreach (name; dump_sections) { + auto sec = doc.sectionByName(name); + if (sec is null) { continue; } + foreach (i, obj; *sec) { + out_ ~= "object "; out_ ~= name; out_ ~= " "; + out_ ~= i.to!string; out_ ~= "\n"; + kn("ocn", obj.ocn); + kv("is_a", obj.is_a); + kv("own_section", obj.own_section); + kv("level", obj.heading_level); + kv("identifier", obj.identifier); + kv("part", obj.part); + kv("is_of_type", obj.is_of_type); + kv("segment_html_is", obj.segment_html_is); + kv("segment_epub", obj.segment_epub); + kv("segment_epub_is", obj.segment_epub_is); + kn("parent", obj.parent_ocn); + kn("last_descendant", obj.last_descendant_ocn); + kn("ocn_range_first", obj.ocn_range_first); + kn("ocn_range_last", obj.ocn_range_last); + kv("children", ints(obj.children)); + kv("ancestors", ints(obj.ancestors)); + kb("dummy", obj.dummy_heading); + kb("ocn_off", obj.ocn_off); + kn("indent_base", obj.indent_base); + kn("indent_hang", obj.indent_hang); + kb("bullet", obj.bullet); + kv("lang", obj.lang); + kb("has_links", obj.has_links); + kb("has_notes_reg", obj.has_notes_reg); + kb("has_notes_star", obj.has_notes_star); + kb("has_images", obj.has_images); + kv("segment", obj.segment); + kv("segment_prev", obj.segment_prev); + kv("segment_next", obj.segment_next); + kv("anchor", obj.anchor); + kv("heading_lev_anchor", obj.heading_lev_anchor); + kn("table_cols", obj.table_cols); + kv("table_widths", obj.table_widths); + kv("table_aligns", obj.table_aligns); + kb("table_header", obj.table_header); + kv("code_syntax", obj.code_syntax); + kb("code_linenumbers", obj.code_linenumbers); + kv("heading_ancestors_text", obj.heading_ancestors_text); + kv("sha256", obj.sha256); + foreach (url; obj.stow_link) { kv("stow_link", url); } + foreach (a; obj.anchor_tags) { kv("anchor_tag", a); } + foreach (img; obj.images) { + /+ ↓ the name last, because a filename may hold a space where + none of the other fields can +/ + kv("image", img.sha256 ~ " " ~ img.bytes.to!string ~ " " + ~ img.width.to!string ~ "x" ~ img.height.to!string ~ " " + ~ (img.missing ? "1" : "0") ~ " " ~ img.name); + foreach (key; img.fields.keys.dup.sort) { + kv("image_field", key ~ " " ~ img.fields[key]); + } + } + /+ ↓ text, when there is any. An object whose only text line is + blank holds "" here, indistinguishable from no text at all, + because the model joins the lines into one string. Both + libraries lose the same distinction in the same place, which is + what makes them comparable; it is recorded rather than hidden. + +/ + if (obj.text.length > 0) { + auto lines = obj.text.splitter("\n").array; + kv("text", lines.length.to!string); + foreach (line; lines) { + out_ ~= "| "; out_ ~= line; out_ ~= "\n"; + } + } + } + } + return out_.data; +} + +/+ ↓ what the reader met and did not understand, sorted. Separate from the + document dump: it describes the reader, not the document, and the + .ocda.db loader has nothing to report where the .ssp loader does. ++/ +string dumpUnknown(ref OcdaDocument doc) { + auto out_ = appender!string; + void counts(string prefix, size_t[string] d) { + foreach (key; d.keys.dup.sort) { + out_ ~= prefix; out_ ~= " "; out_ ~= key; out_ ~= " "; + out_ ~= d[key].to!string; out_ ~= "\n"; + } + } + counts("property", doc.unknown.property); + counts("block", doc.unknown.block); + counts("line", doc.unknown.line); + return out_.data; +} + +private string ints(int[] xs) { + if (xs.length == 0) { return ""; } + string[] parts; + foreach (x; xs) { parts ~= x.to!string; } + return parts.join(","); +} diff --git a/dlang/src/ocda/images.d b/dlang/src/ocda/images.d new file mode 100644 index 0000000..f6d2541 --- /dev/null +++ b/dlang/src/ocda/images.d @@ -0,0 +1,149 @@ +/+ +- Name: SisuDoc Spine, Doc Reform [a part of] + - Description: documents, structuring, processing, publishing, search + - static content generator + + - Author: Ralph Amissah + [ralph.amissah@gmail.com] + + - Copyright: (C) 2015 (continuously updated, current 2026) Ralph Amissah, All Rights Reserved. + + - License: AGPL 3 or later: + + Spine (SiSU), a framework for document structuring, publishing and + search + + Copyright (C) Ralph Amissah + + This program is free software: you can redistribute it and/or modify it + under the terms of the GNU AFERO General Public License as published by the + Free Software Foundation, either version 3 of the License, or (at your + option) any later version. + + This program is distributed in the hope that it will be useful, but WITHOUT + ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + more details. + + You should have received a copy of the GNU General Public License along with + this program. If not, see [https://www.gnu.org/licenses/]. + + If you have Internet connection, the latest version of the AGPL should be + available at these locations: + [https://www.fsf.org/licensing/licenses/agpl.html] + [https://www.gnu.org/licenses/agpl.html] + + - Spine (by Doc Reform, related to SiSU) uses standard: + - docReform markup syntax + - standard SiSU markup syntax with modified headers and minor modifications + - docReform object numbering + - standard SiSU object citation numbering & system + + - Homepages: + [https://www.sisudoc.org] + [https://www.doc-reform.org] + + - Git + [https://git.sisudoc.org/] + ++/ +module ocda.images; + +/+ ↓ an image's bytes, from whichever source the document came from. + . + The model *names* images, with digest, byte size and pixel size; it does + not hold them. That is right - an abstraction of a document is not a + bundle of its assets - but a writer of any format that shows images + needs the bytes, so something has to know where they are. This is that + something, and it is in the library rather than in a writer because the + answer depends on the *input form* and not on the output. + . + Three answers, and one refusal: + . + .ocda.db the files table, role 'image', by name + .sisupod the archive entry whose path ends in /image/<name> + .zip the same + .ssp beside it: a pod puts the abstraction in + media/abstraction/ and its images in media/image/, so the + path is documented rather than guessed + . + A .ssp that is not in a pod has nowhere to look, and that is reported as + absent rather than as an error: a reader that cannot find an image + should say so and carry on, because the document is still readable. ++/ + +import std.algorithm : endsWith; +import std.array : array; +import std.file : exists, isFile, read; +import std.path : buildPath, dirName; +import ocda.load; +import ocda.sqlite; +import ocda.zip; +import ocda.zstd; + +/+ ↓ the bytes, or an empty slice when the source has no such image +/ +ubyte[] imageBytes(string source, string name) { + final switch (sourceOf(source)) { + case Source.ocda_db: return fromDb(source, name); + case Source.pod: return fromArchive(source, name); + case Source.ssp: return fromBeside(source, name); + case Source.unknown: return []; + } +} + +/+ ↓ can this source give images at all? + A writer asks before it decides whether to emit them, because an <img> + pointing at a file it did not include is worse than no image. ++/ +bool canGiveImages(string source) { + final switch (sourceOf(source)) { + case Source.ocda_db: return true; + case Source.pod: return true; + case Source.ssp: return imageDirBeside(source).length > 0; + case Source.unknown: return false; + } +} + +private ubyte[] fromDb(string path, string name) { + auto db = Db.open(path); + scope(exit) { db.close(); } + if (!db.has("files")) { return []; } + /+ ↓ the name is from the abstraction, not from a caller, and the quote + doubling is what keeps it a value rather than syntax +/ + return db.blob("SELECT data FROM files WHERE role = 'image' AND name = '" + ~ sqlQuote(name) ~ "'"); +} + +private ubyte[] fromArchive(string path, string name) { + auto bytes = cast(ubyte[]) read(path); + if (zstdIsFrame(bytes)) { bytes = zstdDecompress(bytes, 512 * 1024 * 1024); } + auto entries = zipEntries(bytes); + foreach (e; entries) { + if (e.name.endsWith("/image/" ~ name)) { return zipRead(bytes, e); } + } + return []; +} + +private ubyte[] fromBeside(string path, string name) { + string dir = imageDirBeside(path); + if (dir.length == 0) { return []; } + string file = buildPath(dir, name); + return file.exists && file.isFile ? cast(ubyte[]) read(file) : []; +} + +/+ ↓ media/abstraction/<x>.ssp -> media/image/ +/ +private string imageDirBeside(string path) { + string dir = path.dirName; + if (!dir.endsWith("abstraction")) { return ""; } + string image_dir = buildPath(dir.dirName, "image"); + return image_dir.exists ? image_dir : ""; +} + +/+ ↓ a single quote doubled is how sql says a single quote +/ +private string sqlQuote(string s) { + string out_; + foreach (c; s) { + out_ ~= (c == '\'') ? "''" : [c]; + } + return out_; +} diff --git a/dlang/src/ocda/inventory.d b/dlang/src/ocda/inventory.d new file mode 100644 index 0000000..32b076b --- /dev/null +++ b/dlang/src/ocda/inventory.d @@ -0,0 +1,166 @@ +/+ +- Name: SisuDoc Spine, Doc Reform [a part of] + - Description: documents, structuring, processing, publishing, search + - static content generator + + - Author: Ralph Amissah + [ralph.amissah@gmail.com] + + - Copyright: (C) 2015 (continuously updated, current 2026) Ralph Amissah, All Rights Reserved. + + - License: AGPL 3 or later: + + Spine (SiSU), a framework for document structuring, publishing and + search + + Copyright (C) Ralph Amissah + + This program is free software: you can redistribute it and/or modify it + under the terms of the GNU AFERO General Public License as published by the + Free Software Foundation, either version 3 of the License, or (at your + option) any later version. + + This program is distributed in the hope that it will be useful, but WITHOUT + ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + more details. + + You should have received a copy of the GNU General Public License along with + this program. If not, see [https://www.gnu.org/licenses/]. + + If you have Internet connection, the latest version of the AGPL should be + available at these locations: + [https://www.fsf.org/licensing/licenses/agpl.html] + [https://www.gnu.org/licenses/agpl.html] + + - Spine (by Doc Reform, related to SiSU) uses standard: + - docReform markup syntax + - standard SiSU markup syntax with modified headers and minor modifications + - docReform object numbering + - standard SiSU object citation numbering & system + + - Homepages: + [https://www.sisudoc.org] + [https://www.doc-reform.org] + + - Git + [https://git.sisudoc.org/] + ++/ +module ocda.inventory; + +/+ ↓ what a source holds, beyond the one document the model is loaded from. + . + The model is one language of one document. Two questions a reader asks + are about the file instead: which languages it holds, and, for a + database, what else it carries. They are here rather than in a reader + because the answer depends on the input form, which is the library's + to know, and so that a reader asking them does not learn which form it + was handed. + . + languages() answers for every form, and answers the same: the + languages in the order the manifest gave them (which every .ssp + records, in @source languages), each with the markup file it was made + from and the sha256 of its .ssp. The database stores that digest; for a + .ssp and a pod it is taken from the bytes, which are the same bytes. + . + carried() is the database's: a .ocda.db holds the markup, images, + configuration and catalogues it was built from, a .ssp holds none of + them, and a pod's are its archive entries rather than anything the + abstraction says. So it answers for a database and says so otherwise. ++/ + +import std.algorithm : endsWith, countUntil; +import std.array : array, split; +import std.conv : to; +import std.digest : toHexString, LetterCase; +import std.digest.sha : sha256Of; +import std.file : read; +import std.string : strip; +import ocda.model; +import ocda.load; +import ocda.load_ssp; +import ocda.load_db; +import ocda.load_pod; +import ocda.sqlite; +import ocda.zip; + +struct OcdaLanguage { + string lang; + string source_filename; // the markup file the abstraction was made from + string ssp_digest; // sha256 of that language's .ssp, upper case hex +} + +struct OcdaCarried { + string role; // source, image, conf, manifest, tools + string compression; // how it is stored: none, zstd + string count; + string bytes; // the files' own size, whatever the storage + string stored; // what they take in the database +} + +OcdaLanguage[] languages(string path) { + final switch (sourceOf(path)) { + case Source.ocda_db: + auto db = Db.open(path); + scope(exit) { db.close(); } + OcdaLanguage[] out_; + foreach (l; dbLanguages(db)) { + out_ ~= OcdaLanguage(l.lang, l.source_filename, l.ssp_digest); + } + return out_; + case Source.ssp: + auto bytes = cast(ubyte[]) read(path); + auto doc = parseSsp(cast(string) bytes.idup); + return [ OcdaLanguage(sourceValue(doc, "language"), doc.source_filename, hexOf(bytes)) ]; + case Source.pod: + auto archive = archiveBytes(path); + OcdaLanguage[string] found; + string[] order; + foreach (e; zipFindSuffix(zipEntries(archive), ".ssp")) { + auto bytes = zipRead(archive, e); + auto doc = parseSsp(cast(string) bytes.idup); + string lang = langOfSspName(e.name); + found[lang] = OcdaLanguage(lang, doc.source_filename, hexOf(bytes)); + if (order.length == 0) { + order = sourceValue(doc, "languages").split; + } + } + /+ ↓ the manifest's order, and anything it did not name after it +/ + OcdaLanguage[] out_; + foreach (l; order) { if (auto p = l in found) { out_ ~= *p; } } + foreach (l, v; found) { if (order.countUntil(l) < 0) { out_ ~= v; } } + return out_; + case Source.unknown: + throw new OcdaLoadError(path ~ ": " ~ sourceName(Source.unknown)); + } +} + +/+ ↓ what a database carries, by role and storage; empty for any other form, + which isCarrier() says in advance +/ +bool isCarrier(string path) { + return sourceOf(path) == Source.ocda_db; +} + +OcdaCarried[] carried(string path) { + if (!isCarrier(path)) { return []; } + auto db = Db.open(path); + scope(exit) { db.close(); } + if (!db.has("files")) { return []; } + OcdaCarried[] out_; + foreach (r; db.rows( + "SELECT role, compression, count(*), sum(bytes), sum(length(data)) + FROM files GROUP BY role, compression ORDER BY role")) { + out_ ~= OcdaCarried(r[0], r[1], r[2], r[3], r[4]); + } + return out_; +} + +private string sourceValue(ref OcdaDocument doc, string key) { + if (auto p = key in doc.source) { return *p; } + return ""; +} + +private string hexOf(const(ubyte)[] bytes) { + return toHexString!(LetterCase.upper)(sha256Of(bytes)).idup; +} diff --git a/dlang/src/ocda/load.d b/dlang/src/ocda/load.d new file mode 100644 index 0000000..4a0e463 --- /dev/null +++ b/dlang/src/ocda/load.d @@ -0,0 +1,118 @@ +/+ +- Name: SisuDoc Spine, Doc Reform [a part of] + - Description: documents, structuring, processing, publishing, search + - static content generator + + - Author: Ralph Amissah + [ralph.amissah@gmail.com] + + - Copyright: (C) 2015 (continuously updated, current 2026) Ralph Amissah, All Rights Reserved. + + - License: AGPL 3 or later: + + Spine (SiSU), a framework for document structuring, publishing and + search + + Copyright (C) Ralph Amissah + + This program is free software: you can redistribute it and/or modify it + under the terms of the GNU AFERO General Public License as published by the + Free Software Foundation, either version 3 of the License, or (at your + option) any later version. + + This program is distributed in the hope that it will be useful, but WITHOUT + ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + more details. + + You should have received a copy of the GNU General Public License along with + this program. If not, see [https://www.gnu.org/licenses/]. + + If you have Internet connection, the latest version of the AGPL should be + available at these locations: + [https://www.fsf.org/licensing/licenses/agpl.html] + [https://www.gnu.org/licenses/agpl.html] + + - Spine (by Doc Reform, related to SiSU) uses standard: + - docReform markup syntax + - standard SiSU markup syntax with modified headers and minor modifications + - docReform object numbering + - standard SiSU object citation numbering & system + + - Homepages: + [https://www.sisudoc.org] + [https://www.doc-reform.org] + + - Git + [https://git.sisudoc.org/] + ++/ +module ocda.load; + +/+ ↓ what a path is, and loading it. + . + One place that decides, so that every reader and every writer accepts + the same inputs and a new input type is added once. This mirrors + spine's own ocda/abstraction/load.d, which names the sources it can + enter the pipeline from and tells them apart by suffix. + . + Deliberately *not* here: .sst and .ssm markup. Reading markup means + implementing the parser, which is spine's work and a far larger thing + than a reader or a sample writer needs. This library reads + abstractions. ++/ + +import std.algorithm : endsWith; +import ocda.model; +import ocda.load_ssp; +import ocda.load_db; +import ocda.load_pod; + +enum Source { + ssp, // .ssp, the abstraction as text + ocda_db, // .ocda.db, the abstraction as sqlite + pod, // .sisupod or .zip, an archive carrying a .ssp + unknown, +} + +Source sourceOf(string path) { + if (path.endsWith(".ocda.db")) { return Source.ocda_db; } + if (path.endsWith(".ssp")) { return Source.ssp; } + /+ ↓ .sisupod is the archive spine writes now; .zip is what it wrote + before, and every pod already published carries that name, so both + are read. What is inside decides how it is opened, not the name. + +/ + if (path.endsWith(".sisupod")) { return Source.pod; } + if (path.endsWith(".zip")) { return Source.pod; } + if (path.endsWith(".db")) { return Source.ocda_db; } + return Source.unknown; +} + +string sourceName(Source s) { + final switch (s) { + case Source.ssp: return ".ssp (the abstraction as text)"; + case Source.ocda_db: return ".ocda.db (the abstraction as sqlite)"; + case Source.pod: return "a pod archive carrying a .ssp"; + case Source.unknown: return "not an abstraction this library reads"; + } +} + +class OcdaLoadError : Exception { + this(string msg) { super(msg); } +} + +/+ ↓ load whatever the path is into the model. + lang names which language to take from a .ocda.db, which holds + every language of its document, and from a pod, which may carry more + than one .ssp. A .ssp is one language by construction and ignores it. ++/ +OcdaDocument load(string path, string lang = "", bool any_version = false) { + final switch (sourceOf(path)) { + case Source.ssp: return parseSspFile(path); + case Source.ocda_db: return loadDbFile(path, lang, any_version); + case Source.pod: return loadPodFile(path, lang); + case Source.unknown: + throw new OcdaLoadError(path ~ ": " ~ sourceName(Source.unknown) + ~ " (expected .ssp, .ocda.db, .sisupod or .zip)"); + } +} diff --git a/dlang/src/ocda/load_db.d b/dlang/src/ocda/load_db.d new file mode 100644 index 0000000..d52b121 --- /dev/null +++ b/dlang/src/ocda/load_db.d @@ -0,0 +1,404 @@ +/+ +- Name: SisuDoc Spine, Doc Reform [a part of] + - Description: documents, structuring, processing, publishing, search + - static content generator + + - Author: Ralph Amissah + [ralph.amissah@gmail.com] + + - Copyright: (C) 2015 (continuously updated, current 2026) Ralph Amissah, All Rights Reserved. + + - License: AGPL 3 or later: + + Spine (SiSU), a framework for document structuring, publishing and + search + + Copyright (C) Ralph Amissah + + This program is free software: you can redistribute it and/or modify it + under the terms of the GNU AFERO General Public License as published by the + Free Software Foundation, either version 3 of the License, or (at your + option) any later version. + + This program is distributed in the hope that it will be useful, but WITHOUT + ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + more details. + + You should have received a copy of the GNU General Public License along with + this program. If not, see [https://www.gnu.org/licenses/]. + + If you have Internet connection, the latest version of the AGPL should be + available at these locations: + [https://www.fsf.org/licensing/licenses/agpl.html] + [https://www.gnu.org/licenses/agpl.html] + + - Spine (by Doc Reform, related to SiSU) uses standard: + - docReform markup syntax + - standard SiSU markup syntax with modified headers and minor modifications + - docReform object numbering + - standard SiSU object citation numbering & system + + - Homepages: + [https://www.sisudoc.org] + [https://www.doc-reform.org] + + - Git + [https://git.sisudoc.org/] + ++/ +module ocda.load_db; + +/+ ↓ build the model from a .ocda.db. + . + The point of this module is that it returns *the same value* + ocda.load_ssp returns from the same document's .ssp. A writer is then + a function from the model to bytes, and cannot tell which + serialisation it was given, exactly as spine's own writers cannot. + . + Written from the sisudoc-abstraction-format specification, + ocda-db-format-specification.md, whose section 4 gives the column by + column correspondence this module implements. ++/ + +import std.array : join, split; +import std.json : JSONType, parseJSON; +import std.conv : to; +import std.algorithm : startsWith; +import std.string : strip; +import ocda.model; +import ocda.sqlite; + +/+ ↓ the schema major this loader knows (specification section 2) +/ +enum schema_major_read = "2"; + +class OcdaDbError : Exception { + this(string msg) { super(msg); } +} + +/+ ↓ every language the database holds, in manifest order +/ +struct DbLanguage { + int doc_id; + string lang; + string source_filename; + string ssp_digest; +} + +DbLanguage[] dbLanguages(ref Db db) { + DbLanguage[] out_; + foreach (r; db.rows( + "SELECT id, lang, coalesce(source_filename,''), coalesce(ssp_digest,'') + FROM documents ORDER BY id")) { + out_ ~= DbLanguage(r[0].to!int, r[1], r[2], r[3]); + } + return out_; +} + +/+ ↓ load one language of a database as the model. + lang == "" takes the first language the manifest gave. ++/ +OcdaDocument loadDbFile(string path, string lang = "", bool any_version = false) { + auto db = Db.open(path); + scope(exit) { db.close(); } + + if (!db.has("documents") || !db.has("objects") || !db.has("metadata")) { + throw new OcdaDbError(path ~ ": no documents/objects/metadata tables;" + ~ " it does not look like a .ocda.db"); + } + + /+ ↓ specification sections 2 and 3.2: the rows describing the file rather + than a document in it are the only ones with doc_id null +/ + string schema_version = db.one( + "SELECT value FROM metadata WHERE doc_id IS NULL AND key = 'schema.version'"); + if (schema_version.length > 0 && !any_version) { + auto major = schema_version.split(".")[0]; + if (major != schema_major_read) { + throw new OcdaDbError(path ~ " is schema v" ~ schema_version + ~ ", this loader knows v" ~ schema_major_read ~ ".x"); + } + } + + auto languages = dbLanguages(db); + if (languages.length == 0) { + throw new OcdaDbError(path ~ ": no documents rows"); + } + DbLanguage chosen = languages[0]; + if (lang.length > 0) { + bool found = false; + foreach (l; languages) { if (l.lang == lang) { chosen = l; found = true; break; } } + if (!found) { + string have; + foreach (i, l; languages) { have ~= (i > 0 ? " " : "") ~ l.lang; } + throw new OcdaDbError(path ~ ": no language '" ~ lang ~ "'; it holds: " ~ have); + } + } + + OcdaDocument doc; + doc.format_version = schema_version; + doc.source_filename = chosen.source_filename; + + /+ ↓ @source of the .ssp corresponds to the documents row and the + source.digest metadata (specification section 5.1 of the .ssp, and + section 7 here) +/ + doc.source["language"] = chosen.lang; + string languages_joined; + foreach (i, l; languages) { languages_joined ~= (i > 0 ? " " : "") ~ l.lang; } + doc.source["languages"] = languages_joined; + auto digest = db.one("SELECT value FROM metadata WHERE doc_id = " + ~ chosen.doc_id.to!string ~ " AND key = 'source.digest'"); + if (digest.length > 0) { doc.source["digest"] = digest; } + + /+ ↓ @meta: every metadata row of this language that is not a source.* + property. source.* describes the artefact rather than the document + and lives in @source in the text form. + +/ + foreach (r; db.rows("SELECT key, value FROM metadata WHERE doc_id = " + ~ chosen.doc_id.to!string ~ " ORDER BY key")) { + string key = r[0]; + if (key.startsWith("source.")) { continue; } + /+ ↓ translation.ocn_aligned says whether the languages of this + document share one numbering. A .ocda.db can know it because it + holds every language; a .ssp is one language and cannot. So it is + a fact about the *file* rather than about the document, and putting + it in the document's metadata would let a writer tell which + serialisation it was handed, which is the one thing this library + exists to prevent. Left out until the model has a place for + collection-wide facts. + +/ + if (key.startsWith("translation.")) { continue; } + /+ ↓ the database has one metadata table where the text form has three + blocks, and tells them apart by a key prefix. Unprefixing them + here is what makes the two loaders give one model: a writer asks + doc.make["footer"] whichever serialisation it was handed, and + never doc.metadata["make.footer"]. + +/ + if (key.startsWith("make.")) { + doc.make[key["make.".length .. $]] = r[1].strip; + } else if (key.startsWith("doc_has.")) { + doc.doc_has[key["doc_has.".length .. $]] = r[1].strip; + } else { + doc.metadata[key] = r[1].strip; + } + } + + /+ ↓ 2.1 added the container range as two columns. A 2.0 database has + neither, and reads as having no range, which is what 2.0 meant. +/ + bool has_range = false; + foreach (c; db.rows("PRAGMA table_info(objects)")) { + if (c[1] == "ocn_range_first") { has_range = true; } + } + string range_cols = has_range + ? "coalesce(ocn_range_first,0), coalesce(ocn_range_last,0)" + : "0, 0"; + + /+ ↓ the objects, in document order within each section +/ + foreach (r; db.rows( + "SELECT section, seq, ocn, is_a, coalesce(is_of_part,''), + coalesce(is_of_section,''), coalesce(is_of_type,''), + coalesce(heading_level,-1), coalesce(identifier,''), + coalesce(parent_ocn,0), coalesce(last_descendant_ocn,0), + coalesce(children,''), coalesce(ancestors,''), + coalesce(dummy_heading,0), coalesce(object_number_off,0), + coalesce(indent_base,0), coalesce(indent_hang,0), + coalesce(bullet,0), coalesce(lang,''), + coalesce(has_links,0), coalesce(has_notes_reg,0), + coalesce(has_notes_star,0), coalesce(has_images,0), + coalesce(segment,''), coalesce(segment_prev,''), + coalesce(segment_next,''), coalesce(anchor,''), + coalesce(segment_html_is,''), + coalesce(table_cols,0), coalesce(table_widths,''), + coalesce(table_aligns,''), coalesce(table_header,0), + coalesce(code_linenumbers,0), + coalesce(heading_ancestors_text,''), coalesce(sha256,''), + coalesce(text,''), id, coalesce(heading_lev_anchor,''), + coalesce(segment_epub,''), coalesce(segment_epub_is,''), " ~ range_cols ~ " + FROM objects WHERE doc_id = " ~ chosen.doc_id.to!string + ~ " ORDER BY CASE section + WHEN 'head' THEN 1 WHEN 'toc' THEN 2 WHEN 'body' THEN 3 + WHEN 'endnotes' THEN 4 WHEN 'glossary' THEN 5 + WHEN 'bibliography' THEN 6 WHEN 'bookindex' THEN 7 + WHEN 'blurb' THEN 8 WHEN 'tail' THEN 9 ELSE 10 END, seq")) { + OcdaObject obj; + obj.section = r[0]; + obj.ocn = r[2].to!int; + obj.is_a = r[3]; + obj.part = r[4]; + obj.own_section = (r[5] == r[0]) ? "" : r[5]; + obj.is_of_type = r[6]; + /+ ↓ the database holds the level as a number, the text form as the + letter or digit the markup wrote. Model section 4 is the mapping. +/ + obj.heading_level = levelName(r[7].to!int); + obj.identifier = r[8]; + obj.parent_ocn = r[9].to!int; + obj.last_descendant_ocn = r[10].to!int; + obj.children = jsonInts(r[11]); + obj.ancestors = jsonInts(r[12]); + obj.dummy_heading = (r[13] != "0"); + obj.ocn_off = (r[14] != "0"); + obj.indent_base = r[15].to!int; + obj.indent_hang = r[16].to!int; + obj.bullet = (r[17] != "0"); + obj.lang = r[18]; + obj.has_links = (r[19] != "0"); + obj.has_notes_reg = (r[20] != "0"); + obj.has_notes_star = (r[21] != "0"); + obj.has_images = (r[22] != "0"); + obj.segment = r[23]; + obj.segment_prev = r[24]; + obj.segment_next = r[25]; + obj.anchor = r[26]; + obj.segment_html_is = r[27]; + obj.heading_lev_anchor = r[37]; + obj.segment_epub = r[38]; + obj.segment_epub_is = r[39]; + obj.ocn_range_first = r[40].to!int; + obj.ocn_range_last = r[41].to!int; + obj.table_cols = r[28].to!int; + obj.table_widths = jsonWords(r[29]); + obj.table_aligns = jsonWords(r[30]); + obj.table_header = (r[31] != "0"); + obj.code_linenumbers = (r[32] != "0"); + obj.heading_ancestors_text = jsonJoined(r[33]); + obj.sha256 = r[34]; + obj.text = r[35]; + + /+ ↓ the four repeatable properties of the text form are child tables + here (specification section 4) +/ + string object_id = r[36]; + foreach (i; db.rows("SELECT name, coalesce(sha256,''), coalesce(bytes,0), + coalesce(width,0), coalesce(height,0), + coalesce(missing,0) + FROM object_images WHERE object_id = " ~ object_id + ~ " ORDER BY seq")) { + OcdaImage img; + img.name = i[0]; + img.sha256 = i[1]; + img.bytes = i[2].to!ulong; + img.width = i[3].to!int; + img.height = i[4].to!int; + img.missing = (i[5] != "0"); + obj.images ~= img; + } + foreach (l; db.rows("SELECT url FROM object_links WHERE object_id = " + ~ object_id ~ " ORDER BY seq")) { + obj.stow_link ~= l[0]; + } + foreach (a; db.rows("SELECT anchor FROM object_anchors WHERE object_id = " + ~ object_id ~ " ORDER BY seq")) { + obj.anchor_tags ~= a[0]; + } + + auto section = doc.sectionByName(r[0]); + if (section !is null) { *section ~= obj; } + } + countUnread(db, chosen.doc_id, doc); + return doc; +} + +/+ ↓ what the database holds and the model does not. + . + The text loader meets every property line and counts those it does not + keep, so a reader can say what was read and not understood. This + loader asks for columns by name, and would never see the ones it did + not ask for: it would report that it had understood everything, which + is the silence the count exists to prevent. So it counts them, on the + conditions under which the .ssp writes the same property (specification + section 6.3), and the report is the same from either form: checked, + 36 of 36. ++/ +private void countUnread(ref Db db, int doc_id, ref OcdaDocument doc) { + string id = doc_id.to!string; + void count(string property, string where) { + auto n = db.one("SELECT count(*) FROM objects WHERE doc_id = " ~ id + ~ " AND " ~ where).to!size_t; + if (n > 0) { doc.unknown.property[property] = n; } + } + count("ancestors_collapsed", "coalesce(ancestors_collapsed,'') != ''"); + count("dom_status", "coalesce(dom_status,'') != ''"); + count("dom_status_collapsed", "coalesce(dom_status_collapsed,'') != ''"); + count("heading_lev_collapsed", "heading_lev_collapsed IS NOT NULL"); + count("parent_lev", "coalesce(parent_lev,0) != 0"); + count("segment_lv4_is", "coalesce(segment_lv4_is,'') != ''"); + count("attrib", "coalesce(attrib,'') != ''"); + if (db.has("object_subtoc")) { + auto n = db.one("SELECT count(*) FROM object_subtoc s JOIN objects o" + ~ " ON o.id = s.object_id WHERE o.doc_id = " ~ id).to!size_t; + if (n > 0) { doc.unknown.property["lev4_subtoc"] = n; } + } +} + +/+ ↓ model section 4: the eight levels and their indices +/ +private string levelName(int index) { + switch (index) { + case 0: return "A"; + case 1: return "B"; + case 2: return "C"; + case 3: return "D"; + case 4: return "1"; + case 5: return "2"; + case 6: return "3"; + case 7: return "4"; + default: return ""; + } +} + +/+ ↓ the JSON columns, read as JSON. + . + The fixed width arrays are JSON in the database and space separated + (or "|" separated) in the text form. This was first written as a + character scan that threw away brackets and quotes, which is wrong the + moment a value holds an escape: Phobos' JSONValue.toString escapes + "/" as "\/", so every heading containing a slash came back with a + backslash in it. 36 of 36 documents differed on + heading_ancestors_text and nothing else had shown it. + . + So: parse. The values are small, there are few of them, and a real + parser is shorter than the scan it replaces. ++/ +private string[] jsonStrings(string v) { + if (v.length == 0) { return []; } + string[] out_; + try { + auto j = parseJSON(v); + if (j.type != JSONType.array) { return []; } + foreach (e; j.array) { + switch (e.type) { + case JSONType.string: out_ ~= e.str; break; + case JSONType.integer: out_ ~= e.integer.to!string; break; + case JSONType.uinteger: out_ ~= e.uinteger.to!string; break; + case JSONType.float_: out_ ~= e.floating.to!string; break; + case JSONType.true_: out_ ~= "true"; break; + case JSONType.false_: out_ ~= "false"; break; + default: out_ ~= ""; break; + } + } + } catch (Exception) { + return []; + } + return out_; +} + +private int[] jsonInts(string v) { + int[] out_; + foreach (s; jsonStrings(v)) { + if (s.length == 0) { continue; } + try { out_ ~= s.to!int; } catch (Exception) {} + } + return out_; +} + +/+ ↓ space separated in the text form: table_widths, table_aligns +/ +private string jsonWords(string v) { + return jsonStrings(v).join(" "); +} + +/+ ↓ heading_ancestors_text is eight slots, joined with "|" in the text + form and a JSON array here. The empty slots are significant: they say + which levels the heading sits under, so the separators carry as much + as the text does. ++/ +private string jsonJoined(string v) { + auto parts = jsonStrings(v); + if (parts.length == 0) { return ""; } + return parts.join("|"); +} diff --git a/dlang/src/ocda/load_pod.d b/dlang/src/ocda/load_pod.d new file mode 100644 index 0000000..a99da0f --- /dev/null +++ b/dlang/src/ocda/load_pod.d @@ -0,0 +1,164 @@ +/+ +- Name: SisuDoc Spine, Doc Reform [a part of] + - Description: documents, structuring, processing, publishing, search + - static content generator + + - Author: Ralph Amissah + [ralph.amissah@gmail.com] + + - Copyright: (C) 2015 (continuously updated, current 2026) Ralph Amissah, All Rights Reserved. + + - License: AGPL 3 or later: + + Spine (SiSU), a framework for document structuring, publishing and + search + + Copyright (C) Ralph Amissah + + This program is free software: you can redistribute it and/or modify it + under the terms of the GNU AFERO General Public License as published by the + Free Software Foundation, either version 3 of the License, or (at your + option) any later version. + + This program is distributed in the hope that it will be useful, but WITHOUT + ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + more details. + + You should have received a copy of the GNU General Public License along with + this program. If not, see [https://www.gnu.org/licenses/]. + + If you have Internet connection, the latest version of the AGPL should be + available at these locations: + [https://www.fsf.org/licensing/licenses/agpl.html] + [https://www.gnu.org/licenses/agpl.html] + + - Spine (by Doc Reform, related to SiSU) uses standard: + - docReform markup syntax + - standard SiSU markup syntax with modified headers and minor modifications + - docReform object numbering + - standard SiSU object citation numbering & system + + - Homepages: + [https://www.sisudoc.org] + [https://www.doc-reform.org] + + - Git + [https://git.sisudoc.org/] + ++/ +module ocda.load_pod; + +/+ ↓ the abstraction out of a pod archive. + . + A .sisupod is one zstd frame wrapping a stored zip; a .zip is the same + archive unwrapped, which is what spine published before it wrapped them. + Either way, a pod built with --pod2 carries the abstraction inside it, + at media/abstraction/<uid>.<lang>.ssp, and that is what is read. + . + So this is a third input form and *not* a third model: the .ssp inside + is parsed by ocda.load_ssp and comes back as the same value. A writer + over the library gains .sisupod without being touched, which is the + whole reason the dispatch is in one place. + . + A pod built without --pod2 carries markup and no abstraction. That is + refused, and the refusal says which flag makes a pod this reader can + read, because "no abstraction in it" is a fact about the pod rather + than about the reader. ++/ + +import std.algorithm : endsWith, sort; +import std.array : array; +import std.file : read; +import std.string : indexOf; +import ocda.model; +import ocda.load_ssp; +import ocda.zip; +import ocda.zstd; + +/+ ↓ a limit on what a frame may expand to. + A reader of an artefact it did not make declines to be told how much + memory to allocate. 512 MiB is far above the largest pod in the + reference collection (4 MiB) and far below anything that matters. ++/ +enum size_t max_pod_size = 512 * 1024 * 1024; + +class OcdaPodError : Exception { + this(string msg) { super(msg); } +} + +/+ ↓ which languages of the abstraction the pod carries, in sorted order +/ +string[] podLanguages(string path) { + auto bytes = archiveBytes(path); + string[] out_; + foreach (e; zipFindSuffix(zipEntries(bytes), ".ssp")) { + string lang = langOfSspName(e.name); + if (lang.length > 0) { out_ ~= lang; } + } + return out_.sort.array; +} + +/+ ↓ load one language of the abstraction a pod carries. + lang == "" takes the only one there is, and where there is a choice it + says so rather than choosing, as the .ocda.db loader does. ++/ +OcdaDocument loadPodFile(string path, string lang = "") { + auto bytes = archiveBytes(path); + auto entries = zipEntries(bytes); + auto ssps = zipFindSuffix(entries, ".ssp"); + if (ssps.length == 0) { + throw new OcdaPodError(path ~ ": carries no .ssp." + ~ " A pod carries the abstraction only when spine was asked for it:" + ~ " spine --pod2 puts it in, spine --source does not."); + } + ZipEntry chosen = ssps[0]; + if (lang.length > 0) { + bool found = false; + foreach (e; ssps) { + if (langOfSspName(e.name) == lang) { chosen = e; found = true; break; } + } + if (!found) { + string have; + foreach (i, e; ssps) { + have ~= (i > 0 ? " " : "") ~ langOfSspName(e.name); + } + throw new OcdaPodError(path ~ ": no language '" ~ lang + ~ "'; it holds: " ~ have); + } + } else if (ssps.length > 1) { + string have; + foreach (i, e; ssps) { + have ~= (i > 0 ? " " : "") ~ langOfSspName(e.name); + } + throw new OcdaPodError(path ~ ": holds more than one language (" + ~ have ~ "); name the one wanted"); + } + auto ssp = zipRead(bytes, chosen); + return parseSsp(cast(string) ssp.idup); +} + +/+ ↓ the archive, unwrapped if it was wrapped. + The bytes decide, not the name: a .sisupod is a zstd frame and a .zip + is not, and a file misnamed either way still reads. ++/ +package ubyte[] archiveBytes(string path) { + auto bytes = cast(ubyte[]) read(path); + return zstdIsFrame(bytes) + ? zstdDecompress(bytes, max_pod_size) + : bytes; +} + +/+ ↓ <uid>.<lang>.ssp -> <lang> +/ +package string langOfSspName(string name) { + if (!name.endsWith(".ssp")) { return ""; } + string stem = name[0 .. $ - 4]; + auto dot = stem.lastIndexOf('.'); + return (dot > 0) ? stem[dot + 1 .. $] : ""; +} + +private ptrdiff_t lastIndexOf(string s, char c) { + for (ptrdiff_t i = s.length - 1; i >= 0; i--) { + if (s[i] == c) { return i; } + } + return -1; +} diff --git a/dlang/src/ocda/load_ssp.d b/dlang/src/ocda/load_ssp.d new file mode 100644 index 0000000..5ceb29e --- /dev/null +++ b/dlang/src/ocda/load_ssp.d @@ -0,0 +1,391 @@ +/+ +- Name: SisuDoc Spine, Doc Reform [a part of] + - Description: documents, structuring, processing, publishing, search + - static content generator + + - Author: Ralph Amissah + [ralph.amissah@gmail.com] + + - Copyright: (C) 2015 (continuously updated, current 2026) Ralph Amissah, All Rights Reserved. + + - License: AGPL 3 or later: + + Spine (SiSU), a framework for document structuring, publishing and + search + + Copyright (C) Ralph Amissah + + This program is free software: you can redistribute it and/or modify it + under the terms of the GNU AFERO General Public License as published by the + Free Software Foundation, either version 3 of the License, or (at your + option) any later version. + + This program is distributed in the hope that it will be useful, but WITHOUT + ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + more details. + + You should have received a copy of the GNU General Public License along with + this program. If not, see [https://www.gnu.org/licenses/]. + + If you have Internet connection, the latest version of the AGPL should be + available at these locations: + [https://www.fsf.org/licensing/licenses/agpl.html] + [https://www.gnu.org/licenses/agpl.html] + + - Spine (by Doc Reform, related to SiSU) uses standard: + - docReform markup syntax + - standard SiSU markup syntax with modified headers and minor modifications + - docReform object numbering + - standard SiSU object citation numbering & system + + - Homepages: + [https://www.sisudoc.org] + [https://www.doc-reform.org] + + - Git + [https://git.sisudoc.org/] + ++/ +module ocda.load_ssp; + +import std.algorithm : splitter, startsWith; +import std.array : array, split; +import std.conv : to, ConvException; +import std.file : readText; +import std.stdio : writeln; +import std.string : strip, stripLeft, indexOf; +import ocda.model; + +OcdaDocument parseSspFile(string filename) { + string content = readText(filename); + return parseSsp(content); +} + +OcdaDocument parseSsp(string content) { + OcdaDocument doc; + OcdaObject current_obj; + bool in_object = false; + /+ ↓ whether a text line has been seen for the object being read. + Not "is the text non-empty": an object whose first text line is blank + (a verse opening on an empty line, which the sample set has seven of) + would then lose it, the next line overwriting rather than appending. + +/ + bool seen_text = false; + string current_section = ""; + string current_block = ""; // meta, make, doc_has, or section name + + foreach (line; content.splitter("\n")) { + /+ ↓ comment. The first line of a .ssp declares the format version, and + it is the one comment that carries meaning: a reader checks the + major part and refuses what it does not know rather than loading + an unfamiliar artefact half populated in silence. + +/ + if (line.length > 0 && line[0] == '%') { + enum _banner = "% SiSU Document Abstraction v"; + enum _source = "% Source: "; + if (doc.format_version.length == 0 && line.startsWith(_banner)) { + doc.format_version = line[_banner.length .. $].strip; + } else if (doc.source_filename.length == 0 && line.startsWith(_source)) { + doc.source_filename = line[_source.length .. $].strip; + } + continue; + } + + /+ ↓ blank line - finalize current object +/ + if (line.length == 0 || line.strip.length == 0) { + if (in_object) { + commitObject(doc, current_section, current_obj); + current_obj = OcdaObject.init; + in_object = false; + seen_text = false; + } + continue; + } + + /+ ↓ section/block open +/ + if (line[0] == '@') { + auto brace_pos = line.indexOf('{'); + if (brace_pos > 0) { + string name = line[1 .. brace_pos].strip; + current_block = name; + if (!isPreambleBlock(name)) { + current_section = name; + /+ ↓ a block that is neither preamble nor a section this reader + knows: counted, so that --report-unknown can say so +/ + if (doc.sectionByName(name) is null) { + doc.unknown.block[name]++; + } + } + } + continue; + } + + /+ ↓ section/block close +/ + if (line[0] == '}') { + if (in_object) { + commitObject(doc, current_section, current_obj); + current_obj = OcdaObject.init; + in_object = false; + seen_text = false; + } + current_block = ""; + continue; + } + + /+ ↓ inside meta/make/doc_has blocks: key-value pairs +/ + if (isPreambleBlock(current_block)) { + string stripped = line.stripLeft; + auto colon_pos = stripped.indexOf(':'); + if (colon_pos > 0) { + string key = stripped[0 .. colon_pos].strip; + string value = (colon_pos + 1 < stripped.length) + ? stripped[colon_pos + 1 .. $].strip : ""; + if (current_block == "meta") { doc.metadata[key] = value; } + else if (current_block == "make") { doc.make[key] = value; } + else if (current_block == "source") { doc.source[key] = value; } + else { doc.doc_has[key] = value; } + } + continue; + } + + /+ ↓ object declaration: [N] type ... +/ + if (line[0] == '[') { + if (in_object) { + commitObject(doc, current_section, current_obj); + } + current_obj = OcdaObject.init; + current_obj.section = current_section; + in_object = true; + seen_text = false; + parseObjectDeclaration(line, current_obj); + continue; + } + + /+ ↓ property: .name: value +/ + if (line[0] == '.') { + if (in_object) { + parseProperty(line, current_obj, doc.unknown); + } + continue; + } + + /+ ↓ text content: | content +/ + if (line.length >= 1 && line[0] == '|') { + if (in_object) { + string text_content = (line.length > 2) ? line[2 .. $] : ""; + if (seen_text) { + current_obj.text ~= "\n" ~ text_content; + } else { + current_obj.text = text_content; + seen_text = true; + } + } + continue; + } + + /+ ↓ no known shape. Counted by its first character so that a reader + can be told the file held something this one did not expect + +/ + doc.unknown.line[line[0 .. 1].idup]++; + } + + /+ ↓ finalize last object +/ + if (in_object) { + commitObject(doc, current_section, current_obj); + } + + return doc; +} + +private void parseObjectDeclaration(string line, ref OcdaObject obj) { + /+ parse: [N] type :level identifier +/ + auto bracket_close = line.indexOf(']'); + if (bracket_close < 0) return; + try { + obj.ocn = line[1 .. bracket_close].strip.to!int; + } catch (ConvException) { + obj.ocn = 0; + } + string rest = (bracket_close + 1 < line.length) + ? line[bracket_close + 1 .. $].strip : ""; + if (rest.length == 0) return; + + auto parts = rest.split(" "); + if (parts.length == 0) return; + obj.is_a = parts[0]; + + if (obj.is_a == "heading" && parts.length >= 2) { + /+ parse :A, :B, :1, etc. +/ + string lev = parts[1]; + if (lev.length >= 2 && lev[0] == ':') { + obj.heading_level = lev[1 .. $]; + } + /+ optional identifier after level +/ + if (parts.length >= 3) { + obj.identifier = parts[2]; + } + } +} + +/+ ↓ the blocks that are document preamble rather than sections of objects +/ +private bool isPreambleBlock(string name) { + return name == "source" || name == "meta" || name == "make" || name == "doc_has"; +} + +private void parseProperty(string line, ref OcdaObject obj, ref OcdaUnknown unknown) { + /+ parse: .name: value +/ + auto colon_pos = line.indexOf(':'); + if (colon_pos < 1) return; + string name = line[1 .. colon_pos].strip; + string value = (colon_pos + 1 < line.length) + ? line[colon_pos + 1 .. $].strip : ""; + + switch (name) { + case "part": obj.part = value; break; + case "is_of_type": obj.is_of_type = value; break; + case "section": obj.own_section = value; break; + case "identifier": obj.identifier = value; break; + case "segment_html_is": obj.segment_html_is = value; break; + /+ ↓ the epub segments at heading levels 1 to 4 where the html + segments at level 4 alone, so the two are different files and + the abstraction records both +/ + case "segment_epub": obj.segment_epub = value; break; + case "segment_epub_is": obj.segment_epub_is = value; break; + case "table_aligns": obj.table_aligns = value; break; + case "stow_link": obj.stow_link ~= value; break; + case "anchor_tag": obj.anchor_tags ~= value; break; + case "parent": obj.parent_ocn = toIntSafe(value); break; + case "last_descendant": obj.last_descendant_ocn = toIntSafe(value); break; + case "ocn_range": { + auto r = parseIntArray(value); + if (r.length == 2) { obj.ocn_range_first = r[0]; obj.ocn_range_last = r[1]; } + break; + } + case "children": obj.children = parseIntArray(value); break; + case "ancestors": obj.ancestors = parseIntArray(value); break; + case "dummy": obj.dummy_heading = (value == "true"); break; + case "ocn_off": obj.ocn_off = (value == "true"); break; + case "indent": + auto parts = value.split(" "); + if (parts.length >= 1) obj.indent_base = toIntSafe(parts[0]); + if (parts.length >= 2) obj.indent_hang = toIntSafe(parts[1]); + break; + case "bullet": obj.bullet = (value == "true"); break; + case "lang": obj.lang = value; break; + case "has": + foreach (flag; value.split(" ")) { + switch (flag) { + case "links": obj.has_links = true; break; + case "notes_reg": obj.has_notes_reg = true; break; + case "notes_star": obj.has_notes_star = true; break; + case "images": obj.has_images = true; break; + default: break; + } + } + break; + case "segment": obj.segment = value; break; + case "segment_prev": obj.segment_prev = value; break; + case "segment_next": obj.segment_next = value; break; + case "anchor": obj.anchor = value; break; + case "heading_lev_anchor": obj.heading_lev_anchor = value; break; + case "table_cols": obj.table_cols = toIntSafe(value); break; + case "table_widths": obj.table_widths = value; break; + case "table_header": obj.table_header = (value == "true"); break; + case "code_syntax": obj.code_syntax = value; break; + case "code_linenumbers": obj.code_linenumbers = (value == "true"); break; + case "heading_ancestors_text": obj.heading_ancestors_text = value; break; + case "sha256": obj.sha256 = value; break; + case "image": obj.images ~= parseImageRecord(value); break; + /+ ↓ a property this reader has no use for, or has never heard of. + Ignored, which is the conforming behaviour within a major + version, but counted rather than passed over in silence + +/ + default: unknown.property[name]++; break; + } +} + +private OcdaImage parseImageRecord(string value) { + /+ parse: <filename> sha256:<HEX> bytes:<n> [key:value ...] + filename first, remaining fields order-independent; unrecognised + fields are kept rather than discarded +/ + OcdaImage img; + bool first = true; + foreach (field; value.split(" ")) { + if (field.length == 0) continue; + if (first) { + img.name = field; + first = false; + continue; + } + auto colon_pos = field.indexOf(':'); + if (colon_pos < 1) continue; + string key = field[0 .. colon_pos]; + string val = (colon_pos + 1 < field.length) ? field[colon_pos + 1 .. $] : ""; + switch (key) { + case "sha256": img.sha256 = val; break; + case "bytes": img.bytes = toUlongSafe(val); break; + case "px": + auto dims = val.split("x"); + if (dims.length == 2) { + img.width = toIntSafe(dims[0]); + img.height = toIntSafe(dims[1]); + } + break; + case "missing": img.missing = (val == "true"); break; + default: img.fields[key] = val; break; + } + } + return img; +} + +/+ ↓ two values the text form leaves out because they can be worked out. + . + The .ssp emits a property only when it carries information, so + .identifier is absent when it is simply the ocn, and .anchor is absent + when it is the segment name. The .ocda.db, being columns rather than + lines, stores both unconditionally. So a loader that takes the text + form literally gives a *different model* from the one the database + gives, and the difference is not small: in the reference collection + the identifier is omitted on all but a handful of objects. + . + The model is the thing both serialisations describe, so the loader + fills them in. Found by dumping both models and diffing them, which no + amount of comparing section counts would have shown. ++/ +private void deriveOmitted(ref OcdaObject obj) { + import std.conv : to; + if (obj.identifier.length == 0 && obj.ocn != 0) { + obj.identifier = obj.ocn.to!string; + } + if (obj.anchor.length == 0 && obj.segment.length > 0) { + obj.anchor = obj.segment; + } +} + +private void commitObject(ref OcdaDocument doc, string section, ref OcdaObject obj) { + deriveOmitted(obj); + auto sec = doc.sectionByName(section); + if (sec !is null) { + *sec ~= obj; + } +} + +private int toIntSafe(string s) { + try { return s.strip.to!int; } + catch (ConvException) { return 0; } +} + +private ulong toUlongSafe(string s) { + try { return s.strip.to!ulong; } + catch (ConvException) { return 0; } +} + +private int[] parseIntArray(string s) { + int[] result; + foreach (part; s.split(" ")) { + auto v = toIntSafe(part); + result ~= v; + } + return result; +} diff --git a/dlang/src/ocda/model.d b/dlang/src/ocda/model.d new file mode 100644 index 0000000..6b6f40b --- /dev/null +++ b/dlang/src/ocda/model.d @@ -0,0 +1,198 @@ +/+ +- Name: SisuDoc Spine, Doc Reform [a part of] + - Description: documents, structuring, processing, publishing, search + - static content generator + + - Author: Ralph Amissah + [ralph.amissah@gmail.com] + + - Copyright: (C) 2015 (continuously updated, current 2026) Ralph Amissah, All Rights Reserved. + + - License: AGPL 3 or later: + + Spine (SiSU), a framework for document structuring, publishing and + search + + Copyright (C) Ralph Amissah + + This program is free software: you can redistribute it and/or modify it + under the terms of the GNU AFERO General Public License as published by the + Free Software Foundation, either version 3 of the License, or (at your + option) any later version. + + This program is distributed in the hope that it will be useful, but WITHOUT + ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + more details. + + You should have received a copy of the GNU General Public License along with + this program. If not, see [https://www.gnu.org/licenses/]. + + If you have Internet connection, the latest version of the AGPL should be + available at these locations: + [https://www.fsf.org/licensing/licenses/agpl.html] + [https://www.gnu.org/licenses/agpl.html] + + - Spine (by Doc Reform, related to SiSU) uses standard: + - docReform markup syntax + - standard SiSU markup syntax with modified headers and minor modifications + - docReform object numbering + - standard SiSU object citation numbering & system + + - Homepages: + [https://www.sisudoc.org] + [https://www.doc-reform.org] + + - Git + [https://git.sisudoc.org/] + ++/ +module ocda.model; + +struct OcdaImage { + string name; // image filename as referenced by the object + string sha256; // hex-encoded (uppercase) digest of the file + ulong bytes; // file size + int width; // file's own pixel width (not display width) + int height; // file's own pixel height + bool missing; // file absent or unreadable, no sha256/bytes + string[string] fields; // any further/unrecognised key:value fields +} + +struct OcdaObject { + int ocn; + string is_a; // heading, para, toc, table, code, group, block, verse, quote, endnote, glossary, bibliography, bookindex, blurb + string section; // head, toc, body, endnotes, glossary, bibliography, bookindex, blurb + string heading_level; // A, B, C, D, 1, 2, 3, 4 (for headings) + string identifier; // heading: from the declaration line; otherwise .identifier + string part; // frontmatter, body, backmatter + string is_of_type; // para, block, comment: how the text behaves + string own_section; // .section, when it differs from the block it sits in + string segment_html_is;// the html segment this object sits in + string segment_epub; // the epub segment this heading opens + string segment_epub_is;// the epub segment this object sits in + string table_aligns; // l c r per column + string[] stow_link; // links extracted from the object + string[] anchor_tags; // named targets the markup placed on this object + int parent_ocn; + int last_descendant_ocn; + int ocn_range_first;// a container (a poem): the first and last + int ocn_range_last; // ocn of what it holds; 0 otherwise (2.1) + int[] children; // direct child heading OCNs + int[] ancestors; // markedup_ancestors[8] + bool dummy_heading; + bool ocn_off; + int indent_base; + int indent_hang; + bool bullet; + string lang; + bool has_links; + bool has_notes_reg; + bool has_notes_star; + bool has_images; + string segment; + string segment_prev; + string segment_next; + string anchor; + string heading_lev_anchor; // a heading's own named target + int table_cols; + string table_widths; + bool table_header; + string code_syntax; + bool code_linenumbers; + string heading_ancestors_text; + string sha256; + OcdaImage[] images; // repeatable .image records, in order + string text; // content (lines joined with newline) + + /+ ↓ whether this object is rendered at all. + is_of_type "comment" is markup commentary: carried by the + abstraction so that nothing is lost, and not put into output. An + output that renders it leaks the author notes into the document, + which is what spine itself does not do. + +/ + bool renders() const { + return is_of_type != "comment"; + } + + /+ ↓ the section this object belongs to, which is the block it sits in + unless it says otherwise +/ + string sectionOf() const { + return (own_section.length > 0) ? own_section : section; + } +} + +/+ ↓ what the reader met and did not understand. + A reader that silently drops what it does not know cannot say whether + it understood the document, and cannot be used to check a + specification against an implementation: silence and understanding + look alike. So the parser counts what it ignored, and + --report-unknown prints it. ++/ +struct OcdaUnknown { + size_t[string] property; // .name -> times seen + size_t[string] block; // @name -> times seen + size_t[string] line; // first char -> times seen, lines of no known shape + + bool any() const { + return property.length > 0 || block.length > 0 || line.length > 0; + } +} + +struct OcdaDocument { + string format_version; // as declared on line 1, "" when absent + string source_filename; // from the "% Source:" line + string[string] source; // @source: language, languages, digest + OcdaUnknown unknown; // what was met and not understood + string[string] metadata; // key -> value + string[string] make; // key -> value + string[string] doc_has; // key -> value + OcdaObject[] head; + OcdaObject[] toc; + OcdaObject[] body; + OcdaObject[] endnotes; + OcdaObject[] glossary; + OcdaObject[] bibliography; + OcdaObject[] bookindex; + OcdaObject[] blurb; + OcdaObject[] tail; + + /+ ↓ the major part of the declared version, "" when none was declared +/ + string formatVersionMajor() const { + import std.string : indexOf; + if (format_version.length == 0) { return ""; } + auto dot = format_version.indexOf('.'); + return (dot > 0) ? format_version[0 .. dot] : format_version; + } + + string meta(string key) { + if (auto p = key in metadata) return *p; + return ""; + } + + OcdaObject[]* sectionByName(string name) { + switch (name) { + case "head": return &head; + case "toc": return &toc; + case "body": return &body; + case "endnotes": return &endnotes; + case "glossary": return &glossary; + case "bibliography": return &bibliography; + case "bookindex": return &bookindex; + case "blurb": return &blurb; + case "tail": return &tail; + default: return null; + } + } +} + +/+ ↓ the format major version this reader knows. + The .ssp and the .ocda.db carry one version between them, declared in + band: the .ssp on its first line, the database in its schema.version + row. The minor part is raised when a property is added, and a reader of + the newer format still reads the older one, so only the major part is + checked here. Raise this when the reader has been brought to a new + major, not before: the point of the gate is to refuse what has not been + looked at. ++/ +enum ssp_format_major_read = "2"; diff --git a/dlang/src/ocda/sqlite.d b/dlang/src/ocda/sqlite.d new file mode 100644 index 0000000..fc6b2be --- /dev/null +++ b/dlang/src/ocda/sqlite.d @@ -0,0 +1,171 @@ +/+ +- Name: SisuDoc Spine, Doc Reform [a part of] + - Description: documents, structuring, processing, publishing, search + - static content generator + + - Author: Ralph Amissah + [ralph.amissah@gmail.com] + + - Copyright: (C) 2015 (continuously updated, current 2026) Ralph Amissah, All Rights Reserved. + + - License: AGPL 3 or later: + + Spine (SiSU), a framework for document structuring, publishing and + search + + Copyright (C) Ralph Amissah + + This program is free software: you can redistribute it and/or modify it + under the terms of the GNU AFERO General Public License as published by the + Free Software Foundation, either version 3 of the License, or (at your + option) any later version. + + This program is distributed in the hope that it will be useful, but WITHOUT + ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + more details. + + You should have received a copy of the GNU General Public License along with + this program. If not, see [https://www.gnu.org/licenses/]. + + If you have Internet connection, the latest version of the AGPL should be + available at these locations: + [https://www.fsf.org/licensing/licenses/agpl.html] + [https://www.gnu.org/licenses/agpl.html] + + - Spine (by Doc Reform, related to SiSU) uses standard: + - docReform markup syntax + - standard SiSU markup syntax with modified headers and minor modifications + - docReform object numbering + - standard SiSU object citation numbering & system + + - Homepages: + [https://www.sisudoc.org] + [https://www.doc-reform.org] + + - Git + [https://git.sisudoc.org/] + ++/ +module ocda.sqlite; + +/+ ↓ the sqlite3 C API, declared rather than depended upon. + . + A binding from the dub registry would be a dependency to fetch, and + the point of this reader is that it builds in a sandbox with no + network. The dozen functions a reader needs are declared here and the + library is linked directly, which costs a page of extern(C) and saves + a dependency. ++/ + +import std.conv : to; +import std.string : fromStringz, toStringz; + +extern (C) { + struct sqlite3; + struct sqlite3_stmt; + + int sqlite3_open_v2(const(char)* filename, sqlite3** db, int flags, const(char)* vfs); + int sqlite3_close(sqlite3* db); + int sqlite3_prepare_v2(sqlite3* db, const(char)* sql, int nbyte, + sqlite3_stmt** stmt, const(char)** tail); + int sqlite3_step(sqlite3_stmt* stmt); + int sqlite3_finalize(sqlite3_stmt* stmt); + int sqlite3_column_count(sqlite3_stmt* stmt); + const(char)* sqlite3_column_name(sqlite3_stmt* stmt, int col); + const(ubyte)* sqlite3_column_text(sqlite3_stmt* stmt, int col); + int sqlite3_column_int(sqlite3_stmt* stmt, int col); + int sqlite3_column_type(sqlite3_stmt* stmt, int col); + const(char)* sqlite3_errmsg(sqlite3* db); + const(void)* sqlite3_column_blob(sqlite3_stmt* stmt, int col); + int sqlite3_column_bytes(sqlite3_stmt* stmt, int col); +} + +enum SQLITE_OK = 0; +enum SQLITE_ROW = 100; +enum SQLITE_DONE = 101; +enum SQLITE_NULL = 5; +enum SQLITE_OPEN_READONLY = 0x00000001; + +class SqliteError : Exception { + this(string msg) { super(msg); } +} + +/+ ↓ a database, opened read only. + Read only is deliberate: a reader has no business writing, and a + request that opens a database read-write can create an empty one where + a path is mistyped. ++/ +struct Db { + private sqlite3* _handle; + + static Db open(string path) { + sqlite3* handle; + auto rc = sqlite3_open_v2(path.toStringz, &handle, SQLITE_OPEN_READONLY, null); + if (rc != SQLITE_OK) { + string msg = (handle is null) + ? "cannot open " ~ path + : "cannot open " ~ path ~ ": " ~ sqlite3_errmsg(handle).fromStringz.idup; + if (handle !is null) { sqlite3_close(handle); } + throw new SqliteError(msg); + } + Db db; + db._handle = handle; + return db; + } + + void close() { + if (_handle !is null) { sqlite3_close(_handle); _handle = null; } + } + + /+ ↓ every row of a query, each row as its columns in order +/ + string[][] rows(string sql) { + sqlite3_stmt* stmt; + auto rc = sqlite3_prepare_v2(_handle, sql.toStringz, -1, &stmt, null); + if (rc != SQLITE_OK) { + throw new SqliteError(sqlite3_errmsg(_handle).fromStringz.idup ~ " in: " ~ sql); + } + scope(exit) { sqlite3_finalize(stmt); } + string[][] out_rows; + int cols = sqlite3_column_count(stmt); + while (sqlite3_step(stmt) == SQLITE_ROW) { + string[] row; + foreach (c; 0 .. cols) { + row ~= (sqlite3_column_type(stmt, c) == SQLITE_NULL) + ? "" : (cast(const(char)*) sqlite3_column_text(stmt, c)).fromStringz.idup; + } + out_rows ~= row; + } + return out_rows; + } + + /+ ↓ the first column of the first row, "" when there is none +/ + string one(string sql) { + auto r = rows(sql); + return (r.length > 0 && r[0].length > 0) ? r[0][0] : ""; + } + + /+ ↓ the first column of the first row as bytes, for a blob. + Separate from one(), because a blob is not text: an image read as text + stops at the first zero byte, and a png's ninth byte is zero. + +/ + ubyte[] blob(string sql) { + sqlite3_stmt* stmt; + if (sqlite3_prepare_v2(_handle, sql.toStringz, -1, &stmt, null) != SQLITE_OK) { + throw new SqliteError("prepare: " ~ sqlite3_errmsg(_handle).fromStringz.idup + ~ " [" ~ sql ~ "]"); + } + scope(exit) { sqlite3_finalize(stmt); } + if (sqlite3_step(stmt) != SQLITE_ROW) { return []; } + int n = sqlite3_column_bytes(stmt, 0); + if (n <= 0) { return []; } + auto p = cast(const(ubyte)*) sqlite3_column_blob(stmt, 0); + if (p is null) { return []; } + return p[0 .. n].dup; + } + + /+ ↓ does a table or view of this name exist +/ + bool has(string name) { + return one("SELECT count(*) FROM sqlite_master WHERE name = '" ~ name ~ "'") != "0"; + } +} diff --git a/dlang/src/ocda/zip.d b/dlang/src/ocda/zip.d new file mode 100644 index 0000000..1b4041a --- /dev/null +++ b/dlang/src/ocda/zip.d @@ -0,0 +1,164 @@ +/+ +- Name: SisuDoc Spine, Doc Reform [a part of] + - Description: documents, structuring, processing, publishing, search + - static content generator + + - Author: Ralph Amissah + [ralph.amissah@gmail.com] + + - Copyright: (C) 2015 (continuously updated, current 2026) Ralph Amissah, All Rights Reserved. + + - License: AGPL 3 or later: + + Spine (SiSU), a framework for document structuring, publishing and + search + + Copyright (C) Ralph Amissah + + This program is free software: you can redistribute it and/or modify it + under the terms of the GNU AFERO General Public License as published by the + Free Software Foundation, either version 3 of the License, or (at your + option) any later version. + + This program is distributed in the hope that it will be useful, but WITHOUT + ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + more details. + + You should have received a copy of the GNU General Public License along with + this program. If not, see [https://www.gnu.org/licenses/]. + + If you have Internet connection, the latest version of the AGPL should be + available at these locations: + [https://www.fsf.org/licensing/licenses/agpl.html] + [https://www.gnu.org/licenses/agpl.html] + + - Spine (by Doc Reform, related to SiSU) uses standard: + - docReform markup syntax + - standard SiSU markup syntax with modified headers and minor modifications + - docReform object numbering + - standard SiSU object citation numbering & system + + - Homepages: + [https://www.sisudoc.org] + [https://www.doc-reform.org] + + - Git + [https://git.sisudoc.org/] + ++/ +module ocda.zip; + +/+ ↓ enough of the zip format to find one file in a pod and read it. + . + Not a zip library. A pod archive holds a handful of entries, spine + writes them **stored** (uncompressed) so that the outer zstd frame does + the compressing once over the whole thing, and what is wanted from it + here is one named entry. So: read the central directory, find the + entry, return its bytes. + . + Deflate is *refused rather than mishandled*. Phobos can inflate, but a + library that says it read an entry it in fact skipped is worse than one + that says it cannot, and spine's own archives do not need it. ++/ + +import std.algorithm : endsWith; +import std.conv : to; + +class OcdaZipError : Exception { + this(string msg) { super(msg); } +} + +struct ZipEntry { + string name; + uint method; // 0 stored, 8 deflate + uint compressed; + uint uncompressed; + uint local_offset; +} + +/+ ↓ the entries, from the central directory at the end of the file +/ +ZipEntry[] zipEntries(const(ubyte)[] bytes) { + enum uint eocd_sig = 0x06054b50; + enum uint cd_sig = 0x02014b50; + if (bytes.length < 22) { + throw new OcdaZipError("too short to be a zip archive"); + } + /+ ↓ the end of central directory record is last, and may be followed by a + comment of up to 65535 bytes, so it is searched for backwards +/ + ptrdiff_t eocd = -1; + ptrdiff_t lowest = bytes.length > (22 + 65535) ? bytes.length - (22 + 65535) : 0; + for (ptrdiff_t i = bytes.length - 22; i >= lowest; i--) { + if (u32(bytes, i) == eocd_sig) { eocd = i; break; } + } + if (eocd < 0) { + throw new OcdaZipError("no end of central directory record;" + ~ " it does not look like a zip archive"); + } + uint count = u16(bytes, eocd + 10); + uint offset = u32(bytes, eocd + 16); + ZipEntry[] out_; + size_t p = offset; + foreach (_; 0 .. count) { + if (p + 46 > bytes.length || u32(bytes, p) != cd_sig) { + throw new OcdaZipError("central directory entry " + ~ out_.length.to!string ~ " is malformed"); + } + ZipEntry e; + e.method = u16(bytes, p + 10); + e.compressed = u32(bytes, p + 20); + e.uncompressed = u32(bytes, p + 24); + uint name_len = u16(bytes, p + 28); + uint extra_len = u16(bytes, p + 30); + uint cmnt_len = u16(bytes, p + 32); + e.local_offset = u32(bytes, p + 42); + e.name = cast(string) bytes[p + 46 .. p + 46 + name_len].idup; + out_ ~= e; + p += 46 + name_len + extra_len + cmnt_len; + } + return out_; +} + +/+ ↓ one entry's bytes, from its local header +/ +ubyte[] zipRead(const(ubyte)[] bytes, ZipEntry e) { + enum uint local_sig = 0x04034b50; + size_t p = e.local_offset; + if (p + 30 > bytes.length || u32(bytes, p) != local_sig) { + throw new OcdaZipError(e.name ~ ": local header is malformed"); + } + uint name_len = u16(bytes, p + 26); + uint extra_len = u16(bytes, p + 28); + size_t start = p + 30 + name_len + extra_len; + size_t end = start + e.compressed; + if (end > bytes.length) { + throw new OcdaZipError(e.name ~ ": runs past the end of the archive"); + } + if (e.method != 0) { + throw new OcdaZipError(e.name ~ ": compression method " + ~ e.method.to!string ~ "; this reader takes only stored entries," + ~ " which is what spine writes"); + } + return bytes[start .. end].dup; +} + +/+ ↓ the first entry whose name ends this way, or none. + Used to find media/abstraction/<uid>.<lang>.ssp without knowing the uid. ++/ +ZipEntry[] zipFindSuffix(ZipEntry[] entries, string suffix) { + ZipEntry[] out_; + foreach (e; entries) { + if (e.name.endsWith(suffix)) { out_ ~= e; } + } + return out_; +} + +private uint u16(const(ubyte)[] b, size_t i) { + return cast(uint) b[i] | (cast(uint) b[i + 1] << 8); +} + +private uint u32(const(ubyte)[] b, size_t i) { + return cast(uint) b[i] + | (cast(uint) b[i + 1] << 8) + | (cast(uint) b[i + 2] << 16) + | (cast(uint) b[i + 3] << 24); +} diff --git a/dlang/src/ocda/zstd.d b/dlang/src/ocda/zstd.d new file mode 100644 index 0000000..3793f09 --- /dev/null +++ b/dlang/src/ocda/zstd.d @@ -0,0 +1,128 @@ +/+ +- Name: SisuDoc Spine, Doc Reform [a part of] + - Description: documents, structuring, processing, publishing, search + - static content generator + + - Author: Ralph Amissah + [ralph.amissah@gmail.com] + + - Copyright: (C) 2015 (continuously updated, current 2026) Ralph Amissah, All Rights Reserved. + + - License: AGPL 3 or later: + + Spine (SiSU), a framework for document structuring, publishing and + search + + Copyright (C) Ralph Amissah + + This program is free software: you can redistribute it and/or modify it + under the terms of the GNU AFERO General Public License as published by the + Free Software Foundation, either version 3 of the License, or (at your + option) any later version. + + This program is distributed in the hope that it will be useful, but WITHOUT + ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + more details. + + You should have received a copy of the GNU General Public License along with + this program. If not, see [https://www.gnu.org/licenses/]. + + If you have Internet connection, the latest version of the AGPL should be + available at these locations: + [https://www.fsf.org/licensing/licenses/agpl.html] + [https://www.gnu.org/licenses/agpl.html] + + - Spine (by Doc Reform, related to SiSU) uses standard: + - docReform markup syntax + - standard SiSU markup syntax with modified headers and minor modifications + - docReform object numbering + - standard SiSU object citation numbering & system + + - Homepages: + [https://www.sisudoc.org] + [https://www.doc-reform.org] + + - Git + [https://git.sisudoc.org/] + ++/ +module ocda.zstd; + +/+ ↓ libzstd, declared rather than bound. + . + A .sisupod is one zstd frame wrapping a zip archive, so reading one + needs exactly two things from the library: is this a frame, and what is + in it. Both are declared here and the C library is linked, which is the + same arrangement ocda.sqlite uses: no vendored tree, no generated + header, and the prototypes say what is being called. ++/ + +extern (C) @nogc nothrow private { + size_t ZSTD_decompress(void* dst, size_t dstCapacity, + const(void)* src, size_t compressedSize); + ulong ZSTD_getFrameContentSize(const(void)* src, size_t srcSize); + uint ZSTD_isError(size_t code); + const(char)* ZSTD_getErrorName(size_t code); +} + +/+ ↓ the two values ZSTD_getFrameContentSize returns in place of a size +/ +private enum ulong contentSizeUnknown = ulong.max; +private enum ulong contentSizeError = ulong.max - 1; + +/+ ↓ the magic number of a zstd frame, little endian 0xFD2FB528 +/ +private static immutable ubyte[4] zstd_magic = [0x28, 0xB5, 0x2F, 0xFD]; + +class OcdaZstdError : Exception { + this(string msg) { super(msg); } +} + +/+ ↓ does this begin with a zstd frame? + Asked of the bytes rather than of the filename, because what is inside + decides how a thing is opened. A .sisupod is a frame; a .zip published + before spine wrapped them is not, and both are read. ++/ +bool zstdIsFrame(const(ubyte)[] bytes) { + if (bytes.length < zstd_magic.length) { return false; } + foreach (i, b; zstd_magic) { + if (bytes[i] != b) { return false; } + } + return true; +} + +/+ ↓ one frame in, its contents out. + max_size is a refusal rather than a buffer size: a frame declares how + much it expands to, and a frame that claims more than the caller is + willing to hold is not decompressed at all. A reader of an artefact it + did not make wants that limit. ++/ +ubyte[] zstdDecompress(const(ubyte)[] bytes, size_t max_size) { + if (!zstdIsFrame(bytes)) { + throw new OcdaZstdError("not a zstd frame"); + } + ulong declared = ZSTD_getFrameContentSize(bytes.ptr, bytes.length); + if (declared == contentSizeError) { + throw new OcdaZstdError("zstd frame: could not read its content size"); + } + if (declared == contentSizeUnknown) { + throw new OcdaZstdError("zstd frame declares no content size;" + ~ " this reader takes only frames that do"); + } + if (declared > max_size) { + import std.conv : to; + throw new OcdaZstdError("zstd frame expands to " ~ declared.to!string + ~ " bytes, over the " ~ max_size.to!string ~ " byte limit"); + } + auto out_ = new ubyte[](cast(size_t) declared); + size_t got = ZSTD_decompress(out_.ptr, out_.length, bytes.ptr, bytes.length); + if (ZSTD_isError(got)) { + import std.string : fromStringz; + throw new OcdaZstdError("zstd: " + ~ ZSTD_getErrorName(got).fromStringz.idup); + } + if (got != declared) { + throw new OcdaZstdError("zstd frame expanded to a different size" + ~ " than it declared"); + } + return out_; +} |
