summaryrefslogtreecommitdiffhomepage
path: root/gleam/src/ocda/dump.gleam
diff options
context:
space:
mode:
authorRalph Amissah <ralph.amissah@gmail.com>2026-09-26 09:47:07 -0400
committerRalph Amissah <ralph.amissah@gmail.com>2026-09-26 09:47:07 -0400
commitfda48e417435e31a3122bfa7ca563333628aed99 (patch)
treec91f50006de299a9db42247bc21df4f59f9c7a82 /gleam/src/ocda/dump.gleam
format 2.0: abstraction as library, in D and Gleam
The object-centric document abstraction that sisudoc-spine builds from SiSU markup, read as a library: the model, and a loader for each form it is written in - the .ssp (text), the .ocda.db (sqlite), and a pod archive carrying a .ssp (.sisupod, .zip). Begun 2026-09-25; this is the library as it stood at format 2.0. The readers (sisudoc-ssp-reader, sisudoc-ocda-db-reader) each read one form and print. A writer needs neither: it needs the document. So the model and the loaders live here, ocda.model imports nothing, and a program built on the library cannot tell which form it was handed. ocda.load is the one place that decides what a path is, so a new input form is added once and every program gains it. The model and the .ssp loader began as the .ssp reader's code, moved and renamed to the level they belong to; the .ocda.db loader is new. The Gleam library is a deliberate field-for-field mirror of the D one, including where that is not the most idiomatic Gleam: a difference in one language and not the other would show in every comparison and mean nothing. The claim holds only if the loaders agree, so the library asks: ocda-agree <doc>.ssp <doc>.ocda.db [lang] ten fields, per object ocda-dump <file> [lang] the whole model, canonically The dump is an oracle, not an output format, and the D and Gleam dumps are byte identical, so four questions are a diff: D and Gleam, each from the .ssp and from the .ocda.db. Over the markup document samples all four hold, every document-language; a .sisupod gives the same model as the .ssp beside it. None of that held at first, and what the checks found is the record of how the specification became enough: - an object whose first text line is blank lost it (the loader asked whether the text was empty, not whether a line had been seen); seven sample documents open verse that way. Found by agree; fixed here and in the .ssp reader, which carried the same code. - found by the dump: the identifier and the anchor are omitted by the text form when they can be derived and stored by the database always, so the loaders derive them, and the rule went into the specification (derived values); make and doc_has share one table in the database under a key prefix, which the loaders strip; the JSON columns were scanned, not parsed, and Phobos writes "/" as "\/"; and translation.ocn_aligned is a fact about the file, not the document, so neither loader puts it in the model. - in Gleam, a JSON element decoder fell through to a fallback giving "", so a float column read as empty strings: a value that could not be read turned into one that looked read. - found by the writers: anchor_tags, heading_lev_anchor, segment_epub and segment_epub_is were in the specification and read by nothing. ocda.images gives an image's bytes from whichever form the document came from (the database's files table, the archive entry, or beside a .ssp at media/image/), and says beforehand whether it can. The Gleam library reads a .zip pod but not a .sisupod: the zstd frame around it has no implementation on the BEAM that builds sandboxed. It says so, and says zstd -d, rather than calling the file malformed. gleam/ builds with nix-gleam's builder, kept in gleam/nix-gleam/ (MIT) Use: dlang/ nix build ocda-dump, ocda-agree; or a dub path dependency on dlang/ gleam/ nix build ocda-lib dump | agree See README.md, and dlang/README.md and gleam/README.md. (assisted by Claude-Code)
Diffstat (limited to 'gleam/src/ocda/dump.gleam')
-rw-r--r--gleam/src/ocda/dump.gleam247
1 files changed, 247 insertions, 0 deletions
diff --git a/gleam/src/ocda/dump.gleam b/gleam/src/ocda/dump.gleam
new file mode 100644
index 0000000..a75da67
--- /dev/null
+++ b/gleam/src/ocda/dump.gleam
@@ -0,0 +1,247 @@
+//// The model, written out canonically.
+////
+//// Not an output format: an *oracle*. Two libraries in two languages
+//// implement the same model from the same specification, and the cheap way
+//// to ask whether they agree is to have each write the whole loaded
+//// document as text and diff the two. Byte for byte identical to the D
+//// library's `ocda/dump.d`, which is the whole point of it.
+////
+//// The rules, which both implementations follow:
+////
+//// - every line is `<key> <value>`, the value running to end of line,
+//// and only one free-form value per line;
+//// - a field at its default is *omitted*, so the output says what the
+//// document has rather than what the type declares;
+//// - dictionaries come in sorted key order, because neither language's
+//// map has an order worth relying on;
+//// - sections come in the fixed order of the format.
+////
+//// Anything that reads this to reconstruct a document has misunderstood
+//// it. Load the `.ssp` or the `.ocda.db`.
+
+import gleam/dict.{type Dict}
+import gleam/int
+import gleam/list
+import gleam/string
+import ocda/model.{type Document, type Object}
+
+pub fn document(doc: Document) -> String {
+ let lines =
+ []
+ |> kv("version", doc.format_version)
+ |> kv("source_filename", doc.source_filename)
+ |> entries("source", doc.source)
+ |> entries("meta", doc.metadata)
+ |> entries("make", doc.make)
+ |> entries("doc_has", doc.doc_has)
+ |> counts(doc)
+ |> objects(doc)
+ lines |> list.reverse |> join
+}
+
+/// What the loader met and did not understand, sorted.
+///
+/// Separate from the document dump: it describes the loader, not the
+/// document, and the `.ocda.db` loader has nothing to report where the
+/// `.ssp` loader does.
+pub fn unknown(doc: Document) -> String {
+ []
+ |> numbered("property", doc.unknown.property)
+ |> numbered("block", doc.unknown.block)
+ |> numbered("line", doc.unknown.line)
+ |> list.reverse
+ |> join
+}
+
+fn join(lines: List(String)) -> String {
+ case lines {
+ [] -> ""
+ _ -> string.join(lines, "\n") <> "\n"
+ }
+}
+
+/// Accumulated newest-first and reversed once, so that nothing appends to
+/// the tail of a list that grows to a few hundred thousand lines.
+fn kv(acc: List(String), key: String, value: String) -> List(String) {
+ case value {
+ "" -> acc
+ v -> [key <> " " <> v, ..acc]
+ }
+}
+
+fn kn(acc: List(String), key: String, value: Int) -> List(String) {
+ case value {
+ 0 -> acc
+ n -> [key <> " " <> int.to_string(n), ..acc]
+ }
+}
+
+fn kb(acc: List(String), key: String, value: Bool) -> List(String) {
+ case value {
+ False -> acc
+ True -> [key <> " 1", ..acc]
+ }
+}
+
+fn entries(
+ acc: List(String),
+ prefix: String,
+ d: Dict(String, String),
+) -> List(String) {
+ d
+ |> dict.to_list
+ |> list.sort(fn(a, b) { string.compare(a.0, b.0) })
+ |> list.fold(acc, fn(lines, pair) {
+ [prefix <> " " <> pair.0 <> " " <> pair.1, ..lines]
+ })
+}
+
+fn numbered(
+ acc: List(String),
+ prefix: String,
+ d: Dict(String, Int),
+) -> List(String) {
+ d
+ |> dict.to_list
+ |> list.sort(fn(a, b) { string.compare(a.0, b.0) })
+ |> list.fold(acc, fn(lines, pair) {
+ [prefix <> " " <> pair.0 <> " " <> int.to_string(pair.1), ..lines]
+ })
+}
+
+fn counts(acc: List(String), doc: Document) -> List(String) {
+ list.fold(model.section_names, acc, fn(lines, name) {
+ case model.section(doc, name) {
+ [] -> lines
+ objects ->
+ [
+ "section " <> name <> " " <> int.to_string(list.length(objects)),
+ ..lines
+ ]
+ }
+ })
+}
+
+fn objects(acc: List(String), doc: Document) -> List(String) {
+ list.fold(model.section_names, acc, fn(lines, name) {
+ model.section(doc, name)
+ |> list.index_fold(lines, fn(l, obj, i) {
+ object(l, name, i, obj)
+ })
+ })
+}
+
+fn object(
+ acc: List(String),
+ section: String,
+ index: Int,
+ obj: Object,
+) -> List(String) {
+ let acc = ["object " <> section <> " " <> int.to_string(index), ..acc]
+ acc
+ |> kn("ocn", obj.ocn)
+ |> kv("is_a", obj.is_a)
+ |> kv("own_section", obj.own_section)
+ |> kv("level", obj.heading_level)
+ |> kv("identifier", obj.identifier)
+ |> kv("part", obj.part)
+ |> kv("is_of_type", obj.is_of_type)
+ |> kv("segment_html_is", obj.segment_html_is)
+ |> kv("segment_epub", obj.segment_epub)
+ |> kv("segment_epub_is", obj.segment_epub_is)
+ |> kn("parent", obj.parent_ocn)
+ |> kn("last_descendant", obj.last_descendant_ocn)
+ |> kv("children", ints(obj.children))
+ |> kv("ancestors", ints(obj.ancestors))
+ |> kb("dummy", obj.dummy_heading)
+ |> kb("ocn_off", obj.ocn_off)
+ |> kn("indent_base", obj.indent_base)
+ |> kn("indent_hang", obj.indent_hang)
+ |> kb("bullet", obj.bullet)
+ |> kv("lang", obj.lang)
+ |> kb("has_links", obj.has_links)
+ |> kb("has_notes_reg", obj.has_notes_reg)
+ |> kb("has_notes_star", obj.has_notes_star)
+ |> kb("has_images", obj.has_images)
+ |> kv("segment", obj.segment)
+ |> kv("segment_prev", obj.segment_prev)
+ |> kv("segment_next", obj.segment_next)
+ |> kv("anchor", obj.anchor)
+ |> kv("heading_lev_anchor", obj.heading_lev_anchor)
+ |> kn("table_cols", obj.table_cols)
+ |> kv("table_widths", obj.table_widths)
+ |> kv("table_aligns", obj.table_aligns)
+ |> kb("table_header", obj.table_header)
+ |> kv("code_syntax", obj.code_syntax)
+ |> kb("code_linenumbers", obj.code_linenumbers)
+ |> kv("heading_ancestors_text", obj.heading_ancestors_text)
+ |> kv("sha256", obj.sha256)
+ |> links(obj)
+ |> anchor_tags(obj)
+ |> images(obj)
+ |> text(obj)
+}
+
+fn links(acc: List(String), obj: Object) -> List(String) {
+ list.fold(obj.stow_link, acc, fn(lines, url) { kv(lines, "stow_link", url) })
+}
+
+fn anchor_tags(acc: List(String), obj: Object) -> List(String) {
+ list.fold(obj.anchor_tags, acc, fn(lines, a) { kv(lines, "anchor_tag", a) })
+}
+
+fn images(acc: List(String), obj: Object) -> List(String) {
+ list.fold(obj.images, acc, fn(lines, img) {
+ // the name last, because a filename may hold a space where none of the
+ // other fields can
+ let lines =
+ kv(
+ lines,
+ "image",
+ img.sha256
+ <> " "
+ <> int.to_string(img.bytes)
+ <> " "
+ <> int.to_string(img.width)
+ <> "x"
+ <> int.to_string(img.height)
+ <> " "
+ <> case img.missing {
+ True -> "1"
+ False -> "0"
+ }
+ <> " "
+ <> img.name,
+ )
+ img.fields
+ |> dict.to_list
+ |> list.sort(fn(a, b) { string.compare(a.0, b.0) })
+ |> list.fold(lines, fn(l, pair) {
+ kv(l, "image_field", pair.0 <> " " <> pair.1)
+ })
+ })
+}
+
+/// Text, when there is any.
+///
+/// An object whose only text line is blank holds "" here, indistinguishable
+/// from no text at all, because the model joins the lines into one string.
+/// Both libraries lose the same distinction in the same place, which is
+/// what makes them comparable; it is recorded rather than hidden.
+fn text(acc: List(String), obj: Object) -> List(String) {
+ case obj.text {
+ "" -> acc
+ t -> {
+ let lines = string.split(t, on: "\n")
+ let acc = kv(acc, "text", int.to_string(list.length(lines)))
+ list.fold(lines, acc, fn(l, line) { ["| " <> line, ..l] })
+ }
+ }
+}
+
+fn ints(xs: List(Int)) -> String {
+ case xs {
+ [] -> ""
+ _ -> xs |> list.map(int.to_string) |> string.join(",")
+ }
+}