diff options
| author | Ralph Amissah <ralph.amissah@gmail.com> | 2026-09-26 09:47:07 -0400 |
|---|---|---|
| committer | Ralph Amissah <ralph.amissah@gmail.com> | 2026-09-26 09:47:07 -0400 |
| commit | fda48e417435e31a3122bfa7ca563333628aed99 (patch) | |
| tree | c91f50006de299a9db42247bc21df4f59f9c7a82 /gleam/src/ocda/load_ssp.gleam | |
format 2.0: abstraction as library, in D and Gleam
The object-centric document abstraction that sisudoc-spine builds from
SiSU markup, read as a library: the model, and a loader for each form
it is written in - the .ssp (text), the .ocda.db (sqlite), and a pod
archive carrying a .ssp (.sisupod, .zip). Begun 2026-09-25; this is
the library as it stood at format 2.0.
The readers (sisudoc-ssp-reader, sisudoc-ocda-db-reader) each read one
form and print. A writer needs neither: it needs the document. So the
model and the loaders live here, ocda.model imports nothing, and a
program built on the library cannot tell which form it was handed.
ocda.load is the one place that decides what a path is, so a new
input form is added once and every program gains it. The model and
the .ssp loader began as the .ssp reader's code, moved and renamed to
the level they belong to; the .ocda.db loader is new.
The Gleam library is a deliberate field-for-field mirror of the D one,
including where that is not the most idiomatic Gleam: a difference in
one language and not the other would show in every comparison and
mean nothing.
The claim holds only if the loaders agree, so the library asks:
ocda-agree <doc>.ssp <doc>.ocda.db [lang] ten fields, per object
ocda-dump <file> [lang] the whole model,
canonically
The dump is an oracle, not an output format, and the D and Gleam
dumps are byte identical, so four questions are a diff: D and Gleam,
each from the .ssp and from the .ocda.db. Over the markup document
samples all four hold, every document-language; a .sisupod gives the
same model as the .ssp beside it.
None of that held at first, and what the checks found is the record
of how the specification became enough:
- an object whose first text line is blank lost it (the loader asked
whether the text was empty, not whether a line had been seen);
seven sample documents open verse that way. Found by agree; fixed
here and in the .ssp reader, which carried the same code.
- found by the dump: the identifier and the anchor are omitted by the
text form when they can be derived and stored by the database
always, so the loaders derive them, and the rule went into the
specification (derived values); make and doc_has share one table in
the database under a key prefix, which the loaders strip; the JSON
columns were scanned, not parsed, and Phobos writes "/" as "\/";
and translation.ocn_aligned is a fact about the file, not the
document, so neither loader puts it in the model.
- in Gleam, a JSON element decoder fell through to a fallback giving
"", so a float column read as empty strings: a value that could not
be read turned into one that looked read.
- found by the writers: anchor_tags, heading_lev_anchor, segment_epub
and segment_epub_is were in the specification and read by nothing.
ocda.images gives an image's bytes from whichever form the document
came from (the database's files table, the archive entry, or beside
a .ssp at media/image/), and says beforehand whether it can.
The Gleam library reads a .zip pod but not a .sisupod: the zstd frame
around it has no implementation on the BEAM that builds sandboxed. It
says so, and says zstd -d, rather than calling the file malformed.
gleam/ builds with nix-gleam's builder, kept in gleam/nix-gleam/ (MIT)
Use:
dlang/ nix build ocda-dump, ocda-agree; or a dub path
dependency on dlang/
gleam/ nix build ocda-lib dump | agree
See README.md, and dlang/README.md and gleam/README.md.
(assisted by Claude-Code)
Diffstat (limited to 'gleam/src/ocda/load_ssp.gleam')
| -rw-r--r-- | gleam/src/ocda/load_ssp.gleam | 478 |
1 files changed, 478 insertions, 0 deletions
diff --git a/gleam/src/ocda/load_ssp.gleam b/gleam/src/ocda/load_ssp.gleam new file mode 100644 index 0000000..b48e1c5 --- /dev/null +++ b/gleam/src/ocda/load_ssp.gleam @@ -0,0 +1,478 @@ +//// Loading a `.ssp` into the model. +//// +//// The format is line oriented and every line is one of eight shapes told +//// apart by its first character, so the loader is a fold over the lines +//// carrying the state the shapes need. +//// +//// Written against `ssp-format-specification.md`. Where it and the D +//// library differ the D library is the bug, or this is: they are meant to +//// give the same model, and `ocda-dump` says whether they do. + +import gleam/dict.{type Dict} +import gleam/int +import gleam/list +import gleam/option +import gleam/result +import gleam/string +import ocda/model.{type Document, type Image, type Object, Image, Object} +import simplifile + +pub fn load_file(path: String) -> Result(Document, String) { + case simplifile.read(path) { + Ok(content) -> Ok(parse(content)) + Error(e) -> Error(path <> ": " <> simplifile.describe_error(e)) + } +} + +type State { + State( + document: Document, + /// the block currently open, "" when none + block: String, + /// the section currently open, "" when in the preamble + section: String, + /// objects of the current section, newest first + objects: List(Object), + /// the object being read, if any + current: Result(Object, Nil), + /// text lines of the object being read, newest first + text: List(String), + /// whether a text line has been seen for the object being read. + /// Not "is the text non-empty": an object whose first text line is + /// blank, which the reference collection has seven of, would then lose + /// it. + seen_text: Bool, + ) +} + +pub fn parse(content: String) -> Document { + content + |> string.split(on: "\n") + |> list.fold(from: initial(), with: line) + |> finish +} + +fn initial() -> State { + State( + document: model.new_document(), + block: "", + section: "", + objects: [], + current: Error(Nil), + text: [], + seen_text: False, + ) +} + +fn line(state: State, raw: String) -> State { + case raw { + "% SiSU Document Abstraction v" <> version -> + case state.document.format_version { + "" -> + State( + ..state, + document: model.Document( + ..state.document, + format_version: string.trim(version), + ), + ) + _ -> state + } + "% Source: " <> filename -> + case state.document.source_filename { + "" -> + State( + ..state, + document: model.Document( + ..state.document, + source_filename: string.trim(filename), + ), + ) + _ -> state + } + "%" <> _ -> state + "@" <> rest -> open_block(state, rest) + "}" <> _ -> close_block(state) + "[" <> rest -> declaration(state, rest) + "." <> rest -> property(state, rest) + "|" <> rest -> text_line(state, rest) + _ -> + case string.trim(raw) { + // a blank line ends the object being read + "" -> commit(state) + _ -> + case model.is_preamble_block(state.block) { + True -> preamble_pair(state, raw) + False -> unknown_line(state, raw) + } + } + } +} + +fn open_block(state: State, rest: String) -> State { + let name = case string.split_once(rest, "{") { + Ok(#(name, _)) -> string.trim(name) + Error(_) -> string.trim(rest) + } + case model.is_preamble_block(name) { + True -> State(..flush(state), block: name) + False -> { + let state = flush(state) + let document = case model.is_section(name) { + True -> state.document + // a block that is neither preamble nor one of the nine sections: + // counted, so that a reader can say the file held something it did + // not expect + False -> + model.Document( + ..state.document, + unknown: model.Unknown( + ..state.document.unknown, + block: bump(state.document.unknown.block, name), + ), + ) + } + State(..state, block: name, section: name, document: document) + } + } +} + +fn close_block(state: State) -> State { + State(..flush(state), block: "") +} + +/// Commit the object being read, then put the section's objects into the +/// document and close the section. +fn flush(state: State) -> State { + let state = commit(state) + case state.section, state.objects { + "", _ -> state + _, [] -> State(..state, section: "", objects: []) + name, objects -> + State( + ..state, + section: "", + objects: [], + document: model.Document( + ..state.document, + sections: add_section( + state.document.sections, + name, + list.reverse(objects), + ), + ), + ) + } +} + +fn preamble_pair(state: State, raw: String) -> State { + case string.split_once(string.trim_start(raw), ":") { + Error(_) -> state + Ok(#(key, value)) -> { + let key = string.trim(key) + let value = string.trim(value) + let document = state.document + let document = case state.block { + "meta" -> + model.Document( + ..document, + metadata: dict.insert(document.metadata, key, value), + ) + "make" -> + model.Document(..document, make: dict.insert(document.make, key, value)) + "source" -> + model.Document( + ..document, + source: dict.insert(document.source, key, value), + ) + _ -> + model.Document( + ..document, + doc_has: dict.insert(document.doc_has, key, value), + ) + } + State(..state, document: document) + } + } +} + +/// `[<ocn>] <kind>`, and for a heading `[<ocn>] heading :<level> +/// [<identifier>]`. +fn declaration(state: State, rest: String) -> State { + let state = commit(state) + case string.split_once(rest, "]") { + Error(_) -> state + Ok(#(ocn_text, tail)) -> { + let ocn = parse_int(string.trim(ocn_text)) + let object = + Object(..model.new_object(), ocn: ocn, section: state.section) + let object = case string.split(string.trim(tail), on: " ") { + ["heading", level, ..identifier] -> + Object( + ..object, + is_a: "heading", + heading_level: string.drop_start(level, 1), + identifier: string.join(identifier, " "), + ) + [""] -> object + [kind, ..] -> Object(..object, is_a: kind) + [] -> object + } + State(..state, current: Ok(object), text: [], seen_text: False) + } + } +} + +fn property(state: State, rest: String) -> State { + case state.current, string.split_once(rest, ":") { + Ok(object), Ok(#(name, raw_value)) -> { + let name = string.trim(name) + let value = string.trim(raw_value) + let #(object, known) = apply_property(object, name, value) + let document = case known { + True -> state.document + False -> + model.Document( + ..state.document, + unknown: model.Unknown( + ..state.document.unknown, + property: bump(state.document.unknown.property, name), + ), + ) + } + State(..state, current: Ok(object), document: document) + } + _, _ -> state + } +} + +/// The properties this library reads, and what each sets. +/// +/// A property it has no use for is counted rather than passed over: +/// ignoring one is conforming within a major version, doing so in silence +/// is what stops a reader being a check on the specification. +fn apply_property( + object: Object, + name: String, + value: String, +) -> #(Object, Bool) { + case name { + "part" -> #(Object(..object, part: value), True) + "is_of_type" -> #(Object(..object, is_of_type: value), True) + "section" -> #(Object(..object, own_section: value), True) + "identifier" -> #(Object(..object, identifier: value), True) + "segment_html_is" -> #(Object(..object, segment_html_is: value), True) + "segment_epub" -> #(Object(..object, segment_epub: value), True) + "segment_epub_is" -> #(Object(..object, segment_epub_is: value), True) + "table_aligns" -> #(Object(..object, table_aligns: value), True) + "stow_link" -> #( + Object(..object, stow_link: list.append(object.stow_link, [value])), + True, + ) + "anchor_tag" -> #( + Object(..object, anchor_tags: list.append(object.anchor_tags, [value])), + True, + ) + "parent" -> #(Object(..object, parent_ocn: parse_int(value)), True) + "last_descendant" -> #( + Object(..object, last_descendant_ocn: parse_int(value)), + True, + ) + "children" -> #(Object(..object, children: numbers(value)), True) + "ancestors" -> #(Object(..object, ancestors: numbers(value)), True) + "dummy" -> #(Object(..object, dummy_heading: value == "true"), True) + "ocn_off" -> #(Object(..object, ocn_off: value == "true"), True) + "indent" -> { + let parts = string.split(value, on: " ") + let base = case parts { + [first, ..] -> parse_int(first) + [] -> 0 + } + let hang = case parts { + [_, second, ..] -> parse_int(second) + _ -> 0 + } + #(Object(..object, indent_base: base, indent_hang: hang), True) + } + "bullet" -> #(Object(..object, bullet: value == "true"), True) + "lang" -> #(Object(..object, lang: value), True) + "has" -> #(has_flags(object, value), True) + "segment" -> #(Object(..object, segment: value), True) + "segment_prev" -> #(Object(..object, segment_prev: value), True) + "segment_next" -> #(Object(..object, segment_next: value), True) + "anchor" -> #(Object(..object, anchor: value), True) + "heading_lev_anchor" -> #( + Object(..object, heading_lev_anchor: value), + True, + ) + "table_cols" -> #(Object(..object, table_cols: parse_int(value)), True) + "table_widths" -> #(Object(..object, table_widths: value), True) + "table_header" -> #(Object(..object, table_header: value == "true"), True) + "code_syntax" -> #(Object(..object, code_syntax: value), True) + "code_linenumbers" -> #( + Object(..object, code_linenumbers: value == "true"), + True, + ) + "heading_ancestors_text" -> #( + Object(..object, heading_ancestors_text: value), + True, + ) + "sha256" -> #(Object(..object, sha256: value), True) + "image" -> #( + Object(..object, images: list.append(object.images, [image(value)])), + True, + ) + _ -> #(object, False) + } +} + +fn has_flags(object: Object, value: String) -> Object { + value + |> string.split(on: " ") + |> list.fold(from: object, with: fn(obj, flag) { + case flag { + "links" -> Object(..obj, has_links: True) + "notes_reg" -> Object(..obj, has_notes_reg: True) + "notes_star" -> Object(..obj, has_notes_star: True) + "images" -> Object(..obj, has_images: True) + _ -> obj + } + }) +} + +/// `<filename> sha256:<HEX> bytes:<n> [px:<w>x<h>] [missing:true] [...]` +/// +/// The filename comes first, the rest are order-independent, and a field +/// this library does not recognise is kept rather than discarded. +fn image(value: String) -> Image { + case string.split(value, on: " ") { + [] -> model.new_image() + [name, ..fields] -> + list.fold(fields, model.new_image(), fn(img, field) { + case string.split_once(field, ":") { + Error(_) -> img + Ok(#(key, val)) -> + case key { + "sha256" -> Image(..img, sha256: val) + "bytes" -> Image(..img, bytes: parse_int(val)) + "px" -> + case string.split_once(val, "x") { + Ok(#(w, h)) -> + Image(..img, width: parse_int(w), height: parse_int(h)) + Error(_) -> img + } + "missing" -> Image(..img, missing: val == "true") + _ -> Image(..img, fields: dict.insert(img.fields, key, val)) + } + } + }) + |> fn(img) { Image(..img, name: name) } + } +} + +/// `| <text>`, and `| ` alone for a blank line. Never a bare `|`: the +/// emitter writes the space. +fn text_line(state: State, rest: String) -> State { + case state.current { + Error(_) -> state + Ok(_) -> { + let content = string.drop_start(rest, 1) + State(..state, text: [content, ..state.text], seen_text: True) + } + } +} + +fn unknown_line(state: State, raw: String) -> State { + let first = string.slice(raw, 0, 1) + State( + ..state, + document: model.Document( + ..state.document, + unknown: model.Unknown( + ..state.document.unknown, + line: bump(state.document.unknown.line, first), + ), + ), + ) +} + +fn commit(state: State) -> State { + case state.current { + Error(_) -> state + Ok(object) -> { + let text = case state.seen_text { + True -> string.join(list.reverse(state.text), "\n") + False -> "" + } + let object = derive_omitted(Object(..object, text: text)) + let objects = case model.is_section(state.section) { + True -> [object, ..state.objects] + False -> state.objects + } + State( + ..state, + current: Error(Nil), + text: [], + seen_text: False, + objects: objects, + ) + } + } +} + +/// Two values the text form leaves out because they can be worked out. +/// +/// `.identifier` is absent when it is simply the ocn, `.anchor` when it is +/// the segment name. The `.ocda.db`, being columns rather than lines, +/// stores both unconditionally. So a loader that takes the text form +/// literally gives a different model from the one the database gives, and +/// over one document that is 1,314 identifiers and 69 anchors. +fn derive_omitted(object: Object) -> Object { + let object = case object.identifier, object.ocn { + "", 0 -> object + "", ocn -> Object(..object, identifier: int.to_string(ocn)) + _, _ -> object + } + case object.anchor, object.segment { + "", "" -> object + "", segment -> Object(..object, anchor: segment) + _, _ -> object + } +} + +fn finish(state: State) -> Document { + flush(state).document +} + +fn add_section( + sections: Dict(String, List(Object)), + name: String, + objects: List(Object), +) -> Dict(String, List(Object)) { + dict.upsert(sections, name, fn(existing) { + case existing { + option.Some(previous) -> list.append(previous, objects) + option.None -> objects + } + }) +} + +fn bump(counts: Dict(String, Int), key: String) -> Dict(String, Int) { + dict.upsert(counts, key, fn(existing) { + case existing { + option.Some(n) -> n + 1 + option.None -> 1 + } + }) +} + +fn parse_int(value: String) -> Int { + value |> string.trim |> int.parse |> result.unwrap(0) +} + +fn numbers(value: String) -> List(Int) { + value + |> string.split(on: " ") + |> list.filter(fn(s) { s != "" }) + |> list.map(parse_int) +} |
