path: root/README.md

# sisudoc-ocda-writer

Sample **writers** over the object-centric document abstraction: functions
from the model to bytes.

The repository is named for its role, in the singular, as its siblings are
(`sisudoc-ssp-reader`, `sisudoc-ocda-lib`): one writer, `ocda-write`, in each
language, taking the format as an option. The plural is in the language
directories and in the formats, not in the name.

**One writer set per language, in a directory named for the language.**

Five formats: `--text`, `--md`, `--html`, `--epub`, `--typst`.

|                    |         |
|--------------------|---------|
| [`dlang/`](dlang/) | D       |
| [`gleam/`](gleam/) | Gleam   |
| `ruby/`            | to come |

## What these are, and what they are not

They are **samples**, and they are **checked**. Spine is the output
pipeline; nothing here replaces it. What these demonstrate is that the
abstraction is sufficient: given a `.ssp`, a `.ocda.db` or a `.sisupod` and
nothing else, a program of a few hundred lines makes a complete document in
that format, in the spirit of spine's own.

```
  .ssp      ─┐
             ├─> sisudoc-ocda-lib ─> the model ─> a writer ─> bytes
  .ocda.db  ─┘
```

**A writer is never told which serialisation it was given.** That is the
claim, and every writer here is run against every form of every document to
check it: the outputs must be identical from every form, and the D and Gleam
writers must produce the same bytes. They are not held to spine's bytes.
Each format has its own judge (below), and where the output differs from
spine's it does so to read better, and the code says why at that point.

## The oracle comes first

A writer with no oracle drifts, and "looks about right" is not a check. So
each writer is written **against a thing that can say no**:

| writer    | oracle                                                                                                                               |
|-----------|--------------------------------------------------------------------------------------------------------------------------------------|
| `--text`  | the words of the html writer's objects, as a multiset; every object number once; **D == Gleam byte for byte**. *Not* spine's bytes any more |
| `--md`    | it is CommonMark; every `(#x)` resolves to an `<a id=x>`, no id twice; the words equal the html writer's objects; every OCN anchored        |
| `--html`  | it parses; every internal link resolves; the **objects' text** equals spine's scroll html                                            |
| `--epub`  | **epubcheck**, which read the specification so that the writer need not be trusted to have; and the same bytes from every input form |
| `--typst` | **typst itself**: a link to a label that does not exist is a compile error, so a silent compile means every internal link resolves   |

and from every input form the library accepts, not only the two it started
with: a document renders identically from its `.ssp`, its `.ocda.db` and its
`.sisupod`.

The checks are run over the markup document samples
([sisudoc-spine-samples](https://git.sisudoc.org/sisudoc-spine-samples)),
built by spine with `--show-abstraction --ocda-db --pod2`, which writes each
document's `.ssp`, its `.ocda.db` and its pod. **As at 2026-09-26**, over
every document-language of them:

`--text`, which is **no longer spine's bytes**, by decision (2026-09-27):

| | |
|---|---|
| the same bytes from every input form | **all** |
| **D == Gleam, byte for byte** | **all** |
| the words == the html writer's objects, as a multiset | **all** |
| every object number once, and no other | **all** |
| lines differing from `spine --text` | 131,474, **reported and not required** |

Byte identity was the first oracle and earned its place while there was
nothing else to check against; holding to it meant reproducing a dozen things
no one would choose, and the code said so a dozen times. What replaced it is
stricter where it matters. What changed in the output, and why each is better
rather than merely different, is in the head of `write/text`; the short list is
that a note is written once instead of twice, the author's markup comments and
the dummy headings are not written, an object with no text of its own writes
nothing (a `poem`, a container of its verses: before format 2.1 it shared its
number with its last verse, since 2.1 it has none), an image keeps
its caption, a table column is padded to its *display* width, and the table of
contents indents by ranked level rather than by spine's `indent_hang`.

`--md`, which is new (2026-09-27):

| | |
|---|---|
| the same bytes from every input form | **all** |
| **D == Gleam, byte for byte** | **all** |
| every `(#x)` resolves to an `<a id=x>`, and no id is defined twice | **all** |
| the words == the html writer's objects | **all** |
| every object number anchored, and no other | **all** |

The third is the markdown one: markdown has no anchor syntax, so the anchors
are the two pieces of raw html the writer emits, and a mistake in them is
invisible until a reader clicks.

**`--md` takes one option the others do not**, `--image-prefix <path>`,
defaulting to `image`. A `.md` names an image by path, and the path depends on
where the `.md` is put, which is the caller's business and not the model's. The
writer emits a private-use marker for the directory and substitutes it once, at
the end, so nothing carries the constant through the call graph. spine, whose
markdown goes to `<lang>/markdown/` and shares the images with its html, passes
`../../image`; and **spine's `.md` is byte identical to what this produces with
that prefix, on every document-language.**

`--epub`:

|                                        |                    |
|----------------------------------------|--------------------|
| epubcheck clean                        | **all**       |
| from a `.ocda.db` == from a `.sisupod` | **every pair** |
| from a `.ssp` == from a `.ocda.db`     | **all**       |
| **D == Gleam, byte for byte**          | **all**       |

The third is the interesting one: a `.ssp` carries no images and a `.ocda.db`
and a `.sisupod` do, yet the epub is the same either way. It is the same
because the writer asks the library whether the source *can* give images
before it emits one, and a `.ssp` inside a pod can: the images are beside it,
at `media/image/`. So the three input forms are three ways to the same
publication rather than three publications.

(One pair fewer than there are documents, because spine names a `.sisupod`
after the pod directory and a `.ocda.db` after the document uid, and for one
document those differ, so it has no pair to compare.)

`--html`:

|                                                                    |              |
|--------------------------------------------------------------------|--------------|
| parses (`xmllint --html`)                                          | **all** |
| every `href="#x"` resolves, no id twice, no heading outside h1..h6 | **all** |
| the objects' text == spine's scroll html                           | **all** |
| **D == Gleam, byte for byte**                                      | **all** |

and so the two writers agree with each other, in two languages, without
either having been written from the other. That is the agreement that says
the model is enough.

`--typst`:

|                                                    |              |
|----------------------------------------------------|--------------|
| `typst compile`, silently                          | **all** |
| from a `.ssp` == from a `.ocda.db`                 | **all** |
| the `.typ`'s text == the html writer's object text | **all** |
| every OCN on a page of the pdf                     | **all** |
| the same pdf twice, byte for byte                  | **all** |
| **D == Gleam, byte for byte**                      | **all** |

4,870 pages typeset. The first row is the strong one: **a link to a label
that does not exist is a typst compile error**, so a silent compile means
every internal link in the document resolves - object to note, note back to
object, table of contents, book index, every one. In html that took a checker
of its own.

The second is the same claim the epub makes, from the other side: a `.typ`
*names* its images rather than carrying them, so the two serialisations must
give the same bytes even though only one of them holds the image files.

The third is a **multiset** comparison and not a sequence one, because the
notes are in different places in the two outputs - at the point of reference
here, gathered at the end there - and what must agree is that the same words
are present the same number of times. Its two allowances are computed from
the `.ssp` object by object rather than allowed by pattern: what the `.typ` is
expected to lack (the endnotes section's heading and OCN, the table of
contents line that pointed at it, and each endnote's leading `N.`, which
typst generates from the note's own number) and what it is expected to *have*
that the html has not (see the note-84 paragraph below).

One allowance, with its reason in the sweep script rather than here: note 84
of `the_public_domain.james_boyle` is referenced from inside a verse and is
not gathered into the endnotes section, so nothing carries the anchor
`note_84`. **Spine's own html has the same dangling link.** It is an
abstraction-stage defect and affects every output.

And it affects them *unequally*, which the typst writer showed. An output
that gathers the notes at the end loses that note altogether, because the
reference has nowhere to go; a footnote at the point of reference never needed
the gathered copy, so the typst pdf has all 385 words of it. The sweep
computes that as an allowance from the `.ssp` rather than waving it through,
and the *direction* of the allowance is the finding.

## Running it

```sh
cd dlang
nix build
./result/bin/ocda-write --text <doc>.ssp
./result/bin/ocda-write --text <doc>.ocda.db en
./result/bin/ocda-write --typst <doc>.ocda.db en --out <dir>/<doc>.typ
typst compile --root <dir> <dir>/<doc>.typ <dir>/<doc>.pdf
```

`--typst` and `--epub` take `--out` because they write beside themselves:
the epub is a zip, and a `.typ` names its images by path, so `--out` also
writes the `image/` directory the `.typ` refers to.

Output goes to stdout, so the check that a writer cannot tell the forms
apart is a comparison:

```sh
./result/bin/ocda-write --text <doc>.ssp > a
./result/bin/ocda-write --text <doc>.ocda.db en > b
cmp a b
```

### the library

Each flake takes `sisudoc-ocda-lib` from
`git://git.sisudoc.org/tools/sisudoc-ocda-lib`, and `flake.lock` pins the
commit the writer was built and checked against. To build against a working
copy of the library instead:

```sh
nix build --override-input ocda-lib path:../../sisudoc-ocda-lib
```

## The pipeline

`pipeline/` holds fixed shell scripts that build the library (whose
`ocda-read` is the reader) and the writer in both languages, run all of them
over a directory of abstraction artefacts, and put the output where it can
be read.

```sh
ARTEFACTS=<dir of .ssp and .ocda.db> OUT=<dir> sh pipeline/run.sh
sh pipeline/one.sh md-g <doc>.ocda.db en     # one tool, one document
```

They are not the checking - the sweeps above are - they are the run to do to
*see* all of it. `pipeline/README.md` is written as documentation and answers
"what do I type?".

## Byte identity was a means, and `--text` has outgrown it

**Byte identity with spine is a means, not the goal**, and `--text` no longer
uses it. It was the first oracle and it earned its place while there was
nothing else to check against; holding to it meant reproducing a dozen things
no one would choose:

- the table layout stripped trailing whitespace from the **whole accumulated
  table** after every cell, not once per row, so a row ending in an empty
  cell lost the newline before it;
- a column's width was the widest cell **in bytes** and the padding was
  counted **in codepoints**, so a cell holding `©` was padded one place
  further than one holding `c`;
- a table of contents entry's indent was `""`, `":"`, `"∴"` for the first
  three levels and two spaces per level after that;
- and every note's text was in the file twice.

The four checks that replaced it are in "The oracle comes first" above, and
they are stricter where it matters. The divergence from `spine --text` is
**reported** by the sweep - 131,474 lines - so that it is visible without
anything depending on it.

**`--html` never used it either.** Html is where spine's output carries things
that are wrong rather than merely unusual, and a sample writer that copied them
would be teaching them. So the html writer is held to a different standard,
and a stricter one where it counts: it parses, every internal link
resolves, no id appears twice, and the objects' visible text equals spine's
exactly. What it does differently, each with the reason in the code:

| | |
|---|---|
| headings are `h1`..`h6` | spine writes `<h0>` for a level A heading, which is not an html element; a depth beyond six takes `h6` with `aria-level` |
| one charset declaration | spine writes `Content-Type: text/plain` for an html document, then a second `Content-Type` saying `text/html` |
| no placeholder metadata | spine writes `dc.publisher="Publisher"` and `dc.date.issued="year"` literally where it has no value |
| `span`, not `label`, for the object number | `<label>` is for form controls |
| `section` with a dpub-aria role | spine's roles are right and are kept; the element is the improvement |
| self-contained, small embedded stylesheet | spine links `../../css/html_scroll.css`, which resolves only inside its own output tree |
| no site furniture | a headband of hard-coded links is not the document |
| an id for **every** anchor tag, first occurrence winning | spine makes an id only for an endnote's, so every other tag the markup placed is written as a target that is not one, and its own output has links to them |
| indents and bullets as data attributes and one glyph | spine writes runs of up to eighteen `&nbsp;`, which puts presentation in the content |

and the epub, which is the html plus a container:

| | |
|---|---|
| notes stay where they were gathered | spine copies each note into the segment that refers to it *and* into the endnotes file, so a note's text is in the publication twice and its id is in two files |
| one navigation document, built from the segments | it has to name a file, and a `toc` object's link may point inside one |
| `dcterms:modified` from the document's own date | a writer that stamps the time of its run cannot be compared with itself |
| the identifier is a uuid over the document's digest | so that the same document is the same publication every time it is written |
| the language is a BCP 47 tag | one document declares `pt_BR`, which is a POSIX locale name; epubcheck refuses it and `pt-BR` is what the document means |

and the pdf, which is not spine's pdf at all but its own typesetting:

| | |
|---|---|
| the object number is a `place` at a coordinate inside the object's block | spine's `\ocn` is a `\marginpar`, which fails inside a float or a footnote and migrates to the next page when it will not fit |
| a note is a **real footnote** at the point of reference, carrying the document's own number, with one link back to the object | spine's latex emits 195 footnote anchors per document that nothing links to, so the return path is not wired up |
| eight heading levels, the document's own depth | typst has no six-level ceiling, so unlike the html this needs no `aria-level` |
| the document's own table of contents, not `#outline()` | the document has one, with the objects it names; an outline would be a second one made of page numbers. The pdf's bookmarks come from the headings either way |
| the table of contents locates by **object number**, not by page, with a dotted leader | a page number is a fact about one typesetting: change the paper, the orientation, the type size or the margins and every number moves. The object number does not, and it is the same number the margin carries and a citation uses. The markdown writer does the same |
| no gathered endnotes section | the notes are on the pages that refer to them, so the section has nothing to hold. Spine's `.tex` ends with the word "Endnotes" as bare text outside any sectioning command, with nothing under it |
| `#set document(date: none)` and the document's own metadata | a writer that stamps the time of its run cannot be compared with itself |
| the author's quotation marks and dashes, `smartquote` off | a document is reproduced, not restyled |

Four things follow from typst rather than from this writer, and all four
are measured against spine's own pdf of the same document:

| | spine | here |
|---|---|---|
| toolchain (nix closure) | 6.6 GB (`texlive-combined-full`) | 108 MB |
| passes per document | two | one |
| `pdfinfo ... Tagged:` | `no` | `yes` |
| CJK characters in `live-manual.ja` | **none**, in 92 pages | **56,990**, in 61 |

The last is worth saying plainly: spine's Japanese pdf is 92 pages of Latin
fragments and white space.

If spine's behaviour should change, it changes in spine and these follow.
The findings above are reported, not worked around silently.

## Escaping for a language that continues a call across what follows it

Worth recording, because the failures were not where they were expected.

typst continues a function call across whatever comes next. `#emph[a](b)`
reads `(b)` as a second argument list; `#emph[a].b` reads `.b` as a field
access. The writer puts a call around every emphasis, every link, every note
and every object, so **a "(" or a "." of the author's own, landing after one
of them, is a compile error**.

Guarding the positions after a call is not a closed set, and it kept failing.
What works is blunt: **escape every character typst can read as syntax,
wherever it occurs.**

```
\  #  [  ]  *  _  $  @  <  >  `  ~  (  )  .  /  "
```

`.` is in there for the ellipsis as well; `/` for the line comment `//` -
one sample's text contains `gsub!(//, "")`, and everything after it to the
end of the line, closing bracket included, was a comment; and the quotation
mark is in there although it is *not* typst syntax, so that a bare one in the
written source always delimits a string literal. That last is what lets the
block-start guard leave a code object's text alone.

Three markers are position-dependent and are guarded rather than escaped,
because escaping them everywhere would put a backslash in every hyphen:
`- `, `+ ` and `= ` at the start of a line or a content block, which typst
reads as a list item, an enumeration item and a heading. (`1. ` needs
nothing, since every `.` is escaped already.)

Two further traps, both found by compiling:

- `#raw` takes a **string**, not content, so it cannot carry the nested
  markup an object may have inside a monospace run. Inline monospace is
  `#mono[...]`, a `text(font: ...)` binding of the writer's own; a code block
  is a different thing and does use `raw`.
- the protection that keeps a replacement from being escaped **nests** - a
  link is protected, and a line break inside its text is then protected again
  - so it has to be counted and not flagged. With a flag the inner close read
  as the end of the outer run, and every image caption carrying a line break
  lost the `]` that closed its link.

## What a writer needs from the model, and what it does not

Everything except two things: the **name of the generating program** and its
**home page**, which the text writer puts in its footer. Those describe the
program, not the document, so the model rightly has no place for them and
each writer names itself. They are set here to spine's own values so that
the comparison is of the writer and not of the signature.

## The format

Specified in **sisudoc-abstraction-format**: `ocda-model.md` for the model,
and the two format specifications for the serialisations. At format **2.1**;
the writer's version follows the format's major and minor parts.

## Licence

AGPL-3.0-or-later. See [`COPYRIGHT`](COPYRIGHT).