1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
|
-*- mode: org -*-
#+TITLE: configuration nix
#+DESCRIPTION: configuration nix
#+FILETAGS: :spine:build:tools:
#+AUTHOR: Ralph Amissah
#+EMAIL: [[mailto:ralph.amissah@gmail.com][ralph.amissah@gmail.com]]
#+COPYRIGHT: Copyright (C) 2015 (continuously updated, current 2026) Ralph Amissah
#+LANGUAGE: en
#+STARTUP: show3levels hideblocks hidestars noindent entitiespretty
#+PROPERTY: header-args+ :eval never-export :exports code
#+PROPERTY: header-args+ :noweb yes :padline no
#+PROPERTY: header-args+ :results silent :cache no
#+PROPERTY: header-args+ :mkdirp yes
#+OPTIONS: H:3 num:nil toc:t \n:t ::t |:t ^:nil -:t f:t *:t
- magic single double-quote → " ← FIX changes hilighting behavior (occuring
after it) in org document. INVESTIGATE (org-mode CONFIG?) FIND & FIX
* ./
** README.md
#+HEADER: :tangle ../README.md
#+HEADER: :noweb yes
#+BEGIN_SRC md
# sisudoc-ocda-lib
The **object-centric document abstraction as a library**: the model, and
loaders for both of the forms it is written in.
**One reader per language, in a directory named for the language.**
| | |
|--------------------|---------|
| [`dlang/`](dlang/) | D |
| [`gleam/`](gleam/) | Gleam |
| `ruby/` | to come |
## What it is for
The readers in `sisudoc-ssp-reader` and `sisudoc-ocda-db-reader` each read
one serialisation and print. A **writer** needs neither: it needs the document.
```
.ssp ─┐
├─> the model ─> a writer
.ocda.db ─┘
```
So this library holds the model and both loaders, and **a writer cannot
tell which serialisation it was handed** - exactly as spine's own writers
cannot, since spine 0.25.0.
That is the whole of the idea. The writers are functions from the model to
bytes, and the reader is a program over the model.
## The reader
```sh
ocda-read [--tree|--html|--llm-text|--llm-jsonl|--images|--verify --image-dir=<d>|
--languages|--files|--report-unknown|--validate]
<file>.ssp | <file>.ocda.db | <file>.sisupod [lang]
```
One reader, in each language, where `sisudoc-ssp-reader` and
`sisudoc-ocda-db-reader` were two, each with a parser of its own. Built on
this library, **every mode reads every form and says the same thing of a
document's `.ssp`, `.ocda.db` and `.sisupod`**, 36 of 36 document-languages.
The D reader has every mode of both old readers; the Gleam one the modes the
Gleam readers had (the summary, `--tree`, `--languages`, `--files`,
`--report-unknown`), and in those **its output is the D one's, byte for
byte**, 36 of 36.
It is in each language's `src/read/`, and not part of the library: a
program using the library does not build it (in D, by dub configuration;
in Gleam a dependant compiles it and does not use it, Gleam building
everything in `src/`).
Writing it found one defect in the library: **the database loader could
not say what it had not understood**. It asks for columns by name, and so
never saw the ones it did not ask for, and reported that it had understood
everything. It now counts them on the conditions under which the `.ssp`
writes the same property, and the report is the same from either form.
## The checks that matter
The claim above is only true if the loaders agree, so the library asks. Two
programs, in each language:
```sh
ocda-agree <doc>.ssp <doc>.ocda.db [lang] # do the two loaders agree?
ocda-dump <file>.ssp|<file>.ocda.db [lang] # the whole model, canonically
```
`agree` compares two models within one language. **`dump` is the stronger
of the two**, and the reason it exists: it writes the entire loaded model
out in a fully determined form, and the D and the Gleam library write it
*byte for byte alike*. So four questions can be asked, all of them with
`diff`:
| | |
|-----------------------------|---------------------------------|
| D: `.ssp` == `.ocda.db` | **36 of 36** document-languages |
| Gleam: `.ssp` == `.ocda.db` | **36 of 36** |
| `.ssp`: D == Gleam | **36 of 36** |
| `.ocda.db`: D == Gleam | **36 of 36** |
None of those held at first.
## What the checks found
### the ten-field check: a blank first text line
An object whose **first text line is blank** lost it. The loader decided
whether to start a new line by asking whether the accumulated text was
empty, which is not the same question as whether a line has been seen. Of
the sample collection **seven documents have verse opening on a blank
line**, and in every one of them the blank was dropped. Eight of
thirty-six documents disagreed.
Nothing else had caught it. The two `.ssp` readers had been compared on
*section counts*, which this does not change. It took loading the same
document two ways and comparing the objects.
Fixed here and in `sisudoc-ssp-reader/dlang`, which carried the same code.
### the whole-model dump: four more
The ten-field check was enough for that and not enough for anything
subtler. Dumping the whole model found four further disagreements between
the two loaders:
1. **`.identifier` and `.anchor`.** The `.ssp` emits a property only when it
carries information, so the identifier is absent when it is simply the
ocn and the anchor is absent when it is the segment name; the database
stores both unconditionally. Over one document that is **1,314
identifiers and 69 anchors**. The loaders now derive them, which is what
the model means.
2. **`make` and `doc_has`.** The database has one metadata table where the
text form has three blocks, and tells them apart by a key prefix. The
loaders unprefix them, so a writer asks `make["footer"]` whichever
serialisation it was handed.
3. **The JSON columns were scanned rather than parsed.** Phobos writes `/`
as `\/`, so every heading containing a slash came back with a backslash
in it: **all 36 document-languages** differed on
`heading_ancestors_text` and nothing else had shown it.
4. **`translation.ocn_aligned`** is in the database and cannot be in a
`.ssp`, which is one language and cannot know whether the languages
share a numbering. It is a fact about the *file*, so the loaders leave
it out rather than letting a writer tell which serialisation it was
given.
The first three are defects. The fourth is a real asymmetry between the two
serialisations, and naming it is the useful part.
## What it reads, and what it does not
| | | |
|-----------------------|------------------------------------------------------------------|-------------------|
| `.ssp` | yes | |
| `.ocda.db` | yes, any language of it | |
| `.sisupod` | yes, any language it carries | D only, see below |
| `.zip` pod | yes, any language it carries | |
| `.sst`, `.ssm` markup | **no, and not planned**: that is a parser, which is spine's work | |
| a url | no | |
A pod carries the abstraction only when spine was asked for it: `--pod2`
puts it in, `--source` does not. A pod without one is refused with a message
saying which flag makes a pod this library can read, because that is a fact
about the pod rather than about the library.
**Over the reference collection, a `.sisupod` gives the same model as the
`.ssp` beside it: 36 of 36 document-languages, byte identical dumps.** A
writer over the library therefore gained a third input form without being
touched, which is the point of the dispatch being in one place.
### the one thing the Gleam library cannot do
A `.sisupod` is **one zstd frame wrapping** the zip. The D library reads both
through a declared `libzstd`. The Gleam library reads the **zip and not the
frame**, because the only zstd on the BEAM is a NIF that clones zstd from
github in its own build and so cannot be built in a sandbox without patching
it.
So a `.zip` pod is read in both languages, a `.sisupod` in D only, and the
Gleam refusal says why and says `zstd -d` rather than claiming the file is
malformed. It is a packaging limit, not anything about the format, and it is
the only place the two libraries differ in what they accept.
`ocda/load` is the one place in each language that decides what a path is,
so a new input form is added once and every reader and writer gains it.
## The format
Specified in **sisudoc-abstraction-format**: `ocda-model.md` for the model
this library implements, and the two format specifications for the loaders.
Section 4 of `ocda-db-format-specification.md` gives the column by column
correspondence the `load_db` of each language follows.
At format **2.x**.
#+END_SRC
** git .gitignore :gitignore:
#+HEADER: :tangle "../.gitignore"
#+BEGIN_SRC gitignore
# git ls-files --others --exclude-from=.git/info/exclude
# git ls-files --others --ignored --exclude-standard --directory
# git check-ignore -v flake.lock
# git clean -ix
# find . -name "*_" | xargs rm -rf
# find . -name "*.o" | xargs rm -rf
# Deny everything, then whitelist what belongs in the repository.
#
# The practice is deliberate: a repository should hold what was decided to
# put in it, not whatever happened to be in the directory when someone ran
# "git add .". Build output, editor droppings, downloaded dependencies and
# exported binaries are numerous, are regenerated, and differ between
# machines; source is few and is chosen. So the default is no, and every
# yes below is a decision that can be read.
#
# To see what is being excluded, and why:
# git status --ignored
# git check-ignore -v <path>
# git ls-files --others --ignored --exclude-standard --directory
*
# ── the repository itself ────────────────────────────────────────────────
!.gitignore
!README.md
!LICENSE
# ── directories have to be walked into before a file inside can match ────
!*/
# but not these, whatever they contain
build/
.build/
.direnv/
.gleam/
bin/
result
result-*
# ── documentation and notes, in any language directory ───────────────────
!**/*.md
!**/*.org
# ── nix ──────────────────────────────────────────────────────────────────
!**/*.nix
!**/flake.lock
!**/.envrc
# ── dlang ────────────────────────────────────────────────────────────────
!**/*.d
!**/dub.json
!**/dub.sdl
!**/dub.selections.json
*.o
*.a
*.lst
# ── gleam ────────────────────────────────────────────────────────────────
!**/*.gleam
# Erlang, for what Gleam reaches through FFI (a sha256, from OTP crypto)
!**/src/*.erl
# gleam.toml is the manifest, manifest.toml is the lockfile: both belong
!**/*.toml
*.beam
*.ez
erl_crash.dump
# ── ruby, when it arrives ────────────────────────────────────────────────
!**/*.rb
!**/Gemfile
!**/Gemfile.lock
!**/*.gemspec
# ── continuous integration ───────────────────────────────────────────────
!**/*.yml
!**/*.yaml
#+END_SRC
* org includes
** project version
#+NAME: spine_version
#+HEADER: :noweb yes
#+BEGIN_SRC emacs-lisp
<<./sisudoc_spine_version_info_and_doc_header_including_copyright_and_license.org:spine_project_version()>>
#+END_SRC
** year
#+NAME: year
#+HEADER: :noweb yes
#+BEGIN_SRC emacs-lisp
<<./sisudoc_spine_version_info_and_doc_header_including_copyright_and_license.org:year()>>
#+END_SRC
** document header including copyright & license
#+NAME: doc_header_including_copyright_and_license
#+HEADER: :noweb yes
#+BEGIN_SRC emacs-lisp
<<./sisudoc_spine_version_info_and_doc_header_including_copyright_and_license.org:spine_doc_header_including_copyright_and_license()>>
#+END_SRC
* nixpkgs source
#+NAME: nixpkgs_unstable
#+HEADER: :noweb yes
#+BEGIN_SRC emacs-lisp
<<./sisudoc_spine_version_info_and_doc_header_including_copyright_and_license.org:nixpkgs_unstable()>>
#+END_SRC
* __END__
|