summaryrefslogtreecommitdiffhomepage
diff options
context:
space:
mode:
authorRalph Amissah <ralph.amissah@gmail.com>2026-09-25 22:28:50 -0400
committerRalph Amissah <ralph.amissah@gmail.com>2026-10-03 21:50:59 -0400
commit5a5036e53aa18afa61d6e475336cd96c115dac00 (patch)
tree669a9190703b6833591311c73f4646a123c11dc3
parent.ocda.db specified & the two serialisations agree (diff)
derived values that the model left unstated
The model says nothing about an object's identifier or its anchor, and both are derived: the identifier is the ocn where there is one, the anchor is the segment name. The .ssp omits each where it can be worked out; the .ocda.db states both on every row. This is existing behaviour, not a new rule. Spine's own .ssp reader has always filled both in, which is why the database carries them: the database is written from the .ssp read straight back. It should have been written down first. Two readers written independently from these documents disagreed for want of it, on nearly every object, and nothing short of comparing whole models showed it. ocda-model.md gains 3.1 the identifier, 7.1 the anchor and 9 derived values, with the loading rule in a table; "what the model does not hold" becomes 10. The two format specifications gain what a reader needs and did not have: - the .ssp's property table now says what the absent value is, not only when it is written; - the .ocda.db gains 3.2.1, the metadata key prefix that carries what the .ssp puts in four blocks, which a reader must strip to give one model from either file, and the note that a value here may carry whitespace the text form cannot represent; - 3.3 gains the element types of each JSON column and a warning to parse rather than scan: the encoder writes "/" as "\/" and every document in the reference collection has a heading containing one; - 4.1 names the three places the two files are not symmetric, including translation.ocn_aligned, which a .ssp cannot hold because it is one language, and which a reader should keep out of the document model so that nothing built on the model can tell which file it was given. (assisted by Claude-Code)
-rw-r--r--ocda-db-format-specification.md83
-rw-r--r--ocda-model.md80
-rw-r--r--org/ocda_specification.org260
-rw-r--r--ssp-format-specification.md97
4 files changed, 430 insertions, 90 deletions
diff --git a/ocda-db-format-specification.md b/ocda-db-format-specification.md
index ebeeaa6..eacc618 100644
--- a/ocda-db-format-specification.md
+++ b/ocda-db-format-specification.md
@@ -101,6 +101,33 @@ written with `doc_id` null: `schema.name`, `schema.version` and
`UNIQUE(doc_id, key)` does not constrain them, and a partial unique index
over `key` alone `WHERE doc_id IS NULL` is what makes them singular.
+#### 3.2.1 The key prefix carries what the `.ssp` puts in blocks
+
+The `.ssp` has four preamble blocks, `@source`, `@meta`, `@make` and
+`@doc_has`. The database has **one** `metadata` table, and tells them apart
+by a **prefix on the key**:
+
+| `.ssp` block | database key |
+|--------------|--------------|
+| `@meta` | the key as it stands: `title.main`, `creator.author`, ... |
+| `@make` | `make.` + the key: `make.footer`, `make.doc_type` |
+| `@doc_has` | `doc_has.` + the key: `doc_has.images`, `doc_has.tables` |
+| `@source` | `source.` + the key: `source.language`, `source.digest` |
+
+**A reader must strip the prefix and restore the grouping**, or a writer
+over it has to ask `metadata["make.footer"]` from a database and
+`make["footer"]` from a `.ssp`, which is exactly the difference the two
+serialisations exist not to make.
+
+`source.` also carries two keys with no `@source` counterpart,
+`source.filename` and `source.ssp_digest`; the `.ssp` puts the first in its
+`% Source:` comment and cannot hold the second, which is a digest *of* it.
+
+Values may carry leading or trailing whitespace here where the `.ssp`
+cannot represent it, its `key: value` line being unable to distinguish a
+value that begins with a space from the separator. **Strip on load**, as a
+reader of the text form necessarily does.
+
### 3.3 Fixed-width arrays are JSON
`children`, `ancestors`, `ancestors_collapsed`, `dom_status`,
@@ -108,6 +135,33 @@ over `key` alone `WHERE doc_id IS NULL` is what makes them singular.
`table_aligns` are stored as **JSON arrays**, with validity checks, so that
`json_extract`, `json_each` and `json_array_length` reach inside them.
+> **Parse them as JSON.** A scan that strips `[`, `]`, `,` and `"` looks
+> adequate and is wrong on the first escape: the encoder writes `/` as
+> `\/`, so `FREE/OPEN-SOURCE SOFTWARE` comes back as
+> `FREE\/OPEN-SOURCE SOFTWARE`. All 36 document-languages of the reference
+> collection contain at least one, and the defect is invisible to any check
+> that does not compare the field itself.
+>
+> Note also that `table_widths` is an array of **integers except where the
+> columns divide evenly**, when it is `[33.3333,33.3333,33.3333]`. A decoder
+> that accepts strings and integers and then falls back to a default turns
+> a value it could not read into one that looks read.
+
+The **element types**, so that a decoder can be written to accept exactly
+what occurs:
+
+| column | elements |
+|--------|----------|
+| `children`, `ancestors`, `ancestors_collapsed`, `dom_status`, `dom_status_collapsed` | integers |
+| `table_widths` | integers, or floats where the columns divide evenly |
+| `table_aligns` | strings, one of `l` `c` `r` |
+| `heading_ancestors_text` | strings, eight of them, empty where that level is not an ancestor |
+
+The `.ssp` writes the first three groups **space separated** and
+`heading_ancestors_text` **`|` separated**. The empty slots of the last are
+significant: they say which levels a heading sits under, so the separators
+carry as much as the text does.
+
The open-ended lists are **rows in the four child tables** instead of
columns: images, links, anchor tags and sub-table-of-contents entries. Each
child table carries `seq`, which preserves the order they appeared in.
@@ -159,6 +213,35 @@ object's position within its section, and `(doc_id, section, seq)` is what
identifies an object when `ocn` will not (see
[`ocda-model.md`](ocda-model.md) section 3).
+### 4.1 Where the two are not symmetric
+
+"Nothing is lost in either direction" is true of the model, and there are
+three places where the *files* nonetheless differ. A reader must know all
+three, because each of them made two independently written readers disagree.
+
+**1. The database states what the `.ssp` derives.** `identifier` and
+`anchor` are written on every row. The `.ssp` omits each where it can be
+worked out, the identifier being the ocn and the anchor the segment name
+(sections 6.1 and 6.3 of
+[`ssp-format-specification.md`](ssp-format-specification.md), section 9 of
+[`ocda-model.md`](ocda-model.md)). In `free_culture.lawrence_lessig` that is
+**1,314 identifiers and 69 anchors** stated here and absent there. Neither
+file is the poorer; a reader that does not derive is.
+
+**2. `translation.ocn_aligned` is here and cannot be in a `.ssp`.** It says
+whether the languages of a document share one numbering, and a `.ssp` is one
+language. It is therefore a fact about **the file**, not about the document
+in it, and it is the one such fact either serialisation carries.
+
+A reader should **keep it out of the document model**, for the same reason
+the `schema.*` and `build.*` rows are kept out: a writer over the model must
+not be able to tell which serialisation it was handed, and a field that
+exists only when the input was a database is exactly that tell. The model
+has no place for collection-wide facts yet; when it grows one, this belongs
+there.
+
+**3. Whitespace and grouping**, as section 3.2.1 gives them.
+
## 5. The `files` table: what it carries, and how
| role | named by | `compression` | `data` holds |
diff --git a/ocda-model.md b/ocda-model.md
index f4bf245..065677a 100644
--- a/ocda-model.md
+++ b/ocda-model.md
@@ -39,9 +39,12 @@ the document is made of and the unit it is cited by.
Every object carries:
- an **object citation number** (OCN), section 3;
+- an **identifier**, section 3.1;
- a **kind** and a **type**, section 2.1;
- its **place in the structure**: parent, ancestors, and for a heading its
children and its last descendant, section 4;
+- the **segment** it is in, and for a heading an **anchor**, sections 7
+ and 7.1;
- its **text**, in spine's internal representation;
- a **digest** of that text.
@@ -108,6 +111,24 @@ otherwise:
its verses share one; `through_the_looking_glass` has 21 such cases. Key
on position within a section when one row is wanted.
+### 3.1 The identifier
+
+Every object also carries an **identifier**: the name by which an output
+refers to it. An HTML anchor, a LaTeX `\hypertarget`, a link from a book
+index entry to the object it indexes, are all written from the identifier.
+
+**The identifier of an object with a non-zero OCN is its OCN**, written as
+text. That is the ordinary case and it is the reason a citation to object
+42 can be turned into a link without consulting anything.
+
+An object with OCN `0` has no OCN to take an identifier from, so one is
+assigned: `a1`, `a2`, `a3`, ... in document order over the objects that
+need one. There are few of them, eleven in a book, and they are the
+generated furniture of section 3.
+
+**A reader must arrive at the identifier whether or not the serialisation
+wrote it down**; see section 9.
+
## 4. Levels
A document has **eight heading levels**, in two groups of four:
@@ -188,6 +209,21 @@ both separately because the answers differ:
Without the second, a reader cannot say where a paragraph belongs. The
search database reads the second.
+### 7.1 The anchor
+
+A heading carries an **anchor**: the name an output links to in order to
+arrive at that heading. It is what a table of contents entry and a
+cross reference point at.
+
+**An object's anchor is its segment name**, in the ordinary case, because a
+heading that opens a segment is arrived at by arriving at the segment. It
+differs only where a heading is not the opening of the segment it sits in:
+69 of the 72 anchors in one reference document equal their segment and 3 do
+not.
+
+**A reader must arrive at the anchor whether or not the serialisation wrote
+it down**; see section 9.
+
## 8. Digests
The model is checkable at four levels, and the chain is what lets an
@@ -203,7 +239,49 @@ each image ──> a digest of the image file, with its size in bytes and pixels
An object's digest is over its text alone, so a change to one paragraph
moves one digest and no other.
-## 9. What the model does not hold
+## 9. Derived values, and why a reader must derive them
+
+Two of the values above are **derived**: the identifier from the OCN
+(section 3.1), and the anchor from the segment name (section 7.1). The
+model holds them; a serialisation need not, and the text form does not.
+
+**A serialisation may omit a value it can compute.** The `.ssp` writes a
+property only when it carries information, so it omits `.identifier` where
+the identifier is the OCN and `.anchor` where the anchor is the segment
+name. The `.ocda.db` is columns rather than lines and writes both
+unconditionally.
+
+Both are conforming. What is **not** conforming is a reader that takes the
+text form literally and hands on a model with those fields empty, because
+that model is not the same model the database gives, and a writer over it
+will produce different output from the two serialisations of one document.
+
+The size of the difference is easy to underestimate. In
+`free_culture.lawrence_lessig` the `.ssp` omits **1,314 identifiers and 69
+anchors** that the `.ocda.db` states, which is nearly every object in the
+document.
+
+So, on loading:
+
+| field | when absent | take |
+|-------|-------------|------|
+| identifier | OCN is non-zero | the OCN as text |
+| identifier | OCN is `0` | whatever the serialisation gave; there is nothing to derive from |
+| anchor | the object has a segment | the segment name |
+
+**This is a property of the model, not of either format**, which is why it
+is stated here and not only in the two format specifications. It is stated
+here *because* two readers written independently from these documents
+disagreed for want of it: the rule was implied by the `.ssp`'s emission
+conditions and nowhere asserted.
+
+It is also what **spine's own reader** does, and always did, which is why
+the `.ocda.db` carries the derived values: the database is written from the
+`.ssp` read straight back, and the reading fills both in. So this section
+documents existing behaviour rather than prescribing new. It should have
+been written down first.
+
+## 10. What the model does not hold
- **The markup.** The model is what the markup produced, not the markup
itself. An artefact that carries both is the `.ocda.db`, which carries
diff --git a/org/ocda_specification.org b/org/ocda_specification.org
index 442315a..c1737b1 100644
--- a/org/ocda_specification.org
+++ b/org/ocda_specification.org
@@ -223,9 +223,12 @@ the document is made of and the unit it is cited by.
Every object carries:
- an **object citation number** (OCN), section 3;
+- an **identifier**, section 3.1;
- a **kind** and a **type**, section 2.1;
- its **place in the structure**: parent, ancestors, and for a heading its
children and its last descendant, section 4;
+- the **segment** it is in, and for a heading an **anchor**, sections 7
+ and 7.1;
- its **text**, in spine's internal representation;
- a **digest** of that text.
@@ -292,6 +295,24 @@ otherwise:
its verses share one; `through_the_looking_glass` has 21 such cases. Key
on position within a section when one row is wanted.
+### 3.1 The identifier
+
+Every object also carries an **identifier**: the name by which an output
+refers to it. An HTML anchor, a LaTeX `\hypertarget`, a link from a book
+index entry to the object it indexes, are all written from the identifier.
+
+,**The identifier of an object with a non-zero OCN is its OCN**, written as
+text. That is the ordinary case and it is the reason a citation to object
+42 can be turned into a link without consulting anything.
+
+An object with OCN `0` has no OCN to take an identifier from, so one is
+assigned: `a1`, `a2`, `a3`, ... in document order over the objects that
+need one. There are few of them, eleven in a book, and they are the
+generated furniture of section 3.
+
+,**A reader must arrive at the identifier whether or not the serialisation
+wrote it down**; see section 9.
+
## 4. Levels
A document has **eight heading levels**, in two groups of four:
@@ -372,6 +393,21 @@ both separately because the answers differ:
Without the second, a reader cannot say where a paragraph belongs. The
search database reads the second.
+### 7.1 The anchor
+
+A heading carries an **anchor**: the name an output links to in order to
+arrive at that heading. It is what a table of contents entry and a
+cross reference point at.
+
+,**An object's anchor is its segment name**, in the ordinary case, because a
+heading that opens a segment is arrived at by arriving at the segment. It
+differs only where a heading is not the opening of the segment it sits in:
+69 of the 72 anchors in one reference document equal their segment and 3 do
+not.
+
+,**A reader must arrive at the anchor whether or not the serialisation wrote
+it down**; see section 9.
+
## 8. Digests
The model is checkable at four levels, and the chain is what lets an
@@ -387,7 +423,49 @@ each image ──> a digest of the image file, with its size in bytes and pixels
An object's digest is over its text alone, so a change to one paragraph
moves one digest and no other.
-## 9. What the model does not hold
+## 9. Derived values, and why a reader must derive them
+
+Two of the values above are **derived**: the identifier from the OCN
+(section 3.1), and the anchor from the segment name (section 7.1). The
+model holds them; a serialisation need not, and the text form does not.
+
+,**A serialisation may omit a value it can compute.** The `.ssp` writes a
+property only when it carries information, so it omits `.identifier` where
+the identifier is the OCN and `.anchor` where the anchor is the segment
+name. The `.ocda.db` is columns rather than lines and writes both
+unconditionally.
+
+Both are conforming. What is **not** conforming is a reader that takes the
+text form literally and hands on a model with those fields empty, because
+that model is not the same model the database gives, and a writer over it
+will produce different output from the two serialisations of one document.
+
+The size of the difference is easy to underestimate. In
+`free_culture.lawrence_lessig` the `.ssp` omits **1,314 identifiers and 69
+anchors** that the `.ocda.db` states, which is nearly every object in the
+document.
+
+So, on loading:
+
+| field | when absent | take |
+|-------|-------------|------|
+| identifier | OCN is non-zero | the OCN as text |
+| identifier | OCN is `0` | whatever the serialisation gave; there is nothing to derive from |
+| anchor | the object has a segment | the segment name |
+
+,**This is a property of the model, not of either format**, which is why it
+is stated here and not only in the two format specifications. It is stated
+here *because* two readers written independently from these documents
+disagreed for want of it: the rule was implied by the `.ssp`'s emission
+conditions and nowhere asserted.
+
+It is also what **spine's own reader** does, and always did, which is why
+the `.ocda.db` carries the derived values: the database is written from the
+`.ssp` read straight back, and the reading fills both in. So this section
+documents existing behaviour rather than prescribing new. It should have
+been written down first.
+
+## 10. What the model does not hold
- **The markup.** The model is what the markup produced, not the markup
itself. An artefact that carries both is the `.ocda.db`, which carries
@@ -742,6 +820,15 @@ reference collection), and then its identifier, when it has one
that is not simply the ocn. Every other kind carries an identifier, on the
same condition, as the `.identifier` property.
+> **A reader must derive the identifier where it is absent, and the anchor
+> likewise.** "when it has one that is not simply the ocn" means the
+> identifier *is* the ocn in every other case, and a reader that leaves the
+> field empty gives a different model from the one the `.ocda.db` gives, on
+> nearly every object. Section 9 of [`ocda-model.md`](ocda-model.md) states
+> both derivations and why they belong to the model rather than to either
+> format. Two readers written independently from these documents disagreed
+> for want of it.
+
### 6.2 Text lines
```
@@ -777,50 +864,50 @@ In the table, "emitted when" is the condition under which the property
appears at all. Counts are occurrences across the 36 document reference
set, as a guide to which parts of the format are load bearing.
-| # | property | value | emitted when | count |
-|----|---------------------------|-------------------------------|-------------------------------------------------------------|--------|
-| 1 | `.identifier` | string | a non-heading has an identifier that is not its ocn | 353 |
-| 2 | `.part` | string | the object belongs to a part (`frontmatter`, `body`, ...) | 56,882 |
-| 3 | `.section` | string | the object's own section differs from the block it sits in | 605 |
-| 4 | `.parent` | int | **every object except** the document root and `toc` entries | 53,631 |
-| 5 | `.last_descendant` | int | non-zero: the last ocn under this heading | 3,202 |
-| 6 | `.children` | ints | the heading has child headings | 827 |
-| 7 | `.ancestors` | 8 ints | any is non-zero | 53,595 |
-| 8 | `.ancestors_collapsed` | 8 ints | any is non-zero | 53,595 |
-| 9 | `.dom_status` | 8 ints | any is non-zero | 3,319 |
-| 10 | `.dom_status_collapsed` | 8 ints | any is non-zero | 3,319 |
-| 11 | `.heading_lev_collapsed` | int | it is below 9 | 3,503 |
-| 12 | `.parent_lev` | int | non-zero | 52,848 |
-| 13 | `.dummy` | `true` | the heading was introduced to segment an output | 183 |
-| 14 | `.ocn_off` | `true` | the object has no citation number | 7,543 |
-| 15 | `.is_of_type` | string | always, in practice | 56,882 |
-| 16 | `.attrib` | string | set | 1 |
-| 17 | `.meta_lang` | string | the object carries its own language | 0 |
-| 18 | `.meta_syntax` | string | the object carries its own markup syntax | 0 |
-| 19 | `.sha256` | hex | the object's text digest is non-zero | 51,253 |
-| 20 | `.image` | record | once **per image**, in order of appearance | 63 |
-| 21 | `.indent` | 2 ints | either indent is non-zero: `base hang` | 7,583 |
-| 22 | `.bullet` | `true` | the object is a bullet item | 1,903 |
-| 23 | `.lang` | string | the text attributes carry a language | 0 |
-| 24 | `.has` | flags | any is set | 11,143 |
-| 25 | `.table_cols` | int | the object is a table with columns | 32 |
-| 26 | `.table_widths` | ints | the table has column widths | 32 |
-| 27 | `.table_aligns` | letters | the table has alignments (`l` `c` `r`) | 32 |
-| 28 | `.table_header` | `true` | the table has a header row | 29 |
-| 29 | `.code_linenumbers` | `true` | the code block is numbered | 1 |
-| 30 | `.stow_link` | url | once **per extracted link** | 1,013 |
-| 31 | `.segment` | string | the heading opens a segment | 3,355 |
-| 32 | `.anchor` | string | the anchor differs from the segment name | 68 |
-| 33 | `.segment_prev` | string | there is a previous segment | 1,022 |
-| 34 | `.segment_next` | string | there is a next segment | 1,022 |
-| 35 | `.heading_lev_anchor` | string | the heading has a level anchor tag | 1,854 |
-| 36 | `.segment_epub` | string | the heading opens an epub segment | 3,355 |
-| 37 | `.segment_html_is` | string | the object is inside an html segment | 47,024 |
-| 38 | `.segment_epub_is` | string | the object is inside an epub segment | 47,024 |
-| 39 | `.segment_lv4_is` | string | **and** it differs from `.segment_html_is` | 342 |
-| 40 | `.heading_ancestors_text` | 8 slots, `\|` joined, escaped | any slot is non-empty | 3,319 |
-| 41 | `.lev4_subtoc` | string | once **per sub-table-of-contents entry** | 1,887 |
-| 42 | `.anchor_tag` | string | once **per anchor tag** | 7,883 |
+| # | property | value | emitted when | count |
+|----|---------------------------|-------------------------------|----------------------------------------------------------------------------------------------------|--------|
+| 1 | `.identifier` | string | a non-heading has an identifier that is not its ocn (**otherwise it is the ocn**: model section 9) | 353 |
+| 2 | `.part` | string | the object belongs to a part (`frontmatter`, `body`, ...) | 56,882 |
+| 3 | `.section` | string | the object's own section differs from the block it sits in | 605 |
+| 4 | `.parent` | int | **every object except** the document root and `toc` entries | 53,631 |
+| 5 | `.last_descendant` | int | non-zero: the last ocn under this heading | 3,202 |
+| 6 | `.children` | ints | the heading has child headings | 827 |
+| 7 | `.ancestors` | 8 ints | any is non-zero | 53,595 |
+| 8 | `.ancestors_collapsed` | 8 ints | any is non-zero | 53,595 |
+| 9 | `.dom_status` | 8 ints | any is non-zero | 3,319 |
+| 10 | `.dom_status_collapsed` | 8 ints | any is non-zero | 3,319 |
+| 11 | `.heading_lev_collapsed` | int | it is below 9 | 3,503 |
+| 12 | `.parent_lev` | int | non-zero | 52,848 |
+| 13 | `.dummy` | `true` | the heading was introduced to segment an output | 183 |
+| 14 | `.ocn_off` | `true` | the object has no citation number | 7,543 |
+| 15 | `.is_of_type` | string | always, in practice | 56,882 |
+| 16 | `.attrib` | string | set | 1 |
+| 17 | `.meta_lang` | string | the object carries its own language | 0 |
+| 18 | `.meta_syntax` | string | the object carries its own markup syntax | 0 |
+| 19 | `.sha256` | hex | the object's text digest is non-zero | 51,253 |
+| 20 | `.image` | record | once **per image**, in order of appearance | 63 |
+| 21 | `.indent` | 2 ints | either indent is non-zero: `base hang` | 7,583 |
+| 22 | `.bullet` | `true` | the object is a bullet item | 1,903 |
+| 23 | `.lang` | string | the text attributes carry a language | 0 |
+| 24 | `.has` | flags | any is set | 11,143 |
+| 25 | `.table_cols` | int | the object is a table with columns | 32 |
+| 26 | `.table_widths` | ints | the table has column widths | 32 |
+| 27 | `.table_aligns` | letters | the table has alignments (`l` `c` `r`) | 32 |
+| 28 | `.table_header` | `true` | the table has a header row | 29 |
+| 29 | `.code_linenumbers` | `true` | the code block is numbered | 1 |
+| 30 | `.stow_link` | url | once **per extracted link** | 1,013 |
+| 31 | `.segment` | string | the heading opens a segment | 3,355 |
+| 32 | `.anchor` | string | the anchor differs from the segment name (**otherwise it is the segment name**: model section 9) | 68 |
+| 33 | `.segment_prev` | string | there is a previous segment | 1,022 |
+| 34 | `.segment_next` | string | there is a next segment | 1,022 |
+| 35 | `.heading_lev_anchor` | string | the heading has a level anchor tag | 1,854 |
+| 36 | `.segment_epub` | string | the heading opens an epub segment | 3,355 |
+| 37 | `.segment_html_is` | string | the object is inside an html segment | 47,024 |
+| 38 | `.segment_epub_is` | string | the object is inside an epub segment | 47,024 |
+| 39 | `.segment_lv4_is` | string | **and** it differs from `.segment_html_is` | 342 |
+| 40 | `.heading_ancestors_text` | 8 slots, `\|` joined, escaped | any slot is non-empty | 3,319 |
+| 41 | `.lev4_subtoc` | string | once **per sub-table-of-contents entry** | 1,887 |
+| 42 | `.anchor_tag` | string | once **per anchor tag** | 7,883 |
Three of these need a word here, the first two because the model behind
them is easy to misread (see [`ocda-model.md`](ocda-model.md) sections 5
@@ -1080,6 +1167,33 @@ written with `doc_id` null: `schema.name`, `schema.version` and
`UNIQUE(doc_id, key)` does not constrain them, and a partial unique index
over `key` alone `WHERE doc_id IS NULL` is what makes them singular.
+#### 3.2.1 The key prefix carries what the `.ssp` puts in blocks
+
+The `.ssp` has four preamble blocks, `@source`, `@meta`, `@make` and
+`@doc_has`. The database has **one** `metadata` table, and tells them apart
+by a **prefix on the key**:
+
+| `.ssp` block | database key |
+|--------------|--------------|
+| `@meta` | the key as it stands: `title.main`, `creator.author`, ... |
+| `@make` | `make.` + the key: `make.footer`, `make.doc_type` |
+| `@doc_has` | `doc_has.` + the key: `doc_has.images`, `doc_has.tables` |
+| `@source` | `source.` + the key: `source.language`, `source.digest` |
+
+,**A reader must strip the prefix and restore the grouping**, or a writer
+over it has to ask `metadata["make.footer"]` from a database and
+`make["footer"]` from a `.ssp`, which is exactly the difference the two
+serialisations exist not to make.
+
+`source.` also carries two keys with no `@source` counterpart,
+`source.filename` and `source.ssp_digest`; the `.ssp` puts the first in its
+`% Source:` comment and cannot hold the second, which is a digest *of* it.
+
+Values may carry leading or trailing whitespace here where the `.ssp`
+cannot represent it, its `key: value` line being unable to distinguish a
+value that begins with a space from the separator. **Strip on load**, as a
+reader of the text form necessarily does.
+
### 3.3 Fixed-width arrays are JSON
`children`, `ancestors`, `ancestors_collapsed`, `dom_status`,
@@ -1087,6 +1201,33 @@ over `key` alone `WHERE doc_id IS NULL` is what makes them singular.
`table_aligns` are stored as **JSON arrays**, with validity checks, so that
`json_extract`, `json_each` and `json_array_length` reach inside them.
+> **Parse them as JSON.** A scan that strips `[`, `]`, `,` and `"` looks
+> adequate and is wrong on the first escape: the encoder writes `/` as
+> `\/`, so `FREE/OPEN-SOURCE SOFTWARE` comes back as
+> `FREE\/OPEN-SOURCE SOFTWARE`. All 36 document-languages of the reference
+> collection contain at least one, and the defect is invisible to any check
+> that does not compare the field itself.
+>
+> Note also that `table_widths` is an array of **integers except where the
+> columns divide evenly**, when it is `[33.3333,33.3333,33.3333]`. A decoder
+> that accepts strings and integers and then falls back to a default turns
+> a value it could not read into one that looks read.
+
+The **element types**, so that a decoder can be written to accept exactly
+what occurs:
+
+| column | elements |
+|--------|----------|
+| `children`, `ancestors`, `ancestors_collapsed`, `dom_status`, `dom_status_collapsed` | integers |
+| `table_widths` | integers, or floats where the columns divide evenly |
+| `table_aligns` | strings, one of `l` `c` `r` |
+| `heading_ancestors_text` | strings, eight of them, empty where that level is not an ancestor |
+
+The `.ssp` writes the first three groups **space separated** and
+`heading_ancestors_text` **`|` separated**. The empty slots of the last are
+significant: they say which levels a heading sits under, so the separators
+carry as much as the text does.
+
The open-ended lists are **rows in the four child tables** instead of
columns: images, links, anchor tags and sub-table-of-contents entries. Each
child table carries `seq`, which preserves the order they appeared in.
@@ -1138,6 +1279,35 @@ object's position within its section, and `(doc_id, section, seq)` is what
identifies an object when `ocn` will not (see
[`ocda-model.md`](ocda-model.md) section 3).
+### 4.1 Where the two are not symmetric
+
+"Nothing is lost in either direction" is true of the model, and there are
+three places where the *files* nonetheless differ. A reader must know all
+three, because each of them made two independently written readers disagree.
+
+,**1. The database states what the `.ssp` derives.** `identifier` and
+`anchor` are written on every row. The `.ssp` omits each where it can be
+worked out, the identifier being the ocn and the anchor the segment name
+(sections 6.1 and 6.3 of
+[`ssp-format-specification.md`](ssp-format-specification.md), section 9 of
+[`ocda-model.md`](ocda-model.md)). In `free_culture.lawrence_lessig` that is
+,**1,314 identifiers and 69 anchors** stated here and absent there. Neither
+file is the poorer; a reader that does not derive is.
+
+,**2. `translation.ocn_aligned` is here and cannot be in a `.ssp`.** It says
+whether the languages of a document share one numbering, and a `.ssp` is one
+language. It is therefore a fact about **the file**, not about the document
+in it, and it is the one such fact either serialisation carries.
+
+A reader should **keep it out of the document model**, for the same reason
+the `schema.*` and `build.*` rows are kept out: a writer over the model must
+not be able to tell which serialisation it was handed, and a field that
+exists only when the input was a database is exactly that tell. The model
+has no place for collection-wide facts yet; when it grows one, this belongs
+there.
+
+,**3. Whitespace and grouping**, as section 3.2.1 gives them.
+
## 5. The `files` table: what it carries, and how
| role | named by | `compression` | `data` holds |
diff --git a/ssp-format-specification.md b/ssp-format-specification.md
index 17974a3..ae1ecaf 100644
--- a/ssp-format-specification.md
+++ b/ssp-format-specification.md
@@ -338,6 +338,15 @@ reference collection), and then its identifier, when it has one
that is not simply the ocn. Every other kind carries an identifier, on the
same condition, as the `.identifier` property.
+> **A reader must derive the identifier where it is absent, and the anchor
+> likewise.** "when it has one that is not simply the ocn" means the
+> identifier *is* the ocn in every other case, and a reader that leaves the
+> field empty gives a different model from the one the `.ocda.db` gives, on
+> nearly every object. Section 9 of [`ocda-model.md`](ocda-model.md) states
+> both derivations and why they belong to the model rather than to either
+> format. Two readers written independently from these documents disagreed
+> for want of it.
+
### 6.2 Text lines
```
@@ -373,50 +382,50 @@ In the table, "emitted when" is the condition under which the property
appears at all. Counts are occurrences across the 36 document reference
set, as a guide to which parts of the format are load bearing.
-| # | property | value | emitted when | count |
-|----|---------------------------|-------------------------------|-------------------------------------------------------------|--------|
-| 1 | `.identifier` | string | a non-heading has an identifier that is not its ocn | 353 |
-| 2 | `.part` | string | the object belongs to a part (`frontmatter`, `body`, ...) | 56,882 |
-| 3 | `.section` | string | the object's own section differs from the block it sits in | 605 |
-| 4 | `.parent` | int | **every object except** the document root and `toc` entries | 53,631 |
-| 5 | `.last_descendant` | int | non-zero: the last ocn under this heading | 3,202 |
-| 6 | `.children` | ints | the heading has child headings | 827 |
-| 7 | `.ancestors` | 8 ints | any is non-zero | 53,595 |
-| 8 | `.ancestors_collapsed` | 8 ints | any is non-zero | 53,595 |
-| 9 | `.dom_status` | 8 ints | any is non-zero | 3,319 |
-| 10 | `.dom_status_collapsed` | 8 ints | any is non-zero | 3,319 |
-| 11 | `.heading_lev_collapsed` | int | it is below 9 | 3,503 |
-| 12 | `.parent_lev` | int | non-zero | 52,848 |
-| 13 | `.dummy` | `true` | the heading was introduced to segment an output | 183 |
-| 14 | `.ocn_off` | `true` | the object has no citation number | 7,543 |
-| 15 | `.is_of_type` | string | always, in practice | 56,882 |
-| 16 | `.attrib` | string | set | 1 |
-| 17 | `.meta_lang` | string | the object carries its own language | 0 |
-| 18 | `.meta_syntax` | string | the object carries its own markup syntax | 0 |
-| 19 | `.sha256` | hex | the object's text digest is non-zero | 51,253 |
-| 20 | `.image` | record | once **per image**, in order of appearance | 63 |
-| 21 | `.indent` | 2 ints | either indent is non-zero: `base hang` | 7,583 |
-| 22 | `.bullet` | `true` | the object is a bullet item | 1,903 |
-| 23 | `.lang` | string | the text attributes carry a language | 0 |
-| 24 | `.has` | flags | any is set | 11,143 |
-| 25 | `.table_cols` | int | the object is a table with columns | 32 |
-| 26 | `.table_widths` | ints | the table has column widths | 32 |
-| 27 | `.table_aligns` | letters | the table has alignments (`l` `c` `r`) | 32 |
-| 28 | `.table_header` | `true` | the table has a header row | 29 |
-| 29 | `.code_linenumbers` | `true` | the code block is numbered | 1 |
-| 30 | `.stow_link` | url | once **per extracted link** | 1,013 |
-| 31 | `.segment` | string | the heading opens a segment | 3,355 |
-| 32 | `.anchor` | string | the anchor differs from the segment name | 68 |
-| 33 | `.segment_prev` | string | there is a previous segment | 1,022 |
-| 34 | `.segment_next` | string | there is a next segment | 1,022 |
-| 35 | `.heading_lev_anchor` | string | the heading has a level anchor tag | 1,854 |
-| 36 | `.segment_epub` | string | the heading opens an epub segment | 3,355 |
-| 37 | `.segment_html_is` | string | the object is inside an html segment | 47,024 |
-| 38 | `.segment_epub_is` | string | the object is inside an epub segment | 47,024 |
-| 39 | `.segment_lv4_is` | string | **and** it differs from `.segment_html_is` | 342 |
-| 40 | `.heading_ancestors_text` | 8 slots, `\|` joined, escaped | any slot is non-empty | 3,319 |
-| 41 | `.lev4_subtoc` | string | once **per sub-table-of-contents entry** | 1,887 |
-| 42 | `.anchor_tag` | string | once **per anchor tag** | 7,883 |
+| # | property | value | emitted when | count |
+|----|---------------------------|-------------------------------|----------------------------------------------------------------------------------------------------|--------|
+| 1 | `.identifier` | string | a non-heading has an identifier that is not its ocn (**otherwise it is the ocn**: model section 9) | 353 |
+| 2 | `.part` | string | the object belongs to a part (`frontmatter`, `body`, ...) | 56,882 |
+| 3 | `.section` | string | the object's own section differs from the block it sits in | 605 |
+| 4 | `.parent` | int | **every object except** the document root and `toc` entries | 53,631 |
+| 5 | `.last_descendant` | int | non-zero: the last ocn under this heading | 3,202 |
+| 6 | `.children` | ints | the heading has child headings | 827 |
+| 7 | `.ancestors` | 8 ints | any is non-zero | 53,595 |
+| 8 | `.ancestors_collapsed` | 8 ints | any is non-zero | 53,595 |
+| 9 | `.dom_status` | 8 ints | any is non-zero | 3,319 |
+| 10 | `.dom_status_collapsed` | 8 ints | any is non-zero | 3,319 |
+| 11 | `.heading_lev_collapsed` | int | it is below 9 | 3,503 |
+| 12 | `.parent_lev` | int | non-zero | 52,848 |
+| 13 | `.dummy` | `true` | the heading was introduced to segment an output | 183 |
+| 14 | `.ocn_off` | `true` | the object has no citation number | 7,543 |
+| 15 | `.is_of_type` | string | always, in practice | 56,882 |
+| 16 | `.attrib` | string | set | 1 |
+| 17 | `.meta_lang` | string | the object carries its own language | 0 |
+| 18 | `.meta_syntax` | string | the object carries its own markup syntax | 0 |
+| 19 | `.sha256` | hex | the object's text digest is non-zero | 51,253 |
+| 20 | `.image` | record | once **per image**, in order of appearance | 63 |
+| 21 | `.indent` | 2 ints | either indent is non-zero: `base hang` | 7,583 |
+| 22 | `.bullet` | `true` | the object is a bullet item | 1,903 |
+| 23 | `.lang` | string | the text attributes carry a language | 0 |
+| 24 | `.has` | flags | any is set | 11,143 |
+| 25 | `.table_cols` | int | the object is a table with columns | 32 |
+| 26 | `.table_widths` | ints | the table has column widths | 32 |
+| 27 | `.table_aligns` | letters | the table has alignments (`l` `c` `r`) | 32 |
+| 28 | `.table_header` | `true` | the table has a header row | 29 |
+| 29 | `.code_linenumbers` | `true` | the code block is numbered | 1 |
+| 30 | `.stow_link` | url | once **per extracted link** | 1,013 |
+| 31 | `.segment` | string | the heading opens a segment | 3,355 |
+| 32 | `.anchor` | string | the anchor differs from the segment name (**otherwise it is the segment name**: model section 9) | 68 |
+| 33 | `.segment_prev` | string | there is a previous segment | 1,022 |
+| 34 | `.segment_next` | string | there is a next segment | 1,022 |
+| 35 | `.heading_lev_anchor` | string | the heading has a level anchor tag | 1,854 |
+| 36 | `.segment_epub` | string | the heading opens an epub segment | 3,355 |
+| 37 | `.segment_html_is` | string | the object is inside an html segment | 47,024 |
+| 38 | `.segment_epub_is` | string | the object is inside an epub segment | 47,024 |
+| 39 | `.segment_lv4_is` | string | **and** it differs from `.segment_html_is` | 342 |
+| 40 | `.heading_ancestors_text` | 8 slots, `\|` joined, escaped | any slot is non-empty | 3,319 |
+| 41 | `.lev4_subtoc` | string | once **per sub-table-of-contents entry** | 1,887 |
+| 42 | `.anchor_tag` | string | once **per anchor tag** | 7,883 |
Three of these need a word here, the first two because the model behind
them is easy to misread (see [`ocda-model.md`](ocda-model.md) sections 5