summaryrefslogtreecommitdiffhomepage
path: root/org/ocda_specification.org
diff options
context:
space:
mode:
Diffstat (limited to 'org/ocda_specification.org')
-rw-r--r--org/ocda_specification.org260
1 files changed, 215 insertions, 45 deletions
diff --git a/org/ocda_specification.org b/org/ocda_specification.org
index 6b95365..9ab08a9 100644
--- a/org/ocda_specification.org
+++ b/org/ocda_specification.org
@@ -225,9 +225,12 @@ the document is made of and the unit it is cited by.
Every object carries:
- an **object citation number** (OCN), section 3;
+- an **identifier**, section 3.1;
- a **kind** and a **type**, section 2.1;
- its **place in the structure**: parent, ancestors, and for a heading its
children and its last descendant, section 4;
+- the **segment** it is in, and for a heading an **anchor**, sections 7
+ and 7.1;
- its **text**, in spine's internal representation;
- a **digest** of that text.
@@ -294,6 +297,24 @@ otherwise:
its verses share one; `through_the_looking_glass` has 21 such cases. Key
on position within a section when one row is wanted.
+### 3.1 The identifier
+
+Every object also carries an **identifier**: the name by which an output
+refers to it. An HTML anchor, a LaTeX `\hypertarget`, a link from a book
+index entry to the object it indexes, are all written from the identifier.
+
+,**The identifier of an object with a non-zero OCN is its OCN**, written as
+text. That is the ordinary case and it is the reason a citation to object
+42 can be turned into a link without consulting anything.
+
+An object with OCN `0` has no OCN to take an identifier from, so one is
+assigned: `a1`, `a2`, `a3`, ... in document order over the objects that
+need one. There are few of them, eleven in a book, and they are the
+generated furniture of section 3.
+
+,**A reader must arrive at the identifier whether or not the serialisation
+wrote it down**; see section 9.
+
## 4. Levels
A document has **eight heading levels**, in two groups of four:
@@ -374,6 +395,21 @@ both separately because the answers differ:
Without the second, a reader cannot say where a paragraph belongs. The
search database reads the second.
+### 7.1 The anchor
+
+A heading carries an **anchor**: the name an output links to in order to
+arrive at that heading. It is what a table of contents entry and a
+cross reference point at.
+
+,**An object's anchor is its segment name**, in the ordinary case, because a
+heading that opens a segment is arrived at by arriving at the segment. It
+differs only where a heading is not the opening of the segment it sits in:
+69 of the 72 anchors in one reference document equal their segment and 3 do
+not.
+
+,**A reader must arrive at the anchor whether or not the serialisation wrote
+it down**; see section 9.
+
## 8. Digests
The model is checkable at four levels, and the chain is what lets an
@@ -389,7 +425,49 @@ each image ──> a digest of the image file, with its size in bytes and pixels
An object's digest is over its text alone, so a change to one paragraph
moves one digest and no other.
-## 9. What the model does not hold
+## 9. Derived values, and why a reader must derive them
+
+Two of the values above are **derived**: the identifier from the OCN
+(section 3.1), and the anchor from the segment name (section 7.1). The
+model holds them; a serialisation need not, and the text form does not.
+
+,**A serialisation may omit a value it can compute.** The `.ssp` writes a
+property only when it carries information, so it omits `.identifier` where
+the identifier is the OCN and `.anchor` where the anchor is the segment
+name. The `.ocda.db` is columns rather than lines and writes both
+unconditionally.
+
+Both are conforming. What is **not** conforming is a reader that takes the
+text form literally and hands on a model with those fields empty, because
+that model is not the same model the database gives, and a writer over it
+will produce different output from the two serialisations of one document.
+
+The size of the difference is easy to underestimate. In
+`free_culture.lawrence_lessig` the `.ssp` omits **1,314 identifiers and 69
+anchors** that the `.ocda.db` states, which is nearly every object in the
+document.
+
+So, on loading:
+
+| field | when absent | take |
+|-------|-------------|------|
+| identifier | OCN is non-zero | the OCN as text |
+| identifier | OCN is `0` | whatever the serialisation gave; there is nothing to derive from |
+| anchor | the object has a segment | the segment name |
+
+,**This is a property of the model, not of either format**, which is why it
+is stated here and not only in the two format specifications. It is stated
+here *because* two readers written independently from these documents
+disagreed for want of it: the rule was implied by the `.ssp`'s emission
+conditions and nowhere asserted.
+
+It is also what **spine's own reader** does, and always did, which is why
+the `.ocda.db` carries the derived values: the database is written from the
+`.ssp` read straight back, and the reading fills both in. So this section
+documents existing behaviour rather than prescribing new. It should have
+been written down first.
+
+## 10. What the model does not hold
- **The markup.** The model is what the markup produced, not the markup
itself. An artefact that carries both is the `.ocda.db`, which carries
@@ -744,6 +822,15 @@ reference collection), and then its identifier, when it has one
that is not simply the ocn. Every other kind carries an identifier, on the
same condition, as the `.identifier` property.
+> **A reader must derive the identifier where it is absent, and the anchor
+> likewise.** "when it has one that is not simply the ocn" means the
+> identifier *is* the ocn in every other case, and a reader that leaves the
+> field empty gives a different model from the one the `.ocda.db` gives, on
+> nearly every object. Section 9 of [`ocda-model.md`](ocda-model.md) states
+> both derivations and why they belong to the model rather than to either
+> format. Two readers written independently from these documents disagreed
+> for want of it.
+
### 6.2 Text lines
```
@@ -779,50 +866,50 @@ In the table, "emitted when" is the condition under which the property
appears at all. Counts are occurrences across the 36 document reference
set, as a guide to which parts of the format are load bearing.
-| # | property | value | emitted when | count |
-|----|---------------------------|-------------------------------|-------------------------------------------------------------|--------|
-| 1 | `.identifier` | string | a non-heading has an identifier that is not its ocn | 353 |
-| 2 | `.part` | string | the object belongs to a part (`frontmatter`, `body`, ...) | 56,882 |
-| 3 | `.section` | string | the object's own section differs from the block it sits in | 605 |
-| 4 | `.parent` | int | **every object except** the document root and `toc` entries | 53,631 |
-| 5 | `.last_descendant` | int | non-zero: the last ocn under this heading | 3,202 |
-| 6 | `.children` | ints | the heading has child headings | 827 |
-| 7 | `.ancestors` | 8 ints | any is non-zero | 53,595 |
-| 8 | `.ancestors_collapsed` | 8 ints | any is non-zero | 53,595 |
-| 9 | `.dom_status` | 8 ints | any is non-zero | 3,319 |
-| 10 | `.dom_status_collapsed` | 8 ints | any is non-zero | 3,319 |
-| 11 | `.heading_lev_collapsed` | int | it is below 9 | 3,503 |
-| 12 | `.parent_lev` | int | non-zero | 52,848 |
-| 13 | `.dummy` | `true` | the heading was introduced to segment an output | 183 |
-| 14 | `.ocn_off` | `true` | the object has no citation number | 7,543 |
-| 15 | `.is_of_type` | string | always, in practice | 56,882 |
-| 16 | `.attrib` | string | set | 1 |
-| 17 | `.meta_lang` | string | the object carries its own language | 0 |
-| 18 | `.meta_syntax` | string | the object carries its own markup syntax | 0 |
-| 19 | `.sha256` | hex | the object's text digest is non-zero | 51,253 |
-| 20 | `.image` | record | once **per image**, in order of appearance | 63 |
-| 21 | `.indent` | 2 ints | either indent is non-zero: `base hang` | 7,583 |
-| 22 | `.bullet` | `true` | the object is a bullet item | 1,903 |
-| 23 | `.lang` | string | the text attributes carry a language | 0 |
-| 24 | `.has` | flags | any is set | 11,143 |
-| 25 | `.table_cols` | int | the object is a table with columns | 32 |
-| 26 | `.table_widths` | ints | the table has column widths | 32 |
-| 27 | `.table_aligns` | letters | the table has alignments (`l` `c` `r`) | 32 |
-| 28 | `.table_header` | `true` | the table has a header row | 29 |
-| 29 | `.code_linenumbers` | `true` | the code block is numbered | 1 |
-| 30 | `.stow_link` | url | once **per extracted link** | 1,013 |
-| 31 | `.segment` | string | the heading opens a segment | 3,355 |
-| 32 | `.anchor` | string | the anchor differs from the segment name | 68 |
-| 33 | `.segment_prev` | string | there is a previous segment | 1,022 |
-| 34 | `.segment_next` | string | there is a next segment | 1,022 |
-| 35 | `.heading_lev_anchor` | string | the heading has a level anchor tag | 1,854 |
-| 36 | `.segment_epub` | string | the heading opens an epub segment | 3,355 |
-| 37 | `.segment_html_is` | string | the object is inside an html segment | 47,024 |
-| 38 | `.segment_epub_is` | string | the object is inside an epub segment | 47,024 |
-| 39 | `.segment_lv4_is` | string | **and** it differs from `.segment_html_is` | 342 |
-| 40 | `.heading_ancestors_text` | 8 slots, `\|` joined, escaped | any slot is non-empty | 3,319 |
-| 41 | `.lev4_subtoc` | string | once **per sub-table-of-contents entry** | 1,887 |
-| 42 | `.anchor_tag` | string | once **per anchor tag** | 7,883 |
+| # | property | value | emitted when | count |
+|----|---------------------------|-------------------------------|----------------------------------------------------------------------------------------------------|--------|
+| 1 | `.identifier` | string | a non-heading has an identifier that is not its ocn (**otherwise it is the ocn**: model section 9) | 353 |
+| 2 | `.part` | string | the object belongs to a part (`frontmatter`, `body`, ...) | 56,882 |
+| 3 | `.section` | string | the object's own section differs from the block it sits in | 605 |
+| 4 | `.parent` | int | **every object except** the document root and `toc` entries | 53,631 |
+| 5 | `.last_descendant` | int | non-zero: the last ocn under this heading | 3,202 |
+| 6 | `.children` | ints | the heading has child headings | 827 |
+| 7 | `.ancestors` | 8 ints | any is non-zero | 53,595 |
+| 8 | `.ancestors_collapsed` | 8 ints | any is non-zero | 53,595 |
+| 9 | `.dom_status` | 8 ints | any is non-zero | 3,319 |
+| 10 | `.dom_status_collapsed` | 8 ints | any is non-zero | 3,319 |
+| 11 | `.heading_lev_collapsed` | int | it is below 9 | 3,503 |
+| 12 | `.parent_lev` | int | non-zero | 52,848 |
+| 13 | `.dummy` | `true` | the heading was introduced to segment an output | 183 |
+| 14 | `.ocn_off` | `true` | the object has no citation number | 7,543 |
+| 15 | `.is_of_type` | string | always, in practice | 56,882 |
+| 16 | `.attrib` | string | set | 1 |
+| 17 | `.meta_lang` | string | the object carries its own language | 0 |
+| 18 | `.meta_syntax` | string | the object carries its own markup syntax | 0 |
+| 19 | `.sha256` | hex | the object's text digest is non-zero | 51,253 |
+| 20 | `.image` | record | once **per image**, in order of appearance | 63 |
+| 21 | `.indent` | 2 ints | either indent is non-zero: `base hang` | 7,583 |
+| 22 | `.bullet` | `true` | the object is a bullet item | 1,903 |
+| 23 | `.lang` | string | the text attributes carry a language | 0 |
+| 24 | `.has` | flags | any is set | 11,143 |
+| 25 | `.table_cols` | int | the object is a table with columns | 32 |
+| 26 | `.table_widths` | ints | the table has column widths | 32 |
+| 27 | `.table_aligns` | letters | the table has alignments (`l` `c` `r`) | 32 |
+| 28 | `.table_header` | `true` | the table has a header row | 29 |
+| 29 | `.code_linenumbers` | `true` | the code block is numbered | 1 |
+| 30 | `.stow_link` | url | once **per extracted link** | 1,013 |
+| 31 | `.segment` | string | the heading opens a segment | 3,355 |
+| 32 | `.anchor` | string | the anchor differs from the segment name (**otherwise it is the segment name**: model section 9) | 68 |
+| 33 | `.segment_prev` | string | there is a previous segment | 1,022 |
+| 34 | `.segment_next` | string | there is a next segment | 1,022 |
+| 35 | `.heading_lev_anchor` | string | the heading has a level anchor tag | 1,854 |
+| 36 | `.segment_epub` | string | the heading opens an epub segment | 3,355 |
+| 37 | `.segment_html_is` | string | the object is inside an html segment | 47,024 |
+| 38 | `.segment_epub_is` | string | the object is inside an epub segment | 47,024 |
+| 39 | `.segment_lv4_is` | string | **and** it differs from `.segment_html_is` | 342 |
+| 40 | `.heading_ancestors_text` | 8 slots, `\|` joined, escaped | any slot is non-empty | 3,319 |
+| 41 | `.lev4_subtoc` | string | once **per sub-table-of-contents entry** | 1,887 |
+| 42 | `.anchor_tag` | string | once **per anchor tag** | 7,883 |
Three of these need a word here, the first two because the model behind
them is easy to misread (see [`ocda-model.md`](ocda-model.md) sections 5
@@ -1082,6 +1169,33 @@ written with `doc_id` null: `schema.name`, `schema.version` and
`UNIQUE(doc_id, key)` does not constrain them, and a partial unique index
over `key` alone `WHERE doc_id IS NULL` is what makes them singular.
+#### 3.2.1 The key prefix carries what the `.ssp` puts in blocks
+
+The `.ssp` has four preamble blocks, `@source`, `@meta`, `@make` and
+`@doc_has`. The database has **one** `metadata` table, and tells them apart
+by a **prefix on the key**:
+
+| `.ssp` block | database key |
+|--------------|--------------|
+| `@meta` | the key as it stands: `title.main`, `creator.author`, ... |
+| `@make` | `make.` + the key: `make.footer`, `make.doc_type` |
+| `@doc_has` | `doc_has.` + the key: `doc_has.images`, `doc_has.tables` |
+| `@source` | `source.` + the key: `source.language`, `source.digest` |
+
+,**A reader must strip the prefix and restore the grouping**, or a writer
+over it has to ask `metadata["make.footer"]` from a database and
+`make["footer"]` from a `.ssp`, which is exactly the difference the two
+serialisations exist not to make.
+
+`source.` also carries two keys with no `@source` counterpart,
+`source.filename` and `source.ssp_digest`; the `.ssp` puts the first in its
+`% Source:` comment and cannot hold the second, which is a digest *of* it.
+
+Values may carry leading or trailing whitespace here where the `.ssp`
+cannot represent it, its `key: value` line being unable to distinguish a
+value that begins with a space from the separator. **Strip on load**, as a
+reader of the text form necessarily does.
+
### 3.3 Fixed-width arrays are JSON
`children`, `ancestors`, `ancestors_collapsed`, `dom_status`,
@@ -1089,6 +1203,33 @@ over `key` alone `WHERE doc_id IS NULL` is what makes them singular.
`table_aligns` are stored as **JSON arrays**, with validity checks, so that
`json_extract`, `json_each` and `json_array_length` reach inside them.
+> **Parse them as JSON.** A scan that strips `[`, `]`, `,` and `"` looks
+> adequate and is wrong on the first escape: the encoder writes `/` as
+> `\/`, so `FREE/OPEN-SOURCE SOFTWARE` comes back as
+> `FREE\/OPEN-SOURCE SOFTWARE`. All 36 document-languages of the reference
+> collection contain at least one, and the defect is invisible to any check
+> that does not compare the field itself.
+>
+> Note also that `table_widths` is an array of **integers except where the
+> columns divide evenly**, when it is `[33.3333,33.3333,33.3333]`. A decoder
+> that accepts strings and integers and then falls back to a default turns
+> a value it could not read into one that looks read.
+
+The **element types**, so that a decoder can be written to accept exactly
+what occurs:
+
+| column | elements |
+|--------|----------|
+| `children`, `ancestors`, `ancestors_collapsed`, `dom_status`, `dom_status_collapsed` | integers |
+| `table_widths` | integers, or floats where the columns divide evenly |
+| `table_aligns` | strings, one of `l` `c` `r` |
+| `heading_ancestors_text` | strings, eight of them, empty where that level is not an ancestor |
+
+The `.ssp` writes the first three groups **space separated** and
+`heading_ancestors_text` **`|` separated**. The empty slots of the last are
+significant: they say which levels a heading sits under, so the separators
+carry as much as the text does.
+
The open-ended lists are **rows in the four child tables** instead of
columns: images, links, anchor tags and sub-table-of-contents entries. Each
child table carries `seq`, which preserves the order they appeared in.
@@ -1140,6 +1281,35 @@ object's position within its section, and `(doc_id, section, seq)` is what
identifies an object when `ocn` will not (see
[`ocda-model.md`](ocda-model.md) section 3).
+### 4.1 Where the two are not symmetric
+
+"Nothing is lost in either direction" is true of the model, and there are
+three places where the *files* nonetheless differ. A reader must know all
+three, because each of them made two independently written readers disagree.
+
+,**1. The database states what the `.ssp` derives.** `identifier` and
+`anchor` are written on every row. The `.ssp` omits each where it can be
+worked out, the identifier being the ocn and the anchor the segment name
+(sections 6.1 and 6.3 of
+[`ssp-format-specification.md`](ssp-format-specification.md), section 9 of
+[`ocda-model.md`](ocda-model.md)). In `free_culture.lawrence_lessig` that is
+,**1,314 identifiers and 69 anchors** stated here and absent there. Neither
+file is the poorer; a reader that does not derive is.
+
+,**2. `translation.ocn_aligned` is here and cannot be in a `.ssp`.** It says
+whether the languages of a document share one numbering, and a `.ssp` is one
+language. It is therefore a fact about **the file**, not about the document
+in it, and it is the one such fact either serialisation carries.
+
+A reader should **keep it out of the document model**, for the same reason
+the `schema.*` and `build.*` rows are kept out: a writer over the model must
+not be able to tell which serialisation it was handed, and a field that
+exists only when the input was a database is exactly that tell. The model
+has no place for collection-wide facts yet; when it grows one, this belongs
+there.
+
+,**3. Whitespace and grouping**, as section 3.2.1 gives them.
+
## 5. The `files` table: what it carries, and how
| role | named by | `compression` | `data` holds |