diff options
| -rw-r--r-- | ocda-db-format-specification.md | 83 | ||||
| -rw-r--r-- | ocda-model.md | 80 | ||||
| -rw-r--r-- | org/ocda_specification.org | 260 | ||||
| -rw-r--r-- | ssp-format-specification.md | 97 |
4 files changed, 430 insertions, 90 deletions
diff --git a/ocda-db-format-specification.md b/ocda-db-format-specification.md index ebeeaa6..eacc618 100644 --- a/ocda-db-format-specification.md +++ b/ocda-db-format-specification.md @@ -101,6 +101,33 @@ written with `doc_id` null: `schema.name`, `schema.version` and `UNIQUE(doc_id, key)` does not constrain them, and a partial unique index over `key` alone `WHERE doc_id IS NULL` is what makes them singular. +#### 3.2.1 The key prefix carries what the `.ssp` puts in blocks + +The `.ssp` has four preamble blocks, `@source`, `@meta`, `@make` and +`@doc_has`. The database has **one** `metadata` table, and tells them apart +by a **prefix on the key**: + +| `.ssp` block | database key | +|--------------|--------------| +| `@meta` | the key as it stands: `title.main`, `creator.author`, ... | +| `@make` | `make.` + the key: `make.footer`, `make.doc_type` | +| `@doc_has` | `doc_has.` + the key: `doc_has.images`, `doc_has.tables` | +| `@source` | `source.` + the key: `source.language`, `source.digest` | + +**A reader must strip the prefix and restore the grouping**, or a writer +over it has to ask `metadata["make.footer"]` from a database and +`make["footer"]` from a `.ssp`, which is exactly the difference the two +serialisations exist not to make. + +`source.` also carries two keys with no `@source` counterpart, +`source.filename` and `source.ssp_digest`; the `.ssp` puts the first in its +`% Source:` comment and cannot hold the second, which is a digest *of* it. + +Values may carry leading or trailing whitespace here where the `.ssp` +cannot represent it, its `key: value` line being unable to distinguish a +value that begins with a space from the separator. **Strip on load**, as a +reader of the text form necessarily does. + ### 3.3 Fixed-width arrays are JSON `children`, `ancestors`, `ancestors_collapsed`, `dom_status`, @@ -108,6 +135,33 @@ over `key` alone `WHERE doc_id IS NULL` is what makes them singular. `table_aligns` are stored as **JSON arrays**, with validity checks, so that `json_extract`, `json_each` and `json_array_length` reach inside them. +> **Parse them as JSON.** A scan that strips `[`, `]`, `,` and `"` looks +> adequate and is wrong on the first escape: the encoder writes `/` as +> `\/`, so `FREE/OPEN-SOURCE SOFTWARE` comes back as +> `FREE\/OPEN-SOURCE SOFTWARE`. All 36 document-languages of the reference +> collection contain at least one, and the defect is invisible to any check +> that does not compare the field itself. +> +> Note also that `table_widths` is an array of **integers except where the +> columns divide evenly**, when it is `[33.3333,33.3333,33.3333]`. A decoder +> that accepts strings and integers and then falls back to a default turns +> a value it could not read into one that looks read. + +The **element types**, so that a decoder can be written to accept exactly +what occurs: + +| column | elements | +|--------|----------| +| `children`, `ancestors`, `ancestors_collapsed`, `dom_status`, `dom_status_collapsed` | integers | +| `table_widths` | integers, or floats where the columns divide evenly | +| `table_aligns` | strings, one of `l` `c` `r` | +| `heading_ancestors_text` | strings, eight of them, empty where that level is not an ancestor | + +The `.ssp` writes the first three groups **space separated** and +`heading_ancestors_text` **`|` separated**. The empty slots of the last are +significant: they say which levels a heading sits under, so the separators +carry as much as the text does. + The open-ended lists are **rows in the four child tables** instead of columns: images, links, anchor tags and sub-table-of-contents entries. Each child table carries `seq`, which preserves the order they appeared in. @@ -159,6 +213,35 @@ object's position within its section, and `(doc_id, section, seq)` is what identifies an object when `ocn` will not (see [`ocda-model.md`](ocda-model.md) section 3). +### 4.1 Where the two are not symmetric + +"Nothing is lost in either direction" is true of the model, and there are +three places where the *files* nonetheless differ. A reader must know all +three, because each of them made two independently written readers disagree. + +**1. The database states what the `.ssp` derives.** `identifier` and +`anchor` are written on every row. The `.ssp` omits each where it can be +worked out, the identifier being the ocn and the anchor the segment name +(sections 6.1 and 6.3 of +[`ssp-format-specification.md`](ssp-format-specification.md), section 9 of +[`ocda-model.md`](ocda-model.md)). In `free_culture.lawrence_lessig` that is +**1,314 identifiers and 69 anchors** stated here and absent there. Neither +file is the poorer; a reader that does not derive is. + +**2. `translation.ocn_aligned` is here and cannot be in a `.ssp`.** It says +whether the languages of a document share one numbering, and a `.ssp` is one +language. It is therefore a fact about **the file**, not about the document +in it, and it is the one such fact either serialisation carries. + +A reader should **keep it out of the document model**, for the same reason +the `schema.*` and `build.*` rows are kept out: a writer over the model must +not be able to tell which serialisation it was handed, and a field that +exists only when the input was a database is exactly that tell. The model +has no place for collection-wide facts yet; when it grows one, this belongs +there. + +**3. Whitespace and grouping**, as section 3.2.1 gives them. + ## 5. The `files` table: what it carries, and how | role | named by | `compression` | `data` holds | diff --git a/ocda-model.md b/ocda-model.md index f4bf245..065677a 100644 --- a/ocda-model.md +++ b/ocda-model.md @@ -39,9 +39,12 @@ the document is made of and the unit it is cited by. Every object carries: - an **object citation number** (OCN), section 3; +- an **identifier**, section 3.1; - a **kind** and a **type**, section 2.1; - its **place in the structure**: parent, ancestors, and for a heading its children and its last descendant, section 4; +- the **segment** it is in, and for a heading an **anchor**, sections 7 + and 7.1; - its **text**, in spine's internal representation; - a **digest** of that text. @@ -108,6 +111,24 @@ otherwise: its verses share one; `through_the_looking_glass` has 21 such cases. Key on position within a section when one row is wanted. +### 3.1 The identifier + +Every object also carries an **identifier**: the name by which an output +refers to it. An HTML anchor, a LaTeX `\hypertarget`, a link from a book +index entry to the object it indexes, are all written from the identifier. + +**The identifier of an object with a non-zero OCN is its OCN**, written as +text. That is the ordinary case and it is the reason a citation to object +42 can be turned into a link without consulting anything. + +An object with OCN `0` has no OCN to take an identifier from, so one is +assigned: `a1`, `a2`, `a3`, ... in document order over the objects that +need one. There are few of them, eleven in a book, and they are the +generated furniture of section 3. + +**A reader must arrive at the identifier whether or not the serialisation +wrote it down**; see section 9. + ## 4. Levels A document has **eight heading levels**, in two groups of four: @@ -188,6 +209,21 @@ both separately because the answers differ: Without the second, a reader cannot say where a paragraph belongs. The search database reads the second. +### 7.1 The anchor + +A heading carries an **anchor**: the name an output links to in order to +arrive at that heading. It is what a table of contents entry and a +cross reference point at. + +**An object's anchor is its segment name**, in the ordinary case, because a +heading that opens a segment is arrived at by arriving at the segment. It +differs only where a heading is not the opening of the segment it sits in: +69 of the 72 anchors in one reference document equal their segment and 3 do +not. + +**A reader must arrive at the anchor whether or not the serialisation wrote +it down**; see section 9. + ## 8. Digests The model is checkable at four levels, and the chain is what lets an @@ -203,7 +239,49 @@ each image ──> a digest of the image file, with its size in bytes and pixels An object's digest is over its text alone, so a change to one paragraph moves one digest and no other. -## 9. What the model does not hold +## 9. Derived values, and why a reader must derive them + +Two of the values above are **derived**: the identifier from the OCN +(section 3.1), and the anchor from the segment name (section 7.1). The +model holds them; a serialisation need not, and the text form does not. + +**A serialisation may omit a value it can compute.** The `.ssp` writes a +property only when it carries information, so it omits `.identifier` where +the identifier is the OCN and `.anchor` where the anchor is the segment +name. The `.ocda.db` is columns rather than lines and writes both +unconditionally. + +Both are conforming. What is **not** conforming is a reader that takes the +text form literally and hands on a model with those fields empty, because +that model is not the same model the database gives, and a writer over it +will produce different output from the two serialisations of one document. + +The size of the difference is easy to underestimate. In +`free_culture.lawrence_lessig` the `.ssp` omits **1,314 identifiers and 69 +anchors** that the `.ocda.db` states, which is nearly every object in the +document. + +So, on loading: + +| field | when absent | take | +|-------|-------------|------| +| identifier | OCN is non-zero | the OCN as text | +| identifier | OCN is `0` | whatever the serialisation gave; there is nothing to derive from | +| anchor | the object has a segment | the segment name | + +**This is a property of the model, not of either format**, which is why it +is stated here and not only in the two format specifications. It is stated +here *because* two readers written independently from these documents +disagreed for want of it: the rule was implied by the `.ssp`'s emission +conditions and nowhere asserted. + +It is also what **spine's own reader** does, and always did, which is why +the `.ocda.db` carries the derived values: the database is written from the +`.ssp` read straight back, and the reading fills both in. So this section +documents existing behaviour rather than prescribing new. It should have +been written down first. + +## 10. What the model does not hold - **The markup.** The model is what the markup produced, not the markup itself. An artefact that carries both is the `.ocda.db`, which carries diff --git a/org/ocda_specification.org b/org/ocda_specification.org index 6b95365..9ab08a9 100644 --- a/org/ocda_specification.org +++ b/org/ocda_specification.org @@ -225,9 +225,12 @@ the document is made of and the unit it is cited by. Every object carries: - an **object citation number** (OCN), section 3; +- an **identifier**, section 3.1; - a **kind** and a **type**, section 2.1; - its **place in the structure**: parent, ancestors, and for a heading its children and its last descendant, section 4; +- the **segment** it is in, and for a heading an **anchor**, sections 7 + and 7.1; - its **text**, in spine's internal representation; - a **digest** of that text. @@ -294,6 +297,24 @@ otherwise: its verses share one; `through_the_looking_glass` has 21 such cases. Key on position within a section when one row is wanted. +### 3.1 The identifier + +Every object also carries an **identifier**: the name by which an output +refers to it. An HTML anchor, a LaTeX `\hypertarget`, a link from a book +index entry to the object it indexes, are all written from the identifier. + +,**The identifier of an object with a non-zero OCN is its OCN**, written as +text. That is the ordinary case and it is the reason a citation to object +42 can be turned into a link without consulting anything. + +An object with OCN `0` has no OCN to take an identifier from, so one is +assigned: `a1`, `a2`, `a3`, ... in document order over the objects that +need one. There are few of them, eleven in a book, and they are the +generated furniture of section 3. + +,**A reader must arrive at the identifier whether or not the serialisation +wrote it down**; see section 9. + ## 4. Levels A document has **eight heading levels**, in two groups of four: @@ -374,6 +395,21 @@ both separately because the answers differ: Without the second, a reader cannot say where a paragraph belongs. The search database reads the second. +### 7.1 The anchor + +A heading carries an **anchor**: the name an output links to in order to +arrive at that heading. It is what a table of contents entry and a +cross reference point at. + +,**An object's anchor is its segment name**, in the ordinary case, because a +heading that opens a segment is arrived at by arriving at the segment. It +differs only where a heading is not the opening of the segment it sits in: +69 of the 72 anchors in one reference document equal their segment and 3 do +not. + +,**A reader must arrive at the anchor whether or not the serialisation wrote +it down**; see section 9. + ## 8. Digests The model is checkable at four levels, and the chain is what lets an @@ -389,7 +425,49 @@ each image ──> a digest of the image file, with its size in bytes and pixels An object's digest is over its text alone, so a change to one paragraph moves one digest and no other. -## 9. What the model does not hold +## 9. Derived values, and why a reader must derive them + +Two of the values above are **derived**: the identifier from the OCN +(section 3.1), and the anchor from the segment name (section 7.1). The +model holds them; a serialisation need not, and the text form does not. + +,**A serialisation may omit a value it can compute.** The `.ssp` writes a +property only when it carries information, so it omits `.identifier` where +the identifier is the OCN and `.anchor` where the anchor is the segment +name. The `.ocda.db` is columns rather than lines and writes both +unconditionally. + +Both are conforming. What is **not** conforming is a reader that takes the +text form literally and hands on a model with those fields empty, because +that model is not the same model the database gives, and a writer over it +will produce different output from the two serialisations of one document. + +The size of the difference is easy to underestimate. In +`free_culture.lawrence_lessig` the `.ssp` omits **1,314 identifiers and 69 +anchors** that the `.ocda.db` states, which is nearly every object in the +document. + +So, on loading: + +| field | when absent | take | +|-------|-------------|------| +| identifier | OCN is non-zero | the OCN as text | +| identifier | OCN is `0` | whatever the serialisation gave; there is nothing to derive from | +| anchor | the object has a segment | the segment name | + +,**This is a property of the model, not of either format**, which is why it +is stated here and not only in the two format specifications. It is stated +here *because* two readers written independently from these documents +disagreed for want of it: the rule was implied by the `.ssp`'s emission +conditions and nowhere asserted. + +It is also what **spine's own reader** does, and always did, which is why +the `.ocda.db` carries the derived values: the database is written from the +`.ssp` read straight back, and the reading fills both in. So this section +documents existing behaviour rather than prescribing new. It should have +been written down first. + +## 10. What the model does not hold - **The markup.** The model is what the markup produced, not the markup itself. An artefact that carries both is the `.ocda.db`, which carries @@ -744,6 +822,15 @@ reference collection), and then its identifier, when it has one that is not simply the ocn. Every other kind carries an identifier, on the same condition, as the `.identifier` property. +> **A reader must derive the identifier where it is absent, and the anchor +> likewise.** "when it has one that is not simply the ocn" means the +> identifier *is* the ocn in every other case, and a reader that leaves the +> field empty gives a different model from the one the `.ocda.db` gives, on +> nearly every object. Section 9 of [`ocda-model.md`](ocda-model.md) states +> both derivations and why they belong to the model rather than to either +> format. Two readers written independently from these documents disagreed +> for want of it. + ### 6.2 Text lines ``` @@ -779,50 +866,50 @@ In the table, "emitted when" is the condition under which the property appears at all. Counts are occurrences across the 36 document reference set, as a guide to which parts of the format are load bearing. -| # | property | value | emitted when | count | -|----|---------------------------|-------------------------------|-------------------------------------------------------------|--------| -| 1 | `.identifier` | string | a non-heading has an identifier that is not its ocn | 353 | -| 2 | `.part` | string | the object belongs to a part (`frontmatter`, `body`, ...) | 56,882 | -| 3 | `.section` | string | the object's own section differs from the block it sits in | 605 | -| 4 | `.parent` | int | **every object except** the document root and `toc` entries | 53,631 | -| 5 | `.last_descendant` | int | non-zero: the last ocn under this heading | 3,202 | -| 6 | `.children` | ints | the heading has child headings | 827 | -| 7 | `.ancestors` | 8 ints | any is non-zero | 53,595 | -| 8 | `.ancestors_collapsed` | 8 ints | any is non-zero | 53,595 | -| 9 | `.dom_status` | 8 ints | any is non-zero | 3,319 | -| 10 | `.dom_status_collapsed` | 8 ints | any is non-zero | 3,319 | -| 11 | `.heading_lev_collapsed` | int | it is below 9 | 3,503 | -| 12 | `.parent_lev` | int | non-zero | 52,848 | -| 13 | `.dummy` | `true` | the heading was introduced to segment an output | 183 | -| 14 | `.ocn_off` | `true` | the object has no citation number | 7,543 | -| 15 | `.is_of_type` | string | always, in practice | 56,882 | -| 16 | `.attrib` | string | set | 1 | -| 17 | `.meta_lang` | string | the object carries its own language | 0 | -| 18 | `.meta_syntax` | string | the object carries its own markup syntax | 0 | -| 19 | `.sha256` | hex | the object's text digest is non-zero | 51,253 | -| 20 | `.image` | record | once **per image**, in order of appearance | 63 | -| 21 | `.indent` | 2 ints | either indent is non-zero: `base hang` | 7,583 | -| 22 | `.bullet` | `true` | the object is a bullet item | 1,903 | -| 23 | `.lang` | string | the text attributes carry a language | 0 | -| 24 | `.has` | flags | any is set | 11,143 | -| 25 | `.table_cols` | int | the object is a table with columns | 32 | -| 26 | `.table_widths` | ints | the table has column widths | 32 | -| 27 | `.table_aligns` | letters | the table has alignments (`l` `c` `r`) | 32 | -| 28 | `.table_header` | `true` | the table has a header row | 29 | -| 29 | `.code_linenumbers` | `true` | the code block is numbered | 1 | -| 30 | `.stow_link` | url | once **per extracted link** | 1,013 | -| 31 | `.segment` | string | the heading opens a segment | 3,355 | -| 32 | `.anchor` | string | the anchor differs from the segment name | 68 | -| 33 | `.segment_prev` | string | there is a previous segment | 1,022 | -| 34 | `.segment_next` | string | there is a next segment | 1,022 | -| 35 | `.heading_lev_anchor` | string | the heading has a level anchor tag | 1,854 | -| 36 | `.segment_epub` | string | the heading opens an epub segment | 3,355 | -| 37 | `.segment_html_is` | string | the object is inside an html segment | 47,024 | -| 38 | `.segment_epub_is` | string | the object is inside an epub segment | 47,024 | -| 39 | `.segment_lv4_is` | string | **and** it differs from `.segment_html_is` | 342 | -| 40 | `.heading_ancestors_text` | 8 slots, `\|` joined, escaped | any slot is non-empty | 3,319 | -| 41 | `.lev4_subtoc` | string | once **per sub-table-of-contents entry** | 1,887 | -| 42 | `.anchor_tag` | string | once **per anchor tag** | 7,883 | +| # | property | value | emitted when | count | +|----|---------------------------|-------------------------------|----------------------------------------------------------------------------------------------------|--------| +| 1 | `.identifier` | string | a non-heading has an identifier that is not its ocn (**otherwise it is the ocn**: model section 9) | 353 | +| 2 | `.part` | string | the object belongs to a part (`frontmatter`, `body`, ...) | 56,882 | +| 3 | `.section` | string | the object's own section differs from the block it sits in | 605 | +| 4 | `.parent` | int | **every object except** the document root and `toc` entries | 53,631 | +| 5 | `.last_descendant` | int | non-zero: the last ocn under this heading | 3,202 | +| 6 | `.children` | ints | the heading has child headings | 827 | +| 7 | `.ancestors` | 8 ints | any is non-zero | 53,595 | +| 8 | `.ancestors_collapsed` | 8 ints | any is non-zero | 53,595 | +| 9 | `.dom_status` | 8 ints | any is non-zero | 3,319 | +| 10 | `.dom_status_collapsed` | 8 ints | any is non-zero | 3,319 | +| 11 | `.heading_lev_collapsed` | int | it is below 9 | 3,503 | +| 12 | `.parent_lev` | int | non-zero | 52,848 | +| 13 | `.dummy` | `true` | the heading was introduced to segment an output | 183 | +| 14 | `.ocn_off` | `true` | the object has no citation number | 7,543 | +| 15 | `.is_of_type` | string | always, in practice | 56,882 | +| 16 | `.attrib` | string | set | 1 | +| 17 | `.meta_lang` | string | the object carries its own language | 0 | +| 18 | `.meta_syntax` | string | the object carries its own markup syntax | 0 | +| 19 | `.sha256` | hex | the object's text digest is non-zero | 51,253 | +| 20 | `.image` | record | once **per image**, in order of appearance | 63 | +| 21 | `.indent` | 2 ints | either indent is non-zero: `base hang` | 7,583 | +| 22 | `.bullet` | `true` | the object is a bullet item | 1,903 | +| 23 | `.lang` | string | the text attributes carry a language | 0 | +| 24 | `.has` | flags | any is set | 11,143 | +| 25 | `.table_cols` | int | the object is a table with columns | 32 | +| 26 | `.table_widths` | ints | the table has column widths | 32 | +| 27 | `.table_aligns` | letters | the table has alignments (`l` `c` `r`) | 32 | +| 28 | `.table_header` | `true` | the table has a header row | 29 | +| 29 | `.code_linenumbers` | `true` | the code block is numbered | 1 | +| 30 | `.stow_link` | url | once **per extracted link** | 1,013 | +| 31 | `.segment` | string | the heading opens a segment | 3,355 | +| 32 | `.anchor` | string | the anchor differs from the segment name (**otherwise it is the segment name**: model section 9) | 68 | +| 33 | `.segment_prev` | string | there is a previous segment | 1,022 | +| 34 | `.segment_next` | string | there is a next segment | 1,022 | +| 35 | `.heading_lev_anchor` | string | the heading has a level anchor tag | 1,854 | +| 36 | `.segment_epub` | string | the heading opens an epub segment | 3,355 | +| 37 | `.segment_html_is` | string | the object is inside an html segment | 47,024 | +| 38 | `.segment_epub_is` | string | the object is inside an epub segment | 47,024 | +| 39 | `.segment_lv4_is` | string | **and** it differs from `.segment_html_is` | 342 | +| 40 | `.heading_ancestors_text` | 8 slots, `\|` joined, escaped | any slot is non-empty | 3,319 | +| 41 | `.lev4_subtoc` | string | once **per sub-table-of-contents entry** | 1,887 | +| 42 | `.anchor_tag` | string | once **per anchor tag** | 7,883 | Three of these need a word here, the first two because the model behind them is easy to misread (see [`ocda-model.md`](ocda-model.md) sections 5 @@ -1082,6 +1169,33 @@ written with `doc_id` null: `schema.name`, `schema.version` and `UNIQUE(doc_id, key)` does not constrain them, and a partial unique index over `key` alone `WHERE doc_id IS NULL` is what makes them singular. +#### 3.2.1 The key prefix carries what the `.ssp` puts in blocks + +The `.ssp` has four preamble blocks, `@source`, `@meta`, `@make` and +`@doc_has`. The database has **one** `metadata` table, and tells them apart +by a **prefix on the key**: + +| `.ssp` block | database key | +|--------------|--------------| +| `@meta` | the key as it stands: `title.main`, `creator.author`, ... | +| `@make` | `make.` + the key: `make.footer`, `make.doc_type` | +| `@doc_has` | `doc_has.` + the key: `doc_has.images`, `doc_has.tables` | +| `@source` | `source.` + the key: `source.language`, `source.digest` | + +,**A reader must strip the prefix and restore the grouping**, or a writer +over it has to ask `metadata["make.footer"]` from a database and +`make["footer"]` from a `.ssp`, which is exactly the difference the two +serialisations exist not to make. + +`source.` also carries two keys with no `@source` counterpart, +`source.filename` and `source.ssp_digest`; the `.ssp` puts the first in its +`% Source:` comment and cannot hold the second, which is a digest *of* it. + +Values may carry leading or trailing whitespace here where the `.ssp` +cannot represent it, its `key: value` line being unable to distinguish a +value that begins with a space from the separator. **Strip on load**, as a +reader of the text form necessarily does. + ### 3.3 Fixed-width arrays are JSON `children`, `ancestors`, `ancestors_collapsed`, `dom_status`, @@ -1089,6 +1203,33 @@ over `key` alone `WHERE doc_id IS NULL` is what makes them singular. `table_aligns` are stored as **JSON arrays**, with validity checks, so that `json_extract`, `json_each` and `json_array_length` reach inside them. +> **Parse them as JSON.** A scan that strips `[`, `]`, `,` and `"` looks +> adequate and is wrong on the first escape: the encoder writes `/` as +> `\/`, so `FREE/OPEN-SOURCE SOFTWARE` comes back as +> `FREE\/OPEN-SOURCE SOFTWARE`. All 36 document-languages of the reference +> collection contain at least one, and the defect is invisible to any check +> that does not compare the field itself. +> +> Note also that `table_widths` is an array of **integers except where the +> columns divide evenly**, when it is `[33.3333,33.3333,33.3333]`. A decoder +> that accepts strings and integers and then falls back to a default turns +> a value it could not read into one that looks read. + +The **element types**, so that a decoder can be written to accept exactly +what occurs: + +| column | elements | +|--------|----------| +| `children`, `ancestors`, `ancestors_collapsed`, `dom_status`, `dom_status_collapsed` | integers | +| `table_widths` | integers, or floats where the columns divide evenly | +| `table_aligns` | strings, one of `l` `c` `r` | +| `heading_ancestors_text` | strings, eight of them, empty where that level is not an ancestor | + +The `.ssp` writes the first three groups **space separated** and +`heading_ancestors_text` **`|` separated**. The empty slots of the last are +significant: they say which levels a heading sits under, so the separators +carry as much as the text does. + The open-ended lists are **rows in the four child tables** instead of columns: images, links, anchor tags and sub-table-of-contents entries. Each child table carries `seq`, which preserves the order they appeared in. @@ -1140,6 +1281,35 @@ object's position within its section, and `(doc_id, section, seq)` is what identifies an object when `ocn` will not (see [`ocda-model.md`](ocda-model.md) section 3). +### 4.1 Where the two are not symmetric + +"Nothing is lost in either direction" is true of the model, and there are +three places where the *files* nonetheless differ. A reader must know all +three, because each of them made two independently written readers disagree. + +,**1. The database states what the `.ssp` derives.** `identifier` and +`anchor` are written on every row. The `.ssp` omits each where it can be +worked out, the identifier being the ocn and the anchor the segment name +(sections 6.1 and 6.3 of +[`ssp-format-specification.md`](ssp-format-specification.md), section 9 of +[`ocda-model.md`](ocda-model.md)). In `free_culture.lawrence_lessig` that is +,**1,314 identifiers and 69 anchors** stated here and absent there. Neither +file is the poorer; a reader that does not derive is. + +,**2. `translation.ocn_aligned` is here and cannot be in a `.ssp`.** It says +whether the languages of a document share one numbering, and a `.ssp` is one +language. It is therefore a fact about **the file**, not about the document +in it, and it is the one such fact either serialisation carries. + +A reader should **keep it out of the document model**, for the same reason +the `schema.*` and `build.*` rows are kept out: a writer over the model must +not be able to tell which serialisation it was handed, and a field that +exists only when the input was a database is exactly that tell. The model +has no place for collection-wide facts yet; when it grows one, this belongs +there. + +,**3. Whitespace and grouping**, as section 3.2.1 gives them. + ## 5. The `files` table: what it carries, and how | role | named by | `compression` | `data` holds | diff --git a/ssp-format-specification.md b/ssp-format-specification.md index 17974a3..ae1ecaf 100644 --- a/ssp-format-specification.md +++ b/ssp-format-specification.md @@ -338,6 +338,15 @@ reference collection), and then its identifier, when it has one that is not simply the ocn. Every other kind carries an identifier, on the same condition, as the `.identifier` property. +> **A reader must derive the identifier where it is absent, and the anchor +> likewise.** "when it has one that is not simply the ocn" means the +> identifier *is* the ocn in every other case, and a reader that leaves the +> field empty gives a different model from the one the `.ocda.db` gives, on +> nearly every object. Section 9 of [`ocda-model.md`](ocda-model.md) states +> both derivations and why they belong to the model rather than to either +> format. Two readers written independently from these documents disagreed +> for want of it. + ### 6.2 Text lines ``` @@ -373,50 +382,50 @@ In the table, "emitted when" is the condition under which the property appears at all. Counts are occurrences across the 36 document reference set, as a guide to which parts of the format are load bearing. -| # | property | value | emitted when | count | -|----|---------------------------|-------------------------------|-------------------------------------------------------------|--------| -| 1 | `.identifier` | string | a non-heading has an identifier that is not its ocn | 353 | -| 2 | `.part` | string | the object belongs to a part (`frontmatter`, `body`, ...) | 56,882 | -| 3 | `.section` | string | the object's own section differs from the block it sits in | 605 | -| 4 | `.parent` | int | **every object except** the document root and `toc` entries | 53,631 | -| 5 | `.last_descendant` | int | non-zero: the last ocn under this heading | 3,202 | -| 6 | `.children` | ints | the heading has child headings | 827 | -| 7 | `.ancestors` | 8 ints | any is non-zero | 53,595 | -| 8 | `.ancestors_collapsed` | 8 ints | any is non-zero | 53,595 | -| 9 | `.dom_status` | 8 ints | any is non-zero | 3,319 | -| 10 | `.dom_status_collapsed` | 8 ints | any is non-zero | 3,319 | -| 11 | `.heading_lev_collapsed` | int | it is below 9 | 3,503 | -| 12 | `.parent_lev` | int | non-zero | 52,848 | -| 13 | `.dummy` | `true` | the heading was introduced to segment an output | 183 | -| 14 | `.ocn_off` | `true` | the object has no citation number | 7,543 | -| 15 | `.is_of_type` | string | always, in practice | 56,882 | -| 16 | `.attrib` | string | set | 1 | -| 17 | `.meta_lang` | string | the object carries its own language | 0 | -| 18 | `.meta_syntax` | string | the object carries its own markup syntax | 0 | -| 19 | `.sha256` | hex | the object's text digest is non-zero | 51,253 | -| 20 | `.image` | record | once **per image**, in order of appearance | 63 | -| 21 | `.indent` | 2 ints | either indent is non-zero: `base hang` | 7,583 | -| 22 | `.bullet` | `true` | the object is a bullet item | 1,903 | -| 23 | `.lang` | string | the text attributes carry a language | 0 | -| 24 | `.has` | flags | any is set | 11,143 | -| 25 | `.table_cols` | int | the object is a table with columns | 32 | -| 26 | `.table_widths` | ints | the table has column widths | 32 | -| 27 | `.table_aligns` | letters | the table has alignments (`l` `c` `r`) | 32 | -| 28 | `.table_header` | `true` | the table has a header row | 29 | -| 29 | `.code_linenumbers` | `true` | the code block is numbered | 1 | -| 30 | `.stow_link` | url | once **per extracted link** | 1,013 | -| 31 | `.segment` | string | the heading opens a segment | 3,355 | -| 32 | `.anchor` | string | the anchor differs from the segment name | 68 | -| 33 | `.segment_prev` | string | there is a previous segment | 1,022 | -| 34 | `.segment_next` | string | there is a next segment | 1,022 | -| 35 | `.heading_lev_anchor` | string | the heading has a level anchor tag | 1,854 | -| 36 | `.segment_epub` | string | the heading opens an epub segment | 3,355 | -| 37 | `.segment_html_is` | string | the object is inside an html segment | 47,024 | -| 38 | `.segment_epub_is` | string | the object is inside an epub segment | 47,024 | -| 39 | `.segment_lv4_is` | string | **and** it differs from `.segment_html_is` | 342 | -| 40 | `.heading_ancestors_text` | 8 slots, `\|` joined, escaped | any slot is non-empty | 3,319 | -| 41 | `.lev4_subtoc` | string | once **per sub-table-of-contents entry** | 1,887 | -| 42 | `.anchor_tag` | string | once **per anchor tag** | 7,883 | +| # | property | value | emitted when | count | +|----|---------------------------|-------------------------------|----------------------------------------------------------------------------------------------------|--------| +| 1 | `.identifier` | string | a non-heading has an identifier that is not its ocn (**otherwise it is the ocn**: model section 9) | 353 | +| 2 | `.part` | string | the object belongs to a part (`frontmatter`, `body`, ...) | 56,882 | +| 3 | `.section` | string | the object's own section differs from the block it sits in | 605 | +| 4 | `.parent` | int | **every object except** the document root and `toc` entries | 53,631 | +| 5 | `.last_descendant` | int | non-zero: the last ocn under this heading | 3,202 | +| 6 | `.children` | ints | the heading has child headings | 827 | +| 7 | `.ancestors` | 8 ints | any is non-zero | 53,595 | +| 8 | `.ancestors_collapsed` | 8 ints | any is non-zero | 53,595 | +| 9 | `.dom_status` | 8 ints | any is non-zero | 3,319 | +| 10 | `.dom_status_collapsed` | 8 ints | any is non-zero | 3,319 | +| 11 | `.heading_lev_collapsed` | int | it is below 9 | 3,503 | +| 12 | `.parent_lev` | int | non-zero | 52,848 | +| 13 | `.dummy` | `true` | the heading was introduced to segment an output | 183 | +| 14 | `.ocn_off` | `true` | the object has no citation number | 7,543 | +| 15 | `.is_of_type` | string | always, in practice | 56,882 | +| 16 | `.attrib` | string | set | 1 | +| 17 | `.meta_lang` | string | the object carries its own language | 0 | +| 18 | `.meta_syntax` | string | the object carries its own markup syntax | 0 | +| 19 | `.sha256` | hex | the object's text digest is non-zero | 51,253 | +| 20 | `.image` | record | once **per image**, in order of appearance | 63 | +| 21 | `.indent` | 2 ints | either indent is non-zero: `base hang` | 7,583 | +| 22 | `.bullet` | `true` | the object is a bullet item | 1,903 | +| 23 | `.lang` | string | the text attributes carry a language | 0 | +| 24 | `.has` | flags | any is set | 11,143 | +| 25 | `.table_cols` | int | the object is a table with columns | 32 | +| 26 | `.table_widths` | ints | the table has column widths | 32 | +| 27 | `.table_aligns` | letters | the table has alignments (`l` `c` `r`) | 32 | +| 28 | `.table_header` | `true` | the table has a header row | 29 | +| 29 | `.code_linenumbers` | `true` | the code block is numbered | 1 | +| 30 | `.stow_link` | url | once **per extracted link** | 1,013 | +| 31 | `.segment` | string | the heading opens a segment | 3,355 | +| 32 | `.anchor` | string | the anchor differs from the segment name (**otherwise it is the segment name**: model section 9) | 68 | +| 33 | `.segment_prev` | string | there is a previous segment | 1,022 | +| 34 | `.segment_next` | string | there is a next segment | 1,022 | +| 35 | `.heading_lev_anchor` | string | the heading has a level anchor tag | 1,854 | +| 36 | `.segment_epub` | string | the heading opens an epub segment | 3,355 | +| 37 | `.segment_html_is` | string | the object is inside an html segment | 47,024 | +| 38 | `.segment_epub_is` | string | the object is inside an epub segment | 47,024 | +| 39 | `.segment_lv4_is` | string | **and** it differs from `.segment_html_is` | 342 | +| 40 | `.heading_ancestors_text` | 8 slots, `\|` joined, escaped | any slot is non-empty | 3,319 | +| 41 | `.lev4_subtoc` | string | once **per sub-table-of-contents entry** | 1,887 | +| 42 | `.anchor_tag` | string | once **per anchor tag** | 7,883 | Three of these need a word here, the first two because the model behind them is easy to misread (see [`ocda-model.md`](ocda-model.md) sections 5 |
