summaryrefslogtreecommitdiffhomepage
path: root/ocda-model.md
blob: 1c173b1bfb1c4cb081f9e8272bb81a69e2283f1f (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
# The object-centric document abstraction (OCDA)

**The model.** What a document *is* once spine has parsed it, independent
of any file it is written to.

Two serialisations carry this model, and they carry **one version number
between them**, because neither may express what the other cannot:

|                                                                      |                                |
|----------------------------------------------------------------------|--------------------------------|
| [`ssp-format-specification.md`](ssp-format-specification.md)         | the OCDA as text, `.ssp`       |
| [`ocda-db-format-specification.md`](ocda-db-format-specification.md) | the OCDA as sqlite, `.ocda.db` |

This document is what they share. It describes objects; they describe
writing objects down.

## 1. Three words, and what each names

The terms are used precisely in this repository and in spine, and the
distinction is worth holding:

| level              | term                       | names                                                                                                                                                  |
|--------------------|----------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------|
| the model          | **OCDA**                   | what a document is once parsed: numbered objects with structure, identity and digests. Independent of any file, any stage, any tool                    |
| the stage          | **abstraction**            | the middle of spine's **markup -> abstraction -> output**: the act of building the model, and the point at which a document may now enter the pipeline |
| the serialisations | **`.ssp`**, **`.ocda.db`** | the model written down, as text and as sqlite                                                                                                          |

A thing is named for the level it belongs to. Something true of the model
whatever file it is in takes *ocda*; something about the pipeline stage
takes *abstraction*; something that reads or writes one file form takes
that form's name.

## 2. The object

A document is a sequence of **objects**, grouped into sections. An object
is a paragraph, a heading, a table, an endnote, a line of verse: the unit
the document is made of and the unit it is cited by.

Every object carries:

- an **object citation number** (OCN), section 3;
- an **identifier**, section 3.1;
- a **kind** and a **type**, section 2.1;
- its **place in the structure**: parent, ancestors, and for a heading its
  children and its last descendant, section 4;
- the **segment** it is in, and for a heading an **anchor**, sections 7
  and 7.1;
- its **text**, in spine's internal representation;
- a **digest** of that text.

### 2.1 Kind and type: two axes, not one

An object is classified twice, and the two answer different questions.

**Kind** (`is_a`) says *what the object is*, one of sixteen. Each kind
belongs to one section, except `heading`, which appears in all nine:

| kind           | section      | what it is                                                                     | count  |
|----------------|--------------|--------------------------------------------------------------------------------|--------|
| `para`         | body         | a paragraph, the ordinary unit of a document                                   | 41,885 |
| `heading`      | *all nine*   | a heading at one of the eight levels                                           | 3,503  |
| `toc`          | toc          | an entry in the generated table of contents                                    | 3,215  |
| `bookindex`    | bookindex    | an entry in the generated book index                                           | 3,034  |
| `endnote`      | endnotes     | a note, gathered from where it was called                                      | 2,294  |
| `code`         | body         | a code block                                                                   | 1,623  |
| `bibliography` | bibliography | a bibliography entry                                                           | 585    |
| `comment`      | body         | markup comment, carried and not rendered                                       | 450    |
| `verse`        | body         | one verse of a poem                                                            | 151    |
| `glossary`     | glossary     | a glossary entry                                                               | 56     |
| `quote`        | body         | a block quotation                                                              | 53     |
| `poem`         | body         | a poem: the container of its verses, OCN `0`, cited as their range (section 5) | 48     |
| `block`        | body         | a block of grouped text                                                        | 45     |
| `table`        | body         | a table, with its columns described                                            | 32     |
| `blurb`        | blurb        | a blurb entry                                                                  | 28     |
| `group`        | body         | a group block                                                                  | 16     |

**Type** (`is_of_type`) says *how the object's text behaves*, and takes
only three values:

| type      | means                                                   | kinds                                                                                 |
|-----------|---------------------------------------------------------|---------------------------------------------------------------------------------------|
| `para`    | flowing text, wrapped as the output sees fit            | `para`, `heading`, `toc`, `bookindex`, `endnote`, `bibliography`, `glossary`, `blurb` |
| `block`   | text whose line breaks are significant and must be kept | `code`, `verse`, `quote`, `poem`, `block`, `table`, `group`                           |
| `comment` | not rendered                                            | `comment`                                                                             |

The mapping from kind to type is fixed: every `code` object is of type
`block`, every `para` object of type `para`. **A renderer that has the type
does not need to know all sixteen kinds** to decide whether it may reflow
the text, which is what the second axis is for.

## 3. The object citation number

The OCN is the heart of the model and the reason for it.

**An object's OCN is the same in every format spine writes.** A paragraph
that is object 42 in the HTML is object 42 in the EPUB, in the PDF, in the
plain text, in the search database and in both serialisations of the
abstraction. A citation to object 42 of a document is therefore a citation
to a fixed piece of text, not to a page in one edition or a scroll position
in one rendering.

Three properties follow, and each is a trap for a reader that assumes
otherwise:

- **Not every object has one.** Generated furniture, and objects the markup
  has suppressed, carry `0`. There are many of them: 7,543 across the 36
  document reference collection.
- **`0` is therefore not an identity.** Objects sharing `0` are not the
  same object, and `0` must never be used as a key.
- **A non-zero OCN is unique within a document**, from format 2.1: checked
  over the reference collection, no non-zero OCN occurs twice in any of the
  36 document-languages. **Before 2.1 it was not**: a poem carried the OCN of
  its last verse, so the same number named two objects (48 such pairs in the
  collection, 21 in `through_the_looking_glass`). A reader of an older
  artefact keys on position within a section when one row is wanted.

### 3.1 The identifier

Every object also carries an **identifier**: the name by which an output
refers to it. An HTML anchor, a LaTeX `\hypertarget`, a link from a book
index entry to the object it indexes, are all written from the identifier.

**The identifier of an object with a non-zero OCN is its OCN**, written as
text. That is the ordinary case and it is the reason a citation to object
42 can be turned into a link without consulting anything.

An object with OCN `0` has no OCN to take an identifier from, so one is
assigned: `a1`, `a2`, `a3`, ... in document order over the objects that
need one. There are few of them, eleven in a book, and they are the
generated furniture of section 3.

**A reader must arrive at the identifier whether or not the serialisation
wrote it down**; see section 9.

## 4. Levels

A document has **eight heading levels**, in two groups of four:

| level | index | what it is                                   |
|-------|-------|----------------------------------------------|
| `A`   | 0     | the document title. Exactly one per document |
| `B`   | 1     |                                              |
| `C`   | 2     | the title matter below it                    |
| `D`   | 3     |                                              |
| `1`   | 4     | the top level of the document body           |
| `2`   | 5     |                                              |
| `3`   | 6     | below that                                   |
| `4`   | 7     |                                              |

`A` to `D` are the title levels, `1` to `4` the body levels, and the index
is what the eight-slot arrays of section 5 are indexed by. A heading that
claims no level has 9, which is the "not a heading level" value and is why
`.heading_lev_collapsed` is emitted only when it is below 9.

**Collapsed levels.** A document that uses `A` and then `1`, skipping `B`,
`C` and `D`, has a gap in its levels that is an artefact of the markup
rather than of the document. Every level-indexed array therefore comes in
two forms: as marked up, and **collapsed**, with the unused levels removed
so that the remaining ones are contiguous. Outputs that care about nesting
depth use the collapsed form; anything tracing back to the markup uses the
other.

## 5. Structure

Structure is carried by number, not by nesting: an object names its parent
rather than containing its children. This is what lets the serialisations
be flat and the model be walked in either direction.

- **`parent`** - the OCN of the object this one hangs from.
  **A parent of `0` is not "no parent"**: it is a parent heading the author
  has made non-substantive, its OCN suppressed or a dummy heading
  introduced to segment an output. The distinction is preserved
  deliberately, and a reader walking upward must stop at `0` rather than
  treat it as missing, or the walk will not terminate.
- **`ancestors`** - **eight slots, indexed by level** (section 4): the OCN
  of the heading at each level above this object, `0` where there is none,
  so the path to the root is available without walking it.
  **A heading's own OCN sits in its own level's slot**: checked over the
  reference collection, 3,432 headings of 3,432. So for a heading the array
  is its path including itself, and for any other object it is the path to
  the heading above it.
- **`ancestors_collapsed`** - the same, in collapsed levels.
- **`children`** - a heading's child headings.
- **`last_descendant`** - the last OCN under a heading, which makes a
  subtree a **range** rather than a traversal: everything from the
  heading's OCN to its last descendant is inside it.
- **`ocn_range`** - for a **container**, the first and last OCN of what it
  holds (2.1). A poem is the one container: its verses are the citable
  units, one object each, and the poem is not; it carries OCN `0` and no
  identifier, **comes immediately before its first verse**, and is cited as
  the range of them (`173-180`). So a writer that meets a poem knows what
  follows and where it ends, and can wrap the verses in one pass. The poem
  has no text of its own. `0 0`, or absent, where the object is not a
  container.
  **Before 2.1** the poem came *after* its last verse, carried that verse's
  OCN and identifier, and stated no range.

## 6. Sections

Objects are grouped into nine sections, in this order:

```
head  toc  body  endnotes  glossary  bibliography  bookindex  blurb  tail
```

`head` is the document's title matter, `body` the document proper, and the
rest are generated or gathered. `tail` is the end of document some output
writers use. A section with no objects is not carried.

## 7. Segments

An output may be one file or many. A **segment** is one of the many: a
chapter-sized piece of HTML or EPUB with its own name and its own
navigation.

Two different questions are asked about segments, and the model answers
both separately because the answers differ:

- **which segment does this heading open?** Set on headings only.
- **which segment is this object in?** Set on nearly every object.

Without the second, a reader cannot say where a paragraph belongs. The
search database reads the second.

### 7.1 The anchor

A heading carries an **anchor**: the name an output links to in order to
arrive at that heading. It is what a table of contents entry and a
cross reference point at.

**An object's anchor is its segment name**, in the ordinary case, because a
heading that opens a segment is arrived at by arriving at the segment. It
differs only where a heading is not the opening of the segment it sits in:
69 of the 72 anchors in one reference document equal their segment and 3 do
not.

**A reader must arrive at the anchor whether or not the serialisation wrote
it down**; see section 9.

## 8. Digests

The model is checkable at four levels, and the chain is what lets an
artefact be verified rather than trusted:

```
the markup files ──> a digest over every markup file of that language
the serialisation ──> a digest of the .ssp a database was built from
each object ──> a digest of the object's text
each image ──> a digest of the image file, with its size in bytes and pixels
```

An object's digest is over its text alone, so a change to one paragraph
moves one digest and no other.

## 9. Derived values, and why a reader must derive them

Two of the values above are **derived**: the identifier from the OCN
(section 3.1), and the anchor from the segment name (section 7.1). The
model holds them; a serialisation need not, and the text form does not.

**A serialisation may omit a value it can compute.** The `.ssp` writes a
property only when it carries information, so it omits `.identifier` where
the identifier is the OCN and `.anchor` where the anchor is the segment
name. The `.ocda.db` is columns rather than lines and writes both
unconditionally.

Both are conforming. What is **not** conforming is a reader that takes the
text form literally and hands on a model with those fields empty, because
that model is not the same model the database gives, and a writer over it
will produce different output from the two serialisations of one document.

The size of the difference is easy to underestimate. In
`free_culture.lawrence_lessig` the `.ssp` omits **1,314 identifiers and 69
anchors** that the `.ocda.db` states, which is nearly every object in the
document.

So, on loading:

| field      | when absent              | take                                                             |
|------------|--------------------------|------------------------------------------------------------------|
| identifier | OCN is non-zero          | the OCN as text                                                  |
| identifier | OCN is `0`               | whatever the serialisation gave; there is nothing to derive from |
| anchor     | the object has a segment | the segment name                                                 |

**This is a property of the model, not of either format**, which is why it
is stated here and not only in the two format specifications. It is stated
here *because* two readers written independently from these documents
disagreed for want of it: the rule was implied by the `.ssp`'s emission
conditions and nowhere asserted.

It is also what **spine's own reader** does, and always did, which is why
the `.ocda.db` carries the derived values: the database is written from the
`.ssp` read straight back, and the reading fills both in. So this section
documents existing behaviour rather than prescribing new. It should have
been written down first.

## 10. What the model does not hold

- **The markup.** The model is what the markup produced, not the markup
  itself. An artefact that carries both is the `.ocda.db`, which carries
  the markup alongside the model rather than within it.
- **The images.** The model *names* them, with digest, byte size and pixel
  size, and they must be found beside the file for a document to be
  complete.
- **Any output format.** No HTML, no page numbers, no typesetting.