diff options
| author | Ralph Amissah <ralph.amissah@gmail.com> | 2026-09-24 11:55:13 -0400 |
|---|---|---|
| committer | Ralph Amissah <ralph.amissah@gmail.com> | 2026-10-03 21:50:59 -0400 |
| commit | 0e307f24cfcbcb365a3fe2770a5ad718eeb41ca4 (patch) | |
| tree | 71e0a461c9ab35248924ab2a0e52461c3985ad55 | |
| parent | gaps: kinds, types, levels, make block, markers (diff) | |
.ocda.db specified & the two serialisations agree
Writing a validator from this document alone, as the test of whether
it suffices, turned up two rules that a reader needs and that were
implied rather than stated.
- Repeated occurrences of a property are consecutive: the four
repeatable properties share one position in the order and nothing is
written between them. Without that said, a validator checking order
cannot tell a legitimate repeat from a property out of place.
- And no property follows the text: the record is declaration, then
properties, then text lines, and once a text line has been written no
further property appears.
Both hold over the reference collection, and both are now stated with
the check that establishes them.
The third document, derived from the writer and checked against the
sample set's published databases: eight tables, one virtual table, three
views, fifty columns on objects.
The claim worth making and checked: nothing is lost in either direction
between the two serialisations. Every .ssp property is a column or a
child table, every column is a property, a value from the object
declaration line, or a database key. The mapping is given in full. Six
properties are renamed, the text form preferring the short name and
the database the explicit one; two are expanded, .indent into base and
hang and .has into five columns; and the four repeatable properties
become the four child tables. That is all that is different.
Which follows from how the database is made: the document is emitted
as .ssp, read straight back, and what comes back is written. So the
database cannot carry an object property the .ssp cannot express, and
the two cannot drift. It is also why they share one version number.
The compression column carries the literal none or zstd, so a reader
need not infer it from the role; images are the role that is stored
as read, png and jpeg being compressed already.
(assisted by Claude-Code)
| -rw-r--r-- | README.md | 14 | ||||
| -rw-r--r-- | ocda-db-format-specification.md | 317 | ||||
| -rw-r--r-- | ocda-model.md | 8 | ||||
| -rw-r--r-- | org/ocda_specification.org | 355 | ||||
| -rw-r--r-- | ssp-format-specification.md | 10 |
5 files changed, 686 insertions, 18 deletions
@@ -1,8 +1,12 @@ # sisudoc-abstraction-format -Specification for the **`.ssp`** text format - a human-readable, -PEG-parsable serialisation of the in-memory document abstraction -produced by [sisudoc-spine](https://git.sisudoc.org/sisudoc-spine). +Specification of the **object-centric document abstraction** that +[sisudoc-spine](https://git.sisudoc.org/sisudoc-spine) builds from +SiSU markup, and of the two forms it is written in: **`.ssp`**, text and +PEG-parsable, and **`.ocda.db`**, sqlite. + +The two carry one version number between them, because neither may express +what the other cannot. ## What this is @@ -14,8 +18,8 @@ A specification, not a program. The repository ships: *abstraction* and `.ssp` are told apart. - [`ssp-format-specification.md`](ssp-format-specification.md) - **the `.ssp`**: the model written as text. -- `ocda-db-format-specification.md` - **the `.ocda.db`**: the model written - as sqlite. To come. +- [`ocda-db-format-specification.md`](ocda-db-format-specification.md) - + **the `.ocda.db`**: the model written as sqlite. - [`example-autonomous-contract.ssp`](example-autonomous-contract.ssp) - spine's output for that sample document, complete and unedited. - [`example-wealth-of-networks.ssp`](example-wealth-of-networks.ssp) - diff --git a/ocda-db-format-specification.md b/ocda-db-format-specification.md new file mode 100644 index 0000000..ebeeaa6 --- /dev/null +++ b/ocda-db-format-specification.md @@ -0,0 +1,317 @@ +# The `.ocda.db` format + +**The object-centric document abstraction, written as sqlite. Format +version 2.0**, as spine 0.25.0 emits and reads it. + +This document specifies *one serialisation*. The model it serialises is +specified in [`ocda-model.md`](ocda-model.md), which the text serialisation +shares; read that first if the question is what an object *is* rather than +how it is stored. + +## Status + +Derived from the writer, `sisudoc/outputs/io_out/sqlite_ocda_db.d`, and +checked against the 27 published databases of the project's reference +collection. Where the two could differ the writer is normative and this +document is wrong. + +## 1. What it is, and what makes it different + +One file per **document**, holding **every language of it**: a ten language +document is one database with ten rows in `documents`, written one language +at a time into the one file. + +Two things distinguish it from the `.ssp`: + +- **It carries its own source.** The markup, the images, `conf/document_make`, + `pod.manifest` and the `tools/po4a` catalogues are all inside it, in the + `files` table. A pod can be written back out of it, and spine does exactly + that when asked for an artefact that needs markup. The `.ssp` names its + images and carries none of them. +- **It is queryable without being parsed.** A question about the document is + a SELECT, answered from the file with no spine and no reader. + +### 1.1 It is built from the `.ssp` + +The document is emitted as `.ssp`, **read straight back**, and what comes +back is what is written. So **the database cannot carry an object property +the `.ssp` cannot express**, and the two cannot drift. That is also why they +share one version number: a bump moves both. + +The carried *files* are the deliberate exception. They come from the pod +rather than from the abstraction, each with its digest. + +## 2. Version + +Declared in band, in the `metadata` table: + +```sql +SELECT key, value FROM metadata WHERE doc_id IS NULL; +-- build.spine_version | 0.25.0 +-- schema.name | sisu-abstraction-db +-- schema.version | 2.0 +``` + +The rule is the model's, and identical to the `.ssp`: the minor part is +raised when a property is added, the major when one changes meaning or goes +away, and **a reader checks the major part and refuses what it does not +know**. See [`ssp-format-specification.md`](ssp-format-specification.md) +section 2 for the history, which is shared. + +## 3. The schema + +Eight tables, one virtual table, three views. + +``` + documents id, lang, source_filename, ssp_digest UNIQUE(lang) + │ doc_id + ├─ metadata doc_id, key, value UNIQUE(doc_id, key) + │ + └─ objects id, doc_id, section, seq, ocn, ... 50 columns + UNIQUE(doc_id, section, seq) + ├─ object_images object_id, seq, name, bytes, sha256, + │ width, height, missing + ├─ object_links object_id, seq, url + ├─ object_anchors object_id, seq, anchor + └─ object_subtoc object_id, seq, entry + + files id, role, name, bytes, sha256, width, height, + compression, data UNIQUE(role, name) + + objects_fts fts5 over objects.text, external content over objects.id +``` + +### 3.1 `doc_id` is the language key + +`documents.id`, referred to as `doc_id` everywhere else, is **what tells one +language's rows from another's**. It is the key to reach for in every query. + +`objects.lang` and `objects.meta_lang` are **properties of the object** and +are not it. They are almost always empty (section 6). + +### 3.2 The file-level metadata rows + +`metadata` is keyed `UNIQUE(doc_id, key)` and not on key alone, because +every language has a title and a creator and a key-only key would refuse the +second. + +Three rows describe **the file** rather than a document in it, and are +written with `doc_id` null: `schema.name`, `schema.version` and +`build.spine_version`. In SQL no null equals another null, so +`UNIQUE(doc_id, key)` does not constrain them, and a partial unique index +over `key` alone `WHERE doc_id IS NULL` is what makes them singular. + +### 3.3 Fixed-width arrays are JSON + +`children`, `ancestors`, `ancestors_collapsed`, `dom_status`, +`dom_status_collapsed`, `heading_ancestors_text`, `table_widths` and +`table_aligns` are stored as **JSON arrays**, with validity checks, so that +`json_extract`, `json_each` and `json_array_length` reach inside them. + +The open-ended lists are **rows in the four child tables** instead of +columns: images, links, anchor tags and sub-table-of-contents entries. Each +child table carries `seq`, which preserves the order they appeared in. + +### 3.4 Indexes + +`objects(doc_id, section)`, `objects(ocn)`, `objects(parent_ocn)`, +`objects(is_a)`, `objects(heading_level)` where it is a heading, and +`files(role)`, `object_images(name)`. + +## 4. The relationship to the `.ssp`, column by column + +**Nothing is lost in either direction.** Every `.ssp` property is a column +or a child table; every column is a property, a value from the object +declaration line, or a database key. + +Where the names differ they differ for a reason, the `.ssp` preferring the +short form and the database the explicit one: + +| `.ssp` property | database | +|--------------------|-----------------------| +| `.parent` | `parent_ocn` | +| `.last_descendant` | `last_descendant_ocn` | +| `.dummy` | `dummy_heading` | +| `.ocn_off` | `object_number_off` | +| `.part` | `is_of_part` | +| `.section` | `is_of_section` | + +Two are **expanded**, the `.ssp` folding what the database spreads: + +| `.ssp` property | database | +|---------------------------------------------------------|-----------------------------------------------------------------------------------| +| `.indent: <base> <hang>` | `indent_base`, `indent_hang` | +| `.has: links notes_reg notes_star images images_no_dim` | `has_links`, `has_notes_reg`, `has_notes_star`, `has_images`, `has_images_no_dim` | + +The four **repeatable** properties become the four child tables: + +| `.ssp` property | database | +|-----------------|------------------| +| `.image` | `object_images` | +| `.stow_link` | `object_links` | +| `.anchor_tag` | `object_anchors` | +| `.lev4_subtoc` | `object_subtoc` | + +And four columns come from outside the property list: `ocn`, `is_a` and +`heading_level` from the object declaration line, and `text` from its `|` +lines. `id`, `doc_id` and `seq` are the database's own keys; `seq` is the +object's position within its section, and `(doc_id, section, seq)` is what +identifies an object when `ocn` will not (see +[`ocda-model.md`](ocda-model.md) section 3). + +## 5. The `files` table: what it carries, and how + +| role | named by | `compression` | `data` holds | +|------------|----------------------------|---------------|------------------| +| `image` | its bare filename | `none` | the file as read | +| `source` | `media/text/<lang>/<file>` | `zstd` | a zstd frame | +| `conf` | `conf/document_make` | `zstd` | a zstd frame | +| `manifest` | `pod.manifest` | `zstd` | a zstd frame | +| `tools` | `tools/po4a/...` | `zstd` | a zstd frame | + +The `compression` column carries the literal `none` or `zstd`, so a reader +need not infer it from the role. + +Three rules make the file predictable: + +1. **Compression is decided by role, never by size.** The same kind of file + always stores the same way, so nobody need measure a document to know + what its database will look like. A very small file can come out a few + bytes larger inside a zstd frame; that is accepted rather than fixed with + a threshold, which would make two documents of the same kind store + differently for no reason a reader could see. +2. **Images are not compressed.** png and jpeg are compressed already; zstd + gains nothing and costs a decompression on every read. +3. **`objects.text` is never compressed.** The full text index reads that + column directly, and compressing it would stop search working completely + rather than subtly. + +**`bytes` and `sha256` are always those of the original file**, whatever +`compression` says. So a digest check, a `digests.txt` line and a rebuilt +`source.digest` all keep working, and a reader that does not decompress can +still say what it is looking at. + +## 6. Querying it + +`sqlite3` is all that is needed. The recipes below were run against the +published databases. + +### 6.1 What is this file + +```sql +SELECT key, value FROM metadata WHERE doc_id IS NULL; +SELECT id, lang, source_filename FROM documents ORDER BY id; +SELECT role, count(*), sum(bytes) FROM files GROUP BY role; +``` + +### 6.2 The views + +| view | gives | +|------------------|---------------------------------------------------------------------------------------------| +| `outline` | the headings: lang, section, seq, ocn, both levels, `last_descendant_ocn`, identifier, text | +| `citable` | every object with an ocn, ordered by ocn, language named | +| `document_files` | what the file carries: role, name, bytes, sha256, pixel size | + +**`outline` and `citable` name the language and order by it first.** A view +over a file that may hold ten languages and does not say which would read as +one document and be several. + +**A trap in `citable`:** it is `SELECT d.lang AS lang, o.*`, and `objects` +has a `lang` column of its own, so sqlite renames the second one. The view +has **`lang`, the document's language, and `lang:1`, the object's own**. +Quote the second if you want it: `SELECT "lang:1" FROM citable`. + +### 6.3 Structure + +A heading's subtree is a **range**, not a traversal, because every heading +carries `last_descendant_ocn`: + +```sql +SELECT count(*) FROM objects o, + (SELECT doc_id, ocn, last_descendant_ocn FROM objects + WHERE is_a='heading' AND ocn=66) h + WHERE o.doc_id = h.doc_id + AND o.ocn BETWEEN h.ocn AND h.last_descendant_ocn; +``` + +And the path back up, on `parent_ocn`: + +```sql +WITH RECURSIVE up(ocn, parent_ocn, text) AS ( + SELECT ocn, parent_ocn, text FROM objects WHERE doc_id=1 AND ocn=345 + UNION ALL + SELECT o.ocn, o.parent_ocn, o.text FROM objects o + JOIN up ON o.ocn = up.parent_ocn AND o.doc_id=1 AND o.is_a='heading' + WHERE up.parent_ocn > 0) +SELECT ocn, substr(text,1,44) FROM up; +``` + +**Keep the `parent_ocn > 0` guard.** A parent of 0 is a real, +non-substantive parent and not a missing value, and without the guard the +recursion does not terminate. + +### 6.4 Full text + +```sql +SELECT o.ocn, substr(o.text,1,44) FROM objects_fts f + JOIN objects o ON o.id = f.rowid + JOIN documents d ON d.id = o.doc_id + WHERE f.text MATCH 'commons' AND d.lang='en' AND o.ocn > 0 + ORDER BY rank LIMIT 3; +``` + +fts5 syntax is available in full: `MATCH 'peer AND production'`, +`MATCH '"peer production"'`, `MATCH 'peer NEAR production'`. + +The index is **external content** over `objects.id`: the text is indexed, +not stored twice. + +### 6.5 Three things to know before trusting a query + +1. **An ocn is not unique within a document**, and `0` is shared by many + objects. Key on `objects.id`, or on `(doc_id, section, seq)`. +2. **`objects.text` is internal markup**, neither the author's markup nor + any output. Fine for search and for reading structure, not for display + without decoding. The markers are in + [`ssp-format-specification.md`](ssp-format-specification.md) section 3.2. +3. **A multi-language file answers an unfiltered query with every language + at once.** A single-language document has one `documents` row, so plain + queries read naturally there; anything else wants `WHERE lang='..'` or a + join to `documents`. + +## 7. Digests, and verifying a file + +``` +the markup files ──> metadata source.digest sha256 over every markup file + of that language +the .ssp ──> documents.ssp_digest of the .ssp it was built from +each object ──> objects.sha256 over the object's text +each carried file──> files.sha256 over the original bytes +the spine used ──> metadata build.spine_version +``` + +So `.sst -> .ssp -> .ocda.db` is checkable end to end, and +`spine --ocda-verify=<file>` closes the loop by writing the carried markup +back out and re-parsing it: **does this spine still make this abstraction +from the markup this file carries?** It writes nothing and the exit status +is the answer. + +**Only this artefact can answer that from itself alone**, which is the +strongest sense in which it is the one to work with. + +The database's own digest is taken once, when the last language has been +written, and appears in `digests.txt`. The metadata page links the database +and claims no hash for it, deliberately: the page is written per language +while the database is added to per language, so a digest taken there would +be of a file that no longer exists by the end of the run. + +## 8. Conformance + +A reader **must** check `schema.version`'s major part and refuse a major it +does not know; **must** treat `ocn` as not an identity; **must** stop at +`parent_ocn = 0` when walking upward; and **should** say what it did not +understand, for the reason given in +[`ssp-format-specification.md`](ssp-format-specification.md) section 8.2. + +A writer **must** keep `bytes` and `sha256` those of the original file +whatever `compression` says, and **must not** compress `objects.text`. diff --git a/ocda-model.md b/ocda-model.md index 1c916ca..f4bf245 100644 --- a/ocda-model.md +++ b/ocda-model.md @@ -6,10 +6,10 @@ of any file it is written to. Two serialisations carry this model, and they carry **one version number between them**, because neither may express what the other cannot: -| | | -|--------------------------------------------------------------|--------------------------------------------| -| [`ssp-format-specification.md`](ssp-format-specification.md) | the OCDA as text, `.ssp` | -| `ocda-db-format-specification.md` | the OCDA as sqlite, `.ocda.db` *(to come)* | +| | | +|----------------------------------------------------------------------|--------------------------------| +| [`ssp-format-specification.md`](ssp-format-specification.md) | the OCDA as text, `.ssp` | +| [`ocda-db-format-specification.md`](ocda-db-format-specification.md) | the OCDA as sqlite, `.ocda.db` | This document is what they share. It describes objects; they describe writing objects down. diff --git a/org/ocda_specification.org b/org/ocda_specification.org index bcb4b0b..442315a 100644 --- a/org/ocda_specification.org +++ b/org/ocda_specification.org @@ -21,9 +21,13 @@ #+BEGIN_SRC md # sisudoc-abstraction-format -Specification for the **`.ssp`** text format - a human-readable, -PEG-parsable serialisation of the in-memory document abstraction -produced by [sisudoc-spine](https://git.sisudoc.org/sisudoc-spine). +Specification of the **object-centric document abstraction** that +[sisudoc-spine](https://git.sisudoc.org/sisudoc-spine) builds from +SiSU markup, and of the two forms it is written in: **`.ssp`**, text and +PEG-parsable, and **`.ocda.db`**, sqlite. + +The two carry one version number between them, because neither may express +what the other cannot. ## What this is @@ -35,8 +39,8 @@ A specification, not a program. The repository ships: *abstraction* and `.ssp` are told apart. - [`ssp-format-specification.md`](ssp-format-specification.md) - **the `.ssp`**: the model written as text. -- `ocda-db-format-specification.md` - **the `.ocda.db`**: the model written - as sqlite. To come. +- [`ocda-db-format-specification.md`](ocda-db-format-specification.md) - + **the `.ocda.db`**: the model written as sqlite. - [`example-autonomous-contract.ssp`](example-autonomous-contract.ssp) - spine's output for that sample document, complete and unedited. - [`example-wealth-of-networks.ssp`](example-wealth-of-networks.ssp) - @@ -186,10 +190,10 @@ of any file it is written to. Two serialisations carry this model, and they carry **one version number between them**, because neither may express what the other cannot: -| | | -|--------------------------------------------------------------|--------------------------------------------| -| [`ssp-format-specification.md`](ssp-format-specification.md) | the OCDA as text, `.ssp` | -| `ocda-db-format-specification.md` | the OCDA as sqlite, `.ocda.db` *(to come)* | +| | | +|----------------------------------------------------------------------|--------------------------------| +| [`ssp-format-specification.md`](ssp-format-specification.md) | the OCDA as text, `.ssp` | +| [`ocda-db-format-specification.md`](ocda-db-format-specification.md) | the OCDA as sqlite, `.ocda.db` | This document is what they share. It describes objects; they describe writing objects down. @@ -759,6 +763,16 @@ meaningful and is defined per property in the table. Four properties repeat, appearing once per value rather than carrying a list: `.image`, `.stow_link`, `.lev4_subtoc` and `.anchor_tag`. +Two consequences of the order, which a validator needs stated and which +hold over the whole reference collection: + +- **a repeated property's occurrences are consecutive.** They share one + position in the order, and nothing is written between them. Checked: no + repeat in the collection is separated by another property. +- **no property follows the text.** The record is declaration, then + properties, then text lines, and once a `|` line has been written no + further property appears. Checked: 0 occurrences in the collection. + In the table, "emitted when" is the condition under which the property appears at all. Counts are occurrences across the 36 document reference set, as a guide to which parts of the format are load bearing. @@ -958,3 +972,326 @@ well, which this format does not. Each carries, in its opening comments, the command that regenerates it. #+END_SRC + +** ocda-db-format-specification.md + +#+HEADER: :tangle ../ocda-db-format-specification.md +#+BEGIN_SRC md +# The `.ocda.db` format + +,**The object-centric document abstraction, written as sqlite. Format +version 2.0**, as spine 0.25.0 emits and reads it. + +This document specifies *one serialisation*. The model it serialises is +specified in [`ocda-model.md`](ocda-model.md), which the text serialisation +shares; read that first if the question is what an object *is* rather than +how it is stored. + +## Status + +Derived from the writer, `sisudoc/outputs/io_out/sqlite_ocda_db.d`, and +checked against the 27 published databases of the project's reference +collection. Where the two could differ the writer is normative and this +document is wrong. + +## 1. What it is, and what makes it different + +One file per **document**, holding **every language of it**: a ten language +document is one database with ten rows in `documents`, written one language +at a time into the one file. + +Two things distinguish it from the `.ssp`: + +- **It carries its own source.** The markup, the images, `conf/document_make`, + `pod.manifest` and the `tools/po4a` catalogues are all inside it, in the + `files` table. A pod can be written back out of it, and spine does exactly + that when asked for an artefact that needs markup. The `.ssp` names its + images and carries none of them. +- **It is queryable without being parsed.** A question about the document is + a SELECT, answered from the file with no spine and no reader. + +### 1.1 It is built from the `.ssp` + +The document is emitted as `.ssp`, **read straight back**, and what comes +back is what is written. So **the database cannot carry an object property +the `.ssp` cannot express**, and the two cannot drift. That is also why they +share one version number: a bump moves both. + +The carried *files* are the deliberate exception. They come from the pod +rather than from the abstraction, each with its digest. + +## 2. Version + +Declared in band, in the `metadata` table: + +```sql +SELECT key, value FROM metadata WHERE doc_id IS NULL; +-- build.spine_version | 0.25.0 +-- schema.name | sisu-abstraction-db +-- schema.version | 2.0 +``` + +The rule is the model's, and identical to the `.ssp`: the minor part is +raised when a property is added, the major when one changes meaning or goes +away, and **a reader checks the major part and refuses what it does not +know**. See [`ssp-format-specification.md`](ssp-format-specification.md) +section 2 for the history, which is shared. + +## 3. The schema + +Eight tables, one virtual table, three views. + +``` + documents id, lang, source_filename, ssp_digest UNIQUE(lang) + │ doc_id + ├─ metadata doc_id, key, value UNIQUE(doc_id, key) + │ + └─ objects id, doc_id, section, seq, ocn, ... 50 columns + UNIQUE(doc_id, section, seq) + ├─ object_images object_id, seq, name, bytes, sha256, + │ width, height, missing + ├─ object_links object_id, seq, url + ├─ object_anchors object_id, seq, anchor + └─ object_subtoc object_id, seq, entry + + files id, role, name, bytes, sha256, width, height, + compression, data UNIQUE(role, name) + + objects_fts fts5 over objects.text, external content over objects.id +``` + +### 3.1 `doc_id` is the language key + +`documents.id`, referred to as `doc_id` everywhere else, is **what tells one +language's rows from another's**. It is the key to reach for in every query. + +`objects.lang` and `objects.meta_lang` are **properties of the object** and +are not it. They are almost always empty (section 6). + +### 3.2 The file-level metadata rows + +`metadata` is keyed `UNIQUE(doc_id, key)` and not on key alone, because +every language has a title and a creator and a key-only key would refuse the +second. + +Three rows describe **the file** rather than a document in it, and are +written with `doc_id` null: `schema.name`, `schema.version` and +`build.spine_version`. In SQL no null equals another null, so +`UNIQUE(doc_id, key)` does not constrain them, and a partial unique index +over `key` alone `WHERE doc_id IS NULL` is what makes them singular. + +### 3.3 Fixed-width arrays are JSON + +`children`, `ancestors`, `ancestors_collapsed`, `dom_status`, +`dom_status_collapsed`, `heading_ancestors_text`, `table_widths` and +`table_aligns` are stored as **JSON arrays**, with validity checks, so that +`json_extract`, `json_each` and `json_array_length` reach inside them. + +The open-ended lists are **rows in the four child tables** instead of +columns: images, links, anchor tags and sub-table-of-contents entries. Each +child table carries `seq`, which preserves the order they appeared in. + +### 3.4 Indexes + +`objects(doc_id, section)`, `objects(ocn)`, `objects(parent_ocn)`, +`objects(is_a)`, `objects(heading_level)` where it is a heading, and +`files(role)`, `object_images(name)`. + +## 4. The relationship to the `.ssp`, column by column + +,**Nothing is lost in either direction.** Every `.ssp` property is a column +or a child table; every column is a property, a value from the object +declaration line, or a database key. + +Where the names differ they differ for a reason, the `.ssp` preferring the +short form and the database the explicit one: + +| `.ssp` property | database | +|--------------------|-----------------------| +| `.parent` | `parent_ocn` | +| `.last_descendant` | `last_descendant_ocn` | +| `.dummy` | `dummy_heading` | +| `.ocn_off` | `object_number_off` | +| `.part` | `is_of_part` | +| `.section` | `is_of_section` | + +Two are **expanded**, the `.ssp` folding what the database spreads: + +| `.ssp` property | database | +|---------------------------------------------------------|-----------------------------------------------------------------------------------| +| `.indent: <base> <hang>` | `indent_base`, `indent_hang` | +| `.has: links notes_reg notes_star images images_no_dim` | `has_links`, `has_notes_reg`, `has_notes_star`, `has_images`, `has_images_no_dim` | + +The four **repeatable** properties become the four child tables: + +| `.ssp` property | database | +|-----------------|------------------| +| `.image` | `object_images` | +| `.stow_link` | `object_links` | +| `.anchor_tag` | `object_anchors` | +| `.lev4_subtoc` | `object_subtoc` | + +And four columns come from outside the property list: `ocn`, `is_a` and +`heading_level` from the object declaration line, and `text` from its `|` +lines. `id`, `doc_id` and `seq` are the database's own keys; `seq` is the +object's position within its section, and `(doc_id, section, seq)` is what +identifies an object when `ocn` will not (see +[`ocda-model.md`](ocda-model.md) section 3). + +## 5. The `files` table: what it carries, and how + +| role | named by | `compression` | `data` holds | +|------------|----------------------------|---------------|------------------| +| `image` | its bare filename | `none` | the file as read | +| `source` | `media/text/<lang>/<file>` | `zstd` | a zstd frame | +| `conf` | `conf/document_make` | `zstd` | a zstd frame | +| `manifest` | `pod.manifest` | `zstd` | a zstd frame | +| `tools` | `tools/po4a/...` | `zstd` | a zstd frame | + +The `compression` column carries the literal `none` or `zstd`, so a reader +need not infer it from the role. + +Three rules make the file predictable: + +1. **Compression is decided by role, never by size.** The same kind of file + always stores the same way, so nobody need measure a document to know + what its database will look like. A very small file can come out a few + bytes larger inside a zstd frame; that is accepted rather than fixed with + a threshold, which would make two documents of the same kind store + differently for no reason a reader could see. +2. **Images are not compressed.** png and jpeg are compressed already; zstd + gains nothing and costs a decompression on every read. +3. **`objects.text` is never compressed.** The full text index reads that + column directly, and compressing it would stop search working completely + rather than subtly. + +,**`bytes` and `sha256` are always those of the original file**, whatever +`compression` says. So a digest check, a `digests.txt` line and a rebuilt +`source.digest` all keep working, and a reader that does not decompress can +still say what it is looking at. + +## 6. Querying it + +`sqlite3` is all that is needed. The recipes below were run against the +published databases. + +### 6.1 What is this file + +```sql +SELECT key, value FROM metadata WHERE doc_id IS NULL; +SELECT id, lang, source_filename FROM documents ORDER BY id; +SELECT role, count(*), sum(bytes) FROM files GROUP BY role; +``` + +### 6.2 The views + +| view | gives | +|------------------|---------------------------------------------------------------------------------------------| +| `outline` | the headings: lang, section, seq, ocn, both levels, `last_descendant_ocn`, identifier, text | +| `citable` | every object with an ocn, ordered by ocn, language named | +| `document_files` | what the file carries: role, name, bytes, sha256, pixel size | + +,**`outline` and `citable` name the language and order by it first.** A view +over a file that may hold ten languages and does not say which would read as +one document and be several. + +,**A trap in `citable`:** it is `SELECT d.lang AS lang, o.*`, and `objects` +has a `lang` column of its own, so sqlite renames the second one. The view +has **`lang`, the document's language, and `lang:1`, the object's own**. +Quote the second if you want it: `SELECT "lang:1" FROM citable`. + +### 6.3 Structure + +A heading's subtree is a **range**, not a traversal, because every heading +carries `last_descendant_ocn`: + +```sql +SELECT count(*) FROM objects o, + (SELECT doc_id, ocn, last_descendant_ocn FROM objects + WHERE is_a='heading' AND ocn=66) h + WHERE o.doc_id = h.doc_id + AND o.ocn BETWEEN h.ocn AND h.last_descendant_ocn; +``` + +And the path back up, on `parent_ocn`: + +```sql +WITH RECURSIVE up(ocn, parent_ocn, text) AS ( + SELECT ocn, parent_ocn, text FROM objects WHERE doc_id=1 AND ocn=345 + UNION ALL + SELECT o.ocn, o.parent_ocn, o.text FROM objects o + JOIN up ON o.ocn = up.parent_ocn AND o.doc_id=1 AND o.is_a='heading' + WHERE up.parent_ocn > 0) +SELECT ocn, substr(text,1,44) FROM up; +``` + +,**Keep the `parent_ocn > 0` guard.** A parent of 0 is a real, +non-substantive parent and not a missing value, and without the guard the +recursion does not terminate. + +### 6.4 Full text + +```sql +SELECT o.ocn, substr(o.text,1,44) FROM objects_fts f + JOIN objects o ON o.id = f.rowid + JOIN documents d ON d.id = o.doc_id + WHERE f.text MATCH 'commons' AND d.lang='en' AND o.ocn > 0 + ORDER BY rank LIMIT 3; +``` + +fts5 syntax is available in full: `MATCH 'peer AND production'`, +`MATCH '"peer production"'`, `MATCH 'peer NEAR production'`. + +The index is **external content** over `objects.id`: the text is indexed, +not stored twice. + +### 6.5 Three things to know before trusting a query + +1. **An ocn is not unique within a document**, and `0` is shared by many + objects. Key on `objects.id`, or on `(doc_id, section, seq)`. +2. **`objects.text` is internal markup**, neither the author's markup nor + any output. Fine for search and for reading structure, not for display + without decoding. The markers are in + [`ssp-format-specification.md`](ssp-format-specification.md) section 3.2. +3. **A multi-language file answers an unfiltered query with every language + at once.** A single-language document has one `documents` row, so plain + queries read naturally there; anything else wants `WHERE lang='..'` or a + join to `documents`. + +## 7. Digests, and verifying a file + +``` +the markup files ──> metadata source.digest sha256 over every markup file + of that language +the .ssp ──> documents.ssp_digest of the .ssp it was built from +each object ──> objects.sha256 over the object's text +each carried file──> files.sha256 over the original bytes +the spine used ──> metadata build.spine_version +``` + +So `.sst -> .ssp -> .ocda.db` is checkable end to end, and +`spine --ocda-verify=<file>` closes the loop by writing the carried markup +back out and re-parsing it: **does this spine still make this abstraction +from the markup this file carries?** It writes nothing and the exit status +is the answer. + +,**Only this artefact can answer that from itself alone**, which is the +strongest sense in which it is the one to work with. + +The database's own digest is taken once, when the last language has been +written, and appears in `digests.txt`. The metadata page links the database +and claims no hash for it, deliberately: the page is written per language +while the database is added to per language, so a digest taken there would +be of a file that no longer exists by the end of the run. + +## 8. Conformance + +A reader **must** check `schema.version`'s major part and refuse a major it +does not know; **must** treat `ocn` as not an identity; **must** stop at +`parent_ocn = 0` when walking upward; and **should** say what it did not +understand, for the reason given in +[`ssp-format-specification.md`](ssp-format-specification.md) section 8.2. + +A writer **must** keep `bytes` and `sha256` those of the original file +whatever `compression` says, and **must not** compress `objects.text`. +#+END_SRC diff --git a/ssp-format-specification.md b/ssp-format-specification.md index 1f54aab..17974a3 100644 --- a/ssp-format-specification.md +++ b/ssp-format-specification.md @@ -359,6 +359,16 @@ meaningful and is defined per property in the table. Four properties repeat, appearing once per value rather than carrying a list: `.image`, `.stow_link`, `.lev4_subtoc` and `.anchor_tag`. +Two consequences of the order, which a validator needs stated and which +hold over the whole reference collection: + +- **a repeated property's occurrences are consecutive.** They share one + position in the order, and nothing is written between them. Checked: no + repeat in the collection is separated by another property. +- **no property follows the text.** The record is declaration, then + properties, then text lines, and once a `|` line has been written no + further property appears. Checked: 0 occurrences in the collection. + In the table, "emitted when" is the condition under which the property appears at all. Counts are occurrences across the 36 document reference set, as a guide to which parts of the format are load bearing. |
