From 628540bedcf62044ea5d371d87ae08fdaa83689c Mon Sep 17 00:00:00 2001 From: Ralph Amissah Date: Sat, 18 Apr 2026 20:28:29 -0400 Subject: sisudoc-abstraction-format 2.0: ocda-model & .ssp In an age of malleable, replaceable, improvable text, we continue to seek to represent set text in a fixed, determinate way. This is an effort, using an LLM (Anthropic's Claude Code), to extract and state the specification of the document abstraction of the program spine: - spine can write its document abstraction as a .ssp file, a PEG-parsable serialisation from which other forms of it, such as the sqlite .ocda.db, can be derived https://git.sisudoc.org/sisudoc-spine git clone --depth=1 git://git.sisudoc.org/software/sisudoc-spine - this repository states the specification of that abstraction, as the .ssp writes it, in enough detail and with enough precision for the abstraction to be read and written independently git clone --depth=1 git://git.sisudoc.org/specs/sisudoc-abstraction-format - the claim is tested separately: with readers and writers built from the specification alone, with no other knowledge of the format, that read and write the abstraction exactly The specification is derived from the spine emitter, sisudoc/ocda/abstraction/ssp.d, whose sspObjectRecord is the single definition of what an object looks like, and checked against the document markup sample set. Where the two differ the emitter is normative and the specification is wrong; the specification says so. Three documents, so the model is described once rather than in each format specification: ocda-model.md the model: what an object is, the citation number, structure, sections, segments, digests. The part both serialisations share ssp-format-specification.md the model written as text ocda-db-format-specification.md the model written as sqlite, to come The naming follows the three levels: ocda names the model, abstraction names the pipeline stage, and .ssp and .ocda.db name the two serialisations. A thing is named for the level it belongs to, which is why this file is ssp-format-specification.md and not ocda-format-specification.md: it specifies one serialisation, not the model. The specification states the version and the policy that governs it, and separates conformance into what a reader must do, should do, and what a writer must do. The single "should" is that a reader should report what it ignored: without it, silence cannot be told from understanding. Both worked examples regenerated from spine 0.25.0. The excerpt carries one complete object embedding an image, so the .image record and the image marker in the object text can be read against each other. The name .ssp is recorded: it belongs to the .sst, .ssm, .ssi family and stands loosely for sisu peg or sisu parsable, .ssa for sisu abstraction having been thought taken by other projects. README.md identifies the repository as a specification, not a program, summarises what a .ssp is, gives its line types at a glance, and points to ssp-format-specification.md and the two examples. org/ocda_specification.org is the literate source the documents are tangled from. 2026-09-22 (assisted by Claude-Code) --- org/ocda_specification.org | 821 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 821 insertions(+) create mode 100644 org/ocda_specification.org (limited to 'org') diff --git a/org/ocda_specification.org b/org/ocda_specification.org new file mode 100644 index 0000000..daad493 --- /dev/null +++ b/org/ocda_specification.org @@ -0,0 +1,821 @@ +-*- mode: org -*- +#+TITLE: configuration git +#+DESCRIPTION: documents - structuring, various output representations & search +#+FILETAGS: :spine:hub: +#+AUTHOR: Ralph Amissah +#+EMAIL: [[mailto:ralph.amissah@gmail.com][ralph.amissah@gmail.com]] +#+COPYRIGHT: Copyright (C) 2015 (continuously updated, current 2026) Ralph Amissah +#+LANGUAGE: en +#+STARTUP: content hideblocks hidestars noindent entitiespretty +#+PROPERTY: header-args+ :eval never-export :exports code +#+PROPERTY: header-args+ :noweb yes :padline no +#+PROPERTY: header-args+ :results silent :cache no +#+PROPERTY: header-args+ :mkdirp yes +#+OPTIONS: H:3 num:nil toc:t \n:t ::t |:t ^:nil -:t f:t *:t +- magic single double-quote → " ← FIX changes hilighting behavior (occuring + after it) in org document. INVESTIGATE (org-mode CONFIG?) FIND & FIX + +* ./ +** README.md + +#+HEADER: :tangle ../README.md +#+HEADER: :noweb yes +#+BEGIN_SRC md +# sisudoc-abstraction-format + +Specification for the **`.ssp`** text format - a human-readable, +PEG-parsable serialisation of the in-memory document abstraction +produced by [sisudoc-spine](https://git.sisudoc.org/sisudoc-spine). + +## What this is + +A specification, not a program. The repository ships: + +- [`ocda-model.md`](ocda-model.md) - **the model**: what an object is, + the citation number, structure, sections, segments, digests. The part + both serialisations share, and the place the three terms *ocda*, + *abstraction* and `.ssp` are told apart. +- [`ssp-format-specification.md`](ssp-format-specification.md) - **the + `.ssp`**: the model written as text. +- `ocda-db-format-specification.md` - **the `.ocda.db`**: the model written + as sqlite. To come. +- [`example-autonomous-contract.ssp`](example-autonomous-contract.ssp) - + spine's output for that sample document, complete and unedited. +- [`example-wealth-of-networks.ssp`](example-wealth-of-networks.ssp) - + an excerpt of spine's output for that sample document: verbatim + lines, marked elisions, keeping the preamble, the `@head` section + and a run of body objects that includes an object embedding an + image. The whole file runs to 27,769 lines. + +Both carry, in their opening comment lines, the command that +regenerates them, so they can be diffed against a run rather than +taken on trust. Both are output of **spine 0.25.0, format 2.0**. + +## Version + +The format is at **2.0**, and says so on the first line of every +file: + +``` +% SiSU Document Abstraction v2.0 +``` + +The minor part is raised when a property is added, the major part +when one changes meaning or goes away, and a reader checks the major +part and refuses what it does not know. The history is in the +specification. + +## What is `.ssp`? + +Spine's processing pipeline goes +,**markup -> abstraction -> output**. The abstraction stage builds an +in-memory object model: every paragraph, heading, table, footnote +etc. is a numbered object with parent / sibling / type metadata +("OCN" - Object Citation Numbering). `.ssp` is a faithful text +serialisation of that object model. + +It is designed for: + +- Inspecting and debugging spine's abstraction stage. +- Diff-friendly version control of document structure (a `.ssp` + diff between two markup edits shows exactly which objects + changed). +- Round-trip experiments: markup -> spine -> `.ssp` -> alternative + renderers (HTML, LLM-ready chunks, ...) without rerunning spine. +- A stable contract between spine and downstream consumers that do + not want to link against spine's internal D types. + +Line types (full grammar in +[`format-specification.md`](format-specification.md)): + +``` +@name { section / block open +} section / block close +[N] type ... object declaration (N = OCN, 0 if unnumbered) +.name: value property of the preceding object +| text content text line of the preceding object + blank line (object separator) +% comment comment (ignored by parser) +``` + +## Audience + +- Developers writing readers / converters for spine's abstraction + (the reference implementation is + [ssp-reader](https://git.sisudoc.org/ssp-reader)). +- Anyone studying spine's internal document model without reading + the D source. +- LLM / search / indexing tooling that consumes structured + document data downstream of spine. + +End-user document authors do not need to read this; they write +`.sst` / `.ssm` markup, which spine then turns into `.ssp` along +the way to the final output formats. + +## Quickstart + +Read the spec: + +```sh +$EDITOR format-specification.md +``` + +Look at a real example: + +```sh +$EDITOR example-wealth-of-networks.ssp +``` + +Generate your own `.ssp` from a sample document via spine: + +```sh +# inside sisudoc-spine/ +result/bin/spine --source --pod2 --output=/tmp/ssp path/to/pod +ls /tmp/ssp/*/abstraction/*.ssp +``` + +`--pod2` writes the abstraction as `.ssp` and bundles it into the +source pod. (See sisudoc-spine's CLI help for the current flags.) + +## License + +AGPL-3.0-or-later, matching the parent +[sisudoc-spine](https://git.sisudoc.org/sisudoc-spine) +project. + +## Repository + +- **Canonical URL:** https://git.sisudoc.org/sisudoc-abstraction-format +- **Clone:** `git clone --depth=1 git://git.sisudoc.org/specs/sisudoc-abstraction-format` + +## Related projects + +- [sisudoc-spine](https://git.sisudoc.org/sisudoc-spine) - + the document engine that produces `.ssp` files. +- [ssp-reader](https://git.sisudoc.org/sisudoc-ocda-lib) - +- [ssp-writer](https://git.sisudoc.org/sisudoc-ocda-writer) - + reference reader; renders `.ssp` to tree outline, HTML, plain + text with breadcrumbs, or LLM-ready JSONL. +- [tree-sitter-sisu](https://git.sisudoc.org/tree-sitter-sisu) - + tree-sitter grammar for the upstream `.sst` / `.ssm` markup that + spine consumes before producing `.ssp`. +#+END_SRC + +** .gitignore :gitignore: + +#+HEADER: :tangle "../.gitignore" +#+BEGIN_SRC gitignore +# git ls-files --cached --ignored --exclude-standard (git ls-files -ci --exclude-standard) +# git ls-files --others --ignored --exclude-standard (git ls-files -oi --exclude-standard) +# git ls-files --cached --others --ignored --exclude-standard (git ls-files -cio --exclude-standard) +# git status --ignored +# git ls-files --others --exclude-from=.git/info/exclude +# git ls-files --others --ignored --exclude-standard --directory +# git check-ignore -v flake.lock +# git clean -ix +# find . -name "*_" | xargs rm -rf +# find . -name "*.o" | xargs rm -rf +,* +!.gitignore +!*.nix +!flake.lock +!*.md +!*.ssp +!org/ +!**/*.org +#+END_SRC + +** ocda-model.md + +#+HEADER: :tangle ../ocda-model.md +#+BEGIN_SRC md +# The object-centric document abstraction (OCDA) + +,**The model.** What a document *is* once spine has parsed it, independent +of any file it is written to. + +Two serialisations carry this model, and they carry **one version number +between them**, because neither may express what the other cannot: + +| | | +|--------------------------------------------------------------|--------------------------------------------| +| [`ssp-format-specification.md`](ssp-format-specification.md) | the OCDA as text, `.ssp` | +| `ocda-db-format-specification.md` | the OCDA as sqlite, `.ocda.db` *(to come)* | + +This document is what they share. It describes objects; they describe +writing objects down. + +## 1. Three words, three levels + +The terms are used precisely in this repository and in spine, and the +distinction is worth holding: + +| level | term | names | +|--------------------|----------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------| +| the model | **OCDA** | what a document is once parsed: numbered objects with structure, identity and digests. Independent of any file, any stage, any tool | +| the stage | **abstraction** | the middle of spine's **markup -> abstraction -> output**: the act of building the model, and the point at which a document may now enter the pipeline | +| the serialisations | **`.ssp`**, **`.ocda.db`** | the model written down, as text and as sqlite | + +A thing is named for the level it belongs to. Something true of the model +whatever file it is in takes *ocda*; something about the pipeline stage +takes *abstraction*; something that reads or writes one file form takes +that form's name. + +## 2. The object + +A document is a sequence of **objects**, grouped into sections. An object +is a paragraph, a heading, a table, an endnote, a line of verse: the unit +the document is made of and the unit it is cited by. + +Every object carries: + +- an **object citation number** (OCN), section 3; +- a **kind**, one of sixteen: `para`, `heading`, `toc`, `bookindex`, + `endnote`, `code`, `bibliography`, `comment`, `verse`, `glossary`, + `quote`, `poem`, `block`, `table`, `blurb`, `group`; +- its **place in the structure**: parent, ancestors, and for a heading its + children and its last descendant; +- its **text**, in spine's internal representation; +- a **digest** of that text. + +## 3. The object citation number + +The OCN is the heart of the model and the reason for it. + +,**An object's OCN is the same in every format spine writes.** A paragraph +that is object 42 in the HTML is object 42 in the EPUB, in the PDF, in the +plain text, in the search database and in both serialisations of the +abstraction. A citation to object 42 of a document is therefore a citation +to a fixed piece of text, not to a page in one edition or a scroll position +in one rendering. + +Three properties follow, and each is a trap for a reader that assumes +otherwise: + +- **Not every object has one.** Generated furniture, and objects the markup + has suppressed, carry `0`. There are many of them: 7,543 across the 36 + document reference collection. +- **`0` is therefore not an identity.** Objects sharing `0` are not the + same object, and `0` must never be used as a key. +- **An OCN is not unique within a document even when non-zero.** A poem and + its verses share one; `through_the_looking_glass` has 21 such cases. Key + on position within a section when one row is wanted. + +## 4. Structure + +Structure is carried by number, not by nesting: an object names its parent +rather than containing its children. This is what lets the serialisations +be flat and the model be walked in either direction. + +- **`parent`** - the OCN of the object this one hangs from. + ,**A parent of `0` is not "no parent"**: it is a parent heading the author + has made non-substantive, its OCN suppressed or a dummy heading + introduced to segment an output. The distinction is preserved + deliberately, and a reader walking upward must stop at `0` rather than + treat it as missing, or the walk will not terminate. +- **`ancestors`** - eight slots, the OCN at each level above this object, + so the path to the root is available without walking. +- **`ancestors_collapsed`** - the same with the empty levels removed. +- **`children`** - a heading's child headings. +- **`last_descendant`** - the last OCN under a heading, which makes a + subtree a **range** rather than a traversal: everything from the + heading's OCN to its last descendant is inside it. + +## 5. Sections + +Objects are grouped into nine sections, in this order: + +``` +head toc body endnotes glossary bibliography bookindex blurb tail +``` + +`head` is the document's title matter, `body` the document proper, and the +rest are generated or gathered. `tail` is the end of document some output +writers use. A section with no objects is not carried. + +## 6. Segments + +An output may be one file or many. A **segment** is one of the many: a +chapter-sized piece of HTML or EPUB with its own name and its own +navigation. + +Two different questions are asked about segments, and the model answers +both separately because the answers differ: + +- **which segment does this heading open?** Set on headings only. +- **which segment is this object in?** Set on nearly every object. + +Without the second, a reader cannot say where a paragraph belongs. The +search database reads the second. + +## 7. Digests + +The model is checkable at four levels, and the chain is what lets an +artefact be verified rather than trusted: + +``` +the markup files ──> a digest over every markup file of that language +the serialisation ──> a digest of the .ssp a database was built from +each object ──> a digest of the object's text +each image ──> a digest of the image file, with its size in bytes and pixels +``` + +An object's digest is over its text alone, so a change to one paragraph +moves one digest and no other. + +## 8. What the model does not hold + +- **The markup.** The model is what the markup produced, not the markup + itself. An artefact that carries both is the `.ocda.db`, which carries + the markup alongside the model rather than within it. +- **The images.** The model *names* them, with digest, byte size and pixel + size, and they must be found beside the file for a document to be + complete. +- **Any output format.** No HTML, no page numbers, no typesetting. +#+END_SRC + +** ssp-format-specification.md + +#+HEADER: :tangle ../ssp-format-specification.md +#+BEGIN_SRC md +# The `.ssp` format + +,**The object-centric document abstraction, written as text. Format version +2.0**, as spine 0.25.0 emits and reads it. + +This document specifies *one serialisation*. The model it serialises is +specified in [`ocda-model.md`](ocda-model.md), which the sqlite +serialisation shares; read that first if the question is what an object +,*is* rather than how it is written down. + +## The name + +`.ssp` belongs to the family of SiSU markup extensions, `.sst` (sisu text), +`.ssm` (sisu master) and `.ssi` (sisu insert), and a short extension was +wanted that sat with them. It stands loosely for **sisu peg**, or **sisu +parsable**. `.ssa`, for sisu abstraction, was thought to be taken by other +projects. + +So the extension names the *file family and its parsability*, where +`.ocda.db` names the *model and its container*. The two are not parallel in +construction and are not meant to be. + +## Status + +This specification is derived from the emitter +(`sisudoc/ocda/abstraction/ssp.d` in sisudoc-spine, whose `sspObjectRecord` +is "the single definition of what an object looks like in a `.ssp` file"), +and every claim in it is checked against the project's 36 document +reference set. Where the two could differ, the emitter is normative and +this document is wrong. + +## 1. What a `.ssp` is + +A `.ssp` file is the serialisation, as text, of one language of one +document after spine has parsed it: the object model, not the markup and +not any output format. + +Spine's pipeline is **markup -> abstraction -> output**. A `.ssp` is the +middle stage written down. Since spine 0.25.0 it is also read back, so a +document may enter the pipeline at the abstraction stage instead of being +parsed, and the output writers cannot tell which happened. + +The format is: + +- **line oriented.** Every construct is a whole line. No line continuations. +- **order significant.** Sections appear in a fixed order, objects in + document order, and an object's properties in a fixed order (section 6.3). +- **UTF-8**, without a byte order mark. +- **lossless for the abstraction**, not for the markup: a `.ssp` read back + and re-emitted is byte identical, which the project's test suite asserts + over the whole reference set. + +### 1.1 What it is not + +It does not carry the markup it was made from, and it does not carry its +images: it *names* them, with their digests and pixel sizes, and they must +be found beside it (`../image/` within a pod) for the document to be +complete. An artefact that carries both is the `.ocda.db`, which is built +by reading a `.ssp` back, so it can hold nothing this format cannot express. + +## 2. Version and compatibility + +The version is declared **in band**, on the first line of every file: + +``` +% SiSU Document Abstraction v2.0 +``` + +and by the sibling sqlite serialisation in its `schema.version` metadata +row, so that one bump moves both. + +### 2.1 The rule + +- The **minor** part is raised when a property is **added**. A reader of the + newer format still reads the older one. +- The **major** part is raised when a property **changes meaning or goes + away**. +- A reader **checks the major part and refuses what it does not know**, + rather than loading an unfamiliar artefact half populated in silence. + +### 2.2 History + +| version | date | what moved | +|---------|------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| 2.0 | 2026-09-18 | `source.digest` changes meaning: it covers every markup file the document is made of, where it had been the digest of the master file alone. A change of meaning, so the major part. | +| 1.1 | 2026-09-10 | The `@source` block; the document metadata the output writers read, carried in full (14 `@meta` and 3 `@make` properties added); `.segment_lv4_is`. | +| 0.1 | | The first form. | + +## 3. Lexical structure + +Six line types, distinguished by their first character. Nothing else is a +valid line. + +| line | form | meaning | +|--------------------|-----------------|-----------------------------------------------| +| comment | `% ...` | file header only (section 4) | +| block open | `@name {` | opens a preamble block or a section | +| block close | `}` | closes it | +| key | ` key: value` | inside a preamble block, two spaces of indent | +| object declaration | `[ocn] kind...` | opens an object record (section 6.1) | +| property | `.name: value` | a property of the object just declared | +| text | `\| text` | a line of the object's text | +| blank | | separates records; carries no meaning | + +A value runs to the **end of the line**. It is not quoted, and it is not +trimmed beyond the single space after the colon. A value may therefore +contain `:`, spaces, and any UTF-8 character except a line break. + +### 3.1 Escaping + +Only one value is escaped, because only one value is a **joined list**: +`.heading_ancestors_text` joins eight slots with `|`. A slot that itself +contains `|`, a backslash, or a line break would otherwise be unreadable. + +| in the value | written as | +|-----------------|------------| +| `\` | `\\` | +| `\|` | `\\|` | +| newline | `\n` | +| carriage return | `\r` | + +Unescaping reads left to right: a backslash escapes the next character, +`\n` and `\r` become the control characters, any other escaped character +stands for itself, and an unescaped `|` ends the slot. + +,**No other value is escaped**, and text lines are not escaped at all: a +text line is the object's text verbatim after the two-character `| ` +prefix. + +### 3.2 Object text is internal markup + +The text on `|` lines is spine's **internal** representation, neither the +author's markup nor any output. It carries the markers spine uses to hold +inline structure: + +| marker | means | +|-----------------|---------------------------| +| `【` `】` | an inline note | +| `┥` `┝` `┤` `├` | link text and link target | +| `░` | non-breaking space | +| `┘` | line break | +| `☼` | image | + +A reader that renders text must decode these; a reader that only searches +or counts need not. They are stable and are part of the format. + +## 4. File shape + +In order, and this order is fixed: + +``` +% SiSU Document Abstraction v2.0 +% Source: + +@source { ... } + +@meta { ... } + +@make { ... } + +@doc_has { ... } + +@head { ... } +@toc { ... } +@body { ... } +@endnotes { ... } +@glossary { ... } +@bibliography { ... } +@bookindex { ... } +@blurb { ... } +@tail { ... } +``` + +The two `%` lines are the only comments in the file and appear only here. +The four preamble blocks are always present and always in that order. The +nine section blocks appear in that order. **A section is omitted entirely +when the document has no objects for it**, though in practice that is rare: +all 36 documents of the reference collection carry all nine, several of +them holding a single generated object. + +## 5. The preamble blocks + +Inside a block, every line is ` key: value` with exactly two spaces of +indent. Keys are unique within a block. **A key whose value is empty is +omitted**, so the set of keys present varies by document; the exceptions +are noted below. + +### 5.1 `@source` - what this abstraction was made from + +| key | always? | meaning | +|-------------|----------------------------|-----------------------------------------------------------------------------| +| `language` | yes | the language of *this* file | +| `languages` | when the pod declares them | every language of the document, space separated | +| `digest` | yes | sha256 over **every markup file the document is made of**, in a fixed order | + +The source *filename* is not repeated here; it is on the `% Source:` line. + +`digest` is what changed in 2.0. It had been the digest of the master file +alone, which for a single file document was right and for a multi-file one +was close to meaningless: any chapter could be edited without the digest +moving. + +### 5.2 `@meta` - the document's own metadata + +Dublin-Core shaped, `group.name`. **43 keys are defined**; a given document +carries only those it has. The 28 the reference collection exercises are +marked; the remainder are defined and simply unused by these documents. + +| group | keys | +|---------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `title.` | `main`*, `subtitle`*, `full`*, `language`*, `edition`* | +| `creator.` | `author`*, `author_surname`*, `author_surname_fn`*, `author_email`*, `illustrator`*, `translator`* | +| `date.` | `published`*, `created`*, `issued`*, `available`*, `modified`*, `valid`*, `added_to_site`* | +| `rights.` | `copyright`*, `license`*, `copyright_text`, `copyright_translation`, `copyright_illustrations`, `copyright_photographs`, `copyright_cover`, `copyright_audio`, `copyright_video` | +| `classify.` | `topic_register`*, `subject`, `keywords`, `loc`*, `dewey`* | +| `identifier.` | `isbn`, `oclc` | +| `notes.` | `summary`*, `abstract`*, `description`* | +| `original.` | `source`*, `title`, `language`, `language_char` | +| `language.` | `document_char`*, `document` | + +`classify.topic_register` is structured within its value: topics separated +by `;`, levels within a topic by `:`, and alternatives at a level by `|`. + +### 5.3 `@make` - how the document is to be built + +| key | meaning | +|-------------------------|-----------------------------------------------------| +| `doc_type` | `article`, `book`, ... | +| `auto_num_top_at_level` | the level automatic numbering starts at | +| `auto_num_top_lv` | | +| `auto_num_depth` | how many levels are numbered | +| `breaks` | page break instructions | +| `footer` | footer text, slots joined with `\|` | +| `home_button_text` | markup for the home link, carrying internal markers | + +### 5.4 `@doc_has` - what the document contains, by count + +Ten keys, **all ten always emitted**, zero included. This block is a +summary a reader can use without walking the objects. + +``` +inline_links inline_notes_reg inline_notes_star tables codeblocks +images poems groups blocks quotes +``` + +## 6. Objects + +The nine section blocks hold object records, in document order. An object +record is an object declaration line, then its properties, then its text +lines, then a blank line. + +### 6.1 The declaration line + +``` +[] +[] heading : +[] heading : +``` + +`` is the **object citation number** (see +[`ocda-model.md`](ocda-model.md) section 3). `[0]` means the object has +none, is shared by many objects, and must never be used as a key; key on +position within a section instead. + +`` is one of sixteen: + +``` +para heading toc bookindex endnote code bibliography comment +verse glossary quote poem block table blurb group +``` + +A **heading** is the one kind whose declaration line carries more: its +markup level after a colon (`:A` `:B` `:C` `:D` for the title levels, +`:1`..`:4` for the body levels; `:D` is defined and unexercised by the +reference collection), and then its identifier, when it has one +that is not simply the ocn. Every other kind carries an identifier, on the +same condition, as the `.identifier` property. + +### 6.2 Text lines + +``` +| +``` + +One line per line of the object's text, in order, each prefixed with +`| ` (pipe, space). An object with no text has no text lines. The text is +internal markup (section 3.2) and is not escaped. + +### 6.3 Properties + +,**Properties appear in a fixed order**, the order of the table below, and a +reader may rely on it: checked over the reference set, no object among +56,882 emits a property out of this order. + +,**A property is omitted when it holds its default**, so absence is +meaningful and is defined per property in the table. Four properties +repeat, appearing once per value rather than carrying a list: `.image`, +`.stow_link`, `.lev4_subtoc` and `.anchor_tag`. + +In the table, "emitted when" is the condition under which the property +appears at all. Counts are occurrences across the 36 document reference +set, as a guide to which parts of the format are load bearing. + +| # | property | value | emitted when | count | +|----|---------------------------|-------------------------------|-------------------------------------------------------------|--------| +| 1 | `.identifier` | string | a non-heading has an identifier that is not its ocn | 353 | +| 2 | `.part` | string | the object belongs to a part (`frontmatter`, `body`, ...) | 56,882 | +| 3 | `.section` | string | the object's own section differs from the block it sits in | 605 | +| 4 | `.parent` | int | **every object except** the document root and `toc` entries | 53,631 | +| 5 | `.last_descendant` | int | non-zero: the last ocn under this heading | 3,202 | +| 6 | `.children` | ints | the heading has child headings | 827 | +| 7 | `.ancestors` | 8 ints | any is non-zero | 53,595 | +| 8 | `.ancestors_collapsed` | 8 ints | any is non-zero | 53,595 | +| 9 | `.dom_status` | 8 ints | any is non-zero | 3,319 | +| 10 | `.dom_status_collapsed` | 8 ints | any is non-zero | 3,319 | +| 11 | `.heading_lev_collapsed` | int | it is below 9 | 3,503 | +| 12 | `.parent_lev` | int | non-zero | 52,848 | +| 13 | `.dummy` | `true` | the heading was introduced to segment an output | 183 | +| 14 | `.ocn_off` | `true` | the object has no citation number | 7,543 | +| 15 | `.is_of_type` | string | always, in practice | 56,882 | +| 16 | `.attrib` | string | set | 1 | +| 17 | `.meta_lang` | string | the object carries its own language | 0 | +| 18 | `.meta_syntax` | string | the object carries its own markup syntax | 0 | +| 19 | `.sha256` | hex | the object's text digest is non-zero | 51,253 | +| 20 | `.image` | record | once **per image**, in order of appearance | 63 | +| 21 | `.indent` | 2 ints | either indent is non-zero: `base hang` | 7,583 | +| 22 | `.bullet` | `true` | the object is a bullet item | 1,903 | +| 23 | `.lang` | string | the text attributes carry a language | 0 | +| 24 | `.has` | flags | any is set | 11,143 | +| 25 | `.table_cols` | int | the object is a table with columns | 32 | +| 26 | `.table_widths` | ints | the table has column widths | 32 | +| 27 | `.table_aligns` | letters | the table has alignments (`l` `c` `r`) | 32 | +| 28 | `.table_header` | `true` | the table has a header row | 29 | +| 29 | `.code_linenumbers` | `true` | the code block is numbered | 1 | +| 30 | `.stow_link` | url | once **per extracted link** | 1,013 | +| 31 | `.segment` | string | the heading opens a segment | 3,355 | +| 32 | `.anchor` | string | the anchor differs from the segment name | 68 | +| 33 | `.segment_prev` | string | there is a previous segment | 1,022 | +| 34 | `.segment_next` | string | there is a next segment | 1,022 | +| 35 | `.heading_lev_anchor` | string | the heading has a level anchor tag | 1,854 | +| 36 | `.segment_epub` | string | the heading opens an epub segment | 3,355 | +| 37 | `.segment_html_is` | string | the object is inside an html segment | 47,024 | +| 38 | `.segment_epub_is` | string | the object is inside an epub segment | 47,024 | +| 39 | `.segment_lv4_is` | string | **and** it differs from `.segment_html_is` | 342 | +| 40 | `.heading_ancestors_text` | 8 slots, `\|` joined, escaped | any slot is non-empty | 3,319 | +| 41 | `.lev4_subtoc` | string | once **per sub-table-of-contents entry** | 1,887 | +| 42 | `.anchor_tag` | string | once **per anchor tag** | 7,883 | + +Three of these need a word here, the first two because the model behind +them is easy to misread (see [`ocda-model.md`](ocda-model.md) sections 4 +and 6), the third because it is a property of this table rather than of the +format. + +,**`.parent: 0` is emitted, not omitted.** A parent of 0 is a real, +non-substantive parent and not a missing value, so the property is written +out rather than defaulted away: omitting it would make the two cases +indistinguishable in the file. A reader walking upward stops at 0. + +,**`.segment` and `.segment_html_is` are different questions**, and both are +carried: what a heading opens, and what an object sits in. A reader wanting +to know where a paragraph belongs needs the second. + +,**Count 0 does not mean "not in the format".** `.meta_lang`, `.meta_syntax` +and `.lang` are emitted by spine under the conditions given, and no +document in the reference collection meets them. A reader must handle them; +a test corpus does not exercise them. + +### 6.4 The `.image` record + +``` +.image: sha256: bytes: px:x +.image: missing:true +``` + +Filename first, as the object's handle on the image. The remaining fields +are self describing `key:value` tokens, **so that further fields may be +added without invalidating existing readers**; `px:` is omitted when the +pixel size is unknown. `px:` is the file's own size, not the display size, +which the object's text already carries. + +### 6.5 The `.has` flags + +Space separated, in this order, each present only when set: + +``` +links notes_reg notes_star images images_no_dim +``` + +## 7. Grammar + +A PEG sketch. It is deliberately loose about property *names*, which the +table in section 6.3 fixes, and about the order, which section 6.3 fixes: +a grammar that enumerated 42 alternatives in sequence would be unreadable +and would need editing for every minor version. + +```peg +Document <- Header Source Meta Make DocHas Section* EOF +Header <- "% SiSU Document Abstraction v" Version EOL + "% Source: " Value EOL Blank +Version <- Digit+ "." Digit+ + +Source <- "@source {" EOL Pair+ "}" EOL Blank +Meta <- "@meta {" EOL Pair* "}" EOL Blank +Make <- "@make {" EOL Pair* "}" EOL Blank +DocHas <- "@doc_has {" EOL Pair+ "}" EOL Blank +Pair <- " " Key ": " Value EOL +Key <- [a-z_]+ ("." [a-z_]+)? + +Section <- "@" SectionName " {" EOL Blank Object* "}" EOL Blank +SectionName <- "head" / "toc" / "body" / "endnotes" / "glossary" + / "bibliography" / "bookindex" / "blurb" / "tail" + +Object <- Decl Property* TextLine* Blank +Decl <- "[" Digit+ "] " (Heading / Kind) EOL +Heading <- "heading :" Level (" " Identifier)? +Level <- [A-D] / [1-4] +Kind <- [a-z]+ +Property <- "." Name ": " Value EOL +Name <- [a-z_0-9]+ +TextLine <- "| " Value EOL + +Value <- (!EOL .)* +Blank <- EOL +EOL <- "\n" +Digit <- [0-9] +``` + +## 8. Conformance + +### 8.1 A reader must + +- **Check the major version** on the first line and refuse a major it does + not know. Loading an unknown major half populated, in silence, is the + failure this rule exists to prevent. +- **Tolerate unknown properties** within a known major version, since the + minor part is raised precisely by adding them. Ignoring them is + conforming behaviour. +- **Treat `[0]` as not an identity.** Objects with no citation number are + numerous and share the number. +- **Stop at `.parent: 0`** when walking upward. + +### 8.2 A reader should + +- **Report what it ignored**, at least on request. A reader that silently + drops what it does not understand cannot tell its user whether it + understood the document, and cannot be used to check this specification + against an implementation. + +### 8.3 A writer must + +- Emit the blocks and sections in the order of section 4, an object's + properties in the order of section 6.3, and objects in document order. +- Omit a property holding its default, per section 6.3. +- Escape only what section 3.1 requires, and only where it requires it. + +## 9. Relationship to the model, and to the sqlite serialisation + +The model this format writes down is specified in +[`ocda-model.md`](ocda-model.md): objects, citation numbers, structure, +sections, segments and digests. This document adds nothing to the model and +subtracts nothing from it; it says how the model is written as text. + +The `.ocda.db` is the same abstraction as sqlite, and is **built by reading +a `.ssp` back**: the document is emitted as `.ssp`, read straight in, and +what comes back is written. So the database cannot carry an object property +this format cannot express, and the two cannot drift. They share the +version, which is why a bump moves both. + +The database is not a competitor to this format. It carries every language +of a document in one file, and the markup, images and configuration as +well, which this format does not. + +## 10. Worked examples + +- `example-autonomous-contract.ssp` - one document, complete and unedited. +- `example-wealth-of-networks.ssp` - an excerpt of a large one: verbatim + lines with marked elisions, keeping the preamble, the `@head` section, + and a run of body objects including one that embeds an image. + +Each carries, in its opening comments, the command that regenerates it. +#+END_SRC -- cgit v1.2.3