# SiSU Document Abstraction - Bespoke Text Format Specification ## Purpose A human-readable, PEG-parsable text representation of the in-memory document abstraction produced by `spineAbstraction!()`. Designed for: - Inspecting and debugging the abstraction stage - Diff-friendly version control of document structure - Potential alternative input format (round-trip: markup -> abstraction -> this format -> abstraction -> output) - Understanding what information the abstraction captures ## Overview The format mirrors the `ObjGenericComposite[][string]` return type of `doc.abstraction()`. A file consists of: 1. A metadata preamble (`@meta`, `@make`, `@doc_has`) 2. Ordered sections (`@head`, `@toc`, `@body`, `@endnotes`, `@glossary`, `@bibliography`, `@bookindex`, `@blurb`) 3. Within each section, a sequence of objects Each object has: - An OCN marker and type declaration on the first line - Optional property lines (`.name: value`) - Text content lines (`| content`) ## Line Types Every line in the file is one of: ``` @name { section/block open } section/block close [N] type ... object declaration (N = OCN, 0 if unnumbered) .name: value property (belongs to preceding object) | text content text line (belongs to preceding object) blank line (object separator) % comment comment (ignored by parser) ``` This makes the format unambiguous at the first character of every line. ## Text Content Encoding The `text` field in the abstraction uses Unicode internal markers from `InternalMarkup` (defined in `sisudoc-spine/src/sisudoc/meta/defaults.d`). These are preserved as-is in the format: | Internal Marker | Unicode | Meaning | |---|---|---| | `en_a_o` / `en_a_c` | `【` `】` | Footnote/endnote open/close | | `en_b_o` / `en_b_c` | `〖` `〗` | Secondary note open/close | | `ff_i` | `⑆` | Font face inline marker | | `ff_o` / `ff_c` | `┨` `┣` | Font face open/close | | `lnk_o` / `lnk_c` | `┥` `┝` | Link text open/close | | `url_o` / `url_c` | `┤` `├` | URL open/close | | `quote_o` / `quote_c` | `\u201c` `\u201d` | Smart quotes | | `nbsp` | `░` | Non-breaking space | | `br_line` | `┘` | Line break | | `br_line_inline` | `┙` | Inline line break | | `br_line_spaced` | `┚` | Spaced line break | | `br_page_line` | `┼` | Page break (line) | | `br_page` | `┿` | Page break | | `br_page_new` | `╂` | New page | | `mk_bullet` | `● ` | Bullet marker | | `mono` | `■` | Monospace marker | | `cite` | `‖` | Citation marker | | `img` | `☼` | Image marker | | `on_o` / `on_c` | `「` `」` | Object number open/close | | `tc_s` | `┊` | Table cell separator | | `tc_o` / `tc_c` | `┏` `┚` | Table cell open/close | | `tc_p` | `┆` | Table cell pipe | | `mark_internal_site_lnk` | `¤` | Internal site link | Font face codes within `⑆` markers use single characters: `*` emphasis, `!` bold, `/` italic, `_` underscore, `^` superscript, `,` subscript, `■` mono, `‖` cite These markers are part of the abstraction's canonical representation and must be preserved exactly. ## Format Structure ### Metadata Preamble ``` @meta { title.main: The Wealth of Networks title.subtitle: How Social Production Transforms Markets and Freedom creator.author: Benkler, Yochai creator.author_surname: Benkler date.published: 2006-04-03 date.created: 2006-04-03 rights.copyright: 2006 Yochai Benkler. rights.license: Creative Commons Noncommercial Sharealike classify.topic_register: networks;Internet;economics;society classify.subject: information society, economics, copyright identifier.isbn: 9780300110562 language.document: en notes.summary: ... } @make { doc_type: book auto_num_top_at_level: 1 auto_num_top_lv: 9 auto_num_depth: 2 } @doc_has { inline_links: 342 inline_notes_reg: 891 inline_notes_star: 0 tables: 3 codeblocks: 0 images: 0 poems: 0 groups: 0 blocks: 0 quotes: 12 } ``` ### Sections Sections correspond directly to the keys in `document_the`: ``` @body { ... objects ... } @endnotes { ... objects ... } ``` Available sections (in output order): `head`, `toc`, `body`, `endnotes`, `glossary`, `bibliography`, `bookindex`, `blurb`. ### Object Declarations Each object starts with `[OCN] type` where: - `OCN` is the integer object citation number (0 for unnumbered) - `type` identifies the object kind, matching `metainfo.is_a` **Heading objects** include the markup level: ``` [1] heading :A | title text [5] heading :1 segment_name | Chapter 1 - Introduction [20] heading :2 | THE EMERGENCE OF THE NETWORKED INFORMATION ECONOMY ``` The `:A` through `:D` and `:1` through `:4` correspond to `heading_lev_markup` values 0-7. The optional segment name follows. **Paragraph objects**: ``` [6] para | Information, knowledge, and culture are central to | human freedom and human development. [7] para | A series of changes in the technologies... ``` **Block-type objects**: ``` [30] code | auto doc = spineAbstraction!()(...); | auto matters = doc.matters(); [40] table | Cell1┊Cell2┊Cell3 | Cell4┊Cell5┊Cell6 [50] group | Grouped text content here. [55] verse | First verse of the poem. [60] block | Block-preserved whitespace content. [65] quote | Quoted text here. ``` **Backmatter objects** (in endnotes/glossary/bibliography/bookindex sections): ``` [0] endnote | 【1 Footnote text here.】 [0] glossary | ░░⑆!GPL⑆! An abbreviation... [0] bibliography | au: Benkler, Yochai ... [0] bookindex | access: 16-18, 35 ``` ### Properties Properties appear on lines starting with `.` after the object declaration and before text content. Only non-default values are written. **Structural properties** (from `metainfo`): ``` .part: body % frontmatter, body, backmatter .section: body % toc, body, glossary, bibliography, % bookindex, blurb .parent: 5 % parent OCN, see below .last_descendant: 70 % last descendant OCN .ancestors: 0 0 0 0 5 20 0 0 % markedup_ancestors[8] .ancestors_collapsed: 0 0 0 0 5 20 0 0 .dom_status: 0 0 0 0 1 1 0 0 % dom_structure_markedup_tags_status[8] .dom_status_collapsed: 0 0 0 0 1 1 0 0 .dummy: true % dummy_heading .ocn_off: true % object_number_off .children: 20 45 70 % direct child heading OCNs .heading_lev_collapsed: 2 % written when < 9 .parent_lev: 1 % parent_lev_markup .o_n_type: 1 % object number type .is_of_type: para % para, block .attrib: ... % metainfo.attrib .meta_lang: en % block language (group, block, quote) .meta_syntax: d % codeblock syntax (metainfo.syntax) ``` A heading's identifier is not a property: it follows the level on the object declaration line (`[5] heading :1 ch1_intro`). `.parent` is written for every object, including when the value is 0, with two exceptions: the `:A~` document root, which has no parent, and the entries of the generated `@toc` section, which are furniture rather than authored objects and whose place is already given by their `.indent` level (the `@toc` heading itself does carry `.parent: 1`). A parent of 0 is not "no parent recorded": it is a parent heading the author has made non-substantive, its ocn suppressed with `~#`, or a dummy heading introduced with `-#` to segment an output. Stating it keeps "non-substantive parent" and "no parent" distinguishable, so a reader must not treat a missing `.parent` as 0. **Text attribute properties** (from `attrib`): ``` .indent: 1 0 % indent_base indent_hang .bullet: true .lang: en % language override ``` **Presence flags** (from `has`): ``` .has: links notes_reg notes_star images images_no_dim ``` Only present flags are listed. Absence means false. `images_no_dim` marks an object whose markup gave no image dimensions (`w0h0`), which spine then derived from the file. **Table properties** (from `table`, only for table objects): ``` .table_cols: 3 .table_widths: 40 30 30 .table_aligns: left right right .table_header: true .table_walls: true ``` **Code block properties** (from `code_block`, only for code objects): ``` .code_syntax: d .code_linenumbers: true ``` **Tag properties** (from `tags`): ``` .segment: ch1_intro % in_segment_html .segment_prev: acknowledgments .segment_next: ch2 .segment_epub: ch1_intro % segment_anchor_tag_epub .anchor: ch1_intro % anchor_tag_html, when it differs % from .segment .heading_lev_anchor: h1 % heading_lev_anchor_tag .heading_ancestors_text: |The Wealth of Networks|||||| .anchor_tag: some_tag % repeatable, one per anchor tag .lev4_subtoc: ... % repeatable, one per subtoc entry ``` The `heading_ancestors_text` uses `|` as separator for the 8-element array, with empty strings between consecutive `|` markers. **Digest** (from `metainfo`): ``` .sha256: 7A3B9F... % hex-encoded (uppercase), of object text ``` The digest covers the object's `text` field only (including inline notes), not any file the text references. **Images** (from `metainfo.sha256.images`): An object that embeds one or more images carries one `.image` line per image, in order of appearance within the object text: ``` .image: won_benkler_2_1.png sha256:5BA492C2C040C179FF8E46FC800B5B3AE6F7F082D9532F877141F44A366DD128 bytes:93861 px:420x342 ``` The value is a record: the image filename first (the object's handle on the image, as it appears after the `☼` marker in the text and as the file is named in the pod's `media/image/` directory), followed by whitespace-separated `key:value` fields: | Field | Meaning | |---|---| | `sha256:` | SHA256 digest of the image file, hex-encoded, uppercase | | `bytes:` | image file size in bytes | | `px:` | the file's own pixel dimensions, `x` | | `missing:` | `true` when the referenced file was absent or unreadable | `px:` is the image as stored, read from the file header. It is *not* the same thing as the `w##h##` dimensions carried in the object's text (`☼won_benkler_2_1.png,w420h342`), which are display dimensions: author set in the markup, or where the markup gave `w0h0` derived from the file and capped to a maximum display width. The two commonly differ, e.g. ``` % display 640x323, file 831x420 (capped) .image: di_evh_f10-1.png sha256:... bytes:... px:831x420 % display 82x128, file 328x512 (author set) .image: sm_d_image.jpg sha256:... bytes:... px:328x512 ``` Display dimensions are therefore not duplicated in the record: they are already present, verbatim, in the object's text lines, and they describe the rendering rather than the file. `px:` records what the text cannot, and completes the file's identity alongside `sha256:` and `bytes:`. It is omitted when the dimensions could not be read. An image the abstraction referenced but could not read is recorded as such, rather than being omitted: ``` .image: ffa.png missing:true ``` `missing:true` and `sha256:`/`bytes:` are mutually exclusive: no digest or size could be taken, so neither field is written. The record still tells a reader that the object embeds an image and names it, and tells a verifier that this `.ssp` was produced from an incomplete pod. Rules: - The property is **repeatable**. An object with two images emits two `.image` lines; readers accumulate them into a list. Objects without images emit none. - The filename is a single token (image names match `[a-zA-Z0-9._-]+\.(jpg|gif|png)`, so it cannot contain whitespace, `:` or `\n`) and is always the first field. - Fields after the filename are **order-independent and extensible**. A reader MUST ignore a `key:value` field it does not recognise, and MUST NOT assume a fixed field count. This reserves room for later additions (e.g. `mime:image/png`) without a format break. - Values are the file's own properties, so two objects embedding the same image emit identical `sha256:`/`bytes:`/`px:` values. Display dimensions, which can differ between two such objects, are a property of the reference and stay in the object's text. Rationale: this is verification data (is the image shipped in this pod the image the abstraction was built from?), so the digest of the referenced file belongs with the object that references it, and remains valid when the `.ssp` file travels alone. Packing filename, size and digest into the property *name* (as `.imageSha256_filesize_filename:`) would put the schema in the key, forcing a new property name on every added field; a positional `::`-separated value would fix the arity for the same reason. The pod-level `digests.txt` manifest keeps its own flat `HEX::bytes - filename` convention: that file is a verification report, not a grammar-bound serialisation. **Extracted links** (from `stow`): Repeatable, one line per link: ``` .stow_link: https://example.org .stow_link: https://other.org ``` ### Property Defaults (omission rules) Properties with default values are omitted to reduce verbosity: | Property | Default | Omit when | |---|---|---| | .part | "" | empty | | .section | "" | empty, or same as the enclosing section | | .parent | (none) | the `:A~` document root, and `toc` entries; otherwise always written, 0 included | | .last_descendant | 0 | zero | | .children | (none) | no child headings | | .ancestors | 0 0 0 0 0 0 0 0 | all zeros | | .ancestors_collapsed | 0 0 0 0 0 0 0 0 | all zeros | | .dom_status | 0 0 0 0 0 0 0 0 | all zeros | | .dom_status_collapsed | 0 0 0 0 0 0 0 0 | all zeros | | .heading_lev_collapsed | 9 | 9 or greater | | .parent_lev | 0 | zero | | .o_n_type | 0 | zero | | .is_of_type | "" | empty | | .attrib | "" | empty | | .meta_lang | "" | empty | | .meta_syntax | "" | empty | | .indent | 0 0 | both zero | | .bullet | false | false | | .has | (none) | no flags set | | .dummy | false | false | | .ocn_off | false | false | | .code_syntax | "" | empty | | .code_linenumbers | false | false | | .anchor | "" | empty, or same as .segment | | .stow_link | (none) | no links (repeatable, so no default) | | .image | (none) | object embeds no image (repeatable, so no default) | ### What is NOT serialized - `ptr` (DocObj_Pointer_) - runtime array indices, recomputed on load - Computed properties (object_number(), marked_up_level(), etc.) ### What is optional - `tags` (DocObj_Tags_) - HTML/EPUB-specific segment identifiers. These capture structure decisions used by multiple output generators. Serialized for completeness but could be recomputed from the heading hierarchy. ## PEG Grammar (Pegged syntax sketch) ``` SiSUAbstraction: Document <- MetaBlock MakeBlock? DocHasBlock? Section+ eoi MetaBlock <- '@meta' sp '{' nl MetaLine+ '}' nl+ MakeBlock <- '@make' sp '{' nl MetaLine+ '}' nl+ DocHasBlock <- '@doc_has' sp '{' nl MetaLine+ '}' nl+ MetaLine <- sp+ ~Identifier ':' sp ~Value nl Section <- '@' ~SectionName sp '{' nl Object* '}' nl+ SectionName <- 'head' / 'toc' / 'body' / 'endnotes' / 'glossary' / 'bibliography' / 'bookindex' / 'blurb' Object <- ObjDecl nl Property* TextLine* nl* ObjDecl <- '[' ~Number ']' sp ~ObjType ObjExtra? ObjType <- 'heading' / 'para' / 'toc' / 'table' / 'code' / 'quote' / 'group' / 'block' / 'verse' / 'endnote' / 'glossary' / 'bibliography' / 'bookindex' / 'blurb' ObjExtra <- sp ~(!nl .)+ Property <- ImageProperty / GenericProperty GenericProperty <- '.' ~PropName ':' sp ~PropValue nl PropName <- Identifier PropValue <- (!nl .)+ ImageProperty <- '.image' ':' sp ~ImageFile ImageField* nl ImageFile <- [a-zA-Z0-9._-]+ '.' ('jpg' / 'gif' / 'png') ImageField <- sp ~ImageFieldName ':' ~ImageFieldValue ImageFieldName <- Identifier ImageFieldValue <- (![ \t\n] .)+ TextLine <- '|' sp? ~(!nl .)* nl Comment <- '%' (!nl .)* nl Identifier <- [a-zA-Z_] [a-zA-Z0-9_.]* Number <- [0-9]+ Value <- (!nl .)+ sp <- [ \t]+ nl <- '\n' ``` ## File Extension `.ssp` (SiSU Document Abstraction) ## Relationship to Source Markup ``` source.sst --> spineAbstraction() --> in-memory objects | v document.ssp (this format) | v load & reconstruct objects | v output generators (HTML, EPUB, LaTeX, etc.) ``` The `.ssp` file captures the state AFTER parsing and abstraction but BEFORE output generation. It is independent of any specific output format. ## Design Decisions ### Why `|` for text lines? The pipe prefix makes text content unambiguous at the character level. Alternative considered: indentation-based (4 spaces). Rejected because the abstraction text can contain leading spaces (preserved whitespace in blocks) which would be ambiguous. The `|` character does not conflict with the internal markup Unicode characters. While `|` appears in table cell separators in SiSU source markup, by the abstraction stage these have been converted to `┊` (tc_s). ### Why properties before text? Properties (`.name: value`) appear between the object declaration and the text content. This means scanning an object goes: identity -> metadata -> content. For downstream processing, the metadata is often needed before the text (e.g., to decide how to render it), so this ordering matches the access pattern. ### Why one line per property? Diff-friendliness. When a property changes, only that line shows in a diff. When an object is added or removed, the diff is a contiguous block. ### Why preserve internal Unicode markers? The abstraction's `text` field has already transformed source markup (`/{italic}/`) into internal markers (`⑆/italic⑆`). This is the canonical form at the abstraction stage. Reverting to source markup would lose information about processing decisions (e.g., auto-bolding, substitutions). The format represents the abstraction faithfully, not the source.