summaryrefslogtreecommitdiffhomepage
path: root/format-specification.md
blob: 8aec323b1dd6f103d03a90d79eebfc2799d1d120 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
# SiSU Document Abstraction - Bespoke Text Format Specification

## Purpose

A human-readable, PEG-parsable text representation of the in-memory document
abstraction produced by `spineAbstraction!()`. Designed for:

- Inspecting and debugging the abstraction stage
- Diff-friendly version control of document structure
- Potential alternative input format (round-trip:
  markup -> abstraction -> this format -> abstraction -> output)
- Understanding what information the abstraction captures

## Overview

The format mirrors the `ObjGenericComposite[][string]` return type of
`doc.abstraction()`. A file consists of:

1. A metadata preamble (`@meta`, `@make`, `@doc_has`)
2. Ordered sections (`@head`, `@toc`, `@body`, `@endnotes`, `@glossary`,
   `@bibliography`, `@bookindex`, `@blurb`)
3. Within each section, a sequence of objects

Each object has:
- An OCN marker and type declaration on the first line
- Optional property lines (`.name: value`)
- Text content lines (`| content`)

## Line Types

Every line in the file is one of:

```
@name {           section/block open
}                  section/block close
[N] type ...       object declaration (N = OCN, 0 if unnumbered)
.name: value       property (belongs to preceding object)
| text content     text line (belongs to preceding object)
                   blank line (object separator)
% comment          comment (ignored by parser)
```

This makes the format unambiguous at the first character of every line.

## Text Content Encoding

The `text` field in the abstraction uses Unicode internal markers from
`InternalMarkup` (defined in `sisudoc-spine/src/sisudoc/meta/defaults.d`).
These are preserved as-is in the format:

| Internal Marker | Unicode | Meaning |
|---|---|---|
| `en_a_o` / `en_a_c` | `【` `】` | Footnote/endnote open/close |
| `en_b_o` / `en_b_c` | `〖` `〗` | Secondary note open/close |
| `ff_i` | `⑆` | Font face inline marker |
| `ff_o` / `ff_c` | `┨` `┣` | Font face open/close |
| `lnk_o` / `lnk_c` | `┥` `┝` | Link text open/close |
| `url_o` / `url_c` | `┤` `├` | URL open/close |
| `quote_o` / `quote_c` | `\u201c` `\u201d` | Smart quotes |
| `nbsp` | `░` | Non-breaking space |
| `br_line` | `┘` | Line break |
| `br_line_inline` | `┙` | Inline line break |
| `br_line_spaced` | `┚` | Spaced line break |
| `br_page_line` | `┼` | Page break (line) |
| `br_page` | `┿` | Page break |
| `br_page_new` | `╂` | New page |
| `mk_bullet` | `● ` | Bullet marker |
| `mono` | `■` | Monospace marker |
| `cite` | `‖` | Citation marker |
| `img` | `☼` | Image marker |
| `on_o` / `on_c` | `「` `」` | Object number open/close |
| `tc_s` | `┊` | Table cell separator |
| `tc_o` / `tc_c` | `┏` `┚` | Table cell open/close |
| `tc_p` | `┆` | Table cell pipe |
| `mark_internal_site_lnk` | `¤` | Internal site link |

Font face codes within `⑆` markers use single characters:
`*` emphasis, `!` bold, `/` italic, `_` underscore,
`^` superscript, `,` subscript, `■` mono, `‖` cite

These markers are part of the abstraction's canonical representation and
must be preserved exactly.

## Format Structure

### Metadata Preamble

```
@meta {
  title.main: The Wealth of Networks
  title.subtitle: How Social Production Transforms Markets and Freedom
  creator.author: Benkler, Yochai
  creator.author_surname: Benkler
  date.published: 2006-04-03
  date.created: 2006-04-03
  rights.copyright: 2006 Yochai Benkler.
  rights.license: Creative Commons Noncommercial Sharealike
  classify.topic_register: networks;Internet;economics;society
  classify.subject: information society, economics, copyright
  identifier.isbn: 9780300110562
  language.document: en
  notes.summary: ...
}

@make {
  doc_type: book
  auto_num_top_at_level: 1
  auto_num_top_lv: 9
  auto_num_depth: 2
}

@doc_has {
  inline_links: 342
  inline_notes_reg: 891
  inline_notes_star: 0
  tables: 3
  codeblocks: 0
  images: 0
  poems: 0
  groups: 0
  blocks: 0
  quotes: 12
}
```

### Sections

Sections correspond directly to the keys in `document_the`:

```
@body {
  ... objects ...
}

@endnotes {
  ... objects ...
}
```

Available sections (in output order): `head`, `toc`, `body`, `endnotes`,
`glossary`, `bibliography`, `bookindex`, `blurb`.

### Object Declarations

Each object starts with `[OCN] type` where:
- `OCN` is the integer object citation number (0 for unnumbered)
- `type` identifies the object kind, matching `metainfo.is_a`

**Heading objects** include the markup level:

```
[1] heading :A
| title text

[5] heading :1 segment_name
| Chapter 1 - Introduction

[20] heading :2
| THE EMERGENCE OF THE NETWORKED INFORMATION ECONOMY
```

The `:A` through `:D` and `:1` through `:4` correspond to
`heading_lev_markup` values 0-7. The optional segment name follows.

**Paragraph objects**:

```
[6] para
| Information, knowledge, and culture are central to
| human freedom and human development.

[7] para
| A series of changes in the technologies...
```

**Block-type objects**:

```
[30] code
| auto doc = spineAbstraction!()(...);
| auto matters = doc.matters();

[40] table
| Cell1┊Cell2┊Cell3
| Cell4┊Cell5┊Cell6

[50] group
| Grouped text content here.

[55] verse
| First verse of the poem.

[60] block
| Block-preserved whitespace content.

[65] quote
| Quoted text here.
```

**Backmatter objects** (in endnotes/glossary/bibliography/bookindex sections):

```
[0] endnote
| 【1 Footnote text here.】

[0] glossary
| ░░⑆!GPL⑆! An abbreviation...

[0] bibliography
| au: Benkler, Yochai ...

[0] bookindex
| access: 16-18, 35
```

### Properties

Properties appear on lines starting with `.` after the object declaration
and before text content. Only non-default values are written.

**Structural properties** (from `metainfo`):

```
.part: body                          % frontmatter, body, backmatter
.section: body                       % toc, body, glossary, bibliography,
                                     % bookindex, blurb
.parent: 5                           % parent OCN, see below
.last_descendant: 70                 % last descendant OCN
.ancestors: 0 0 0 0 5 20 0 0        % markedup_ancestors[8]
.ancestors_collapsed: 0 0 0 0 5 20 0 0
.dom_status: 0 0 0 0 1 1 0 0        % dom_structure_markedup_tags_status[8]
.dom_status_collapsed: 0 0 0 0 1 1 0 0
.dummy: true                         % dummy_heading
.ocn_off: true                       % object_number_off
.children: 20 45 70                  % direct child heading OCNs
.heading_lev_collapsed: 2            % written when < 9
.parent_lev: 1                       % parent_lev_markup
.o_n_type: 1                         % object number type
.is_of_type: para                    % para, block
.attrib: ...                         % metainfo.attrib
.meta_lang: en                       % block language (group, block, quote)
.meta_syntax: d                      % codeblock syntax (metainfo.syntax)
```

A heading's identifier is not a property: it follows the level on the
object declaration line (`[5] heading :1 ch1_intro`).

`.parent` is written for every object, including when the value is 0,
with two exceptions: the `:A~` document root, which has no parent, and
the entries of the generated `@toc` section, which are furniture rather
than authored objects and whose place is already given by their
`.indent` level (the `@toc` heading itself does carry `.parent: 1`). A
parent of 0 is not "no parent recorded": it is a parent heading the
author has made non-substantive, its ocn suppressed with `~#`, or a
dummy heading introduced with `-#` to segment an output. Stating it
keeps "non-substantive parent" and "no parent" distinguishable, so a
reader must not treat a missing `.parent` as 0.

**Text attribute properties** (from `attrib`):

```
.indent: 1 0                         % indent_base indent_hang
.bullet: true
.lang: en                             % language override
```

**Presence flags** (from `has`):

```
.has: links notes_reg notes_star images images_no_dim
```

Only present flags are listed. Absence means false. `images_no_dim`
marks an object whose markup gave no image dimensions (`w0h0`), which
spine then derived from the file.

**Table properties** (from `table`, only for table objects):

```
.table_cols: 3
.table_widths: 40 30 30
.table_aligns: left right right
.table_header: true
.table_walls: true
```

**Code block properties** (from `code_block`, only for code objects):

```
.code_syntax: d
.code_linenumbers: true
```

**Tag properties** (from `tags`):

```
.segment: ch1_intro                   % in_segment_html
.segment_prev: acknowledgments
.segment_next: ch2
.segment_epub: ch1_intro              % segment_anchor_tag_epub
.anchor: ch1_intro                    % anchor_tag_html, when it differs
                                      % from .segment
.heading_lev_anchor: h1               % heading_lev_anchor_tag
.heading_ancestors_text: |The Wealth of Networks||||||
.anchor_tag: some_tag                 % repeatable, one per anchor tag
.lev4_subtoc: ...                     % repeatable, one per subtoc entry
```

The `heading_ancestors_text` uses `|` as separator for the 8-element
array, with empty strings between consecutive `|` markers.

**Digest** (from `metainfo`):

```
.sha256: 7A3B9F...                    % hex-encoded (uppercase), of object text
```

The digest covers the object's `text` field only (including inline notes),
not any file the text references.

**Images** (from `metainfo.sha256.images`):

An object that embeds one or more images carries one `.image` line per
image, in order of appearance within the object text:

```
.image: won_benkler_2_1.png sha256:5BA492C2C040C179FF8E46FC800B5B3AE6F7F082D9532F877141F44A366DD128 bytes:93861 px:420x342
```

The value is a record: the image filename first (the object's handle on
the image, as it appears after the `☼` marker in the text and as the file
is named in the pod's `media/image/` directory), followed by
whitespace-separated `key:value` fields:

| Field | Meaning |
|---|---|
| `sha256:` | SHA256 digest of the image file, hex-encoded, uppercase |
| `bytes:` | image file size in bytes |
| `px:` | the file's own pixel dimensions, `<width>x<height>` |
| `missing:` | `true` when the referenced file was absent or unreadable |

`px:` is the image as stored, read from the file header. It is *not* the
same thing as the `w##h##` dimensions carried in the object's text
(`☼won_benkler_2_1.png,w420h342`), which are display dimensions: author
set in the markup, or where the markup gave `w0h0` derived from the file
and capped to a maximum display width. The two commonly differ, e.g.

```
% display 640x323, file 831x420 (capped)
.image: di_evh_f10-1.png sha256:... bytes:... px:831x420
% display 82x128, file 328x512 (author set)
.image: sm_d_image.jpg sha256:... bytes:... px:328x512
```

Display dimensions are therefore not duplicated in the record: they are
already present, verbatim, in the object's text lines, and they describe
the rendering rather than the file. `px:` records what the text cannot,
and completes the file's identity alongside `sha256:` and `bytes:`.
It is omitted when the dimensions could not be read.

An image the abstraction referenced but could not read is recorded as
such, rather than being omitted:

```
.image: ffa.png missing:true
```

`missing:true` and `sha256:`/`bytes:` are mutually exclusive: no digest
or size could be taken, so neither field is written. The record still
tells a reader that the object embeds an image and names it, and tells a
verifier that this `.ssp` was produced from an incomplete pod.

Rules:

- The property is **repeatable**. An object with two images emits two
  `.image` lines; readers accumulate them into a list. Objects without
  images emit none.
- The filename is a single token (image names match
  `[a-zA-Z0-9._-]+\.(jpg|gif|png)`, so it cannot contain whitespace, `:`
  or `\n`) and is always the first field.
- Fields after the filename are **order-independent and extensible**.
  A reader MUST ignore a `key:value` field it does not recognise, and
  MUST NOT assume a fixed field count. This reserves room for later
  additions (e.g. `mime:image/png`) without a format break.
- Values are the file's own properties, so two objects embedding the same
  image emit identical `sha256:`/`bytes:`/`px:` values. Display
  dimensions, which can differ between two such objects, are a property
  of the reference and stay in the object's text.

Rationale: this is verification data (is the image shipped in this pod the
image the abstraction was built from?), so the digest of the referenced
file belongs with the object that references it, and remains valid when
the `.ssp` file travels alone. Packing filename, size and digest into the
property *name* (as `.imageSha256_filesize_filename:`) would put the
schema in the key, forcing a new property name on every added field; a
positional `::`-separated value would fix the arity for the same reason.
The pod-level `digests.txt` manifest keeps its own flat
`HEX::bytes - filename` convention: that file is a verification report,
not a grammar-bound serialisation.

**Extracted links** (from `stow`):

Repeatable, one line per link:

```
.stow_link: https://example.org
.stow_link: https://other.org
```

### Property Defaults (omission rules)

Properties with default values are omitted to reduce verbosity:

| Property | Default | Omit when |
|---|---|---|
| .part | "" | empty |
| .section | "" | empty, or same as the enclosing section |
| .parent | (none) | the `:A~` document root, and `toc` entries; otherwise always written, 0 included |
| .last_descendant | 0 | zero |
| .children | (none) | no child headings |
| .ancestors | 0 0 0 0 0 0 0 0 | all zeros |
| .ancestors_collapsed | 0 0 0 0 0 0 0 0 | all zeros |
| .dom_status | 0 0 0 0 0 0 0 0 | all zeros |
| .dom_status_collapsed | 0 0 0 0 0 0 0 0 | all zeros |
| .heading_lev_collapsed | 9 | 9 or greater |
| .parent_lev | 0 | zero |
| .o_n_type | 0 | zero |
| .is_of_type | "" | empty |
| .attrib | "" | empty |
| .meta_lang | "" | empty |
| .meta_syntax | "" | empty |
| .indent | 0 0 | both zero |
| .bullet | false | false |
| .has | (none) | no flags set |
| .dummy | false | false |
| .ocn_off | false | false |
| .code_syntax | "" | empty |
| .code_linenumbers | false | false |
| .anchor | "" | empty, or same as .segment |
| .stow_link | (none) | no links (repeatable, so no default) |
| .image | (none) | object embeds no image (repeatable, so no default) |

### What is NOT serialized

- `ptr` (DocObj_Pointer_) - runtime array indices, recomputed on load
- Computed properties (object_number(), marked_up_level(), etc.)

### What is optional

- `tags` (DocObj_Tags_) - HTML/EPUB-specific segment identifiers. These
  capture structure decisions used by multiple output generators. Serialized
  for completeness but could be recomputed from the heading hierarchy.

## PEG Grammar (Pegged syntax sketch)

```
SiSUAbstraction:
  Document     <- MetaBlock MakeBlock? DocHasBlock? Section+ eoi

  MetaBlock    <- '@meta' sp '{' nl MetaLine+ '}' nl+
  MakeBlock    <- '@make' sp '{' nl MetaLine+ '}' nl+
  DocHasBlock  <- '@doc_has' sp '{' nl MetaLine+ '}' nl+
  MetaLine     <- sp+ ~Identifier ':' sp ~Value nl

  Section      <- '@' ~SectionName sp '{' nl Object* '}' nl+
  SectionName  <- 'head' / 'toc' / 'body' / 'endnotes'
                / 'glossary' / 'bibliography' / 'bookindex' / 'blurb'

  Object       <- ObjDecl nl Property* TextLine* nl*
  ObjDecl      <- '[' ~Number ']' sp ~ObjType ObjExtra?
  ObjType      <- 'heading' / 'para' / 'toc' / 'table' / 'code'
                / 'quote' / 'group' / 'block' / 'verse'
                / 'endnote' / 'glossary' / 'bibliography' / 'bookindex'
                / 'blurb'
  ObjExtra     <- sp ~(!nl .)+

  Property     <- ImageProperty / GenericProperty
  GenericProperty
               <- '.' ~PropName ':' sp ~PropValue nl
  PropName     <- Identifier
  PropValue    <- (!nl .)+

  ImageProperty
               <- '.image' ':' sp ~ImageFile ImageField* nl
  ImageFile    <- [a-zA-Z0-9._-]+ '.' ('jpg' / 'gif' / 'png')
  ImageField   <- sp ~ImageFieldName ':' ~ImageFieldValue
  ImageFieldName
               <- Identifier
  ImageFieldValue
               <- (![ \t\n] .)+

  TextLine     <- '|' sp? ~(!nl .)* nl

  Comment      <- '%' (!nl .)* nl

  Identifier   <- [a-zA-Z_] [a-zA-Z0-9_.]*
  Number       <- [0-9]+
  Value        <- (!nl .)+
  sp           <- [ \t]+
  nl           <- '\n'
```

## File Extension

`.ssp` (SiSU Document Abstraction)

## Relationship to Source Markup

```
source.sst  -->  spineAbstraction()  -->  in-memory objects
                                              |
                                              v
                                    document.ssp  (this format)
                                              |
                                              v
                                    load & reconstruct objects
                                              |
                                              v
                                    output generators
                                    (HTML, EPUB, LaTeX, etc.)
```

The `.ssp` file captures the state AFTER parsing and abstraction but BEFORE
output generation. It is independent of any specific output format.

## Design Decisions

### Why `|` for text lines?

The pipe prefix makes text content unambiguous at the character level.
Alternative considered: indentation-based (4 spaces). Rejected because
the abstraction text can contain leading spaces (preserved whitespace in
blocks) which would be ambiguous.

The `|` character does not conflict with the internal markup Unicode
characters. While `|` appears in table cell separators in SiSU source
markup, by the abstraction stage these have been converted to `┊` (tc_s).

### Why properties before text?

Properties (`.name: value`) appear between the object declaration and the
text content. This means scanning an object goes: identity -> metadata ->
content. For downstream processing, the metadata is often needed before
the text (e.g., to decide how to render it), so this ordering matches the
access pattern.

### Why one line per property?

Diff-friendliness. When a property changes, only that line shows in a diff.
When an object is added or removed, the diff is a contiguous block.

### Why preserve internal Unicode markers?

The abstraction's `text` field has already transformed source markup
(`/{italic}/`) into internal markers (`⑆/italic⑆`). This is the canonical
form at the abstraction stage. Reverting to source markup would lose
information about processing decisions (e.g., auto-bolding, substitutions).
The format represents the abstraction faithfully, not the source.