1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
|
-*- mode: org -*-
#+TITLE: sisu project bespoke homepage
#+DESCRIPTION: documents - structuring, various output representations & search
#+FILETAGS: :spine:hub:
#+AUTHOR: Ralph Amissah
#+EMAIL: [[mailto:ralph.amissah@gmail.com][ralph.amissah@gmail.com]]
#+COPYRIGHT: Copyright (C) 2015 - 2024 Ralph Amissah
#+LANGUAGE: en
#+STARTUP: content hideblocks hidestars noindent entitiespretty
#+PROPERTY: header-args+ :eval never-export :exports code
#+PROPERTY: header-args+ :noweb yes :padline no
#+PROPERTY: header-args+ :results silent :cache no
#+PROPERTY: header-args+ :mkdirp yes
#+OPTIONS: H:3 num:nil toc:t \n:t ::t |:t ^:nil -:t f:t *:t
- magic single double-quote → " ← FIX changes hilighting behavior (occuring
after it) in org document. INVESTIGATE (org-mode CONFIG?) FIND & FIX
* homepage index.html
#+HEADER: :tangle "../markup/sisudoc-spine-bespoke-output/html/homepage.index.html"
#+BEGIN_SRC html
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<title>≅ SiSU project sisudoc.org</title>
<link href="./css/html_seg.css" rel="stylesheet" />
</head>
<body>
<h1>≅ SiSU - lightweight markup, object-centric documents,
multiple outputs & search</h1>
<p>
SiSU parses a lightweight-markup source into an abstract document object
model. Every substantive element (paragraph, heading, table, verse, image)
becomes a typed object carrying its position in the document's sequence
and in its heading hierarchy, and a stable citation number. From that single
abstraction it emits multiple output formats - HTML (segmented and scroll),
EPUB3, LaTeX (then PDF via xelatex), ODT, plain text, and an SQLite full-text
search database. Each object's number stays stable across every output
format and across translations of the same document.
</p>
<p>
The processing pipeline is <b>markup → abstraction →
output</b>.
</p>
<p>
<b>Object-Centric Document Abstraction</b>. The abstraction stage builds an
in-memory object model along two bound dimensions: <b>document structure</b>
(the heading hierarchy - title, section, part, chapter, sub-headings) and
<b>document objects</b> (every paragraph, heading, table, footnote and so on,
as a typed unit). The two are mapped to each other - each object knows its
place in the heading hierarchy, and each structural node knows the objects
that belong to it. Substantive objects also carry a stable citation number
(OCN, Object Citation Numbering). Every output format is generated from this
single abstraction, so all formats share the same object identifiers and the
same structural shape. The abstraction can also be written out as a
human-readable, PEG-parsable text format (<code>.ssp</code>) that other tools
can consume directly.
</p>
<h2>ℹ - How this differs from a typical "markup → HTML"
pipeline</h2>
<p>
<ul>
<li><b>Citation that survives format conversion.</b> Quote object 412 and the
reference is meaningful in the HTML, the EPUB, the PDF, the plain text and the
SQLite search results - and in any translation, because OCN is a property of
the abstraction, not of pagination or layout.</li>
<li><b>Object-granular search.</b> The SQLite database is populated at object
granularity. A query reports not just "this document matches" but "object 412
in this document matches" - and links straight back to that object in the
published HTML.</li>
<li><b>Inspectable intermediate form.</b> The document abstraction has a
human-readable, PEG-parsable text serialisation (<code>.ssp</code>). Other
tools - in any language - can consume the abstraction without re-implementing
the parser. This is also what lets the abstraction stage be reasoned about,
diffed, fed to embedding pipelines, or used as the input to custom
renderers.</li>
<li><b>Deterministic and reproducible.</b> The same markup source produces the
same OCN sequence and the same outputs every time. Per-object content hashes
can be exposed for content identification or verification without disclosing
the content itself.</li>
<li><b>Designed for finished, "published" works.</b> SiSU is aimed at writings
that are published as a stable artefact (books, essays, articles, legal and
regulatory texts), where a fixed citable reference of object-level granularity
is more valuable than the flexibility of fluid text.</li>
<li><b>Static output, optional search.</b> Generated content is static HTML /
EPUB / PDF / text - trivial to host and to archive. The SQLite + CGI search is
an opt-in component that adds object-granular full-text query without changing
the publishing model.</li>
</ul>
</p>
<h2>⌘ - See it in action</h2>
<p>
A single document - <i>The Wealth of Networks</i>, Yochai Benkler -
shown in every output format SiSU Spine produces. The same OCN
identifies the same object in each:
</p>
<ul>
<li><a href="./en/html/the_wealth_of_networks.yochai_benkler/toc.html">
HTML (segmented, one page per chapter)</a></li>
<li><a href="./en/html/the_wealth_of_networks.yochai_benkler.html">
HTML (single scroll)</a></li>
<li><a href="./en/epub/the_wealth_of_networks.yochai_benkler.en.epub">
EPUB</a></li>
<li><a href="./pdf/the_wealth_of_networks.yochai_benkler.en.a4.portrait.pdf">
PDF (LaTeX → xelatex, A4)</a></li>
<li><a href="./en/html/metadata.the_wealth_of_networks.yochai_benkler.html">
Metadata page</a></li>
<li><a href="./spine_search?fn=the_wealth_of_networks.yochai_benkler&rt=txt&ec=on&url=on&sml=1000">
Search within this document (object-granular)</a></li>
<li><a href="./pod/the_wealth_of_networks.yochai_benkler/">
Source pod (markup + assets + manifest)</a></li>
</ul>
<h2>⌘ - Browse and search the sample collection</h2>
<p>
<a href="./authors.html">⌘ Authors</a>
-
<a href="./topics.html">⌘ Topics</a>
-
<a href="./spine_search">፨ Search</a>
<br>
(Authors and Topics are software-curated from each document's
header metadata. Search is object-granular.)
</p>
<div class="p">
<!-- SiSU Spine Search -->
<form action="https://sisudoc.org/spine_search" target="_top"
method="POST" accept-charset="UTF-8" id="search">
<input type="text" name="sf" size="32" maxlength="255"
placeholder="search the collection...">
<input type="hidden" name="db" value="spine.search.db">
<input type="hidden" name="sml" value="1000">
<input type="hidden" name="ec" value="on">
<input type="hidden" name="url" value="on">
<button type="submit" form="search"> ㏈ ፨ </button>
</form>
<!-- SiSU Spine Search -->
</div>
<p>
The collection contains 25+ documents released under various Creative Commons
licences, in the public domain, or as the author's own work (with one
GPL-licensed exception and the abandoned Debian live-manual). A specialised
collection would benefit from a consistently applied bespoke ontology or
thesaurus.
</p>
<h2>Δ - Source repositories</h2>
<p>
All project repositories are at
<a href="https://git.sisudoc.org">https://git.sisudoc.org</a>:
</p>
<ul>
<li><b>sisudoc-spine</b> (D) - the current generator
<br><code>git clone git://git.sisudoc.org/software/sisudoc-spine</code></li>
<li><b>sisudoc-spine-search-cgi</b> (D) - object-granular CGI search
<br><code>git clone git://git.sisudoc.org/software/sisudoc-spine-search-cgi</code></li>
<li><b>sisudoc-spine-samples</b> - 25+ marked-up sample documents
<br><code>git clone git://git.sisudoc.org/markup/sisudoc-spine-samples</code></li>
<li><b>sisu</b> (Ruby, original/antecedent) - the original generator
<br><code>git clone git://git.sisudoc.org/software/sisu</code></li>
<li><b>sisu-markup-samples</b> - samples for the original sisu
<br><code>git clone git://git.sisudoc.org/markup/sisu-markup-samples</code></li>
<li><b>tree-sitter-sisu</b> - tree sitter for sisu markup
<br><code>git clone git://git.sisudoc.org/tools/tree-sitter-sisu</code></li>
</ul>
<h2>ℹ - Spine vs. the original sisu</h2>
<p>
Spine (D) and the original sisu (Ruby) share the same lightweight body markup;
spine moves the document header to YAML where the original uses a bespoke header
dialect. Spine is roughly 60x faster on equivalent inputs (a one-minute Ruby run
is about a one-second D run). Spine emits HTML, EPUB, LaTeX, ODT, plain text and
the SQLite search database; PDF is delegated to an external xelatex pass (slower
but produces excellent output). For output formats both produce, spine's
representations are generally more up to date. Spine was released publicly under
AGPLv3 on 2024-05-01.
</p>
<hr>
<!-- Below the fold: long-form material wrapped in <details> so the
homepage does not have to render or scroll past it on first
paint. The content is unchanged, just moved. -->
<details>
<summary><b>ℹ - A longer description (design and intent)</b></summary>
<p>
<b>Summary.</b> An object is a unit of text within a document, the most common
being a paragraph. Objects include individual headings, paragraphs, tables,
and grouped text of various types such as code blocks and (within poems)
verse. Objects have properties and attributes; of particular significance are
headings and their levels, which provide document structure. A heading is an
object with a hierarchical value that conceptually contains other objects
(such as paragraphs and possibly sub-headings). Objects are tracked
sequentially as they relate to each other within a document, and substantive
objects are numbered sequentially for citation purposes. Notably, footnotes
are not objects in themselves - they belong to the object from which they are
referenced, and follow their own numbering sequence. From heading objects,
linked tables of content may be generated; and if additional metadata is
provided, book-style indexes can be generated that link back to the objects to
which they relate.
</p>
<p>
<b>Object-centricity.</b> In SiSU, objects are the fundamental unit from which
larger constructs and the document itself are built. Breaking the document
into objects provides interesting possibilities.
</p>
<p>
<b>Objects are fundamental building blocks.</b> Objects are usually blocks of
text - paragraphs, headings, tables, grouped text of various types including
code blocks and verse - and may also be, for example, images. Objects can be
formatted and placed as needed, enabling multiple types of representation
across disparate formats and text receptacles: HTML, EPUB, LaTeX, (in the
past, mind-maps) and SQL (populated at object level, so that search has that
degree of granularity).
</p>
<p>
<b>Sequence.</b> Objects have sequence - this follows authorship and is part
of how a document conveys meaning.
</p>
<p>
<b>Object numbers and citation.</b> Substantive objects are numbered
sequentially and can be referenced for citation purposes. Object numbers
locate text precisely across different document formats and different
languages (assuming the document has been translated). For search, they
identify precisely where within each document the search criteria are met - in
the form of an index, or by surfacing the matching text objects so a reader
can decide which documents are of interest before opening them. Object
numbering also frees the representation of each format to be whatever is most
suitable to that format, while structural and citation integrity are retained.
</p>
<p>
<b>Characteristics.</b> Objects have properties (the fundamental type:
heading, paragraph, table, verse, etc.) and may carry attributes (e.g.
indentation, language, programming language for a code block).
</p>
<p>
<b>Document structure.</b> Headings hold the document's structure through
their heading-level property. Headings are individual objects like any other,
with the additional properties that (i) they may be regarded as containing the
other objects following them sequentially (until the next heading of similar
or higher level), and (ii) they have a hierarchy, the root being the document
title. To give greater flexibility across output formats, headings have two
sets of levels: the level under which substantive text occurs (chapter or
segment), and above that, optional document section separators (book, section,
part).
</p>
<p>
<b>Non-objects.</b> Footnotes are not objects in themselves; they belong to
the referencing object and follow their own numbering sequence. Tables of
content may be generated from heading objects; book-style indexes may be
generated when the required metadata is provided.
</p>
<p>
<b>The document header.</b> A SiSU document has a header carrying document
metadata - at a minimum, title and author. The header may also carry markup
instructions (e.g. how to identify headings within the document, so that those
headings do not need to be inferred).
</p>
</details>
<details>
<summary><b>ℹ - Historical description (original sisu)</b></summary>
<p>
With minimal preparation of a plain-text (UTF-8) file using SiSU markup syntax
in your text editor of choice, SiSU can generate various document formats,
most of which share a common object numbering system for locating content -
plain text, HTML, XHTML, XML, EPUB, OpenDocument text (ODT), LaTeX, PDF - and
populate an SQL database with objects (roughly paragraph-sized chunks) so
searches may be performed and matches returned with that degree of
granularity. Think of being able to finely match text across different output
formats (same object identifier for PDF, EPUB or HTML) and across languages
where translations exist (same object identifier across languages). For
search, your criteria are met by these documents at these locations within
each document (equally relevant across different output formats and
languages). Page numbers provide none of this functionality. Object numbering
is particularly suitable for "published" works (finalised texts as opposed to
works that are frequently changed or updated), for which it provides a fixed
means of reference of content. Document outputs can also share provided
semantic metadata.
</p>
<p>
SiSU is less about document layout than about finding a way, using little
markup, to construct an abstract representation of a document that makes it
possible to produce multiple representations - which may be rather different
from each other and used for different purposes - whether layout and
publishing, scrollworthy online viewing, or content search. The aim is to take
advantage, from a minimal-preparation starting point, of some of the strengths
of rather different established ways of representing documents for different
purposes: search (relational database, or indexed flat files of complete
documents or files made up of objects), online or electronic viewing (HTML,
XML, EPUB), or paper publication (PDF via LaTeX).
</p>
<p>
The solution arrived at is to extract structural information about the
document (sections and headings, available through pattern matching or markup)
and to track objects (defined units of text such as paragraphs, headings,
tables, verse, etc., but also images), which can then be reconstituted as the
same document with relevant object identification numbers - so text (objects)
can be referenced across different output formats and presentations.
</p>
<p>
SiSU generates tables of content and, through its markup, the means for
metadata to be provided for the generation of book-style indexes for a
document (that, again, due to document object numbers, are the same and
equally relevant across all output formats). Per-document
classifying/organizing metadata can also be provided for automated document
curation.
</p>
<p>
There have also been working experiments with SiSU-markup source: two-way
conversion/representation in mind-mapping software (kdissert / semantik, for
its strong focus on producing documents); and po4a (for translators) has been
used successfully in its regular text mode for SiSU markup in translation -
which is more an attribute of po4a than of SiSU, but of interest due to
SiSU/spine's object citation numbering being available across translations.
ODT has been an output, but much more interesting (and requested by potential
users) would be the ability of a word processor to save text in SiSU markup,
making alternative document processing and presentations with SiSU possible.
</p>
<p>
Also worth mention: in the relatively long history of this project there has
been work on extracting hash representations of each object that could
hypothetically be shared to prove the content of a document without sharing
its content, or to identify which objects change. These hashes can also be
used as unique identifiers in a database, or as filenames if individual
objects are saved.
</p>
</details>
<details>
<summary><b>ℹ - From a 2004 evaluation (IBM Software
Innovations)</b></summary>
<p>
SiSU has evolved; the current implementation focuses on one primary use-case,
books and literary writings. The concept, however, has wider application. The
following is a souvenir from an encounter with an IBM software evaluator in
London in June 2004, set up after a chance meeting with an IBM manager at a
Linux Expo who was curious about my interest in GNU/Linux given my legal
background - on hearing that I also wrote software, he suggested IBM should
have a look. The evaluator's response after the meeting:
</p>
<p>
"Ralph<br>
Good to meet with you today, I was very impressed with your software.<br>
<i>[colleague's name (also posted to an IBM colleague)]</i> - in summary -
Ralph has built an application that runs on linux and takes ASCII documents
and pulls them apart in to the smallest constituent parts, storing them as
XML, PDF and HTML; the HTML are hyperlinked up so the document can be browsed
in its full form. The format and text data created is stored in a
database. <br>This has potential in any place that needs the power of full
text search whilst holding the structural concepts of the document i.e. legal,
pharma, education, research.. which ones we need to figure out, ..."
</p>
<p>
Special interest was expressed in the search implications of SiSU. To
paraphrase: the company has document management systems dealing with hundreds
of thousands of texts; these tell you which documents match your search
criteria, but cannot inform you where within a text these matches were found
without opening the documents. SiSU addresses this by defining document
objects and making them the building block of the document - trackable objects
that can be placed back in the context of the document or corpus of documents
if part of a collection. SiSU's early design was to abstract documents to
their structure and identified objects, numbered in a citable way (as the
evaluator pointed out, document-object hashes can be of use for the purpose).
</p>
</details>
<details>
<summary><b>ℹ - Some observations</b></summary>
<p>
SiSU is more suited to finalised / stratified / published writings (articles,
books) that are to remain and be referenced as published - works set at a
given time. (As opposed to the increasingly prevalent and important forms of
fluid text.)
</p>
<p>
Trained AI could likely assist in the preparation of documents with SiSU
markup, with resulting deterministic and reproducible outputs (for substantive
document objects). Caveats: where text objects may be in blocks (or not),
there is some room for discretion and ambiguity in the markup, with resulting
possibility of differences in presentation. Book indexes are another
markup-intensive area; unless following an already published index, they can
be prepared differently and possibly improved over time, and for specialised
subject collections could potentially be prepared against a thesaurus.
</p>
</details>
<!--
<hr>
<p>
<a href="./links.html">Personal-interest external links</a> (toolchain,
distributions, editors, forges).
</p>
--!>
<hr>
<p class="tiny"><i>
ralph.amissah - www since 1993 ;-)
</i></p>
</body>
</html>
#+END_SRC
|
