|
|
|
SiSU
Description
Ralph Amissah
copy @ SiSU
SiSU - Description,
|
1 |
SiSU an attempt to describe |
2 |
1. Description |
1. Description |
3 |
4 |
SiSU is a flexible document preparation, generation publishing and search system. 1 |
5 |
SiSU ("SiSU information Structuring Universe" or "Structured information, Serialized Units"), 2 is a Unix command line oriented framework for document structuring, publishing and search. Featuring minimalistic markup, multiple standard outputs, a common citation system, and granular search. |
6 |
Using markup applied to a document, SiSU can produce plain text, HTML, XHTML, XML, OpenDocument, LaTeX or PDF files, and populate an SQL database with objects 3 (equating generally to paragraph-sized chunks) so searches may be performed and matches returned with that degree of granularity (e.g. your search criteria is met by these documents and at these locations within each document). Document output formats share a common object numbering system for locating content. This is particularly suitable for "published" works (finalized texts as opposed to works that are frequently changed or updated) for which it provides a fixed means of reference of content. |
7 |
SiSU is the data/information structuring and transforming tool, that has resulted from work on one of the oldest law web projects. It makes possible the one time, simple human readable markup of documents, that SiSU can then publish in various forms, suitable for paper 4 , web 5 and relational database 6 presentations, retaining common data-structure and meta-information across the output/presentation formats. Several requirements of legal and scholarly publication on the web have been addressed, including the age old need to be able to reliably cite/pinpoint text within a document, to easily make footnotes/endnotes, to allow for semantic document meta-tagging, and to keep required markup to a minimum. These and other features of interest are listed and described below. A few points are worth making early (and will be repeated a number of times): |
8 |
(i) The SiSU document generator was the first to place material on the web with a system that makes possible citation across different document types, with paragraph, or rather object citation numbering 7 a text positioning system, available for the pinpointing of text, 1997, a simple idea from which much benefit, and SiSU remains today, to the best of my knowledge, the only multiple format e-book/ electronic-document system on the web that gives you this possibility (including for relational databases). |
9 |
10 |
(iv) SiSU is a batch processor, dealing with as many files as you need to generate at a time. |
12 |
(v) Scalability is dependent on your file system (in my case Reiserfs), the database (currently Postgresql and/or SQLite) and your hardware. |
13 |
SiSU Sabaki 8 (or just SiSU) is the provisional name given to the software described here that helps structure documents for web and other publication. The name SiSU is a loose anagram for something along the lines of "SiSU is structuring unit", or "SiSU, information structuring unit" or the more descriptive "Structured information, Serialized Units" or "simple - information structuring unit" or the more descriptive "Structured information, Serialized Units" or what it may be directed towards "semantic and information structuring universe" , 9 tongue in cheek, only just. Guess I'll get away with "Simple - information Structuring Universe". SiSU is also a Finnish word roughly meaning guts, inner strength and perseverance. 10 |
14 |
SiSU was born of the need to find a way, with minimal effort, and for as wide a range of document types as possible, to produce high quality publishing output in a variety of document formats. As such it was necessary to find a simple document representation that would work across a large number of document types, and the most convenient way(s) to produce acceptable output formats. The project leading to this program was started in 1993 (together with the trade law project now known as Lex Mercatoria) as an investigation of how to effectively/efficiently place documents on the web. The unified document handling, together with features such as paragraph numbering, endnote handling and tables... appeared in 1996/97. SiSU was originally written in Perl, 11 and converted to Ruby, 12 in 2000, one of the most impressive programming languages in existence! In its current form it has been written to run on the Gnu /Linux platform, and in particular on Debian, 13 taking advantage of many of the wonderful projects that are available there. |
15 |
SiSU markup is based on requiring the minimum markup needed to determine the structure of a document. (This can be as little as saying in a header to look for the word Book at a specified level and the word Chapter at another level). SiSU then breaks a document into its smallest parts (at a heading, and paragraph level) while retaining all structural information. This break up of the document and information on its structure is taken advantage of in the transformations made in generating the very different output types that can be created, and in providing as much as can be for what each output type is best at doing, e.g. LaTeX (professional document typesetting, easy conversion to pdf or Postscript), XML (in this case, structural representation), ODF (OpenDocument [experimental]), SQL (e.g. document search; representing constituent parts of documents based on their structure, headings, chapters, paragraphs as required; user control). 14 |
16 |
From markup that is simpler and more sparse than html you get: |
17 |
|
18 |
|
19 |
|
20 |
21 |
SiSU processes files with minimal tagging to produce various document outputs including html, LaTeX or lout (which is converted to pdf) and if required loads the structured information into an SQL database (PostgreSQL and SQLite have been used for this). SiSU produces an intermediate processing format. 15 |
22 |
SiSU is used in constructing Lex Mercatoria <http://lexmercatoria.org/> or <http://www.jus.uio.no/lm/> (one of the oldest law web sites), and considerable thought went into producing output that would be suitable for legal and academic writings (that do not have formulae) given the limitations of html, and publication in a wide variety of "formats", in particular in relation to the convenient and accurate citation of text. However, the construction of Lex Mercatoria uses only a fraction of the features available from SiSU today, vis generation of flat file structures, rather than in addition the building of ("granular") SQL database content, (at an object level with relevant relational tables, and other outputs also available). |
23 |
24 |
(i) markup syntax: (a) simpler than html, (b) mnemonic, influenced by mail/messaging/wiki markup practices, (c) human readable, and easily writable, |
25 |
(ii) (a) minimal markup requirement, (b) single file marked up for multiple outputs, |
26 |
27 |
(vi) use of semantic meta-tags in headers permit the addition of semantic information on documents, (the available fields are easily extended) |
34 |
(viii) "Concordance file" wordmap, consisting of all the words in a document and their (text/ object) locations within the text, (and the possibility of adding vocabularies), |
36 |
(x) SiSU's minimalist markup makes for meaningful "diffing" of the substantive content of markup-files, |
38 |
(xi) easily skinnable, document appearance on a project/site wide, directory wide, or document instance level easily controlled/changed, |
39 |
there is a considerable degree of future-proofing, output representations are "upgradeable", and new document formats may be added. |
43 |
(xvi) scalability, dependent on your file-system (ext3, Reiserfs, XFS, whatever) and on the relational database used (currently Postgresql and SQLite), and your hardware, |
45 |
(xvii) only marked up files need be backed up, to secure the larger document set produced, |
46 |
47 |
(xix) Syntax highlighting for SiSU markup is available for a number of text editors. |
48 |
52 |
SiSU has been developed and has been in use for several years. Requirements to cover a wide range of documents within its use domain have been explored. |
54 |
56 |
SiSU markup is fairly minimalistic, it consists of: a (largely optional) document header, made up of information about the document (such as when it was published, who authored it, and granting what rights) and any processing instructions; and markup within text which is related to document structure and typeface. SiSU must be able to discern the structure of a document, (text headings and their levels in relation to each other), either from information provided in the instruction header or from markup within the text (or from a combination of both). Processing is done against an abstraction of the document comprising of information on the document's structure and its objects, 16 which the program serializes (providing the object numbers) and which are assigned hash sum values based on their content. This abstraction of information about document structure, objects, (and hash sums), provides considerable flexibility in representing documents different ways and for different purposes (e.g. search, document layout, publishing, content certification, concordance etc.), and makes it possible to take advantage of some of the strengths of established ways of representing documents, (or indeed to create new ones). |
57 |
58 |
SiSU markup is based on requiring the minimum markup needed to determine the structure of a document. (This can be as little as saying in a header to look for the word Book at a specified level and the word Chapter at another level). SiSU then breaks a document into its smallest parts (at a heading, and paragraph level) while retaining all structural information. This break up of the document and information on its structure is taken advantage of in the transformations made in generating the very different output types that can be created, and in providing as much as can be for what each output type is best at doing, e.g. LaTeX (professional document typesetting, easy conversion to pdf or Postscript), XML (in this case, structural representation), ODF (OpenDocument), SQL (e.g. document search; representing constituent parts of documents based on their structure, headings, chapters, paragraphs as required; user control). 17 |
59 |
1.4.1 Sparse markup requirement, try to get the most out of markup |
60 |
One of its strengths is that very small amounts of initial tagging is required for the program to generate its output. |
61 |
62 |
|
63 |
|
64 |
Emphasis has been on simplicity and minimalism in markup requirements. Design philosophy is to try keep the amount of markup required low, for whatever has been determined to be acceptable output. 20 |
65 |
67 |
For each document, there is only one (input, minimalistically marked up) file from which all the available output types are generated. 21 |
68 |
69 |
|
70 |
|
71 |
72 |
|
73 |
|
74 |
|
75 |
|
76 |
|
77 |
|
78 |
|
79 |
|
80 |
(and in addition to these: PostgreSQL, SQLite, texinfo and |
81 |
82 |
Syntax is kept simple and mnemonic. 33 |
83 |
1.4.4 Kept simple by having a limited publishing feature set, and features identified as most important, are available across several document types |
84 |
86 |
87 |
Output is designed to be uniform, easy to read, navigate and cite. |
88 |
89 |
Code 34 is separated from content. This means that when changes are desired in the output presentation, the code that produces them, and not the marked up text data set (which could be thousands of documents) is modified. Separating code from content makes large scale changes to output appearance trivial, and permits the easy addition of new output modules. |
90 |
1.7 Object citation numbering, a text or object positioning / citation system - "paragraph" (or text object) numbering, that remains same and usable across all output formats by people and machine |
91 |
|
93 |
|
94 |
|
95 |
An advantage is that the numbering remains the same regardless of document structure. |
101 |
Text object ("paragraph") numbering is the same for all output versions of the same document, vis html, pdf, pgsql, yaml etc. |
102 |
1.8 Handling of Dublin Core meta-tags making use of the Resource Description Framework |
105 |
SiSU is able to use meta tags based on the Dublin Core 36 and Resource Description Framework 37 |
106 |
This information is provided both in html metatags, and (where available) under the section titled "Document Information - MetaData", near the end of a document, for example in the segmented html version of this text at: <http://www.jus.uio.no/sisu/SiSU/metadata.html> |
108 |
109 |
1. Directory file association, skins and special image management, made simpler. 38 |
110 |
/www/docs |
112 |
all are placed in their own directories within the directory structure created. Similar rules are used in the creation of sql type databases (though they can be overridden). |
113 |
There are a couple of further associations with these directories. |
114 |
115 |
116 |
118 |
119 |
c. document skin (takes precedence wherever document requests a specific skin) |
120 |
3. If the working directory has within it a sub-directory called image_local, the images within that directory are used for references to images, that are not part of the default site build. |
122 |
123 |
124 |
Permits the inclusion of document version control information to the document body and metatags. 39 This provides a much more certain method of referring to the exact version of a particular document, (assuming that the document is from a trusted source, that will retain earlier versions of a document). 40 |
125 |
This information (where available) is provided under the section of the document titled "Document Information - MetaData", near the end of a document, for example in the segmented html version of this text at: <http://www.jus.uio.no/sisu/SiSU/metadata.html> |
126 |
127 |
SiSU produces a rudimentary a table of contents based on document headings. |
128 |
129 |
Headings can be automatically numbered, (and automatically named for hyper-linking) |
130 |
131 |
SiSU can automatically number footnotes/endnotes. This is the default operation where no number is provided. |
132 |
Footnotes/endnotes may also be manually numbered. Where a number, or numbers are provided for a footnote/endnote, this does not increment the automatic footnote/endnote number counter. |
133 |
In the html output footnotes/endnotes are cross-hyper-linked (to their reference point and vice versa). In th pdf output footnotes are linked from their reference point only. |
134 |
135 |
Whilst it is skinnable, the default output styles are selected to work across the widest possible range of document types. |
137 |
138 |
From markup that is simpler and more sparse than html you get: |
139 |
|
140 |
|
141 |
|
142 |
1.15.1 html - several presentations: full length & segmented; css & table based |
144 |
Most documents are produced in single and segmented html versions, described below: |
145 |
146 |
The full length of the text in a single scrollable document. 42 As a rule the files they are saved in are named: doc or more precisely doc.html |
147 |
For various reasons texts may only be provided in this form (such as this one which is short), though most are also provided as segmented texts. |
148 |
"Scroll" is a reference to the historical scroll, a single long document/ parchment, and also no doubt to what you will have to do to get to the bottom of the text. 43 |
149 |
150 |
The text divided into segments (such as articles or chapters depending on the text) 44 As a rule the files they are saved in are named: toc and index or more precisely toc.html and index.html |
151 |
If you know exactly what you are looking for, loading a segment of text is faster (the segments being smaller). Occasionally longer documents such as the WTA 1994 <http://www.jus.uio.no/lm/wta.1994/toc> are only provided in segmented form. |
152 |
153 |
SiSU outputs html, two current standard forms available are: |
154 |
155 |
156 |
table based [largely discontinued ] 45 |
157 |
158 |
I like to remind you that there are other excellent browsers out there, many of which have long supported practical features like tabbing. |
159 |
160 |
|
161 |
|
162 |
|
163 |
|
164 |
|
165 |
|
166 |
|
167 |
168 |
|
169 |
|
170 |
|
171 |
172 |
|
173 |
|
174 |
|
175 |
The html tables output is rendered more accurately across a wider variety set and older versions of browsers (than the html css output). |
176 |
177 |
179 |
180 |
1.15.4 PDF - portrait and landscape, (through the generation of LaTeX output which is then transformed to pdf) |
181 |
SiSU outputs LaTeX if required which is easily transformed to PDF. 59 PDF documents are generated on the site from the same source files and Ruby program that produce html. Landscape oriented pdf introduced, providing easier screen viewing, they are also (paper saving, being currently) formatted to have fewer pages than their portrait equivalents. |
182 |
|
183 |
|
184 |
|
185 |
1.15.5 Search - loading/populating of relational database while retaining document structure information, object citation numbering and other features (currently PostgreSQL and/or SQLite) |
186 |
SiSU (from the same markup input file) automatically feeds into PostgreSQL 63 and/or SQLite 64 database (could be any other of the better relational databases) 65 - together with all additional information related to document structure, and the alternative ways in which it is generated on the site retained. As regards scaling of the database, it is as scalable as the database (here Postgresql or SQLite) and hardware allow. I will prune the images later. |
187 |
|
189 |
|
191 |
|
192 |
There is of course the possibility to add further structures. |
193 |
This is a larger scale project, (with little development on the front end largely ignored), though the "infrastructure" has been in place since 2002. |
197 |
1.15.6 Search - database frontend sample, utilising database and SiSU features, including object citation numbering (backend currently PostgreSQL) |
198 |
Sample search frontend 66 A small database and sample query front-end (search from) that makes use of the citation system, object citation numbering to demonstrates functionality. 67 |
199 |
(further work needs to be done on the sample search form, which is rudimentary and only passes simple booleans correctly at present to the SQL engine) |
201 |
202 |
203 |
204 |
205 |
206 |
Note that the searches done in this form are case sensitive. |
207 |
Expand those same searches, showing the matching text in each document: |
208 |
209 |
210 |
211 |
212 |
Note you may set results either for documents matched and object number locations within each matched document meeting the search criteria; or display the names of the documents matched along with the objects (paragraphs) that meet the search criteria. 68 |
213 |
OCN index mode, (object citation number) the numbers displayed are relevant (and may be used to reference the match) in any sisu generated rendition of the text 69 the links provided are to the locations of matches within the html generated by SiSU. |
214 |
Paragraph mode, you may alternatively display the text of each paragraph in which the match was made, again the object/paragraph numbers are relevant to any SiSU generated/published text. |
215 |
Earlier version of the search frontend - Simple search, results with files in which search found, and locations where found within files. |
217 |
Simple search, results with files in which search found, and text object (paragraph or endnote) where found within files. |
218 |
219 |
221 |
Concordance /WordMaps: 70 SiSU produces a rudimentary index based on the words within the text, making use of paragraph numbers to identify text locations. This is generated in html and hyper-linked but identifies these words locations in the other document formats. Though it is possible to search using a search engine, this is a means for browsing an alphabetical list of words which may suggest other useful content. |
222 |
1.17 Managed (document) directory, database, or site structure |
223 |
225 |
SiSU is a batch processing tool, handling and transforming multiple (or individual) documents (in many ways) with a single instruction. |
226 |
227 |
229 |
Unix provides many tools for version control. For documents Subversion, CVS and even the old RCS are useful for the per-document histories they provide. |
230 |
Several backup tools exist. At the base level I tend to use rdiff. |
232 |
233 |
SiSU documents are prepared / marked up in utf-8 text you are free to use the text editor of your choice. |
234 |
Syntax highlighting for a number of editors are provided. Amongst them Vim, Kwrite, Kate, Gedit and diakonos. These may be found with configuration instructions at <http://www.jus.uio.no/sisu/syntax_highlight>. Vim 71 as of version 7 has built in sytax highlighting for SiSU. |
235 |
236 |
Need a new output format that does not already exist, write a new module. |
237 |
1. This information was first placed on the web 12 November 2002; with predating material taken from <http://www.jus.uio.no/lm/lm.information/toc.html> part of a site started and developed since 1993. See document metadata section <http://www.jus.uio.no/sisu/SiSU/metadata.html> for information on this version. Dates related to the development of SiSU are mostly contained within the Chronology section of this document, e.g. <http://www.jus.uio.no/sisu/sisu_chronology> |
2. also chosen for the meaning of the Finnish term "sisu". |
3. objects include: headings, paragraphs, verse, tables, images, but not footnotes/endnotes which are numbered separately and tied to the object from which they are referenced. |
4. pdf via LaTeX or lout |
5. currently html (two forms of html presentation one based on css the other on tables), and PHP; potentially structured XML |
6. any SQL - currently PostgreSQL and sqlite (for portability, testing and development) |
7. previously called "text object numbering" |
8. SiSU Sabaki, release version. Pre-release version SiSU Scribe, and version prior to that SiSU nicknamed Scribbler. Pre-release versions go back several years. Both Scribbler and Scribe (still maintained) made system calls to SiSU's various parts, instead of using libraries. |
9. A little universe it may be, but semantic you may have a hard time getting away with, given the meaning the word has taken on with markup. On a document wide basis semantic information may be provided, which can be really useful, (and meaningful, especially) if you have a large document set, and use this with rss feeds or in an sql database etc. On a markup level, I have little inclination to add semantic markup formally beyond references, title, author [Dublin Core entities? addresses?] etc. Actually this deserves a bit of thought possibly use letter tags (including letter alias/synonyms for font faces) to create a small set of default semantic tags, with the possibility for per document adjustments. Will seek to permit XML entity tagging, within SiSU markup and have that ignored/removed by the parts of the program that have no use for it. |
10. "Sisu refers not to the courage of optimism, but to a concept of life that says, 'I may not win, but I will gladly give my life for what I believe.'" Aini Rajanen, Of Finnish Ways, 1981, p. 10. |
"Every Finn has his own pet definition. To me, sisu means patience without passion. But there are many varieties of sisu. Sisu can be a sudden outburst or it can be the kind that lasts. A man can have both kinds. It is outside reason. It is something in the soul. It comes from oneself. For instance, it makes a soldier do things because he himself must, not because he has been told." Paavo Nurmi |
<http://personalweb.smcvt.edu/tmatikainen/finnishtraditions.htm> |
14. where explicit structure is provided through the use of tagging headings, it could be reduced (still) further, for example by reducing the number of characters used to identify heading levels; but in many cases even that information is not required as regular expressions can be used to extract the implicit structure. |
15. This proved to be the easiest way to develop syntax, changes could be made, or alternatives provided for the markup syntax whilst the intermediate markup syntax was largely held constant. There is actually an optional second intermediate markup format in YAML <http://www.yaml.org/> |
16. objects include: headings, paragraphs, verse, tables, images, but not footnotes/endnotes which are numbered separately and tied to the object from which they are referenced. |
17. where explicit structure is provided through the use of tagging headings, it could be reduced (still) further, for example by reducing the number of characters used to identify heading levels; but in many cases even that information is not required as regular expressions can be used to extract the implicit structure. |
18. <http://www.jus.uio.no/sisu/sample/markup/un_contracts_international_sale_of_goods_convention_1980.sst> output provided as example in the next section |
19. <http://www.jus.uio.no/sisu/sample/syntax/un_contracts_international_sale_of_goods_convention_1980.sst.html> as it would appear with syntax highlighting (by vim) |
20. seems there are several "smart ASCIIs" available, primarily for ascii to html conversion, that make this, and reasonable looking ascii their goal |
21. These include richly laid out and linked html (table or css variants), PHP, LaTeX (from which pdf portrait and landscape documents are produced), texinfo (for info files etc.), and PostgreSQL and/or SQLite. And the opportunity to fairly easily build additional modules, such as XML. See the examples provided in this document. |
22. <http://www.jus.uio.no/sisu/sample/markup/un_contracts_international_sale_of_goods_convention_1980.sst> |
23. <http://www.jus.uio.no/sisu/sample/syntax/un_contracts_international_sale_of_goods_convention_1980.sst.html> |
24. <http://www.jus.uio.no/sisu/un_contracts_international_sale_of_goods_convention_1980/toc.html> |
25. <http://www.jus.uio.no/sisu/un_contracts_international_sale_of_goods_convention_1980/doc.html> |
26. <http://www.jus.uio.no/sisu/un_contracts_international_sale_of_goods_convention_1980/landscape.pdf> |
27. <http://www.jus.uio.no/sisu/un_contracts_international_sale_of_goods_convention_1980/portrait.pdf> |
28. <http://www.jus.uio.no/sisu/un_contracts_international_sale_of_goods_convention_1980/plain.txt> |
29. <http://www.jus.uio.no/sisu/un_contracts_international_sale_of_goods_convention_1980/sax.xml> |
30. <http://www.jus.uio.no/sisu/un_contracts_international_sale_of_goods_convention_1980/dom.xml> |
31. <http://www.jus.uio.no/sisu/un_contracts_international_sale_of_goods_convention_1980/concordance.html> |
32. discontinued for the time being |
33. SiSU markup syntax, an incomplete summary: <http://www.jus.uio.no/sisu/sisu_markup_table/doc.html#h200306> |
Visual check of elementary font face modifiers: bold bold emphasis italics underscore |
34. the program that generates the documents |
38. The previous way was directory associations for file output were set up in the configuration file. The present system is a more natural way to work requireing less configuration. |
39. from a version control system such as CVS |
40. The version control system must be run, so the version number is obtained, prior to the SiSU document generation, and subsequent posting of the document. |
41. e.g. LaTeX (professional document typesetting, easy conversion to pdf or Postscript), XML (in this case, structural representation), SQL (e.g. document set searches; representation of the constituent parts of documents based on their structure, headings, chapters, paragraphs as desired; control of use) |
42. CISG <http://www.jus.uio.no/lm/un_contracts_international_sale_of_goods_convention_1980/doc> |
The Unidroit Contract Principles <http://www.jus.uio.no/lm/unidroit.contract.principles.1994/doc> or |
The Autonomous Contract <http://www.jus.uio.no/lm/autonomous.contract.2000.amissah/doc> |
43. Scrolling is not however necessarily confined to full length documents as you will have to scroll to get to the bottom of any long segment (eg. chapter) of a segmented text. |
44. CISG <http://www.jus.uio.no/sisu/un_contracts_international_sale_of_goods_convention_1980> |
The Unidroit Principles <http://www.jus.uio.no/lm/unidroit.contract.principles.1994> |
The Autonomous Contract <http://www.jus.uio.no/sisu/the.autonomous.contract.2000.amissah> or |
WTA 1994 <http://www.jus.uio.no/lm/wta.1994> |
45. formatting possibility still exists in code tree but maintenance has been largely discontinuted. |
51. <http://home.netscape.com/comprod/mirror/client_download.html> |
59. LaTeX and pdf features introduced 18th June 2001, Landscape and portrait pdfs introduced 7th October 2001., Lout is a more recent addition 22th April 2003 |
65. Relational database features retaining document structure and citation introduced 15th July 2002 |
67. (which could be extended further with current back-end). As regards scaling of the database, it is as scalable as the database (here Postgresql) and hardware allow. |
68. of this feature when demonstrated to an IBM software innovations evaluator in 2004 he said to paraphrase: this could be of interest to us. We have large document management systems, you can search hundreds of thousands of documents and we can tell you which documents meet your search criteria, but there is no way we can tell you without opening each document where within each your matches are found. |
69. OCN are provided for HTML, XML, pdf ... though currently omitted in plain-text and opendocument format output |
70. Concordance/ WordMaps introduced 15th August 2002 |
|
|
|
Output generated by
SiSU
0.59.0 2007-09-23 (2007w38/0)
|
SiSU using:
| |
SiSU is released under GPLv3 or later, <http://www.gnu.org/licenses/gpl.html> |
SiSU, developed using
Ruby
on
Debian/Gnu/Linux
software infrastructure,
with the usual GPL (or OSS) suspects.
|