1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
|
<!-- manual page source format generated by PolyglotMan v3.2, -->
<!-- available at http://polyglotman.sourceforge.net/ -->
<html>
<head>
<title>"sisu_search"("1") manual page</title>
</head>
<body bgcolor='white'>
<a href='#toc'>Table of Contents</a><p>
SISU - SEARCH, RALPH AMISSAH
<p> SISU SEARCH
<p> 1. SISU SEARCH - INTRODUCTION
<p> <b>SiSU</b> output can easily and conveniently be indexed by a number of standalone
indexing tools, such as Lucene, Hyperestraier.
<p> Because the document structure
of sites created is clearly defined, and the text object citation system
is available hypothetically at least, for all forms of output, it is possible
to search the sql database, and either read results from that database,
or just as simply map the results to the html output, which has richer
text markup.
<p> In addition to this <b>SiSU</b> has the ability to populate a relational
sql type database with documents at an object level, with objects numbers
that are shared across different output types, which make them searchable
with that degree of granularity. Basically, your match criteria is met by
these documents and at these locations within each document, which can
be viewed within the database directly or in various output formats.
<p> 2.
SQL
<p> 2.1 POPULATING SQL TYPE DATABASES
<p> <b>SiSU</b> feeds sisu markupd documents
into sql type databases PostgreSQL[^1] and/or SQLite[^2] database together
with information related to document structure.
<p> This is one of the more
interesting output forms, as all the structural data of the documents are
retained (though can be ignored by the user of the database should they
so choose). All site texts/documents are (currently) streamed to four tables:
<p> * one containing semantic (and other) headers, including, title, author,<br>
subject, (the Dublin Core...);<br>
<p> * another the substantive texts by individual<br>
along with structural information, each paragraph being identifiable
by its<br>
paragraph number (if it has one which almost all of them do), and the<br>
substantive text of each paragraph quite naturally being searchable
(both in<br>
formatted and clean text versions for searching); and<br>
<p> * a third containing endnotes cross-referenced back to the paragraph
from<br>
which they are referenced (both in formatted and clean text versions
for<br>
searching).<br>
<p> * a fourth table with a one to one relation with the headers table
contains<br>
full text versions of output, eg. pdf, html, xml, and ascii.<br>
<p> There is of course the possibility to add further structures.
<p> At this
level <b>SiSU</b> loads a relational database with documents chunked into objects,
their smallest logical structurally constituent parts, as text objects,
with their object citation number and all other structural information
needed to construct the document. Text is stored (at this text object level)
with and without elementary markup tagging, the stripped version being
so as to facilitate ease of searching.
<p> Being able to search a relational
database at an object level with the <b>SiSU</b> citation system is an effective
way of locating content generated by <b>SiSU</b> object numbers, and all versions
of the document have the same numbering, complex searches can be tailored
to return just the locations of the search results relevant for all available
output formats, with live links to the precise locations in the database
or in html/xml documents; or, the structural information provided makes
it possible to search the full contents of the database and have headings
in which search content appears, or to search only headings etc. (as the
Dublin Core is incorporated it is easy to make use of that as well).
<p> 3.
POSTGRESQL
<p> 3.1 NAME
<p> <b>SiSU</b> - Structured information, Serialized Units -
a document publishing system, postgresql dependency package
<p> 3.2 DESCRIPTION
<p> Information related to using postgresql with sisu (and related to the
sisu_postgresql dependency package, which is a dummy package to install
dependencies needed for <b>SiSU</b> to populate a postgresql database, this being
part of <b>SiSU</b> - man sisu).
<p> 3.3 SYNOPSIS
<p> sisu -D [instruction] [filename/wildcard
if required]<br>
<p> sisu -D --pg --[instruction] [filename/wildcard if required]<br>
<p> 3.4 COMMANDS
<p> Mappings to two databases are provided by default, postgresql
and sqlite, the same commands are used within sisu to construct and populate
databases however -d (lowercase) denotes sqlite and -D (uppercase) denotes
postgresql, alternatively --sqlite or --pgsql may be used
<p> <b>-D or --pgsql</b> may
be used interchangeably.
<p> 3.4.1 CREATE AND DESTROY DATABASE
<p>
<dl>
<dt><b> --pgsql --createall</b>
</dt>
<dd> initial step, creates required relations (tables, indexes) in existing
(postgresql) database (a database should be created manually and given
the same name as working directory, as requested) (rb.dbi) the same name
as working directory, as
<p> </dd>
<dt><b> sisu -D --createdb</b> </dt>
<dd> creates database where no database
existed before as
<p> </dd>
<dt><b> sisu -D --create</b> </dt>
<dd> creates database tables where no database
tables existed before database tables where no database tables existed
<p> </dd>
<dt><b> sisu -D --Dropall</b> </dt>
<dd> destroys database (including all its content)! kills data
and drops tables, indexes and database associated with a given directory
(and directories of the same name). a
<p> </dd>
<dt><b> sisu -D --recreate</b> </dt>
<dd> destroys existing
<p> database and builds a new empty database structure
<p> </dd>
</dl>
3.4.2 IMPORT AND REMOVE
<p>DOCUMENTS
<p>
<dl>
<dt><b> sisu -D --import -v [filename/wildcard]</b> </dt>
<dd>populates database with
the contents of the file. Imports documents(s) specified to a postgresql
database (at an object level).
<p> </dd>
<dt><b> sisu -D --update -v [filename/wildcard]</b> </dt>
<dd>updates
<p>file contents in database
<p> </dd>
<dt><b> sisu -D --remove -v [filename/wildcard]</b> </dt>
<dd>removes
specified document from postgresql database.
<p> </dd>
</dl>
4. SQLITE
<p> 4.1 NAME
<p> <b>SiSU</b>
- Structured information, Serialized Units - a document publishing system.
<p> 4.2 DESCRIPTION
<p> Information related to using sqlite with sisu (and related
to the sisu_sqlite dependency package, which is a dummy package to install
dependencies needed for <b>SiSU</b> to populate an sqlite database, this being
part of <b>SiSU</b> - man sisu).
<p> 4.3 SYNOPSIS
<p> sisu -d [instruction] [filename/wildcard
if required]<br>
<p> sisu -d --(sqlite|pg) --[instruction] [filename/wildcard if <br>
required]<br>
<p> 4.4 COMMANDS
<p> Mappings to two databases are provided by default, postgresql
and sqlite, the same commands are used within sisu to construct and populate
databases however -d (lowercase) denotes sqlite and -D (uppercase) denotes
postgresql, alternatively --sqlite or --pgsql may be used
<p> <b>-d or --sqlite</b> may
be used interchangeably.
<p> 4.4.1 CREATE AND DESTROY DATABASE
<p>
<dl>
<dt><b> --sqlite --createall</b>
</dt>
<dd> initial step, creates required relations (tables, indexes) in existing
(sqlite) database (a database should be created as requested) (rb.dbi) the
same name as working directory, as
<p> </dd>
<dt><b> sisu -d --createdb</b> </dt>
<dd> creates database where
no database existed before as
<p> </dd>
<dt><b> sisu -d --create</b> </dt>
<dd> creates database tables where
no database tables existed before database tables where no database tables
existed
<p> </dd>
<dt><b> sisu -d --dropall</b> </dt>
<dd> destroys database (including all its content)!
kills data and drops tables, indexes and database associated with a given
directory (and directories of the same name). a
<p> </dd>
<dt><b> sisu -d --recreate</b> </dt>
<dd> destroys
<p> existing database and builds a new empty database structure
<p> </dd>
</dl>
4.4.2 IMPORT
<p>AND REMOVE DOCUMENTS
<p>
<dl>
<dt><b> sisu -d --import -v [filename/wildcard]</b> </dt>
<dd>populates database
with the contents of the file. Imports documents(s) specified to an sqlite
database (at an object level).
<p> </dd>
<dt><b> sisu -d --update -v [filename/wildcard]</b> </dt>
<dd>updates
<p>file contents in database
<p> </dd>
<dt><b> sisu -d --remove -v [filename/wildcard]</b> </dt>
<dd>removes
specified document from sqlite database.
<p> </dd>
</dl>
5. INTRODUCTION
<p> 5.1 SEARCH - DATABASE
FRONTEND SAMPLE, UTILISING DATABASE AND SISU FEATURES, INCLUDING OBJECT
CITATION NUMBERING (BACKEND CURRENTLY POSTGRESQL)
<p> Sample search frontend
<<a href='http://search.sisudoc.org'>http://search.sisudoc.org</a>
> [^3] A small database and sample query front-end
(search from) that makes use of the citation system, <i>object</i> citation numbering
to demonstrates functionality.[^4]
<p> <b>SiSU</b> can provide information on which
documents are matched and at what locations within each document the matches
are found. These results are relevant across all outputs using object citation
numbering, which includes html, XML, LaTeX, PDF and indeed the SQL database.
You can then refer to one of the other outputs or in the SQL database expand
the text within the matched objects (paragraphs) in the documents matched.
<p> Note you may set results either for documents matched and object number
locations within each matched document meeting the search criteria; or
display the names of the documents matched along with the objects (paragraphs)
that meet the search criteria.[^5]
<p>
<dl>
<dt><b> sisu -F --webserv-webrick</b> </dt>
<dd> builds a cgi web
<p> search frontend for the database created
<p> The following is feedback on
the setup on a machine provided by the help command:
<p> sisu --help sql<br>
<p>
<p> <br>
<pre> Postgresql
user: ralph
current db set: SiSU_sisu
port: 5432
dbi connect: DBI:Pg:database=SiSU_sisu;port=5432
sqlite
current db set: /home/ralph/sisu_www/sisu/sisu_sqlite.db
dbi connect DBI:SQLite:/home/ralph/sisu_www/sisu/sisu_sqlite.db
</pre>
<p> Note on databases built
<p> By default, [unless otherwise specified] databases
are built on a directory basis, from collections of documents within that
directory. The name of the directory you choose to work from is used as
the database name, i.e. if you are working in a directory called /home/ralph/ebook
the database SiSU_ebook is used. [otherwise a manual mapping for the collection
is
<p> </dd>
</dl>
5.2 SEARCH FORM
<p>
<dl>
<dt><b> sisu -F</b> </dt>
<dd> generates a sample search form, which must be
copied to which must be copied to
<p> </dd>
<dt><b> sisu -F --webserv-webrick</b> </dt>
<dd> generates a sample
search form for use with the webrick which must be copied to the web-server
cgi directory which must be copied to the web-server cgi directory
<p> </dd>
<dt><b> sisu
-Fv</b> </dt>
<dd> as above, and provides some information on setting up
<p> </dd>
<dt><b> sisu -W</b> </dt>
<dd> starts
<p> the webrick server which should be available
<p> The generated search form
<p>must be copied manually to the webserver directory as instructed
<p> </dd>
</dl>
6. HYPERESTRAIER
<p> See the documentation for hyperestraier:
<p> <<a href='http://hyperestraier.sourceforge.net/'>http://hyperestraier.sourceforge.net/</a>
><br>
<p> /usr/share/doc/hyperestraier/index.html<br>
<p> man estcmd<br>
<p> on sisu_hyperestraier:
<p> man sisu_hyperestraier<br>
<p> /usr/share/doc/sisu/sisu_markup/sisu_hyperestraier/index.html<br>
<p> NOTE: the examples that follow assume that sisu output is placed in
<p>the directory /home/ralph/sisu_www
<p> (A) to generate the index within the
webserver directory to be indexed:
<p> estcmd gather -sd [index name] [directory
path to index]<br>
<p> the following are examples that will need to be tailored according to
your needs:
<p> cd /home/ralph/sisu_www<br>
<p> estcmd gather -sd casket /home/ralph/sisu_www<br>
<p> you may use the ’find’ command together with ’egrep’ to limit indexing to
particular document collection directories within the web server directory:
<p> find /home/ralph/sisu_www -type f | egrep<br>
’/home/ralph/sisu_www/sisu/.+?.html$’ |estcmd gather -sd casket -<br>
<p> Check which directories in the webserver/output directory (~/sisu_www
or elsewhere depending on configuration) you wish to include in the search
index.
<p> As sisu duplicates output in multiple file formats, it it is probably
preferable to limit the estraier index to html output, and as it may also
be desirable to exclude files ’plain.txt’, ’toc.html’ and ’concordance.html’, as
these duplicate information held in other html output e.g.
<p> find /home/ralph/sisu_www
-type f | egrep<br>
’/sisu_www/(sisu|bookmarks)/.+?.html$’ | egrep -v<br>
’(doc|concordance).html$’ |estcmd gather -sd casket -<br>
<p> from your current document preparation/markup directory, you would construct
a rune along the following lines:
<p> find /home/ralph/sisu_www -type f
| egrep ’/home/ralph/sisu_www/([specify <br>
first directory for inclusion]|[specify second directory for <br>
inclusion]|[another directory for inclusion? ...])/.+?.html$’ |<br>
egrep -v ’(doc|concordance).html$’ |estcmd gather -sd<br>
/home/ralph/sisu_www/casket -<br>
<p> (B) to set up the search form
<p> (i) copy estseek.cgi to your cgi directory
and set file permissions to 755:
<p> sudo cp -vi /usr/lib/estraier/estseek.cgi
/usr/lib/cgi-bin<br>
<p> sudo chmod -v 755 /usr/lib/cgi-bin/estseek.cgi<br>
<p> sudo cp -v /usr/share/hyperestraier/estseek.* /usr/lib/cgi-bin<br>
<p> [see estraier documentation for paths]<br>
<p> (ii) edit estseek.conf, with attention to the lines starting ’indexname:’
and ’replace:’:
<p> indexname: /home/ralph/sisu_www/casket<br>
<p> replace: ^file:///home/ralph/sisu_www{{!}}<a href='http://localhost'>http://localhost</a>
<br>
<p> replace: /index.html?${{!}}/<br>
<p> (C) to test using webrick, start webrick:
<p> sisu -W<br>
<p> and try open the url: <<a href='http://localhost:8081/cgi-bin/estseek.cgi'>http://localhost:8081/cgi-bin/estseek.cgi</a>
>
<p> DOCUMENT
INFORMATION (METADATA)
<p> METADATA
<p> Document Manifest @ <<a href='http://www.jus.uio.no/sisu/sisu_manual/sisu_search/sisu_manifest.html'>http://www.jus.uio.no/sisu/sisu_manual/sisu_search/sisu_manifest.html</a>
>
<p> <b>Dublin Core</b> (DC)
<p> <i>DC</i> tags included with this document are provided here.
<p> DC Title: <i>SiSU</i> - Search
<p> DC Creator: <i>Ralph</i> Amissah
<p> DC Rights: <i>Copyright</i>
(C) Ralph Amissah 2007, part of SiSU documentation, License GPL 3
<p> DC
Type: <i>information</i>
<p> DC Date created: <i>2002-08-28</i>
<p> DC Date issued: <i>2002-08-28</i>
<p> DC Date available: <i>2002-08-28</i>
<p> DC Date modified: <i>2007-09-16</i>
<p> DC Date: <i>2007-09-16</i>
<p> <b>Version Information</b>
<p> Sourcefile: <i>sisu_search._sst</i>
<p> Filetype: <i>SiSU</i> text
<p>insert 0.58
<p> Sourcefile Digest, MD5(sisu_search._sst)= <i>c085c2eb6d68f1b7d50435f673ede407</i>
<p> Skin_Digest: MD5(/home/ralph/grotto/theatre/dbld/sisu-dev/sisu/data/doc/sisu/sisu_markup_samples/sisu_manual/_sisu/skin/doc/skin_sisu_manual.rb)=
<p><i>20fc43cf3eb6590bc3399a1aef65c5a9</i>
<p> <b>Generated</b>
<p> Document (metaverse) last
generated: <i>Mon</i> Sep 24 15:36:19 +0100 2007
<p> Generated by: <i>SiSU</i> <i>0.59.0</i> of
2007w38/0 (2007-09-23)
<p> Ruby version: <i>ruby</i> 1.8.6 (2007-06-07 patchlevel 36)
[i486-linux]
<p>
<ol>
<b>.</b><li><<a href='http://www.postgresql.org/'>http://www.postgresql.org/</a>
> <<a href='http://advocacy.postgresql.org/'>http://advocacy.postgresql.org/</a>
><br>
<<a href='http://en.wikipedia.org/wiki/Postgresql'>http://en.wikipedia.org/wiki/Postgresql</a>
><br>
</li><b>.</b><li><<a href='http://www.hwaci.com/sw/sqlite/'>http://www.hwaci.com/sw/sqlite/</a>
> <<a href='http://en.wikipedia.org/wiki/Sqlite'>http://en.wikipedia.org/wiki/Sqlite</a>
><br>
</li><b>.</b><li><<a href='http://search.sisudoc.org'>http://search.sisudoc.org</a>
> </li><b>.</b><li>(which could be extended further with current
back-end). As regards scaling of the database, it is as scalable as the database
(here Postgresql) and hardware allow. </li><b>.</b><li>of this feature when demonstrated
to an IBM software innovations evaluator in 2004 he said to paraphrase:
this could be of interest to us. We have large document management systems,
you can search hundreds of thousands of documents and we can tell you which
documents meet your search criteria, but there is no way we can tell you
without opening each document where within each your matches are found.
<p> </dd>
<dt>Other versions of this document: </dt>
<dd></dd>
<dt>manifest: <<a href='http://www.jus.uio.no/sisu/sisu_search/sisu_manifest.html'><a href='http://www.jus.uio.no/sisu/sisu_search/sisu_manifest.html'>http://www.jus.uio.no/sisu/sisu_search/sisu_manifest.html</a>
</a>
>
</dt>
<dd></dd>
<dt>html: <<a href='http://www.jus.uio.no/sisu/sisu_search/toc.html'><a href='http://www.jus.uio.no/sisu/sisu_search/toc.html'>http://www.jus.uio.no/sisu/sisu_search/toc.html</a>
</a>
> </dt>
<dd></dd>
<dt>pdf: <<a href='http://www.jus.uio.no/sisu/sisu_search/portrait.pdf'><a href='http://www.jus.uio.no/sisu/sisu_search/portrait.pdf'>http://www.jus.uio.no/sisu/sisu_search/portrait.pdf</a>
</a>
>
</dt>
<dd></dd>
<dt>pdf: <<a href='http://www.jus.uio.no/sisu/sisu_search/landscape.pdf'><a href='http://www.jus.uio.no/sisu/sisu_search/landscape.pdf'>http://www.jus.uio.no/sisu/sisu_search/landscape.pdf</a>
</a>
> </dt>
<dd> </dd>
<dt>at: <<a href='http://www.jus.uio.no/sisu'><a href='http://www.jus.uio.no/sisu'>http://www.jus.uio.no/sisu</a>
</a>
>
</dt>
<dd></dd>
<dt>* Generated by: SiSU 0.59.0 of 2007w38/0 (2007-09-23) </dt>
<dd></dd>
<dt>* Ruby version: ruby
1.8.6 (2007-06-07 patchlevel 36) [i486-linux] </dt>
<dd></dd>
<dt>* Last Generated on: Mon Sep 24
15:36:32 +0100 2007 </dt>
<dd></dd>
<dt>* SiSU <a href='http://www.jus.uio.no/sisu'>http://www.jus.uio.no/sisu</a>
</dt>
<dd></dd>
</dl>
<p>
</body>
</html>
|
