SiSU - SiSU information Structuring Universe - Search [0.58]
Ralph Amissah
Copyright (C) Ralph Amissah 2007, part of SiSU documentation, License GPL 3
information
ebook, epublishing, electronic book, electronic publishing, electronic document, electronic citation, data structure, citation systems, search
2007-09-16
SiSU - SiSU information Structuring Universe - Search [0.58],
1
SiSU Search
2
1. SiSU Search - Introduction
3
SiSU output can easily and conveniently be indexed by a number
of standalone indexing tools, such as Lucene, Hyperestraier.
4
Because the document structure of sites created is clearly defined, and
the text object citation system is available hypothetically at least,
for all forms of output, it is possible to search the sql database, and
either read results from that database, or just as simply map the
results to the html output, which has richer text markup.
5
In addition to this SiSU has the ability to populate a
relational sql type database with documents at an object level, with
objects numbers that are shared across different output types, which
make them searchable with that degree of granularity. Basically, your
match criteria is met by these documents and at these locations within
each document, which can be viewed within the database directly or in
various output formats.
6
2. SQL
7
2.1 populating SQL type databases
8
SiSU feeds sisu markupd documents into sql type databases
PostgreSQL1 and/or SQLite2 database together with
information related to document structure.
1. <
2. <
9
This is one of the more interesting output forms, as all the structural
data of the documents are retained (though can be ignored by the user
of the database should they so choose). All site texts/documents are
(currently) streamed to four tables:
10
one containing semantic (and other) headers, including, title,
author, subject, (the Dublin Core...);
11
another the substantive texts by individual "paragraph" (or
object) - along with structural information, each paragraph being
identifiable by its paragraph number (if it has one which almost all of
them do), and the substantive text of each paragraph quite naturally
being searchable (both in formatted and clean text versions for
searching); and
12
a third containing endnotes cross-referenced back to the
paragraph from which they are referenced (both in formatted and clean
text versions for searching).
13
a fourth table with a one to one relation with the headers table
contains full text versions of output, eg. pdf, html, xml, and ascii.
14
There is of course the possibility to add further structures.
15
At this level SiSU loads a relational database with documents
chunked into objects, their smallest logical structurally constituent
parts, as text objects, with their object citation number and all other
structural information needed to construct the document. Text is stored
(at this text object level) with and without elementary markup tagging,
the stripped version being so as to facilitate ease of searching.
16
Being able to search a relational database at an object level with the
SiSU citation system is an effective way of locating content
generated by SiSU . As individual text objects of a document
stored (and indexed) together with object numbers, and all versions of
the document have the same numbering, complex searches can be tailored
to return just the locations of the search results relevant for all
available output formats, with live links to the precise locations in
the database or in html/xml documents; or, the structural information
provided makes it possible to search the full contents of the database
and have headings in which search content appears, or to search only
headings etc. (as the Dublin Core is incorporated it is easy to make
use of that as well).
17
3. Postgresql
18
3.1 Name
19
SiSU - Structured information, Serialized Units - a document
publishing system, postgresql dependency package
20
3.2 Description
21
Information related to using postgresql with sisu (and related to the
sisu_postgresql dependency package, which is a dummy package to install
dependencies needed for SiSU to populate a postgresql database,
this being part of SiSU - man sisu).
22
3.3 Synopsis
23
sisu -D [instruction] [filename/wildcard if required]
24
sisu -D --pg --[instruction] [filename/wildcard if required]
25
3.4 Commands
26
Mappings to two databases are provided by default, postgresql and
sqlite, the same commands are used within sisu to construct and
populate databases however -d (lowercase) denotes sqlite and -D
(uppercase) denotes postgresql, alternatively --sqlite or --pgsql may
be used
27
-D or --pgsql may be used interchangeably.
28
3.4.1 create and destroy database
29
--pgsql --createall
30
sisu -D --createdb
31
sisu -D --create
32
sisu -D --Dropall
33
sisu -D --recreate
34
3.4.2 import and remove documents
35
sisu -D --import -v [filename/wildcard]
36
sisu -D --update -v [filename/wildcard]
37
sisu -D --remove -v [filename/wildcard]
38
4. Sqlite
39
4.1 Name
40
SiSU - Structured information, Serialized Units - a document
publishing system.
41
4.2 Description
42
Information related to using sqlite with sisu (and related to the
sisu_sqlite dependency package, which is a dummy package to install
dependencies needed for SiSU to populate an sqlite database,
this being part of SiSU - man sisu).
43
4.3 Synopsis
44
sisu -d [instruction] [filename/wildcard if required]
45
sisu -d --(sqlite|pg) --[instruction] [filename/wildcard if
required]
46
4.4 Commands
47
Mappings to two databases are provided by default, postgresql and
sqlite, the same commands are used within sisu to construct and
populate databases however -d (lowercase) denotes sqlite and -D
(uppercase) denotes postgresql, alternatively --sqlite or --pgsql may
be used
48
-d or --sqlite may be used interchangeably.
49
4.4.1 create and destroy database
50
--sqlite --createall
51
sisu -d --createdb
52
sisu -d --create
53
sisu -d --dropall
54
sisu -d --recreate
55
4.4.2 import and remove documents
56
sisu -d --import -v [filename/wildcard]
57
sisu -d --update -v [filename/wildcard]
58
sisu -d --remove -v [filename/wildcard]
59
5. Introduction
60
5.1 Search - database frontend sample, utilising database and SiSU
features, including object citation numbering (backend currently
PostgreSQL)
61
3 A small database and sample query front-end (search from)
that makes use of the citation system, object citation numbering
to demonstrates functionality.4
3. <
4. (which could be extended further with current back-end). As regards
scaling of the database, it is as scalable as the database (here
Postgresql) and hardware allow.
62
SiSU can provide information on which documents are matched and
at what locations within each document the matches are found. These
results are relevant across all outputs using object citation
numbering, which includes html, XML, LaTeX, PDF and indeed the SQL
database. You can then refer to one of the other outputs or in the SQL
database expand the text within the matched objects (paragraphs) in the
documents matched.
63
Note you may set results either for documents matched and object number
locations within each matched document meeting the search criteria; or
display the names of the documents matched along with the objects
(paragraphs) that meet the search criteria.5
5. of this feature when demonstrated to an IBM software innovations
evaluator in 2004 he said to paraphrase: this could be of interest to
us. We have large document management systems, you can search hundreds
of thousands of documents and we can tell you which documents meet your
search criteria, but there is no way we can tell you without opening
each document where within each your matches are found.
64
sisu -F --webserv-webrick
65
The following is feedback on the setup on a machine provided by the
help command:
66
sisu --help sql
67
68
Postgresql
Note on databases built
69
By default, [unless otherwise specified] databases are built on a
directory basis, from collections of documents within that directory.
The name of the directory you choose to work from is used as the
database name, i.e. if you are working in a directory called
/home/ralph/ebook the database SiSU_ebook is used. [otherwise a manual
mapping for the collection is necessary]
70
5.2 Search Form
71
sisu -F
72
sisu -F --webserv-webrick
73
sisu -Fv
74
sisu -W
75
The generated search form must be copied manually to the webserver
directory as instructed
76
6. Hyperestraier
77
See the documentation for hyperestraier:
78
<
79
/usr/share/doc/hyperestraier/index.html
80
man estcmd
81
on sisu_hyperestraier:
82
man sisu_hyperestraier
83
/usr/share/doc/sisu/sisu_markup/sisu_hyperestraier/index.html
84
NOTE: the examples that follow assume that sisu output is placed in the
directory /home/ralph/sisu_www
85
(A) to generate the index within the webserver directory to be indexed:
86
estcmd gather -sd [index name] [directory path to index]
87
the following are examples that will need to be tailored according to
your needs:
88
cd /home/ralph/sisu_www
89
estcmd gather -sd casket /home/ralph/sisu_www
90
you may use the 'find' command together with 'egrep' to limit indexing
to particular document collection directories within the web server
directory:
91
find /home/ralph/sisu_www -type f | egrep
'/home/ralph/sisu_www/sisu/.+?.html$' |estcmd gather -sd casket -
92
Check which directories in the webserver/output directory
(~/sisu_www or elsewhere depending on configuration) you wish to
include in the search index.
93
As sisu duplicates output in multiple file formats, it it is probably
preferable to limit the estraier index to html output, and as it may
also be desirable to exclude files 'plain.txt', 'toc.html' and
'concordance.html', as these duplicate information held in other html
output e.g.
94
find /home/ralph/sisu_www -type f | egrep
'/sisu_www/(sisu|bookmarks)/.+?.html$' | egrep -v
'(doc|concordance).html$' |estcmd gather -sd casket -
95
from your current document preparation/markup directory, you would
construct a rune along the following lines:
96
find /home/ralph/sisu_www -type f | egrep
'/home/ralph/sisu_www/([specify first directory for inclusion]|[specify
second directory for inclusion]|[another directory for inclusion?
...])/.+?.html$' | egrep -v '(doc|concordance).html$' |estcmd gather
-sd /home/ralph/sisu_www/casket -
97
(B) to set up the search form
98
(i) copy estseek.cgi to your cgi directory and set file permissions to
755:
99
sudo cp -vi /usr/lib/estraier/estseek.cgi /usr/lib/cgi-bin
100
sudo chmod -v 755 /usr/lib/cgi-bin/estseek.cgi
101
sudo cp -v /usr/share/hyperestraier/estseek.* /usr/lib/cgi-bin
102
[see estraier documentation for paths]
103
(ii) edit estseek.conf, with attention to the lines starting
'indexname:' and 'replace:':
104
indexname: /home/ralph/sisu_www/casket
105
replace: ^file:///home/ralph/sisu_www{{!}}http://localhost
106
replace: /index.html?${{!}}/
107
(C) to test using webrick, start webrick:
108
sisu -W
109
and try open the url: <
110
Endnotes
0