diff options
Diffstat (limited to 'ocda-db-format-specification.org')
| -rw-r--r-- | ocda-db-format-specification.org | 283 |
1 files changed, 144 insertions, 139 deletions
diff --git a/ocda-db-format-specification.org b/ocda-db-format-specification.org index 324babd..7655ad2 100644 --- a/ocda-db-format-specification.org +++ b/ocda-db-format-specification.org @@ -9,7 +9,7 @@ version 2.1*, as spine emits and reads it after 0.25.1. This document specifies /one serialisation/. The model it serialises is -specified in [[file:ocda-model.org][=ocda-model.org=]], which the text serialisation +specified in [[file:ocda-model.org][~ocda-model.org~]], which the text serialisation shares; read that first if the question is what an object /is/ rather than how it is stored. @@ -17,7 +17,7 @@ how it is stored. :PROPERTIES: :CUSTOM_ID: status :END: -Derived from the writer, =sisudoc/outputs/io_out/sqlite_ocda_db.d=, and +Derived from the writer, ~sisudoc/outputs/io_out/sqlite_ocda_db.d~, and checked against the 27 published databases of the project's reference collection. Where the two could differ the writer is normative and this document is wrong. @@ -27,26 +27,26 @@ document is wrong. :CUSTOM_ID: sec-1 :END: One file per *document*, holding *every language of it*: a ten language -document is one database with ten rows in =documents=, written one language +document is one database with ten rows in ~documents~, written one language at a time into the one file. -Two things distinguish it from the =.ssp=: +Two things distinguish it from the ~.ssp~: -- *It carries its own source.* The markup, the images, =conf/document_make=, - =pod.manifest= and the =tools/po4a= catalogues are all inside it, in the - =files= table. A pod can be written back out of it, and spine does exactly - that when asked for an artefact that needs markup. The =.ssp= names its +- *It carries its own source.* The markup, the images, ~conf/document_make~, + ~pod.manifest~ and the ~tools/po4a~ catalogues are all inside it, in the + ~files~ table. A pod can be written back out of it, and spine does exactly + that when asked for an artefact that needs markup. The ~.ssp~ names its images and carries none of them. - *It is queryable without being parsed.* A question about the document is a SELECT, answered from the file with no spine and no reader. -** 1.1 It is built from the =.ssp= +** 1.1 It is built from the ~.ssp~ :PROPERTIES: :CUSTOM_ID: sec-1-1 :END: -The document is emitted as =.ssp=, *read straight back*, and what comes +The document is emitted as ~.ssp~, *read straight back*, and what comes back is what is written. So *the database cannot carry an object property -the =.ssp= cannot express*, and the two cannot drift. That is also why they +the ~.ssp~ cannot express*, and the two cannot drift. That is also why they share one version number: a bump moves both. The carried /files/ are the deliberate exception. They come from the pod @@ -56,19 +56,20 @@ rather than from the abstraction, each with its digest. :PROPERTIES: :CUSTOM_ID: sec-2 :END: -Declared in band, in the =metadata= table: +Declared in band, in the ~metadata~ table: -#+begin_src sql +#+NAME: ocda-db-query-1 +#+begin_src sqlite :tangle ocda-db-queries.sql SELECT key, value FROM metadata WHERE doc_id IS NULL; -- build.spine_version | 0.25.0 -- schema.name | sisu-abstraction-db -- schema.version | 2.1 #+end_src -The rule is the model's, and identical to the =.ssp=: the minor part is +The rule is the model's, and identical to the ~.ssp~: the minor part is raised when a property is added, the major when one changes meaning or goes away, and *a reader checks the major part and refuses what it does not -know*. See [[file:ssp-format-specification.org][=ssp-format-specification.org=]] +know*. See [[file:ssp-format-specification.org][~ssp-format-specification.org~]] [[file:ssp-format-specification.org::#sec-2][section 2]] for the history, which is shared. * 3. The schema @@ -96,56 +97,56 @@ Eight tables, one virtual table, three views. objects_fts fts5 over objects.text, external content over objects.id #+end_example -** 3.1 =doc_id= is the language key +** 3.1 ~doc_id~ is the language key :PROPERTIES: :CUSTOM_ID: sec-3-1 :END: -=documents.id=, referred to as =doc_id= everywhere else, is *what tells one +~documents.id~, referred to as ~doc_id~ everywhere else, is *what tells one language's rows from another's*. It is the key to reach for in every query. -=objects.lang= and =objects.meta_lang= are *properties of the object* and +~objects.lang~ and ~objects.meta_lang~ are *properties of the object* and are not it. They are almost always empty ([[#sec-6][section 6]]). ** 3.2 The file-level metadata rows :PROPERTIES: :CUSTOM_ID: sec-3-2 :END: -=metadata= is keyed =UNIQUE(doc_id, key)= and not on key alone, because +~metadata~ is keyed ~UNIQUE(doc_id, key)~ and not on key alone, because every language has a title and a creator and a key-only key would refuse the second. Three rows describe *the file* rather than a document in it, and are -written with =doc_id= null: =schema.name=, =schema.version= and -=build.spine_version=. In SQL no null equals another null, so -=UNIQUE(doc_id, key)= does not constrain them, and a partial unique index -over =key= alone =WHERE doc_id IS NULL= is what makes them singular. +written with ~doc_id~ null: ~schema.name~, ~schema.version~ and +~build.spine_version~. In SQL no null equals another null, so +~UNIQUE(doc_id, key)~ does not constrain them, and a partial unique index +over ~key~ alone ~WHERE doc_id IS NULL~ is what makes them singular. -*** 3.2.1 The key prefix carries what the =.ssp= puts in blocks +*** 3.2.1 The key prefix carries what the ~.ssp~ puts in blocks :PROPERTIES: :CUSTOM_ID: sec-3-2-1 :END: -The =.ssp= has four preamble blocks, =@source=, =@meta=, =@make= and -=@doc_has=. The database has *one* =metadata= table, and tells them apart +The ~.ssp~ has four preamble blocks, ~@source~, ~@meta~, ~@make~ and +~@doc_has~. The database has *one* ~metadata~ table, and tells them apart by a *prefix on the key*: -| =.ssp= block | database key | +| ~.ssp~ block | database key | |--------------+-----------------------------------------------------------| -| =@meta= | the key as it stands: =title.main=, =creator.author=, ... | -| =@make= | =make.= + the key: =make.footer=, =make.doc_type= | -| =@doc_has= | =doc_has.= + the key: =doc_has.images=, =doc_has.tables= | -| =@source= | =source.= + the key: =source.language=, =source.digest= | +| ~@meta~ | the key as it stands: ~title.main~, ~creator.author~, ... | +| ~@make~ | ~make.~ + the key: ~make.footer~, ~make.doc_type~ | +| ~@doc_has~ | ~doc_has.~ + the key: ~doc_has.images~, ~doc_has.tables~ | +| ~@source~ | ~source.~ + the key: ~source.language~, ~source.digest~ | *A reader must strip the prefix and restore the grouping*, or a writer -over it has to ask =metadata["make.footer"]= from a database and -=make["footer"]= from a =.ssp=, which is exactly the difference the two +over it has to ask ~metadata["make.footer"]~ from a database and +~make["footer"]~ from a ~.ssp~, which is exactly the difference the two serialisations exist not to make. -=source.= also carries two keys with no =@source= counterpart, -=source.filename= and =source.ssp_digest=; the =.ssp= puts the first in its -=% Source:= comment and cannot hold the second, which is a digest /of/ it. +~source.~ also carries two keys with no ~@source~ counterpart, +~source.filename~ and ~source.ssp_digest~; the ~.ssp~ puts the first in its +~% Source:~ comment and cannot hold the second, which is a digest /of/ it. -Values may carry leading or trailing whitespace here where the =.ssp= -cannot represent it, its =key: value= line being unable to distinguish a +Values may carry leading or trailing whitespace here where the ~.ssp~ +cannot represent it, its ~key: value~ line being unable to distinguish a value that begins with a space from the separator. *Strip on load*, as a reader of the text form necessarily does. @@ -153,21 +154,21 @@ reader of the text form necessarily does. :PROPERTIES: :CUSTOM_ID: sec-3-3 :END: -=children=, =ancestors=, =ancestors_collapsed=, =dom_status=, -=dom_status_collapsed=, =heading_ancestors_text=, =table_widths= and -=table_aligns= are stored as *JSON arrays*, with validity checks, so that -=json_extract=, =json_each= and =json_array_length= reach inside them. +~children~, ~ancestors~, ~ancestors_collapsed~, ~dom_status~, +~dom_status_collapsed~, ~heading_ancestors_text~, ~table_widths~ and +~table_aligns~ are stored as *JSON arrays*, with validity checks, so that +~json_extract~, ~json_each~ and ~json_array_length~ reach inside them. #+begin_quote -*Parse them as JSON.* A scan that strips =[=, =]=, =,= and ="= looks -adequate and is wrong on the first escape: the encoder writes =/= as -=\/=, so =FREE/OPEN-SOURCE SOFTWARE= comes back as -=FREE\/OPEN-SOURCE SOFTWARE=. All 36 document-languages of the reference +*Parse them as JSON.* A scan that strips ~[~, ~]~, ~,~ and ~"~ looks +adequate and is wrong on the first escape: the encoder writes ~/~ as +~\/~, so ~FREE/OPEN-SOURCE SOFTWARE~ comes back as +~FREE\/OPEN-SOURCE SOFTWARE~. All 36 document-languages of the reference collection contain at least one, and the defect is invisible to any check that does not compare the field itself. -Note also that =table_widths= is an array of *integers except where the -columns divide evenly*, when it is =[33.3333,33.3333,33.3333]=. A decoder +Note also that ~table_widths~ is an array of *integers except where the +columns divide evenly*, when it is ~[33.3333,33.3333,33.3333]~. A decoder that accepts strings and integers and then falls back to a default turns a value it could not read into one that looks read. #+end_quote @@ -177,71 +178,71 @@ what occurs: | column | elements | |--------------------------------------------------------------------------------------+-------------------------------------------------------------------| -| =children=, =ancestors=, =ancestors_collapsed=, =dom_status=, =dom_status_collapsed= | integers | -| =table_widths= | integers, or floats where the columns divide evenly | -| =table_aligns= | strings, one of =l= =c= =r= | -| =heading_ancestors_text= | strings, eight of them, empty where that level is not an ancestor | +| ~children~, ~ancestors~, ~ancestors_collapsed~, ~dom_status~, ~dom_status_collapsed~ | integers | +| ~table_widths~ | integers, or floats where the columns divide evenly | +| ~table_aligns~ | strings, one of ~l~ ~c~ ~r~ | +| ~heading_ancestors_text~ | strings, eight of them, empty where that level is not an ancestor | -The =.ssp= writes the first three groups *space separated* and -=heading_ancestors_text= *=|= separated*. The empty slots of the last are +The ~.ssp~ writes the first three groups *space separated* and +~heading_ancestors_text~ *~|~ separated*. The empty slots of the last are significant: they say which levels a heading sits under, so the separators carry as much as the text does. The open-ended lists are *rows in the four child tables* instead of columns: images, links, anchor tags and sub-table-of-contents entries. Each -child table carries =seq=, which preserves the order they appeared in. +child table carries ~seq~, which preserves the order they appeared in. ** 3.4 Indexes :PROPERTIES: :CUSTOM_ID: sec-3-4 :END: -=objects(doc_id, section)=, =objects(ocn)=, =objects(parent_ocn)=, -=objects(is_a)=, =objects(heading_level)= where it is a heading, and -=files(role)=, =object_images(name)=. +~objects(doc_id, section)~, ~objects(ocn)~, ~objects(parent_ocn)~, +~objects(is_a)~, ~objects(heading_level)~ where it is a heading, and +~files(role)~, ~object_images(name)~. -* 4. The relationship to the =.ssp=, column by column +* 4. The relationship to the ~.ssp~, column by column :PROPERTIES: :CUSTOM_ID: sec-4 :END: -*Nothing is lost in either direction.* Every =.ssp= property is a column +*Nothing is lost in either direction.* Every ~.ssp~ property is a column or a child table; every column is a property, a value from the object declaration line, or a database key. -Where the names differ they differ for a reason, the =.ssp= preferring the +Where the names differ they differ for a reason, the ~.ssp~ preferring the short form and the database the explicit one: -| =.ssp= property | database | +| ~.ssp~ property | database | |--------------------+-----------------------| -| =.parent= | =parent_ocn= | -| =.last_descendant= | =last_descendant_ocn= | -| =.dummy= | =dummy_heading= | -| =.ocn_off= | =object_number_off= | -| =.part= | =is_of_part= | -| =.section= | =is_of_section= | +| [[file:ssp-format-specification.org::#prop-parent][~.parent~]] | ~parent_ocn~ | +| [[file:ssp-format-specification.org::#prop-last-descendant][~.last_descendant~]] | ~last_descendant_ocn~ | +| [[file:ssp-format-specification.org::#prop-dummy][~.dummy~]] | ~dummy_heading~ | +| [[file:ssp-format-specification.org::#prop-ocn-off][~.ocn_off~]] | ~object_number_off~ | +| [[file:ssp-format-specification.org::#prop-part][~.part~]] | ~is_of_part~ | +| [[file:ssp-format-specification.org::#prop-section][~.section~]] | ~is_of_section~ | -Three are *expanded*, the =.ssp= folding what the database spreads: +Three are *expanded*, the ~.ssp~ folding what the database spreads: -| =.ssp= property | database | +| ~.ssp~ property | database | |---------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------| -| =.indent: <base> <hang>= | =indent_base=, =indent_hang= | -| =.ocn_range: <first> <last>= | =ocn_range_first=, =ocn_range_last= (2.1: =0= where the object is not a container; a 2.0 database has neither column, and reads as no range) | -| =.has: links notes_reg notes_star images images_no_dim= | =has_links=, =has_notes_reg=, =has_notes_star=, =has_images=, =has_images_no_dim= | +| [[file:ssp-format-specification.org::#prop-indent][~.indent: <base> <hang>~]] | ~indent_base~, ~indent_hang~ | +| [[file:ssp-format-specification.org::#prop-ocn-range][~.ocn_range: <first> <last>~]] | ~ocn_range_first~, ~ocn_range_last~ (2.1: ~0~ where the object is not a container; a 2.0 database has neither column, and reads as no range) | +| [[file:ssp-format-specification.org::#prop-has][~.has: links notes_reg notes_star images images_no_dim~]] | ~has_links~, ~has_notes_reg~, ~has_notes_star~, ~has_images~, ~has_images_no_dim~ | The four *repeatable* properties become the four child tables: -| =.ssp= property | database | +| ~.ssp~ property | database | |-----------------+------------------| -| =.image= | =object_images= | -| =.stow_link= | =object_links= | -| =.anchor_tag= | =object_anchors= | -| =.lev4_subtoc= | =object_subtoc= | - -And four columns come from outside the property list: =ocn=, =is_a= and -=heading_level= from the object declaration line, and =text= from its =|= -lines. =id=, =doc_id= and =seq= are the database's own keys; =seq= is the -object's position within its section, and =(doc_id, section, seq)= is what -identifies an object when =ocn= will not: =0=, or, before 2.1, a poem -sharing its last verse's number (see [[file:ocda-model.org][=ocda-model.org=]] +| [[file:ssp-format-specification.org::#prop-image][~.image~]] | ~object_images~ | +| [[file:ssp-format-specification.org::#prop-stow-link][~.stow_link~]] | ~object_links~ | +| [[file:ssp-format-specification.org::#prop-anchor-tag][~.anchor_tag~]] | ~object_anchors~ | +| [[file:ssp-format-specification.org::#prop-lev4-subtoc][~.lev4_subtoc~]] | ~object_subtoc~ | + +And four columns come from outside the property list: ~ocn~, ~is_a~ and +~heading_level~ from the object declaration line, and ~text~ from its ~|~ +lines. ~id~, ~doc_id~ and ~seq~ are the database's own keys; ~seq~ is the +object's position within its section, and ~(doc_id, section, seq)~ is what +identifies an object when ~ocn~ will not: ~0~, or, before 2.1, a poem +sharing its last verse's number (see [[file:ocda-model.org][~ocda-model.org~]] [[file:ocda-model.org::#sec-3][section 3]]). ** 4.1 Where the two are not symmetric @@ -252,22 +253,22 @@ sharing its last verse's number (see [[file:ocda-model.org][=ocda-model.org=]] three places where the /files/ nonetheless differ. A reader must know all three, because each of them made two independently written readers disagree. -*1. The database states what the =.ssp= derives.* =identifier= and -=anchor= are written on every row. The =.ssp= omits each where it can be +*1. The database states what the ~.ssp~ derives.* ~identifier~ and +~anchor~ are written on every row. The ~.ssp~ omits each where it can be worked out, the identifier being the ocn and the anchor the segment name ([[file:ssp-format-specification.org::#sec-6-1][sections 6.1]] and [[file:ssp-format-specification.org::#sec-6-3][6.3]] of -[[file:ssp-format-specification.org][=ssp-format-specification.org=]], [[file:ocda-model.org::#sec-9][section 9]] of -[[file:ocda-model.org][=ocda-model.org=]]). In =free_culture.lawrence_lessig= that is +[[file:ssp-format-specification.org][~ssp-format-specification.org~]], [[file:ocda-model.org::#sec-9][section 9]] of +[[file:ocda-model.org][~ocda-model.org~]]). In ~free_culture.lawrence_lessig~ that is *1,314 identifiers and 69 anchors* stated here and absent there. Neither file is the poorer; a reader that does not derive is. -*2. =translation.ocn_aligned= is here and cannot be in a =.ssp=.* It says -whether the languages of a document share one numbering, and a =.ssp= is one +*2. ~translation.ocn_aligned~ is here and cannot be in a ~.ssp~.* It says +whether the languages of a document share one numbering, and a ~.ssp~ is one language. It is therefore a fact about *the file*, not about the document in it, and it is the one such fact either serialisation carries. A reader should *keep it out of the document model*, for the same reason -the =schema.*= and =build.*= rows are kept out: a writer over the model must +the ~schema.*~ and ~build.*~ rows are kept out: a writer over the model must not be able to tell which serialisation it was handed, and a field that exists only when the input was a database is exactly that tell. The model has no place for collection-wide facts yet; when it grows one, this belongs @@ -275,19 +276,19 @@ there. *3. Whitespace and grouping*, as [[#sec-3-2-1][section 3.2.1]] gives them. -* 5. The =files= table: what it carries, and how +* 5. The ~files~ table: what it carries, and how :PROPERTIES: :CUSTOM_ID: sec-5 :END: -| role | named by | =compression= | =data= holds | +| role | named by | ~compression~ | ~data~ holds | |------------+----------------------------+---------------+------------------| -| =image= | its bare filename | =none= | the file as read | -| =source= | =media/text/<lang>/<file>= | =zstd= | a zstd frame | -| =conf= | =conf/document_make= | =zstd= | a zstd frame | -| =manifest= | =pod.manifest= | =zstd= | a zstd frame | -| =tools= | =tools/po4a/...= | =zstd= | a zstd frame | +| ~image~ | its bare filename | ~none~ | the file as read | +| ~source~ | ~media/text/<lang>/<file>~ | ~zstd~ | a zstd frame | +| ~conf~ | ~conf/document_make~ | ~zstd~ | a zstd frame | +| ~manifest~ | ~pod.manifest~ | ~zstd~ | a zstd frame | +| ~tools~ | ~tools/po4a/...~ | ~zstd~ | a zstd frame | -The =compression= column carries the literal =none= or =zstd=, so a reader +The ~compression~ column carries the literal ~none~ or ~zstd~, so a reader need not infer it from the role. Three rules make the file predictable: @@ -300,27 +301,28 @@ Three rules make the file predictable: differently for no reason a reader could see. 2. *Images are not compressed.* png and jpeg are compressed already; zstd gains nothing and costs a decompression on every read. -3. *=objects.text= is never compressed.* The full text index reads that +3. *~objects.text~ is never compressed.* The full text index reads that column directly, and compressing it would stop search working completely rather than subtly. -*=bytes= and =sha256= are always those of the original file*, whatever -=compression= says. So a digest check, a =digests.txt= line and a rebuilt -=source.digest= all keep working, and a reader that does not decompress can +*~bytes~ and ~sha256~ are always those of the original file*, whatever +~compression~ says. So a digest check, a ~digests.txt~ line and a rebuilt +~source.digest~ all keep working, and a reader that does not decompress can still say what it is looking at. * 6. Querying it :PROPERTIES: :CUSTOM_ID: sec-6 :END: -=sqlite3= is all that is needed. The recipes below were run against the +~sqlite3~ is all that is needed. The recipes below were run against the published databases. ** 6.1 What is this file :PROPERTIES: :CUSTOM_ID: sec-6-1 :END: -#+begin_src sql +#+NAME: ocda-db-query-2 +#+begin_src sqlite :tangle ocda-db-queries.sql SELECT key, value FROM metadata WHERE doc_id IS NULL; SELECT id, lang, source_filename FROM documents ORDER BY id; SELECT role, count(*), sum(bytes) FROM files GROUP BY role; @@ -332,27 +334,28 @@ SELECT role, count(*), sum(bytes) FROM files GROUP BY role; :END: | view | gives | |------------------+---------------------------------------------------------------------------------------------| -| =outline= | the headings: lang, section, seq, ocn, both levels, =last_descendant_ocn=, identifier, text | -| =citable= | every object with an ocn, ordered by ocn, language named | -| =document_files= | what the file carries: role, name, bytes, sha256, pixel size | +| ~outline~ | the headings: lang, section, seq, ocn, both levels, ~last_descendant_ocn~, identifier, text | +| ~citable~ | every object with an ocn, ordered by ocn, language named | +| ~document_files~ | what the file carries: role, name, bytes, sha256, pixel size | -*=outline= and =citable= name the language and order by it first.* A view +*~outline~ and ~citable~ name the language and order by it first.* A view over a file that may hold ten languages and does not say which would read as one document and be several. -*A trap in =citable=:* it is =SELECT d.lang AS lang, o.*=, and =objects= -has a =lang= column of its own, so sqlite renames the second one. The view -has *=lang=, the document's language, and =lang:1=, the object's own*. -Quote the second if you want it: =SELECT "lang:1" FROM citable=. +*A trap in ~citable~:* it is ~SELECT d.lang AS lang, o.*~, and ~objects~ +has a ~lang~ column of its own, so sqlite renames the second one. The view +has *~lang~, the document's language, and ~lang:1~, the object's own*. +Quote the second if you want it: ~SELECT "lang:1" FROM citable~. ** 6.3 Structure :PROPERTIES: :CUSTOM_ID: sec-6-3 :END: A heading's subtree is a *range*, not a traversal, because every heading -carries =last_descendant_ocn=: +carries ~last_descendant_ocn~: -#+begin_src sql +#+NAME: ocda-db-query-3 +#+begin_src sqlite :tangle ocda-db-queries.sql SELECT count(*) FROM objects o, (SELECT doc_id, ocn, last_descendant_ocn FROM objects WHERE is_a='heading' AND ocn=66) h @@ -360,9 +363,10 @@ SELECT count(*) FROM objects o, AND o.ocn BETWEEN h.ocn AND h.last_descendant_ocn; #+end_src -And the path back up, on =parent_ocn=: +And the path back up, on ~parent_ocn~: -#+begin_src sql +#+NAME: ocda-db-query-4 +#+begin_src sqlite :tangle ocda-db-queries.sql WITH RECURSIVE up(ocn, parent_ocn, text) AS ( SELECT ocn, parent_ocn, text FROM objects WHERE doc_id=1 AND ocn=345 UNION ALL @@ -372,7 +376,7 @@ WITH RECURSIVE up(ocn, parent_ocn, text) AS ( SELECT ocn, substr(text,1,44) FROM up; #+end_src -*Keep the =parent_ocn > 0= guard.* A parent of 0 is a real, +*Keep the ~parent_ocn > 0~ guard.* A parent of 0 is a real, non-substantive parent and not a missing value, and without the guard the recursion does not terminate. @@ -380,7 +384,8 @@ recursion does not terminate. :PROPERTIES: :CUSTOM_ID: sec-6-4 :END: -#+begin_src sql +#+NAME: ocda-db-query-5 +#+begin_src sqlite :tangle ocda-db-queries.sql SELECT o.ocn, substr(o.text,1,44) FROM objects_fts f JOIN objects o ON o.id = f.rowid JOIN documents d ON d.id = o.doc_id @@ -388,26 +393,26 @@ SELECT o.ocn, substr(o.text,1,44) FROM objects_fts f ORDER BY rank LIMIT 3; #+end_src -fts5 syntax is available in full: =MATCH 'peer AND production'=, -=MATCH '"peer production"'=, =MATCH 'peer NEAR production'=. +fts5 syntax is available in full: ~MATCH 'peer AND production'~, +~MATCH '"peer production"'~, ~MATCH 'peer NEAR production'~. -The index is *external content* over =objects.id=: the text is indexed, +The index is *external content* over ~objects.id~: the text is indexed, not stored twice. ** 6.5 Three things to know before trusting a query :PROPERTIES: :CUSTOM_ID: sec-6-5 :END: -1. *An ocn is not unique within a document*, and =0= is shared by many - objects. Key on =objects.id=, or on =(doc_id, section, seq)=. -2. *=objects.text= is internal markup*, neither the author's markup nor +1. *An ocn is not unique within a document*, and ~0~ is shared by many + objects. Key on ~objects.id~, or on ~(doc_id, section, seq)~. +2. *~objects.text~ is internal markup*, neither the author's markup nor any output. Fine for search and for reading structure, not for display without decoding. The markers are in - [[file:ssp-format-specification.org][=ssp-format-specification.org=]] [[file:ssp-format-specification.org::#sec-3-2][section 3.2]]. + [[file:ssp-format-specification.org][~ssp-format-specification.org~]] [[file:ssp-format-specification.org::#sec-3-2][section 3.2]]. 3. *A multi-language file answers an unfiltered query with every language - at once.* A single-language document has one =documents= row, so plain - queries read naturally there; anything else wants =WHERE lang='..'= or a - join to =documents=. + at once.* A single-language document has one ~documents~ row, so plain + queries read naturally there; anything else wants ~WHERE lang~'..'= or a + join to ~documents~. * 7. Digests, and verifying a file :PROPERTIES: @@ -422,8 +427,8 @@ each carried file──> files.sha256 over the original bytes the spine used ──> metadata build.spine_version #+end_example -So =.sst -> .ssp -> .ocda.db= is checkable end to end, and -=spine --ocda-verify=<file>= closes the loop by writing the carried markup +So ~.sst -> .ssp -> .ocda.db~ is checkable end to end, and +~spine --ocda-verify=<file>~ closes the loop by writing the carried markup back out and re-parsing it: *does this spine still make this abstraction from the markup this file carries?* It writes nothing and the exit status is the answer. @@ -432,7 +437,7 @@ is the answer. strongest sense in which it is the one to work with. The database's own digest is taken once, when the last language has been -written, and appears in =digests.txt=. The metadata page links the database +written, and appears in ~digests.txt~. The metadata page links the database and claims no hash for it, deliberately: the page is written per language while the database is added to per language, so a digest taken there would be of a file that no longer exists by the end of the run. @@ -441,11 +446,11 @@ be of a file that no longer exists by the end of the run. :PROPERTIES: :CUSTOM_ID: sec-8 :END: -A reader *must* check =schema.version='s major part and refuse a major it -does not know; *must* treat =ocn= as not an identity; *must* stop at -=parent_ocn = 0= when walking upward; and *should* say what it did not +A reader *must* check ~schema.version~'s major part and refuse a major it +does not know; *must* treat ~ocn~ as not an identity; *must* stop at +~parent_ocn = 0~ when walking upward; and *should* say what it did not understand, for the reason given in -[[file:ssp-format-specification.org][=ssp-format-specification.org=]] [[file:ssp-format-specification.org::#sec-8-2][section 8.2]]. +[[file:ssp-format-specification.org][~ssp-format-specification.org~]] [[file:ssp-format-specification.org::#sec-8-2][section 8.2]]. -A writer *must* keep =bytes= and =sha256= those of the original file -whatever =compression= says, and *must not* compress =objects.text=. +A writer *must* keep ~bytes~ and ~sha256~ those of the original file +whatever ~compression~ says, and *must not* compress ~objects.text~. |
