diff options
| author | Ralph Amissah <ralph.amissah@gmail.com> | 2026-09-26 09:47:07 -0400 |
|---|---|---|
| committer | Ralph Amissah <ralph.amissah@gmail.com> | 2026-09-26 09:47:07 -0400 |
| commit | fda48e417435e31a3122bfa7ca563333628aed99 (patch) | |
| tree | c91f50006de299a9db42247bc21df4f59f9c7a82 | |
format 2.0: abstraction as library, in D and Gleam
The object-centric document abstraction that sisudoc-spine builds from
SiSU markup, read as a library: the model, and a loader for each form
it is written in - the .ssp (text), the .ocda.db (sqlite), and a pod
archive carrying a .ssp (.sisupod, .zip). Begun 2026-09-25; this is
the library as it stood at format 2.0.
The readers (sisudoc-ssp-reader, sisudoc-ocda-db-reader) each read one
form and print. A writer needs neither: it needs the document. So the
model and the loaders live here, ocda.model imports nothing, and a
program built on the library cannot tell which form it was handed.
ocda.load is the one place that decides what a path is, so a new
input form is added once and every program gains it. The model and
the .ssp loader began as the .ssp reader's code, moved and renamed to
the level they belong to; the .ocda.db loader is new.
The Gleam library is a deliberate field-for-field mirror of the D one,
including where that is not the most idiomatic Gleam: a difference in
one language and not the other would show in every comparison and
mean nothing.
The claim holds only if the loaders agree, so the library asks:
ocda-agree <doc>.ssp <doc>.ocda.db [lang] ten fields, per object
ocda-dump <file> [lang] the whole model,
canonically
The dump is an oracle, not an output format, and the D and Gleam
dumps are byte identical, so four questions are a diff: D and Gleam,
each from the .ssp and from the .ocda.db. Over the markup document
samples all four hold, every document-language; a .sisupod gives the
same model as the .ssp beside it.
None of that held at first, and what the checks found is the record
of how the specification became enough:
- an object whose first text line is blank lost it (the loader asked
whether the text was empty, not whether a line had been seen);
seven sample documents open verse that way. Found by agree; fixed
here and in the .ssp reader, which carried the same code.
- found by the dump: the identifier and the anchor are omitted by the
text form when they can be derived and stored by the database
always, so the loaders derive them, and the rule went into the
specification (derived values); make and doc_has share one table in
the database under a key prefix, which the loaders strip; the JSON
columns were scanned, not parsed, and Phobos writes "/" as "\/";
and translation.ocn_aligned is a fact about the file, not the
document, so neither loader puts it in the model.
- in Gleam, a JSON element decoder fell through to a fallback giving
"", so a float column read as empty strings: a value that could not
be read turned into one that looked read.
- found by the writers: anchor_tags, heading_lev_anchor, segment_epub
and segment_epub_is were in the specification and read by nothing.
ocda.images gives an image's bytes from whichever form the document
came from (the database's files table, the archive entry, or beside
a .ssp at media/image/), and says beforehand whether it can.
The Gleam library reads a .zip pod but not a .sisupod: the zstd frame
around it has no implementation on the BEAM that builds sandboxed. It
says so, and says zstd -d, rather than calling the file malformed.
gleam/ builds with nix-gleam's builder, kept in gleam/nix-gleam/ (MIT)
Use:
dlang/ nix build ocda-dump, ocda-agree; or a dub path
dependency on dlang/
gleam/ nix build ocda-lib dump | agree
See README.md, and dlang/README.md and gleam/README.md.
(assisted by Claude-Code)
46 files changed, 7136 insertions, 0 deletions
diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..12a3507 --- /dev/null +++ b/.gitignore @@ -0,0 +1,76 @@ +# Deny everything, then whitelist what belongs in the repository. +# +# The practice is deliberate: a repository should hold what was decided to +# put in it, not whatever happened to be in the directory when someone ran +# "git add .". Build output, editor droppings, downloaded dependencies and +# exported binaries are numerous, are regenerated, and differ between +# machines; source is few and is chosen. So the default is no, and every +# yes below is a decision that can be read. +# +# To see what is being excluded, and why: +# git status --ignored +# git check-ignore -v <path> +# git ls-files --others --ignored --exclude-standard --directory + +* + +# ── the repository itself ──────────────────────────────────────────────── +!.gitignore +!README.md +!LICENSE +!**/LICENSE + +# ── directories have to be walked into before a file inside can match ──── +!gleam/ +!dlang/ +!org/ +!gleam/**/ +!dlang/**/ +!org/**/ +# but not these, whatever they contain +build/ +.build/ +.direnv/ +.gleam/ +bin/ +result +result-* + +# ── documentation and notes, in any language directory ─────────────────── +!**/COPYRIGHT +!**/*.md +!**/*.org + +# ── nix ────────────────────────────────────────────────────────────────── +!**/*.nix +!**/flake.lock +!**/.envrc + +# ── dlang ──────────────────────────────────────────────────────────────── +!**/*.d +!**/dub.json +!**/dub.sdl +!**/dub.selections.json +*.o +*.a +*.lst + +# ── gleam ──────────────────────────────────────────────────────────────── +!**/*.gleam +# Erlang, for what Gleam reaches through FFI (a sha256, from OTP crypto) +!**/src/*.erl +# gleam.toml is the manifest, manifest.toml is the lockfile: both belong +!**/*.toml +*.beam +*.ez +erl_crash.dump + +# ── ruby, when it arrives ──────────────────────────────────────────────── +!**/*.rb +!**/Gemfile +!**/Gemfile.lock +!**/*.gemspec + +# ── continuous integration ─────────────────────────────────────────────── +!**/*.yml +!**/*.yaml diff --git a/COPYRIGHT b/COPYRIGHT new file mode 100644 index 0000000..7e7dd56 --- /dev/null +++ b/COPYRIGHT @@ -0,0 +1,45 @@ +- Name: sisudoc-ocda-lib + - Description: part of SiSU spine (Doc Reform): documents, structuring, + processing, publishing and search + - [https://www.sisudoc.org] + + - Author: Ralph Amissah + [ralph.amissah@gmail.com] + + - Copyright: (C) 2026 Ralph Amissah, All Rights Reserved. + + - License: AGPL 3 or later, for everything in this repository unless a file or + a directory specifies otherwise.: + + sisudoc-ocda-lib + + part of SiSU spine (Doc Reform): documents, structuring, + processing, publishing and search + + Copyright (C) Ralph Amissah + + This program is free software: you can redistribute it and/or modify it + under the terms of the GNU Affero General Public License (AGPLv3) as + published by the Free Software Foundation, either version 3 of the License, + or (at your option) any later version. + + This program is distributed in the hope that it will be useful, but + WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY + or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public + License for more details. + + You should have received a copy of the GNU Affero General Public License + along with this program. If not, see [https://www.gnu.org/licenses/]. + + If you have Internet connection, the latest version of the AGPL should be + available at: [https://www.gnu.org/licenses/agpl-3.0.html] + + - Spine, Doc Reform (related to SiSU) uses standard: + - docReform markup syntax (based on SiSU markup) + - standard SiSU markup syntax with modified headers and minor + modifications + - docReform object numbering (based on SiSU object citation numbering) + - standard SiSU document object numbering + + - Homepages: + [https://www.sisudoc.org] @@ -0,0 +1,660 @@ +# GNU AFFERO GENERAL PUBLIC LICENSE + +Version 3, 19 November 2007 + +Copyright (C) 2007 Free Software Foundation, Inc. +<https://fsf.org/> + +Everyone is permitted to copy and distribute verbatim copies of this +license document, but changing it is not allowed. + +## Preamble + +The GNU Affero General Public License is a free, copyleft license for +software and other kinds of works, specifically designed to ensure +cooperation with the community in the case of network server software. + +The licenses for most software and other practical works are designed +to take away your freedom to share and change the works. By contrast, +our General Public Licenses are intended to guarantee your freedom to +share and change all versions of a program--to make sure it remains +free software for all its users. + +When we speak of free software, we are referring to freedom, not +price. Our General Public Licenses are designed to make sure that you +have the freedom to distribute copies of free software (and charge for +them if you wish), that you receive source code or can get it if you +want it, that you can change the software or use pieces of it in new +free programs, and that you know you can do these things. + +Developers that use our General Public Licenses protect your rights +with two steps: (1) assert copyright on the software, and (2) offer +you this License which gives you legal permission to copy, distribute +and/or modify the software. + +A secondary benefit of defending all users' freedom is that +improvements made in alternate versions of the program, if they +receive widespread use, become available for other developers to +incorporate. Many developers of free software are heartened and +encouraged by the resulting cooperation. However, in the case of +software used on network servers, this result may fail to come about. +The GNU General Public License permits making a modified version and +letting the public access it on a server without ever releasing its +source code to the public. + +The GNU Affero General Public License is designed specifically to +ensure that, in such cases, the modified source code becomes available +to the community. It requires the operator of a network server to +provide the source code of the modified version running there to the +users of that server. Therefore, public use of a modified version, on +a publicly accessible server, gives the public access to the source +code of the modified version. + +An older license, called the Affero General Public License and +published by Affero, was designed to accomplish similar goals. This is +a different license, not a version of the Affero GPL, but Affero has +released a new version of the Affero GPL which permits relicensing +under this license. + +The precise terms and conditions for copying, distribution and +modification follow. + +## TERMS AND CONDITIONS + +### 0. Definitions. + +"This License" refers to version 3 of the GNU Affero General Public +License. + +"Copyright" also means copyright-like laws that apply to other kinds +of works, such as semiconductor masks. + +"The Program" refers to any copyrightable work licensed under this +License. Each licensee is addressed as "you". "Licensees" and +"recipients" may be individuals or organizations. + +To "modify" a work means to copy from or adapt all or part of the work +in a fashion requiring copyright permission, other than the making of +an exact copy. The resulting work is called a "modified version" of +the earlier work or a work "based on" the earlier work. + +A "covered work" means either the unmodified Program or a work based +on the Program. + +To "propagate" a work means to do anything with it that, without +permission, would make you directly or secondarily liable for +infringement under applicable copyright law, except executing it on a +computer or modifying a private copy. Propagation includes copying, +distribution (with or without modification), making available to the +public, and in some countries other activities as well. + +To "convey" a work means any kind of propagation that enables other +parties to make or receive copies. Mere interaction with a user +through a computer network, with no transfer of a copy, is not +conveying. + +An interactive user interface displays "Appropriate Legal Notices" to +the extent that it includes a convenient and prominently visible +feature that (1) displays an appropriate copyright notice, and (2) +tells the user that there is no warranty for the work (except to the +extent that warranties are provided), that licensees may convey the +work under this License, and how to view a copy of this License. If +the interface presents a list of user commands or options, such as a +menu, a prominent item in the list meets this criterion. + +### 1. Source Code. + +The "source code" for a work means the preferred form of the work for +making modifications to it. "Object code" means any non-source form of +a work. + +A "Standard Interface" means an interface that either is an official +standard defined by a recognized standards body, or, in the case of +interfaces specified for a particular programming language, one that +is widely used among developers working in that language. + +The "System Libraries" of an executable work include anything, other +than the work as a whole, that (a) is included in the normal form of +packaging a Major Component, but which is not part of that Major +Component, and (b) serves only to enable use of the work with that +Major Component, or to implement a Standard Interface for which an +implementation is available to the public in source code form. A +"Major Component", in this context, means a major essential component +(kernel, window system, and so on) of the specific operating system +(if any) on which the executable work runs, or a compiler used to +produce the work, or an object code interpreter used to run it. + +The "Corresponding Source" for a work in object code form means all +the source code needed to generate, install, and (for an executable +work) run the object code and to modify the work, including scripts to +control those activities. However, it does not include the work's +System Libraries, or general-purpose tools or generally available free +programs which are used unmodified in performing those activities but +which are not part of the work. For example, Corresponding Source +includes interface definition files associated with source files for +the work, and the source code for shared libraries and dynamically +linked subprograms that the work is specifically designed to require, +such as by intimate data communication or control flow between those +subprograms and other parts of the work. + +The Corresponding Source need not include anything that users can +regenerate automatically from other parts of the Corresponding Source. + +The Corresponding Source for a work in source code form is that same +work. + +### 2. Basic Permissions. + +All rights granted under this License are granted for the term of +copyright on the Program, and are irrevocable provided the stated +conditions are met. This License explicitly affirms your unlimited +permission to run the unmodified Program. The output from running a +covered work is covered by this License only if the output, given its +content, constitutes a covered work. This License acknowledges your +rights of fair use or other equivalent, as provided by copyright law. + +You may make, run and propagate covered works that you do not convey, +without conditions so long as your license otherwise remains in force. +You may convey covered works to others for the sole purpose of having +them make modifications exclusively for you, or provide you with +facilities for running those works, provided that you comply with the +terms of this License in conveying all material for which you do not +control copyright. Those thus making or running the covered works for +you must do so exclusively on your behalf, under your direction and +control, on terms that prohibit them from making any copies of your +copyrighted material outside their relationship with you. + +Conveying under any other circumstances is permitted solely under the +conditions stated below. Sublicensing is not allowed; section 10 makes +it unnecessary. + +### 3. Protecting Users' Legal Rights From Anti-Circumvention Law. + +No covered work shall be deemed part of an effective technological +measure under any applicable law fulfilling obligations under article +11 of the WIPO copyright treaty adopted on 20 December 1996, or +similar laws prohibiting or restricting circumvention of such +measures. + +When you convey a covered work, you waive any legal power to forbid +circumvention of technological measures to the extent such +circumvention is effected by exercising rights under this License with +respect to the covered work, and you disclaim any intention to limit +operation or modification of the work as a means of enforcing, against +the work's users, your or third parties' legal rights to forbid +circumvention of technological measures. + +### 4. Conveying Verbatim Copies. + +You may convey verbatim copies of the Program's source code as you +receive it, in any medium, provided that you conspicuously and +appropriately publish on each copy an appropriate copyright notice; +keep intact all notices stating that this License and any +non-permissive terms added in accord with section 7 apply to the code; +keep intact all notices of the absence of any warranty; and give all +recipients a copy of this License along with the Program. + +You may charge any price or no price for each copy that you convey, +and you may offer support or warranty protection for a fee. + +### 5. Conveying Modified Source Versions. + +You may convey a work based on the Program, or the modifications to +produce it from the Program, in the form of source code under the +terms of section 4, provided that you also meet all of these +conditions: + +- a) The work must carry prominent notices stating that you modified + it, and giving a relevant date. +- b) The work must carry prominent notices stating that it is + released under this License and any conditions added under + section 7. This requirement modifies the requirement in section 4 + to "keep intact all notices". +- c) You must license the entire work, as a whole, under this + License to anyone who comes into possession of a copy. This + License will therefore apply, along with any applicable section 7 + additional terms, to the whole of the work, and all its parts, + regardless of how they are packaged. This License gives no + permission to license the work in any other way, but it does not + invalidate such permission if you have separately received it. +- d) If the work has interactive user interfaces, each must display + Appropriate Legal Notices; however, if the Program has interactive + interfaces that do not display Appropriate Legal Notices, your + work need not make them do so. + +A compilation of a covered work with other separate and independent +works, which are not by their nature extensions of the covered work, +and which are not combined with it such as to form a larger program, +in or on a volume of a storage or distribution medium, is called an +"aggregate" if the compilation and its resulting copyright are not +used to limit the access or legal rights of the compilation's users +beyond what the individual works permit. Inclusion of a covered work +in an aggregate does not cause this License to apply to the other +parts of the aggregate. + +### 6. Conveying Non-Source Forms. + +You may convey a covered work in object code form under the terms of +sections 4 and 5, provided that you also convey the machine-readable +Corresponding Source under the terms of this License, in one of these +ways: + +- a) Convey the object code in, or embodied in, a physical product + (including a physical distribution medium), accompanied by the + Corresponding Source fixed on a durable physical medium + customarily used for software interchange. +- b) Convey the object code in, or embodied in, a physical product + (including a physical distribution medium), accompanied by a + written offer, valid for at least three years and valid for as + long as you offer spare parts or customer support for that product + model, to give anyone who possesses the object code either (1) a + copy of the Corresponding Source for all the software in the + product that is covered by this License, on a durable physical + medium customarily used for software interchange, for a price no + more than your reasonable cost of physically performing this + conveying of source, or (2) access to copy the Corresponding + Source from a network server at no charge. +- c) Convey individual copies of the object code with a copy of the + written offer to provide the Corresponding Source. This + alternative is allowed only occasionally and noncommercially, and + only if you received the object code with such an offer, in accord + with subsection 6b. +- d) Convey the object code by offering access from a designated + place (gratis or for a charge), and offer equivalent access to the + Corresponding Source in the same way through the same place at no + further charge. You need not require recipients to copy the + Corresponding Source along with the object code. If the place to + copy the object code is a network server, the Corresponding Source + may be on a different server (operated by you or a third party) + that supports equivalent copying facilities, provided you maintain + clear directions next to the object code saying where to find the + Corresponding Source. Regardless of what server hosts the + Corresponding Source, you remain obligated to ensure that it is + available for as long as needed to satisfy these requirements. +- e) Convey the object code using peer-to-peer transmission, + provided you inform other peers where the object code and + Corresponding Source of the work are being offered to the general + public at no charge under subsection 6d. + +A separable portion of the object code, whose source code is excluded +from the Corresponding Source as a System Library, need not be +included in conveying the object code work. + +A "User Product" is either (1) a "consumer product", which means any +tangible personal property which is normally used for personal, +family, or household purposes, or (2) anything designed or sold for +incorporation into a dwelling. In determining whether a product is a +consumer product, doubtful cases shall be resolved in favor of +coverage. For a particular product received by a particular user, +"normally used" refers to a typical or common use of that class of +product, regardless of the status of the particular user or of the way +in which the particular user actually uses, or expects or is expected +to use, the product. A product is a consumer product regardless of +whether the product has substantial commercial, industrial or +non-consumer uses, unless such uses represent the only significant +mode of use of the product. + +"Installation Information" for a User Product means any methods, +procedures, authorization keys, or other information required to +install and execute modified versions of a covered work in that User +Product from a modified version of its Corresponding Source. The +information must suffice to ensure that the continued functioning of +the modified object code is in no case prevented or interfered with +solely because modification has been made. + +If you convey an object code work under this section in, or with, or +specifically for use in, a User Product, and the conveying occurs as +part of a transaction in which the right of possession and use of the +User Product is transferred to the recipient in perpetuity or for a +fixed term (regardless of how the transaction is characterized), the +Corresponding Source conveyed under this section must be accompanied +by the Installation Information. But this requirement does not apply +if neither you nor any third party retains the ability to install +modified object code on the User Product (for example, the work has +been installed in ROM). + +The requirement to provide Installation Information does not include a +requirement to continue to provide support service, warranty, or +updates for a work that has been modified or installed by the +recipient, or for the User Product in which it has been modified or +installed. Access to a network may be denied when the modification +itself materially and adversely affects the operation of the network +or violates the rules and protocols for communication across the +network. + +Corresponding Source conveyed, and Installation Information provided, +in accord with this section must be in a format that is publicly +documented (and with an implementation available to the public in +source code form), and must require no special password or key for +unpacking, reading or copying. + +### 7. Additional Terms. + +"Additional permissions" are terms that supplement the terms of this +License by making exceptions from one or more of its conditions. +Additional permissions that are applicable to the entire Program shall +be treated as though they were included in this License, to the extent +that they are valid under applicable law. If additional permissions +apply only to part of the Program, that part may be used separately +under those permissions, but the entire Program remains governed by +this License without regard to the additional permissions. + +When you convey a copy of a covered work, you may at your option +remove any additional permissions from that copy, or from any part of +it. (Additional permissions may be written to require their own +removal in certain cases when you modify the work.) You may place +additional permissions on material, added by you to a covered work, +for which you have or can give appropriate copyright permission. + +Notwithstanding any other provision of this License, for material you +add to a covered work, you may (if authorized by the copyright holders +of that material) supplement the terms of this License with terms: + +- a) Disclaiming warranty or limiting liability differently from the + terms of sections 15 and 16 of this License; or +- b) Requiring preservation of specified reasonable legal notices or + author attributions in that material or in the Appropriate Legal + Notices displayed by works containing it; or +- c) Prohibiting misrepresentation of the origin of that material, + or requiring that modified versions of such material be marked in + reasonable ways as different from the original version; or +- d) Limiting the use for publicity purposes of names of licensors + or authors of the material; or +- e) Declining to grant rights under trademark law for use of some + trade names, trademarks, or service marks; or +- f) Requiring indemnification of licensors and authors of that + material by anyone who conveys the material (or modified versions + of it) with contractual assumptions of liability to the recipient, + for any liability that these contractual assumptions directly + impose on those licensors and authors. + +All other non-permissive additional terms are considered "further +restrictions" within the meaning of section 10. If the Program as you +received it, or any part of it, contains a notice stating that it is +governed by this License along with a term that is a further +restriction, you may remove that term. If a license document contains +a further restriction but permits relicensing or conveying under this +License, you may add to a covered work material governed by the terms +of that license document, provided that the further restriction does +not survive such relicensing or conveying. + +If you add terms to a covered work in accord with this section, you +must place, in the relevant source files, a statement of the +additional terms that apply to those files, or a notice indicating +where to find the applicable terms. + +Additional terms, permissive or non-permissive, may be stated in the +form of a separately written license, or stated as exceptions; the +above requirements apply either way. + +### 8. Termination. + +You may not propagate or modify a covered work except as expressly +provided under this License. Any attempt otherwise to propagate or +modify it is void, and will automatically terminate your rights under +this License (including any patent licenses granted under the third +paragraph of section 11). + +However, if you cease all violation of this License, then your license +from a particular copyright holder is reinstated (a) provisionally, +unless and until the copyright holder explicitly and finally +terminates your license, and (b) permanently, if the copyright holder +fails to notify you of the violation by some reasonable means prior to +60 days after the cessation. + +Moreover, your license from a particular copyright holder is +reinstated permanently if the copyright holder notifies you of the +violation by some reasonable means, this is the first time you have +received notice of violation of this License (for any work) from that +copyright holder, and you cure the violation prior to 30 days after +your receipt of the notice. + +Termination of your rights under this section does not terminate the +licenses of parties who have received copies or rights from you under +this License. If your rights have been terminated and not permanently +reinstated, you do not qualify to receive new licenses for the same +material under section 10. + +### 9. Acceptance Not Required for Having Copies. + +You are not required to accept this License in order to receive or run +a copy of the Program. Ancillary propagation of a covered work +occurring solely as a consequence of using peer-to-peer transmission +to receive a copy likewise does not require acceptance. However, +nothing other than this License grants you permission to propagate or +modify any covered work. These actions infringe copyright if you do +not accept this License. Therefore, by modifying or propagating a +covered work, you indicate your acceptance of this License to do so. + +### 10. Automatic Licensing of Downstream Recipients. + +Each time you convey a covered work, the recipient automatically +receives a license from the original licensors, to run, modify and +propagate that work, subject to this License. You are not responsible +for enforcing compliance by third parties with this License. + +An "entity transaction" is a transaction transferring control of an +organization, or substantially all assets of one, or subdividing an +organization, or merging organizations. If propagation of a covered +work results from an entity transaction, each party to that +transaction who receives a copy of the work also receives whatever +licenses to the work the party's predecessor in interest had or could +give under the previous paragraph, plus a right to possession of the +Corresponding Source of the work from the predecessor in interest, if +the predecessor has it or can get it with reasonable efforts. + +You may not impose any further restrictions on the exercise of the +rights granted or affirmed under this License. For example, you may +not impose a license fee, royalty, or other charge for exercise of +rights granted under this License, and you may not initiate litigation +(including a cross-claim or counterclaim in a lawsuit) alleging that +any patent claim is infringed by making, using, selling, offering for +sale, or importing the Program or any portion of it. + +### 11. Patents. + +A "contributor" is a copyright holder who authorizes use under this +License of the Program or a work on which the Program is based. The +work thus licensed is called the contributor's "contributor version". + +A contributor's "essential patent claims" are all patent claims owned +or controlled by the contributor, whether already acquired or +hereafter acquired, that would be infringed by some manner, permitted +by this License, of making, using, or selling its contributor version, +but do not include claims that would be infringed only as a +consequence of further modification of the contributor version. For +purposes of this definition, "control" includes the right to grant +patent sublicenses in a manner consistent with the requirements of +this License. + +Each contributor grants you a non-exclusive, worldwide, royalty-free +patent license under the contributor's essential patent claims, to +make, use, sell, offer for sale, import and otherwise run, modify and +propagate the contents of its contributor version. + +In the following three paragraphs, a "patent license" is any express +agreement or commitment, however denominated, not to enforce a patent +(such as an express permission to practice a patent or covenant not to +sue for patent infringement). To "grant" such a patent license to a +party means to make such an agreement or commitment not to enforce a +patent against the party. + +If you convey a covered work, knowingly relying on a patent license, +and the Corresponding Source of the work is not available for anyone +to copy, free of charge and under the terms of this License, through a +publicly available network server or other readily accessible means, +then you must either (1) cause the Corresponding Source to be so +available, or (2) arrange to deprive yourself of the benefit of the +patent license for this particular work, or (3) arrange, in a manner +consistent with the requirements of this License, to extend the patent +license to downstream recipients. "Knowingly relying" means you have +actual knowledge that, but for the patent license, your conveying the +covered work in a country, or your recipient's use of the covered work +in a country, would infringe one or more identifiable patents in that +country that you have reason to believe are valid. + +If, pursuant to or in connection with a single transaction or +arrangement, you convey, or propagate by procuring conveyance of, a +covered work, and grant a patent license to some of the parties +receiving the covered work authorizing them to use, propagate, modify +or convey a specific copy of the covered work, then the patent license +you grant is automatically extended to all recipients of the covered +work and works based on it. + +A patent license is "discriminatory" if it does not include within the +scope of its coverage, prohibits the exercise of, or is conditioned on +the non-exercise of one or more of the rights that are specifically +granted under this License. You may not convey a covered work if you +are a party to an arrangement with a third party that is in the +business of distributing software, under which you make payment to the +third party based on the extent of your activity of conveying the +work, and under which the third party grants, to any of the parties +who would receive the covered work from you, a discriminatory patent +license (a) in connection with copies of the covered work conveyed by +you (or copies made from those copies), or (b) primarily for and in +connection with specific products or compilations that contain the +covered work, unless you entered into that arrangement, or that patent +license was granted, prior to 28 March 2007. + +Nothing in this License shall be construed as excluding or limiting +any implied license or other defenses to infringement that may +otherwise be available to you under applicable patent law. + +### 12. No Surrender of Others' Freedom. + +If conditions are imposed on you (whether by court order, agreement or +otherwise) that contradict the conditions of this License, they do not +excuse you from the conditions of this License. If you cannot convey a +covered work so as to satisfy simultaneously your obligations under +this License and any other pertinent obligations, then as a +consequence you may not convey it at all. For example, if you agree to +terms that obligate you to collect a royalty for further conveying +from those to whom you convey the Program, the only way you could +satisfy both those terms and this License would be to refrain entirely +from conveying the Program. + +### 13. Remote Network Interaction; Use with the GNU General Public License. + +Notwithstanding any other provision of this License, if you modify the +Program, your modified version must prominently offer all users +interacting with it remotely through a computer network (if your +version supports such interaction) an opportunity to receive the +Corresponding Source of your version by providing access to the +Corresponding Source from a network server at no charge, through some +standard or customary means of facilitating copying of software. This +Corresponding Source shall include the Corresponding Source for any +work covered by version 3 of the GNU General Public License that is +incorporated pursuant to the following paragraph. + +Notwithstanding any other provision of this License, you have +permission to link or combine any covered work with a work licensed +under version 3 of the GNU General Public License into a single +combined work, and to convey the resulting work. The terms of this +License will continue to apply to the part which is the covered work, +but the work with which it is combined will remain governed by version +3 of the GNU General Public License. + +### 14. Revised Versions of this License. + +The Free Software Foundation may publish revised and/or new versions +of the GNU Affero General Public License from time to time. Such new +versions will be similar in spirit to the present version, but may +differ in detail to address new problems or concerns. + +Each version is given a distinguishing version number. If the Program +specifies that a certain numbered version of the GNU Affero General +Public License "or any later version" applies to it, you have the +option of following the terms and conditions either of that numbered +version or of any later version published by the Free Software +Foundation. If the Program does not specify a version number of the +GNU Affero General Public License, you may choose any version ever +published by the Free Software Foundation. + +If the Program specifies that a proxy can decide which future versions +of the GNU Affero General Public License can be used, that proxy's +public statement of acceptance of a version permanently authorizes you +to choose that version for the Program. + +Later license versions may give you additional or different +permissions. However, no additional obligations are imposed on any +author or copyright holder as a result of your choosing to follow a +later version. + +### 15. Disclaimer of Warranty. + +THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY +APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT +HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT +WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND +PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE +DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR +CORRECTION. + +### 16. Limitation of Liability. + +IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING +WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR +CONVEYS THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, +INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES +ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT +NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR +LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM +TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER +PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. + +### 17. Interpretation of Sections 15 and 16. + +If the disclaimer of warranty and limitation of liability provided +above cannot be given local legal effect according to their terms, +reviewing courts shall apply local law that most closely approximates +an absolute waiver of all civil liability in connection with the +Program, unless a warranty or assumption of liability accompanies a +copy of the Program in return for a fee. + +END OF TERMS AND CONDITIONS + +## How to Apply These Terms to Your New Programs + +If you develop a new program, and you want it to be of the greatest +possible use to the public, the best way to achieve this is to make it +free software which everyone can redistribute and change under these +terms. + +To do so, attach the following notices to the program. It is safest to +attach them to the start of each source file to most effectively state +the exclusion of warranty; and each file should have at least the +"copyright" line and a pointer to where the full notice is found. + + <one line to give the program's name and a brief idea of what it does.> + Copyright (C) <year> <name of author> + + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU Affero General Public License as + published by the Free Software Foundation, either version 3 of the + License, or (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU Affero General Public License for more details. + + You should have received a copy of the GNU Affero General Public License + along with this program. If not, see <https://www.gnu.org/licenses/>. + +Also add information on how to contact you by electronic and paper +mail. + +If your software can interact with users remotely through a computer +network, you should also make sure that it provides a way for users to +get its source. For example, if your program is a web application, its +interface could display a "Source" link that leads users to an archive +of the code. There are many ways you could offer source, and different +solutions will be better for different programs; see section 13 for +the specific requirements. + +You should also get your employer (if you work as a programmer) or +school, if any, to sign a "copyright disclaimer" for the program, if +necessary. For more information on this, and how to apply and follow +the GNU AGPL, see <https://www.gnu.org/licenses/>. diff --git a/README.md b/README.md new file mode 100644 index 0000000..5e328c0 --- /dev/null +++ b/README.md @@ -0,0 +1,150 @@ +# sisudoc-ocda-lib + +The **object-centric document abstraction as a library**: the model, and +loaders for both of the forms it is written in. + +Written by Claude (Anthropic), working in Claude Code from the specification, +at the direction of and reviewed by Ralph Amissah. + +**One reader per language, in a directory named for the language.** + +| | | +|--------------------|---------| +| [`dlang/`](dlang/) | D | +| [`gleam/`](gleam/) | Gleam | +| `ruby/` | to come | + +## What it is for + +The readers in `sisudoc-ssp-reader` and `sisudoc-ocda-db-reader` each read +one serialisation and print. A **writer** needs neither: it needs the document. + +``` + .ssp ─┐ + ├─> the model ─> a writer + .ocda.db ─┘ +``` + +So this library holds the model and both loaders, and **a writer cannot +tell which serialisation it was handed** - exactly as spine's own writers +cannot, since spine 0.25.0. + +That is the whole of the idea. The readers become thin programs over this, +and the writers are functions from the model to bytes. + +## The checks that matter + +The claim above is only true if the loaders agree, so the library asks. Two +programs, in each language: + +```sh +ocda-agree <doc>.ssp <doc>.ocda.db [lang] # do the two loaders agree? +ocda-dump <file>.ssp|<file>.ocda.db [lang] # the whole model, canonically +``` + +`agree` compares two models within one language. **`dump` is the stronger +of the two**, and the reason it exists: it writes the entire loaded model +out in a fully determined form, and the D and the Gleam library write it +*byte for byte alike*. So four questions can be asked, all of them with +`diff`: + +| | | +|-----------------------------|---------------------------------| +| D: `.ssp` == `.ocda.db` | **36 of 36** document-languages | +| Gleam: `.ssp` == `.ocda.db` | **36 of 36** | +| `.ssp`: D == Gleam | **36 of 36** | +| `.ocda.db`: D == Gleam | **36 of 36** | + +None of those held at first. + +## What the checks found + +### the ten-field check: a blank first text line + +An object whose **first text line is blank** lost it. The loader decided +whether to start a new line by asking whether the accumulated text was +empty, which is not the same question as whether a line has been seen. Of +the sample collection **seven documents have verse opening on a blank +line**, and in every one of them the blank was dropped. Eight of +thirty-six documents disagreed. + +Nothing else had caught it. The two `.ssp` readers had been compared on +*section counts*, which this does not change. It took loading the same +document two ways and comparing the objects. + +Fixed here and in `sisudoc-ssp-reader/dlang`, which carried the same code. + +### the whole-model dump: four more + +The ten-field check was enough for that and not enough for anything +subtler. Dumping the whole model found four further disagreements between +the two loaders: + +1. **`.identifier` and `.anchor`.** The `.ssp` emits a property only when it + carries information, so the identifier is absent when it is simply the + ocn and the anchor is absent when it is the segment name; the database + stores both unconditionally. Over one document that is **1,314 + identifiers and 69 anchors**. The loaders now derive them, which is what + the model means. +2. **`make` and `doc_has`.** The database has one metadata table where the + text form has three blocks, and tells them apart by a key prefix. The + loaders unprefix them, so a writer asks `make["footer"]` whichever + serialisation it was handed. +3. **The JSON columns were scanned rather than parsed.** Phobos writes `/` + as `\/`, so every heading containing a slash came back with a backslash + in it: **all 36 document-languages** differed on + `heading_ancestors_text` and nothing else had shown it. +4. **`translation.ocn_aligned`** is in the database and cannot be in a + `.ssp`, which is one language and cannot know whether the languages + share a numbering. It is a fact about the *file*, so the loaders leave + it out rather than letting a writer tell which serialisation it was + given. + +The first three are defects. The fourth is a real asymmetry between the two +serialisations, and naming it is the useful part. + +## What it reads, and what it does not + +| | | | +|-----------------------|------------------------------------------------------------------|-------------------| +| `.ssp` | yes | | +| `.ocda.db` | yes, any language of it | | +| `.sisupod` | yes, any language it carries | D only, see below | +| `.zip` pod | yes, any language it carries | | +| `.sst`, `.ssm` markup | **no, and not planned**: that is a parser, which is spine's work | | +| a url | no | | + +A pod carries the abstraction only when spine was asked for it: `--pod2` +puts it in, `--source` does not. A pod without one is refused with a message +saying which flag makes a pod this library can read, because that is a fact +about the pod rather than about the library. + +**Over the reference collection, a `.sisupod` gives the same model as the +`.ssp` beside it: 36 of 36 document-languages, byte identical dumps.** A +writer over the library therefore gained a third input form without being +touched, which is the point of the dispatch being in one place. + +### the one thing the Gleam library cannot do + +A `.sisupod` is **one zstd frame wrapping** the zip. The D library reads both +through a declared `libzstd`. The Gleam library reads the **zip and not the +frame**, because the only zstd on the BEAM is a NIF that clones zstd from +github in its own build and so cannot be built in a sandbox without patching +it. + +So a `.zip` pod is read in both languages, a `.sisupod` in D only, and the +Gleam refusal says why and says `zstd -d` rather than claiming the file is +malformed. It is a packaging limit, not anything about the format, and it is +the only place the two libraries differ in what they accept. + +`ocda/load` is the one place in each language that decides what a path is, +so a new input form is added once and every reader and writer gains it. + +## The format + +Specified in **sisudoc-abstraction-format**: `ocda-model.md` for the model +this library implements, and the two format specifications for the loaders. +Section 4 of `ocda-db-format-specification.md` gives the column by column +correspondence the `load_db` of each language follows. + +At format **2.x**. diff --git a/dlang/COPYRIGHT b/dlang/COPYRIGHT new file mode 100644 index 0000000..7e7dd56 --- /dev/null +++ b/dlang/COPYRIGHT @@ -0,0 +1,45 @@ +- Name: sisudoc-ocda-lib + - Description: part of SiSU spine (Doc Reform): documents, structuring, + processing, publishing and search + - [https://www.sisudoc.org] + + - Author: Ralph Amissah + [ralph.amissah@gmail.com] + + - Copyright: (C) 2026 Ralph Amissah, All Rights Reserved. + + - License: AGPL 3 or later, for everything in this repository unless a file or + a directory specifies otherwise.: + + sisudoc-ocda-lib + + part of SiSU spine (Doc Reform): documents, structuring, + processing, publishing and search + + Copyright (C) Ralph Amissah + + This program is free software: you can redistribute it and/or modify it + under the terms of the GNU Affero General Public License (AGPLv3) as + published by the Free Software Foundation, either version 3 of the License, + or (at your option) any later version. + + This program is distributed in the hope that it will be useful, but + WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY + or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public + License for more details. + + You should have received a copy of the GNU Affero General Public License + along with this program. If not, see [https://www.gnu.org/licenses/]. + + If you have Internet connection, the latest version of the AGPL should be + available at: [https://www.gnu.org/licenses/agpl-3.0.html] + + - Spine, Doc Reform (related to SiSU) uses standard: + - docReform markup syntax (based on SiSU markup) + - standard SiSU markup syntax with modified headers and minor + modifications + - docReform object numbering (based on SiSU object citation numbering) + - standard SiSU document object numbering + + - Homepages: + [https://www.sisudoc.org] diff --git a/dlang/README.md b/dlang/README.md new file mode 100644 index 0000000..124e4d6 --- /dev/null +++ b/dlang/README.md @@ -0,0 +1,62 @@ +# ocda-lib, in D + +The abstraction's model and both its loaders, as a dub library. + +## Modules + +| | | +|-----------------|---------------------------------------------------------------| +| `ocda.model` | the document, the object, the eight sections. Imports nothing | +| `ocda.load_ssp` | text -> model | +| `ocda.load_db` | sqlite -> model | +| `ocda.load` | what a path is, and loading it | +| `ocda.dump` | the model written out canonically, as an oracle | +| `ocda.sqlite` | the sqlite3 C API, declared rather than depended upon | + +`ocda.model` importing nothing is the property that matters: a writer +depends on the model and on neither loader. + +## Use + +```json +"dependencies": { "ocda-lib": { "path": "../../sisudoc-ocda-lib/dlang" } } +``` + +```d +import ocda.load; +auto doc = load("document.ocda.db", "de"); // or "document.ssp" +foreach (obj; doc.body) { ... } +``` + +Link `sqlite3`; the library declares the dozen functions it needs rather +than taking a binding from the registry, so there is nothing to fetch. + +## The two checks + +```sh +nix build .#ocda-agree && ./result/bin/ocda-agree <doc>.ssp <doc>.ocda.db [lang] +nix build .#ocda-dump && ./result/bin/ocda-dump [--unknown] <file> [lang] +``` + +`agree` loads both and compares ten fields of each object, naming the field +where they part company. `agree` found the blank-first-text-line defect and +was not enough for anything subtler. + +**`ocda-dump` is the stronger check**: the whole loaded model, every field, +written out in a form fully determined by the model, so that two +implementations can be compared with `diff`. It is an oracle, not an output +format; anything reading it to reconstruct a document has misunderstood it. +It agrees byte for byte with the Gleam library's `ocda-lib dump`, which is +the point of having it. + +It found four disagreements `agree` could not see. They are in the +repository README and in the history. + +## Build + +```sh +nix build .#ocda-agree # the agreement check +nix build .#ocda-dump # the canonical dump +nix develop # ldc, dub, dtools, sqlite +dub build # the library +``` diff --git a/dlang/dub.json b/dlang/dub.json new file mode 100644 index 0000000..4f59aec --- /dev/null +++ b/dlang/dub.json @@ -0,0 +1,13 @@ +{ + "authors": [ "Ralph Amissah" ], + "copyright": "Copyright (C) 2026 Ralph Amissah", + "name": "ocda-lib", + "version": "2.0.0", + "description": "the object-centric document abstraction as a library: the model, and loaders for its two serialisations", + "homepage": "https://sisudoc.org", + "license": "AGPL-3.0+", + "targetType": "library", + "sourcePaths": [ "./src" ], + "importPaths": [ "./src" ], + "libs": [ "sqlite3", "zstd" ] +} diff --git a/dlang/dub.selections.json b/dlang/dub.selections.json new file mode 100644 index 0000000..84a60aa --- /dev/null +++ b/dlang/dub.selections.json @@ -0,0 +1 @@ +{ "fileVersion": 1, "versions": {} } diff --git a/dlang/flake.lock b/dlang/flake.lock new file mode 100644 index 0000000..5f236ab --- /dev/null +++ b/dlang/flake.lock @@ -0,0 +1,27 @@ +{ + "nodes": { + "nixpkgs": { + "locked": { + "lastModified": 1790185690, + "narHash": "sha256-xJ+X4hBtOcAFGBOe5nAMyMUeF9foJBmIOu3NjBqBycU=", + "owner": "NixOS", + "repo": "nixpkgs", + "rev": "4975466d324710c576dc11ad614684e6bd8cad8e", + "type": "github" + }, + "original": { + "owner": "NixOS", + "ref": "nixos-unstable", + "repo": "nixpkgs", + "type": "github" + } + }, + "root": { + "inputs": { + "nixpkgs": "nixpkgs" + } + } + }, + "root": "root", + "version": 7 +} diff --git a/dlang/flake.nix b/dlang/flake.nix new file mode 100644 index 0000000..3aa9a60 --- /dev/null +++ b/dlang/flake.nix @@ -0,0 +1,63 @@ +{ + description = "ocda-lib: the object-centric document abstraction as a D library"; + + inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable"; + + outputs = { self, nixpkgs }: + let + system = "x86_64-linux"; # adjust for aarch64-linux / darwin + pkgs = import nixpkgs { inherit system; }; + + /* A library, so there is nothing to install as a binary. What is + built here are the two programs that put the library to the + question: the agreement check, which loads a document's .ssp and + its .ocda.db and compares the two models, and the dump, which + writes a loaded model out canonically so that an implementation in + another language can be diffed against this one. + */ + mkTool = { tool, binary, what, compiler, compilerBin }: + pkgs.stdenv.mkDerivation { + pname = "ocda-lib-${tool}"; + version = "2.0.0"; + src = self; + nativeBuildInputs = [ compiler ]; + buildInputs = [ pkgs.sqlite pkgs.zstd ]; + buildPhase = '' + runHook preBuild + export HOME=$TMPDIR + ${compilerBin} -I=src -of=${tool} src/ocda/*.d test/${tool}.d -L-lsqlite3 -L-lzstd + runHook postBuild + ''; + installPhase = '' + runHook preInstall + mkdir -p $out/bin + install -m755 ${tool} $out/bin/${binary} + runHook postInstall + ''; + meta = { + description = what; + mainProgram = binary; + }; + }; + + ldc = { compiler = pkgs.ldc; compilerBin = "ldmd2"; }; + in { + packages.${system} = { + default = self.packages.${system}.ocda-agree; + + ocda-agree = mkTool (ldc // { + tool = "agree"; binary = "ocda-agree"; + what = "checks that the .ssp and .ocda.db loaders give one model"; + }); + + ocda-dump = mkTool (ldc // { + tool = "dump"; binary = "ocda-dump"; + what = "writes a loaded abstraction out canonically, as an oracle"; + }); + }; + + devShells.${system}.default = pkgs.mkShell { + packages = [ pkgs.ldc pkgs.dub pkgs.dtools pkgs.sqlite pkgs.zstd ]; + }; + }; +} diff --git a/dlang/src/COPYRIGHT b/dlang/src/COPYRIGHT new file mode 100644 index 0000000..7e7dd56 --- /dev/null +++ b/dlang/src/COPYRIGHT @@ -0,0 +1,45 @@ +- Name: sisudoc-ocda-lib + - Description: part of SiSU spine (Doc Reform): documents, structuring, + processing, publishing and search + - [https://www.sisudoc.org] + + - Author: Ralph Amissah + [ralph.amissah@gmail.com] + + - Copyright: (C) 2026 Ralph Amissah, All Rights Reserved. + + - License: AGPL 3 or later, for everything in this repository unless a file or + a directory specifies otherwise.: + + sisudoc-ocda-lib + + part of SiSU spine (Doc Reform): documents, structuring, + processing, publishing and search + + Copyright (C) Ralph Amissah + + This program is free software: you can redistribute it and/or modify it + under the terms of the GNU Affero General Public License (AGPLv3) as + published by the Free Software Foundation, either version 3 of the License, + or (at your option) any later version. + + This program is distributed in the hope that it will be useful, but + WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY + or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public + License for more details. + + You should have received a copy of the GNU Affero General Public License + along with this program. If not, see [https://www.gnu.org/licenses/]. + + If you have Internet connection, the latest version of the AGPL should be + available at: [https://www.gnu.org/licenses/agpl-3.0.html] + + - Spine, Doc Reform (related to SiSU) uses standard: + - docReform markup syntax (based on SiSU markup) + - standard SiSU markup syntax with modified headers and minor + modifications + - docReform object numbering (based on SiSU object citation numbering) + - standard SiSU document object numbering + + - Homepages: + [https://www.sisudoc.org] diff --git a/dlang/src/ocda/dump.d b/dlang/src/ocda/dump.d new file mode 100644 index 0000000..7a4f11a --- /dev/null +++ b/dlang/src/ocda/dump.d @@ -0,0 +1,164 @@ +module ocda.dump; + +/+ ↓ the model, written out canonically. + . + Not an output format: an *oracle*. Two libraries in two languages + implement the same model from the same specification, and the cheap + way to ask whether they agree is to have each write the whole loaded + document as text and diff the two. That is only useful if the text is + fully determined by the model, so: + . + - every line is "<key> <value>", the value running to end of line, + and only one free-form value per line; + - a field at its default is *omitted*, so the output says what the + document has rather than what the struct declares; + - dictionaries are emitted in sorted key order, because neither + language's hash map has an order worth relying on; + - sections come in the fixed order of the format, not + alphabetically. + . + Anything that reads this to reconstruct a document has misunderstood + it. Load the .ssp or the .ocda.db. ++/ + +import std.algorithm : sort, splitter; +import std.array : appender, array, join; +import std.conv : to; +import ocda.model; + +/+ ↓ the nine sections, in format order +/ +static immutable string[] dump_sections = ["head", "toc", "body", "endnotes", + "glossary", "bibliography", "bookindex", "blurb", "tail"]; + +string dumpDocument(ref OcdaDocument doc) { + auto out_ = appender!string; + + void kv(string key, string value) { + if (value.length == 0) { return; } + out_ ~= key; out_ ~= " "; out_ ~= value; out_ ~= "\n"; + } + void kn(string key, long value) { + if (value == 0) { return; } + kv(key, value.to!string); + } + void kb(string key, bool value) { + if (!value) { return; } + kv(key, "1"); + } + void dict(string prefix, string[string] d) { + foreach (key; d.keys.dup.sort) { + out_ ~= prefix; out_ ~= " "; out_ ~= key; out_ ~= " "; + out_ ~= d[key]; out_ ~= "\n"; + } + } + + kv("version", doc.format_version); + kv("source_filename", doc.source_filename); + dict("source", doc.source); + dict("meta", doc.metadata); + dict("make", doc.make); + dict("doc_has", doc.doc_has); + + foreach (name; dump_sections) { + auto sec = doc.sectionByName(name); + if (sec is null || (*sec).length == 0) { continue; } + kv("section", name ~ " " ~ (*sec).length.to!string); + } + + foreach (name; dump_sections) { + auto sec = doc.sectionByName(name); + if (sec is null) { continue; } + foreach (i, obj; *sec) { + out_ ~= "object "; out_ ~= name; out_ ~= " "; + out_ ~= i.to!string; out_ ~= "\n"; + kn("ocn", obj.ocn); + kv("is_a", obj.is_a); + kv("own_section", obj.own_section); + kv("level", obj.heading_level); + kv("identifier", obj.identifier); + kv("part", obj.part); + kv("is_of_type", obj.is_of_type); + kv("segment_html_is", obj.segment_html_is); + kv("segment_epub", obj.segment_epub); + kv("segment_epub_is", obj.segment_epub_is); + kn("parent", obj.parent_ocn); + kn("last_descendant", obj.last_descendant_ocn); + kv("children", ints(obj.children)); + kv("ancestors", ints(obj.ancestors)); + kb("dummy", obj.dummy_heading); + kb("ocn_off", obj.ocn_off); + kn("indent_base", obj.indent_base); + kn("indent_hang", obj.indent_hang); + kb("bullet", obj.bullet); + kv("lang", obj.lang); + kb("has_links", obj.has_links); + kb("has_notes_reg", obj.has_notes_reg); + kb("has_notes_star", obj.has_notes_star); + kb("has_images", obj.has_images); + kv("segment", obj.segment); + kv("segment_prev", obj.segment_prev); + kv("segment_next", obj.segment_next); + kv("anchor", obj.anchor); + kv("heading_lev_anchor", obj.heading_lev_anchor); + kn("table_cols", obj.table_cols); + kv("table_widths", obj.table_widths); + kv("table_aligns", obj.table_aligns); + kb("table_header", obj.table_header); + kv("code_syntax", obj.code_syntax); + kb("code_linenumbers", obj.code_linenumbers); + kv("heading_ancestors_text", obj.heading_ancestors_text); + kv("sha256", obj.sha256); + foreach (url; obj.stow_link) { kv("stow_link", url); } + foreach (a; obj.anchor_tags) { kv("anchor_tag", a); } + foreach (img; obj.images) { + /+ ↓ the name last, because a filename may hold a space where + none of the other fields can +/ + kv("image", img.sha256 ~ " " ~ img.bytes.to!string ~ " " + ~ img.width.to!string ~ "x" ~ img.height.to!string ~ " " + ~ (img.missing ? "1" : "0") ~ " " ~ img.name); + foreach (key; img.fields.keys.dup.sort) { + kv("image_field", key ~ " " ~ img.fields[key]); + } + } + /+ ↓ text, when there is any. An object whose only text line is + blank holds "" here, indistinguishable from no text at all, + because the model joins the lines into one string. Both + libraries lose the same distinction in the same place, which is + what makes them comparable; it is recorded rather than hidden. + +/ + if (obj.text.length > 0) { + auto lines = obj.text.splitter("\n").array; + kv("text", lines.length.to!string); + foreach (line; lines) { + out_ ~= "| "; out_ ~= line; out_ ~= "\n"; + } + } + } + } + return out_.data; +} + +/+ ↓ what the reader met and did not understand, sorted. Separate from the + document dump: it describes the reader, not the document, and the + .ocda.db loader has nothing to report where the .ssp loader does. ++/ +string dumpUnknown(ref OcdaDocument doc) { + auto out_ = appender!string; + void counts(string prefix, size_t[string] d) { + foreach (key; d.keys.dup.sort) { + out_ ~= prefix; out_ ~= " "; out_ ~= key; out_ ~= " "; + out_ ~= d[key].to!string; out_ ~= "\n"; + } + } + counts("property", doc.unknown.property); + counts("block", doc.unknown.block); + counts("line", doc.unknown.line); + return out_.data; +} + +private string ints(int[] xs) { + if (xs.length == 0) { return ""; } + string[] parts; + foreach (x; xs) { parts ~= x.to!string; } + return parts.join(","); +} diff --git a/dlang/src/ocda/images.d b/dlang/src/ocda/images.d new file mode 100644 index 0000000..b3a9652 --- /dev/null +++ b/dlang/src/ocda/images.d @@ -0,0 +1,100 @@ +module ocda.images; + +/+ ↓ an image's bytes, from whichever source the document came from. + . + The model *names* images, with digest, byte size and pixel size; it does + not hold them. That is right - an abstraction of a document is not a + bundle of its assets - but a writer of any format that shows images + needs the bytes, so something has to know where they are. This is that + something, and it is in the library rather than in a writer because the + answer depends on the *input form* and not on the output. + . + Three answers, and one refusal: + . + .ocda.db the files table, role 'image', by name + .sisupod the archive entry whose path ends in /image/<name> + .zip the same + .ssp beside it: a pod puts the abstraction in + media/abstraction/ and its images in media/image/, so the + path is documented rather than guessed + . + A .ssp that is not in a pod has nowhere to look, and that is reported as + absent rather than as an error: a reader that cannot find an image + should say so and carry on, because the document is still readable. ++/ + +import std.algorithm : endsWith; +import std.array : array; +import std.file : exists, isFile, read; +import std.path : buildPath, dirName; +import ocda.load; +import ocda.sqlite; +import ocda.zip; +import ocda.zstd; + +/+ ↓ the bytes, or an empty slice when the source has no such image +/ +ubyte[] imageBytes(string source, string name) { + final switch (sourceOf(source)) { + case Source.ocda_db: return fromDb(source, name); + case Source.pod: return fromArchive(source, name); + case Source.ssp: return fromBeside(source, name); + case Source.unknown: return []; + } +} + +/+ ↓ can this source give images at all? + A writer asks before it decides whether to emit them, because an <img> + pointing at a file it did not include is worse than no image. ++/ +bool canGiveImages(string source) { + final switch (sourceOf(source)) { + case Source.ocda_db: return true; + case Source.pod: return true; + case Source.ssp: return imageDirBeside(source).length > 0; + case Source.unknown: return false; + } +} + +private ubyte[] fromDb(string path, string name) { + auto db = Db.open(path); + scope(exit) { db.close(); } + if (!db.has("files")) { return []; } + /+ ↓ the name is from the abstraction, not from a caller, and the quote + doubling is what keeps it a value rather than syntax +/ + return db.blob("SELECT data FROM files WHERE role = 'image' AND name = '" + ~ sqlQuote(name) ~ "'"); +} + +private ubyte[] fromArchive(string path, string name) { + auto bytes = cast(ubyte[]) read(path); + if (zstdIsFrame(bytes)) { bytes = zstdDecompress(bytes, 512 * 1024 * 1024); } + auto entries = zipEntries(bytes); + foreach (e; entries) { + if (e.name.endsWith("/image/" ~ name)) { return zipRead(bytes, e); } + } + return []; +} + +private ubyte[] fromBeside(string path, string name) { + string dir = imageDirBeside(path); + if (dir.length == 0) { return []; } + string file = buildPath(dir, name); + return file.exists && file.isFile ? cast(ubyte[]) read(file) : []; +} + +/+ ↓ media/abstraction/<x>.ssp -> media/image/ +/ +private string imageDirBeside(string path) { + string dir = path.dirName; + if (!dir.endsWith("abstraction")) { return ""; } + string image_dir = buildPath(dir.dirName, "image"); + return image_dir.exists ? image_dir : ""; +} + +/+ ↓ a single quote doubled is how sql says a single quote +/ +private string sqlQuote(string s) { + string out_; + foreach (c; s) { + out_ ~= (c == '\'') ? "''" : [c]; + } + return out_; +} diff --git a/dlang/src/ocda/load.d b/dlang/src/ocda/load.d new file mode 100644 index 0000000..89c8dcc --- /dev/null +++ b/dlang/src/ocda/load.d @@ -0,0 +1,69 @@ +module ocda.load; + +/+ ↓ what a path is, and loading it. + . + One place that decides, so that every reader and every writer accepts + the same inputs and a new input type is added once. This mirrors + spine's own ocda/abstraction/load.d, which names the sources it can + enter the pipeline from and tells them apart by suffix. + . + Deliberately *not* here: .sst and .ssm markup. Reading markup means + implementing the parser, which is spine's work and a far larger thing + than a reader or a sample writer needs. This library reads + abstractions. ++/ + +import std.algorithm : endsWith; +import ocda.model; +import ocda.load_ssp; +import ocda.load_db; +import ocda.load_pod; + +enum Source { + ssp, // .ssp, the abstraction as text + ocda_db, // .ocda.db, the abstraction as sqlite + pod, // .sisupod or .zip, an archive carrying a .ssp + unknown, +} + +Source sourceOf(string path) { + if (path.endsWith(".ocda.db")) { return Source.ocda_db; } + if (path.endsWith(".ssp")) { return Source.ssp; } + /+ ↓ .sisupod is the archive spine writes now; .zip is what it wrote + before, and every pod already published carries that name, so both + are read. What is inside decides how it is opened, not the name. + +/ + if (path.endsWith(".sisupod")) { return Source.pod; } + if (path.endsWith(".zip")) { return Source.pod; } + if (path.endsWith(".db")) { return Source.ocda_db; } + return Source.unknown; +} + +string sourceName(Source s) { + final switch (s) { + case Source.ssp: return ".ssp (the abstraction as text)"; + case Source.ocda_db: return ".ocda.db (the abstraction as sqlite)"; + case Source.pod: return "a pod archive carrying a .ssp"; + case Source.unknown: return "not an abstraction this library reads"; + } +} + +class OcdaLoadError : Exception { + this(string msg) { super(msg); } +} + +/+ ↓ load whatever the path is into the model. + lang names which language to take from a .ocda.db, which holds + every language of its document, and from a pod, which may carry more + than one .ssp. A .ssp is one language by construction and ignores it. ++/ +OcdaDocument load(string path, string lang = "", bool any_version = false) { + final switch (sourceOf(path)) { + case Source.ssp: return parseSspFile(path); + case Source.ocda_db: return loadDbFile(path, lang, any_version); + case Source.pod: return loadPodFile(path, lang); + case Source.unknown: + throw new OcdaLoadError(path ~ ": " ~ sourceName(Source.unknown) + ~ " (expected .ssp, .ocda.db, .sisupod or .zip)"); + } +} diff --git a/dlang/src/ocda/load_db.d b/dlang/src/ocda/load_db.d new file mode 100644 index 0000000..dec0626 --- /dev/null +++ b/dlang/src/ocda/load_db.d @@ -0,0 +1,310 @@ +module ocda.load_db; + +/+ ↓ build the model from a .ocda.db. + . + The point of this module is that it returns *the same value* + ocda.load_ssp returns from the same document's .ssp. A writer is then + a function from the model to bytes, and cannot tell which + serialisation it was given, exactly as spine's own writers cannot. + . + Written from the sisudoc-abstraction-format specification, + ocda-db-format-specification.md, whose section 4 gives the column by + column correspondence this module implements. ++/ + +import std.array : join, split; +import std.json : JSONType, parseJSON; +import std.conv : to; +import std.algorithm : startsWith; +import std.string : strip; +import ocda.model; +import ocda.sqlite; + +/+ ↓ the schema major this loader knows (specification section 2) +/ +enum schema_major_read = "2"; + +class OcdaDbError : Exception { + this(string msg) { super(msg); } +} + +/+ ↓ every language the database holds, in manifest order +/ +struct DbLanguage { + int doc_id; + string lang; + string source_filename; + string ssp_digest; +} + +DbLanguage[] dbLanguages(ref Db db) { + DbLanguage[] out_; + foreach (r; db.rows( + "SELECT id, lang, coalesce(source_filename,''), coalesce(ssp_digest,'') + FROM documents ORDER BY id")) { + out_ ~= DbLanguage(r[0].to!int, r[1], r[2], r[3]); + } + return out_; +} + +/+ ↓ load one language of a database as the model. + lang == "" takes the first language the manifest gave. ++/ +OcdaDocument loadDbFile(string path, string lang = "", bool any_version = false) { + auto db = Db.open(path); + scope(exit) { db.close(); } + + if (!db.has("documents") || !db.has("objects") || !db.has("metadata")) { + throw new OcdaDbError(path ~ ": no documents/objects/metadata tables;" + ~ " it does not look like a .ocda.db"); + } + + /+ ↓ specification sections 2 and 3.2: the rows describing the file rather + than a document in it are the only ones with doc_id null +/ + string schema_version = db.one( + "SELECT value FROM metadata WHERE doc_id IS NULL AND key = 'schema.version'"); + if (schema_version.length > 0 && !any_version) { + auto major = schema_version.split(".")[0]; + if (major != schema_major_read) { + throw new OcdaDbError(path ~ " is schema v" ~ schema_version + ~ ", this loader knows v" ~ schema_major_read ~ ".x"); + } + } + + auto languages = dbLanguages(db); + if (languages.length == 0) { + throw new OcdaDbError(path ~ ": no documents rows"); + } + DbLanguage chosen = languages[0]; + if (lang.length > 0) { + bool found = false; + foreach (l; languages) { if (l.lang == lang) { chosen = l; found = true; break; } } + if (!found) { + string have; + foreach (i, l; languages) { have ~= (i > 0 ? " " : "") ~ l.lang; } + throw new OcdaDbError(path ~ ": no language '" ~ lang ~ "'; it holds: " ~ have); + } + } + + OcdaDocument doc; + doc.format_version = schema_version; + doc.source_filename = chosen.source_filename; + + /+ ↓ @source of the .ssp corresponds to the documents row and the + source.digest metadata (specification section 5.1 of the .ssp, and + section 7 here) +/ + doc.source["language"] = chosen.lang; + string languages_joined; + foreach (i, l; languages) { languages_joined ~= (i > 0 ? " " : "") ~ l.lang; } + doc.source["languages"] = languages_joined; + auto digest = db.one("SELECT value FROM metadata WHERE doc_id = " + ~ chosen.doc_id.to!string ~ " AND key = 'source.digest'"); + if (digest.length > 0) { doc.source["digest"] = digest; } + + /+ ↓ @meta: every metadata row of this language that is not a source.* + property. source.* describes the artefact rather than the document + and lives in @source in the text form. + +/ + foreach (r; db.rows("SELECT key, value FROM metadata WHERE doc_id = " + ~ chosen.doc_id.to!string ~ " ORDER BY key")) { + string key = r[0]; + if (key.startsWith("source.")) { continue; } + /+ ↓ translation.ocn_aligned says whether the languages of this + document share one numbering. A .ocda.db can know it because it + holds every language; a .ssp is one language and cannot. So it is + a fact about the *file* rather than about the document, and putting + it in the document's metadata would let a writer tell which + serialisation it was handed, which is the one thing this library + exists to prevent. Left out until the model has a place for + collection-wide facts. + +/ + if (key.startsWith("translation.")) { continue; } + /+ ↓ the database has one metadata table where the text form has three + blocks, and tells them apart by a key prefix. Unprefixing them + here is what makes the two loaders give one model: a writer asks + doc.make["footer"] whichever serialisation it was handed, and + never doc.metadata["make.footer"]. + +/ + if (key.startsWith("make.")) { + doc.make[key["make.".length .. $]] = r[1].strip; + } else if (key.startsWith("doc_has.")) { + doc.doc_has[key["doc_has.".length .. $]] = r[1].strip; + } else { + doc.metadata[key] = r[1].strip; + } + } + + /+ ↓ the objects, in document order within each section +/ + foreach (r; db.rows( + "SELECT section, seq, ocn, is_a, coalesce(is_of_part,''), + coalesce(is_of_section,''), coalesce(is_of_type,''), + coalesce(heading_level,-1), coalesce(identifier,''), + coalesce(parent_ocn,0), coalesce(last_descendant_ocn,0), + coalesce(children,''), coalesce(ancestors,''), + coalesce(dummy_heading,0), coalesce(object_number_off,0), + coalesce(indent_base,0), coalesce(indent_hang,0), + coalesce(bullet,0), coalesce(lang,''), + coalesce(has_links,0), coalesce(has_notes_reg,0), + coalesce(has_notes_star,0), coalesce(has_images,0), + coalesce(segment,''), coalesce(segment_prev,''), + coalesce(segment_next,''), coalesce(anchor,''), + coalesce(segment_html_is,''), + coalesce(table_cols,0), coalesce(table_widths,''), + coalesce(table_aligns,''), coalesce(table_header,0), + coalesce(code_linenumbers,0), + coalesce(heading_ancestors_text,''), coalesce(sha256,''), + coalesce(text,''), id, coalesce(heading_lev_anchor,''), + coalesce(segment_epub,''), coalesce(segment_epub_is,'') + FROM objects WHERE doc_id = " ~ chosen.doc_id.to!string + ~ " ORDER BY CASE section + WHEN 'head' THEN 1 WHEN 'toc' THEN 2 WHEN 'body' THEN 3 + WHEN 'endnotes' THEN 4 WHEN 'glossary' THEN 5 + WHEN 'bibliography' THEN 6 WHEN 'bookindex' THEN 7 + WHEN 'blurb' THEN 8 WHEN 'tail' THEN 9 ELSE 10 END, seq")) { + OcdaObject obj; + obj.section = r[0]; + obj.ocn = r[2].to!int; + obj.is_a = r[3]; + obj.part = r[4]; + obj.own_section = (r[5] == r[0]) ? "" : r[5]; + obj.is_of_type = r[6]; + /+ ↓ the database holds the level as a number, the text form as the + letter or digit the markup wrote. Model section 4 is the mapping. +/ + obj.heading_level = levelName(r[7].to!int); + obj.identifier = r[8]; + obj.parent_ocn = r[9].to!int; + obj.last_descendant_ocn = r[10].to!int; + obj.children = jsonInts(r[11]); + obj.ancestors = jsonInts(r[12]); + obj.dummy_heading = (r[13] != "0"); + obj.ocn_off = (r[14] != "0"); + obj.indent_base = r[15].to!int; + obj.indent_hang = r[16].to!int; + obj.bullet = (r[17] != "0"); + obj.lang = r[18]; + obj.has_links = (r[19] != "0"); + obj.has_notes_reg = (r[20] != "0"); + obj.has_notes_star = (r[21] != "0"); + obj.has_images = (r[22] != "0"); + obj.segment = r[23]; + obj.segment_prev = r[24]; + obj.segment_next = r[25]; + obj.anchor = r[26]; + obj.segment_html_is = r[27]; + obj.heading_lev_anchor = r[37]; + obj.segment_epub = r[38]; + obj.segment_epub_is = r[39]; + obj.table_cols = r[28].to!int; + obj.table_widths = jsonWords(r[29]); + obj.table_aligns = jsonWords(r[30]); + obj.table_header = (r[31] != "0"); + obj.code_linenumbers = (r[32] != "0"); + obj.heading_ancestors_text = jsonJoined(r[33]); + obj.sha256 = r[34]; + obj.text = r[35]; + + /+ ↓ the four repeatable properties of the text form are child tables + here (specification section 4) +/ + string object_id = r[36]; + foreach (i; db.rows("SELECT name, coalesce(sha256,''), coalesce(bytes,0), + coalesce(width,0), coalesce(height,0), + coalesce(missing,0) + FROM object_images WHERE object_id = " ~ object_id + ~ " ORDER BY seq")) { + OcdaImage img; + img.name = i[0]; + img.sha256 = i[1]; + img.bytes = i[2].to!ulong; + img.width = i[3].to!int; + img.height = i[4].to!int; + img.missing = (i[5] != "0"); + obj.images ~= img; + } + foreach (l; db.rows("SELECT url FROM object_links WHERE object_id = " + ~ object_id ~ " ORDER BY seq")) { + obj.stow_link ~= l[0]; + } + foreach (a; db.rows("SELECT anchor FROM object_anchors WHERE object_id = " + ~ object_id ~ " ORDER BY seq")) { + obj.anchor_tags ~= a[0]; + } + + auto section = doc.sectionByName(r[0]); + if (section !is null) { *section ~= obj; } + } + return doc; +} + +/+ ↓ model section 4: the eight levels and their indices +/ +private string levelName(int index) { + switch (index) { + case 0: return "A"; + case 1: return "B"; + case 2: return "C"; + case 3: return "D"; + case 4: return "1"; + case 5: return "2"; + case 6: return "3"; + case 7: return "4"; + default: return ""; + } +} + +/+ ↓ the JSON columns, read as JSON. + . + The fixed width arrays are JSON in the database and space separated + (or "|" separated) in the text form. This was first written as a + character scan that threw away brackets and quotes, which is wrong the + moment a value holds an escape: Phobos' JSONValue.toString escapes + "/" as "\/", so every heading containing a slash came back with a + backslash in it. 36 of 36 documents differed on + heading_ancestors_text and nothing else had shown it. + . + So: parse. The values are small, there are few of them, and a real + parser is shorter than the scan it replaces. ++/ +private string[] jsonStrings(string v) { + if (v.length == 0) { return []; } + string[] out_; + try { + auto j = parseJSON(v); + if (j.type != JSONType.array) { return []; } + foreach (e; j.array) { + switch (e.type) { + case JSONType.string: out_ ~= e.str; break; + case JSONType.integer: out_ ~= e.integer.to!string; break; + case JSONType.uinteger: out_ ~= e.uinteger.to!string; break; + case JSONType.float_: out_ ~= e.floating.to!string; break; + case JSONType.true_: out_ ~= "true"; break; + case JSONType.false_: out_ ~= "false"; break; + default: out_ ~= ""; break; + } + } + } catch (Exception) { + return []; + } + return out_; +} + +private int[] jsonInts(string v) { + int[] out_; + foreach (s; jsonStrings(v)) { + if (s.length == 0) { continue; } + try { out_ ~= s.to!int; } catch (Exception) {} + } + return out_; +} + +/+ ↓ space separated in the text form: table_widths, table_aligns +/ +private string jsonWords(string v) { + return jsonStrings(v).join(" "); +} + +/+ ↓ heading_ancestors_text is eight slots, joined with "|" in the text + form and a JSON array here. The empty slots are significant: they say + which levels the heading sits under, so the separators carry as much + as the text does. ++/ +private string jsonJoined(string v) { + auto parts = jsonStrings(v); + if (parts.length == 0) { return ""; } + return parts.join("|"); +} diff --git a/dlang/src/ocda/load_pod.d b/dlang/src/ocda/load_pod.d new file mode 100644 index 0000000..75993be --- /dev/null +++ b/dlang/src/ocda/load_pod.d @@ -0,0 +1,115 @@ +module ocda.load_pod; + +/+ ↓ the abstraction out of a pod archive. + . + A .sisupod is one zstd frame wrapping a stored zip; a .zip is the same + archive unwrapped, which is what spine published before it wrapped them. + Either way, a pod built with --pod2 carries the abstraction inside it, + at media/abstraction/<uid>.<lang>.ssp, and that is what is read. + . + So this is a third input form and *not* a third model: the .ssp inside + is parsed by ocda.load_ssp and comes back as the same value. A writer + over the library gains .sisupod without being touched, which is the + whole reason the dispatch is in one place. + . + A pod built without --pod2 carries markup and no abstraction. That is + refused, and the refusal says which flag makes a pod this reader can + read, because "no abstraction in it" is a fact about the pod rather + than about the reader. ++/ + +import std.algorithm : endsWith, sort; +import std.array : array; +import std.file : read; +import std.string : indexOf; +import ocda.model; +import ocda.load_ssp; +import ocda.zip; +import ocda.zstd; + +/+ ↓ a limit on what a frame may expand to. + A reader of an artefact it did not make declines to be told how much + memory to allocate. 512 MiB is far above the largest pod in the + reference collection (4 MiB) and far below anything that matters. ++/ +enum size_t max_pod_size = 512 * 1024 * 1024; + +class OcdaPodError : Exception { + this(string msg) { super(msg); } +} + +/+ ↓ which languages of the abstraction the pod carries, in sorted order +/ +string[] podLanguages(string path) { + auto bytes = archiveBytes(path); + string[] out_; + foreach (e; zipFindSuffix(zipEntries(bytes), ".ssp")) { + string lang = langOfSspName(e.name); + if (lang.length > 0) { out_ ~= lang; } + } + return out_.sort.array; +} + +/+ ↓ load one language of the abstraction a pod carries. + lang == "" takes the only one there is, and where there is a choice it + says so rather than choosing, as the .ocda.db loader does. ++/ +OcdaDocument loadPodFile(string path, string lang = "") { + auto bytes = archiveBytes(path); + auto entries = zipEntries(bytes); + auto ssps = zipFindSuffix(entries, ".ssp"); + if (ssps.length == 0) { + throw new OcdaPodError(path ~ ": carries no .ssp." + ~ " A pod carries the abstraction only when spine was asked for it:" + ~ " spine --pod2 puts it in, spine --source does not."); + } + ZipEntry chosen = ssps[0]; + if (lang.length > 0) { + bool found = false; + foreach (e; ssps) { + if (langOfSspName(e.name) == lang) { chosen = e; found = true; break; } + } + if (!found) { + string have; + foreach (i, e; ssps) { + have ~= (i > 0 ? " " : "") ~ langOfSspName(e.name); + } + throw new OcdaPodError(path ~ ": no language '" ~ lang + ~ "'; it holds: " ~ have); + } + } else if (ssps.length > 1) { + string have; + foreach (i, e; ssps) { + have ~= (i > 0 ? " " : "") ~ langOfSspName(e.name); + } + throw new OcdaPodError(path ~ ": holds more than one language (" + ~ have ~ "); name the one wanted"); + } + auto ssp = zipRead(bytes, chosen); + return parseSsp(cast(string) ssp.idup); +} + +/+ ↓ the archive, unwrapped if it was wrapped. + The bytes decide, not the name: a .sisupod is a zstd frame and a .zip + is not, and a file misnamed either way still reads. ++/ +private ubyte[] archiveBytes(string path) { + auto bytes = cast(ubyte[]) read(path); + return zstdIsFrame(bytes) + ? zstdDecompress(bytes, max_pod_size) + : bytes; +} + +/+ ↓ <uid>.<lang>.ssp -> <lang> +/ +private string langOfSspName(string name) { + if (!name.endsWith(".ssp")) { return ""; } + string stem = name[0 .. $ - 4]; + auto dot = stem.lastIndexOf('.'); + return (dot > 0) ? stem[dot + 1 .. $] : ""; +} + +private ptrdiff_t lastIndexOf(string s, char c) { + for (ptrdiff_t i = s.length - 1; i >= 0; i--) { + if (s[i] == c) { return i; } + } + return -1; +} diff --git a/dlang/src/ocda/load_ssp.d b/dlang/src/ocda/load_ssp.d new file mode 100644 index 0000000..34d156c --- /dev/null +++ b/dlang/src/ocda/load_ssp.d @@ -0,0 +1,337 @@ +module ocda.load_ssp; + +import std.algorithm : splitter, startsWith; +import std.array : array, split; +import std.conv : to, ConvException; +import std.file : readText; +import std.stdio : writeln; +import std.string : strip, stripLeft, indexOf; +import ocda.model; + +OcdaDocument parseSspFile(string filename) { + string content = readText(filename); + return parseSsp(content); +} + +OcdaDocument parseSsp(string content) { + OcdaDocument doc; + OcdaObject current_obj; + bool in_object = false; + /+ ↓ whether a text line has been seen for the object being read. + Not "is the text non-empty": an object whose first text line is blank + (a verse opening on an empty line, which the sample set has seven of) + would then lose it, the next line overwriting rather than appending. + +/ + bool seen_text = false; + string current_section = ""; + string current_block = ""; // meta, make, doc_has, or section name + + foreach (line; content.splitter("\n")) { + /+ ↓ comment. The first line of a .ssp declares the format version, and + it is the one comment that carries meaning: a reader checks the + major part and refuses what it does not know rather than loading + an unfamiliar artefact half populated in silence. + +/ + if (line.length > 0 && line[0] == '%') { + enum _banner = "% SiSU Document Abstraction v"; + enum _source = "% Source: "; + if (doc.format_version.length == 0 && line.startsWith(_banner)) { + doc.format_version = line[_banner.length .. $].strip; + } else if (doc.source_filename.length == 0 && line.startsWith(_source)) { + doc.source_filename = line[_source.length .. $].strip; + } + continue; + } + + /+ ↓ blank line - finalize current object +/ + if (line.length == 0 || line.strip.length == 0) { + if (in_object) { + commitObject(doc, current_section, current_obj); + current_obj = OcdaObject.init; + in_object = false; + seen_text = false; + } + continue; + } + + /+ ↓ section/block open +/ + if (line[0] == '@') { + auto brace_pos = line.indexOf('{'); + if (brace_pos > 0) { + string name = line[1 .. brace_pos].strip; + current_block = name; + if (!isPreambleBlock(name)) { + current_section = name; + /+ ↓ a block that is neither preamble nor a section this reader + knows: counted, so that --report-unknown can say so +/ + if (doc.sectionByName(name) is null) { + doc.unknown.block[name]++; + } + } + } + continue; + } + + /+ ↓ section/block close +/ + if (line[0] == '}') { + if (in_object) { + commitObject(doc, current_section, current_obj); + current_obj = OcdaObject.init; + in_object = false; + seen_text = false; + } + current_block = ""; + continue; + } + + /+ ↓ inside meta/make/doc_has blocks: key-value pairs +/ + if (isPreambleBlock(current_block)) { + string stripped = line.stripLeft; + auto colon_pos = stripped.indexOf(':'); + if (colon_pos > 0) { + string key = stripped[0 .. colon_pos].strip; + string value = (colon_pos + 1 < stripped.length) + ? stripped[colon_pos + 1 .. $].strip : ""; + if (current_block == "meta") { doc.metadata[key] = value; } + else if (current_block == "make") { doc.make[key] = value; } + else if (current_block == "source") { doc.source[key] = value; } + else { doc.doc_has[key] = value; } + } + continue; + } + + /+ ↓ object declaration: [N] type ... +/ + if (line[0] == '[') { + if (in_object) { + commitObject(doc, current_section, current_obj); + } + current_obj = OcdaObject.init; + current_obj.section = current_section; + in_object = true; + seen_text = false; + parseObjectDeclaration(line, current_obj); + continue; + } + + /+ ↓ property: .name: value +/ + if (line[0] == '.') { + if (in_object) { + parseProperty(line, current_obj, doc.unknown); + } + continue; + } + + /+ ↓ text content: | content +/ + if (line.length >= 1 && line[0] == '|') { + if (in_object) { + string text_content = (line.length > 2) ? line[2 .. $] : ""; + if (seen_text) { + current_obj.text ~= "\n" ~ text_content; + } else { + current_obj.text = text_content; + seen_text = true; + } + } + continue; + } + + /+ ↓ no known shape. Counted by its first character so that a reader + can be told the file held something this one did not expect + +/ + doc.unknown.line[line[0 .. 1].idup]++; + } + + /+ ↓ finalize last object +/ + if (in_object) { + commitObject(doc, current_section, current_obj); + } + + return doc; +} + +private void parseObjectDeclaration(string line, ref OcdaObject obj) { + /+ parse: [N] type :level identifier +/ + auto bracket_close = line.indexOf(']'); + if (bracket_close < 0) return; + try { + obj.ocn = line[1 .. bracket_close].strip.to!int; + } catch (ConvException) { + obj.ocn = 0; + } + string rest = (bracket_close + 1 < line.length) + ? line[bracket_close + 1 .. $].strip : ""; + if (rest.length == 0) return; + + auto parts = rest.split(" "); + if (parts.length == 0) return; + obj.is_a = parts[0]; + + if (obj.is_a == "heading" && parts.length >= 2) { + /+ parse :A, :B, :1, etc. +/ + string lev = parts[1]; + if (lev.length >= 2 && lev[0] == ':') { + obj.heading_level = lev[1 .. $]; + } + /+ optional identifier after level +/ + if (parts.length >= 3) { + obj.identifier = parts[2]; + } + } +} + +/+ ↓ the blocks that are document preamble rather than sections of objects +/ +private bool isPreambleBlock(string name) { + return name == "source" || name == "meta" || name == "make" || name == "doc_has"; +} + +private void parseProperty(string line, ref OcdaObject obj, ref OcdaUnknown unknown) { + /+ parse: .name: value +/ + auto colon_pos = line.indexOf(':'); + if (colon_pos < 1) return; + string name = line[1 .. colon_pos].strip; + string value = (colon_pos + 1 < line.length) + ? line[colon_pos + 1 .. $].strip : ""; + + switch (name) { + case "part": obj.part = value; break; + case "is_of_type": obj.is_of_type = value; break; + case "section": obj.own_section = value; break; + case "identifier": obj.identifier = value; break; + case "segment_html_is": obj.segment_html_is = value; break; + /+ ↓ the epub segments at heading levels 1 to 4 where the html + segments at level 4 alone, so the two are different files and + the abstraction records both +/ + case "segment_epub": obj.segment_epub = value; break; + case "segment_epub_is": obj.segment_epub_is = value; break; + case "table_aligns": obj.table_aligns = value; break; + case "stow_link": obj.stow_link ~= value; break; + case "anchor_tag": obj.anchor_tags ~= value; break; + case "parent": obj.parent_ocn = toIntSafe(value); break; + case "last_descendant": obj.last_descendant_ocn = toIntSafe(value); break; + case "children": obj.children = parseIntArray(value); break; + case "ancestors": obj.ancestors = parseIntArray(value); break; + case "dummy": obj.dummy_heading = (value == "true"); break; + case "ocn_off": obj.ocn_off = (value == "true"); break; + case "indent": + auto parts = value.split(" "); + if (parts.length >= 1) obj.indent_base = toIntSafe(parts[0]); + if (parts.length >= 2) obj.indent_hang = toIntSafe(parts[1]); + break; + case "bullet": obj.bullet = (value == "true"); break; + case "lang": obj.lang = value; break; + case "has": + foreach (flag; value.split(" ")) { + switch (flag) { + case "links": obj.has_links = true; break; + case "notes_reg": obj.has_notes_reg = true; break; + case "notes_star": obj.has_notes_star = true; break; + case "images": obj.has_images = true; break; + default: break; + } + } + break; + case "segment": obj.segment = value; break; + case "segment_prev": obj.segment_prev = value; break; + case "segment_next": obj.segment_next = value; break; + case "anchor": obj.anchor = value; break; + case "heading_lev_anchor": obj.heading_lev_anchor = value; break; + case "table_cols": obj.table_cols = toIntSafe(value); break; + case "table_widths": obj.table_widths = value; break; + case "table_header": obj.table_header = (value == "true"); break; + case "code_syntax": obj.code_syntax = value; break; + case "code_linenumbers": obj.code_linenumbers = (value == "true"); break; + case "heading_ancestors_text": obj.heading_ancestors_text = value; break; + case "sha256": obj.sha256 = value; break; + case "image": obj.images ~= parseImageRecord(value); break; + /+ ↓ a property this reader has no use for, or has never heard of. + Ignored, which is the conforming behaviour within a major + version, but counted rather than passed over in silence + +/ + default: unknown.property[name]++; break; + } +} + +private OcdaImage parseImageRecord(string value) { + /+ parse: <filename> sha256:<HEX> bytes:<n> [key:value ...] + filename first, remaining fields order-independent; unrecognised + fields are kept rather than discarded +/ + OcdaImage img; + bool first = true; + foreach (field; value.split(" ")) { + if (field.length == 0) continue; + if (first) { + img.name = field; + first = false; + continue; + } + auto colon_pos = field.indexOf(':'); + if (colon_pos < 1) continue; + string key = field[0 .. colon_pos]; + string val = (colon_pos + 1 < field.length) ? field[colon_pos + 1 .. $] : ""; + switch (key) { + case "sha256": img.sha256 = val; break; + case "bytes": img.bytes = toUlongSafe(val); break; + case "px": + auto dims = val.split("x"); + if (dims.length == 2) { + img.width = toIntSafe(dims[0]); + img.height = toIntSafe(dims[1]); + } + break; + case "missing": img.missing = (val == "true"); break; + default: img.fields[key] = val; break; + } + } + return img; +} + +/+ ↓ two values the text form leaves out because they can be worked out. + . + The .ssp emits a property only when it carries information, so + .identifier is absent when it is simply the ocn, and .anchor is absent + when it is the segment name. The .ocda.db, being columns rather than + lines, stores both unconditionally. So a loader that takes the text + form literally gives a *different model* from the one the database + gives, and the difference is not small: in the reference collection + the identifier is omitted on all but a handful of objects. + . + The model is the thing both serialisations describe, so the loader + fills them in. Found by dumping both models and diffing them, which no + amount of comparing section counts would have shown. ++/ +private void deriveOmitted(ref OcdaObject obj) { + import std.conv : to; + if (obj.identifier.length == 0 && obj.ocn != 0) { + obj.identifier = obj.ocn.to!string; + } + if (obj.anchor.length == 0 && obj.segment.length > 0) { + obj.anchor = obj.segment; + } +} + +private void commitObject(ref OcdaDocument doc, string section, ref OcdaObject obj) { + deriveOmitted(obj); + auto sec = doc.sectionByName(section); + if (sec !is null) { + *sec ~= obj; + } +} + +private int toIntSafe(string s) { + try { return s.strip.to!int; } + catch (ConvException) { return 0; } +} + +private ulong toUlongSafe(string s) { + try { return s.strip.to!ulong; } + catch (ConvException) { return 0; } +} + +private int[] parseIntArray(string s) { + int[] result; + foreach (part; s.split(" ")) { + auto v = toIntSafe(part); + result ~= v; + } + return result; +} diff --git a/dlang/src/ocda/model.d b/dlang/src/ocda/model.d new file mode 100644 index 0000000..403502a --- /dev/null +++ b/dlang/src/ocda/model.d @@ -0,0 +1,147 @@ +module ocda.model; + +struct OcdaImage { + string name; // image filename as referenced by the object + string sha256; // hex-encoded (uppercase) digest of the file + ulong bytes; // file size + int width; // file's own pixel width (not display width) + int height; // file's own pixel height + bool missing; // file absent or unreadable, no sha256/bytes + string[string] fields; // any further/unrecognised key:value fields +} + +struct OcdaObject { + int ocn; + string is_a; // heading, para, toc, table, code, group, block, verse, quote, endnote, glossary, bibliography, bookindex, blurb + string section; // head, toc, body, endnotes, glossary, bibliography, bookindex, blurb + string heading_level; // A, B, C, D, 1, 2, 3, 4 (for headings) + string identifier; // heading: from the declaration line; otherwise .identifier + string part; // frontmatter, body, backmatter + string is_of_type; // para, block, comment: how the text behaves + string own_section; // .section, when it differs from the block it sits in + string segment_html_is;// the html segment this object sits in + string segment_epub; // the epub segment this heading opens + string segment_epub_is;// the epub segment this object sits in + string table_aligns; // l c r per column + string[] stow_link; // links extracted from the object + string[] anchor_tags; // named targets the markup placed on this object + int parent_ocn; + int last_descendant_ocn; + int[] children; // direct child heading OCNs + int[] ancestors; // markedup_ancestors[8] + bool dummy_heading; + bool ocn_off; + int indent_base; + int indent_hang; + bool bullet; + string lang; + bool has_links; + bool has_notes_reg; + bool has_notes_star; + bool has_images; + string segment; + string segment_prev; + string segment_next; + string anchor; + string heading_lev_anchor; // a heading's own named target + int table_cols; + string table_widths; + bool table_header; + string code_syntax; + bool code_linenumbers; + string heading_ancestors_text; + string sha256; + OcdaImage[] images; // repeatable .image records, in order + string text; // content (lines joined with newline) + + /+ ↓ whether this object is rendered at all. + is_of_type "comment" is markup commentary: carried by the + abstraction so that nothing is lost, and not put into output. An + output that renders it leaks the author notes into the document, + which is what spine itself does not do. + +/ + bool renders() const { + return is_of_type != "comment"; + } + + /+ ↓ the section this object belongs to, which is the block it sits in + unless it says otherwise +/ + string sectionOf() const { + return (own_section.length > 0) ? own_section : section; + } +} + +/+ ↓ what the reader met and did not understand. + A reader that silently drops what it does not know cannot say whether + it understood the document, and cannot be used to check a + specification against an implementation: silence and understanding + look alike. So the parser counts what it ignored, and + --report-unknown prints it. ++/ +struct OcdaUnknown { + size_t[string] property; // .name -> times seen + size_t[string] block; // @name -> times seen + size_t[string] line; // first char -> times seen, lines of no known shape + + bool any() const { + return property.length > 0 || block.length > 0 || line.length > 0; + } +} + +struct OcdaDocument { + string format_version; // as declared on line 1, "" when absent + string source_filename; // from the "% Source:" line + string[string] source; // @source: language, languages, digest + OcdaUnknown unknown; // what was met and not understood + string[string] metadata; // key -> value + string[string] make; // key -> value + string[string] doc_has; // key -> value + OcdaObject[] head; + OcdaObject[] toc; + OcdaObject[] body; + OcdaObject[] endnotes; + OcdaObject[] glossary; + OcdaObject[] bibliography; + OcdaObject[] bookindex; + OcdaObject[] blurb; + OcdaObject[] tail; + + /+ ↓ the major part of the declared version, "" when none was declared +/ + string formatVersionMajor() const { + import std.string : indexOf; + if (format_version.length == 0) { return ""; } + auto dot = format_version.indexOf('.'); + return (dot > 0) ? format_version[0 .. dot] : format_version; + } + + string meta(string key) { + if (auto p = key in metadata) return *p; + return ""; + } + + OcdaObject[]* sectionByName(string name) { + switch (name) { + case "head": return &head; + case "toc": return &toc; + case "body": return &body; + case "endnotes": return &endnotes; + case "glossary": return &glossary; + case "bibliography": return &bibliography; + case "bookindex": return &bookindex; + case "blurb": return &blurb; + case "tail": return &tail; + default: return null; + } + } +} + +/+ ↓ the format major version this reader knows. + The .ssp and the .ocda.db carry one version between them, declared in + band: the .ssp on its first line, the database in its schema.version + row. The minor part is raised when a property is added, and a reader of + the newer format still reads the older one, so only the major part is + checked here. Raise this when the reader has been brought to a new + major, not before: the point of the gate is to refuse what has not been + looked at. ++/ +enum ssp_format_major_read = "2"; diff --git a/dlang/src/ocda/sqlite.d b/dlang/src/ocda/sqlite.d new file mode 100644 index 0000000..6f3753c --- /dev/null +++ b/dlang/src/ocda/sqlite.d @@ -0,0 +1,122 @@ +module ocda.sqlite; + +/+ ↓ the sqlite3 C API, declared rather than depended upon. + . + A binding from the dub registry would be a dependency to fetch, and + the point of this reader is that it builds in a sandbox with no + network. The dozen functions a reader needs are declared here and the + library is linked directly, which costs a page of extern(C) and saves + a dependency. ++/ + +import std.conv : to; +import std.string : fromStringz, toStringz; + +extern (C) { + struct sqlite3; + struct sqlite3_stmt; + + int sqlite3_open_v2(const(char)* filename, sqlite3** db, int flags, const(char)* vfs); + int sqlite3_close(sqlite3* db); + int sqlite3_prepare_v2(sqlite3* db, const(char)* sql, int nbyte, + sqlite3_stmt** stmt, const(char)** tail); + int sqlite3_step(sqlite3_stmt* stmt); + int sqlite3_finalize(sqlite3_stmt* stmt); + int sqlite3_column_count(sqlite3_stmt* stmt); + const(char)* sqlite3_column_name(sqlite3_stmt* stmt, int col); + const(ubyte)* sqlite3_column_text(sqlite3_stmt* stmt, int col); + int sqlite3_column_int(sqlite3_stmt* stmt, int col); + int sqlite3_column_type(sqlite3_stmt* stmt, int col); + const(char)* sqlite3_errmsg(sqlite3* db); + const(void)* sqlite3_column_blob(sqlite3_stmt* stmt, int col); + int sqlite3_column_bytes(sqlite3_stmt* stmt, int col); +} + +enum SQLITE_OK = 0; +enum SQLITE_ROW = 100; +enum SQLITE_DONE = 101; +enum SQLITE_NULL = 5; +enum SQLITE_OPEN_READONLY = 0x00000001; + +class SqliteError : Exception { + this(string msg) { super(msg); } +} + +/+ ↓ a database, opened read only. + Read only is deliberate: a reader has no business writing, and a + request that opens a database read-write can create an empty one where + a path is mistyped. ++/ +struct Db { + private sqlite3* _handle; + + static Db open(string path) { + sqlite3* handle; + auto rc = sqlite3_open_v2(path.toStringz, &handle, SQLITE_OPEN_READONLY, null); + if (rc != SQLITE_OK) { + string msg = (handle is null) + ? "cannot open " ~ path + : "cannot open " ~ path ~ ": " ~ sqlite3_errmsg(handle).fromStringz.idup; + if (handle !is null) { sqlite3_close(handle); } + throw new SqliteError(msg); + } + Db db; + db._handle = handle; + return db; + } + + void close() { + if (_handle !is null) { sqlite3_close(_handle); _handle = null; } + } + + /+ ↓ every row of a query, each row as its columns in order +/ + string[][] rows(string sql) { + sqlite3_stmt* stmt; + auto rc = sqlite3_prepare_v2(_handle, sql.toStringz, -1, &stmt, null); + if (rc != SQLITE_OK) { + throw new SqliteError(sqlite3_errmsg(_handle).fromStringz.idup ~ " in: " ~ sql); + } + scope(exit) { sqlite3_finalize(stmt); } + string[][] out_rows; + int cols = sqlite3_column_count(stmt); + while (sqlite3_step(stmt) == SQLITE_ROW) { + string[] row; + foreach (c; 0 .. cols) { + row ~= (sqlite3_column_type(stmt, c) == SQLITE_NULL) + ? "" : (cast(const(char)*) sqlite3_column_text(stmt, c)).fromStringz.idup; + } + out_rows ~= row; + } + return out_rows; + } + + /+ ↓ the first column of the first row, "" when there is none +/ + string one(string sql) { + auto r = rows(sql); + return (r.length > 0 && r[0].length > 0) ? r[0][0] : ""; + } + + /+ ↓ the first column of the first row as bytes, for a blob. + Separate from one(), because a blob is not text: an image read as text + stops at the first zero byte, and a png's ninth byte is zero. + +/ + ubyte[] blob(string sql) { + sqlite3_stmt* stmt; + if (sqlite3_prepare_v2(_handle, sql.toStringz, -1, &stmt, null) != SQLITE_OK) { + throw new SqliteError("prepare: " ~ sqlite3_errmsg(_handle).fromStringz.idup + ~ " [" ~ sql ~ "]"); + } + scope(exit) { sqlite3_finalize(stmt); } + if (sqlite3_step(stmt) != SQLITE_ROW) { return []; } + int n = sqlite3_column_bytes(stmt, 0); + if (n <= 0) { return []; } + auto p = cast(const(ubyte)*) sqlite3_column_blob(stmt, 0); + if (p is null) { return []; } + return p[0 .. n].dup; + } + + /+ ↓ does a table or view of this name exist +/ + bool has(string name) { + return one("SELECT count(*) FROM sqlite_master WHERE name = '" ~ name ~ "'") != "0"; + } +} diff --git a/dlang/src/ocda/zip.d b/dlang/src/ocda/zip.d new file mode 100644 index 0000000..9dff6f4 --- /dev/null +++ b/dlang/src/ocda/zip.d @@ -0,0 +1,115 @@ +module ocda.zip; + +/+ ↓ enough of the zip format to find one file in a pod and read it. + . + Not a zip library. A pod archive holds a handful of entries, spine + writes them **stored** (uncompressed) so that the outer zstd frame does + the compressing once over the whole thing, and what is wanted from it + here is one named entry. So: read the central directory, find the + entry, return its bytes. + . + Deflate is *refused rather than mishandled*. Phobos can inflate, but a + library that says it read an entry it in fact skipped is worse than one + that says it cannot, and spine's own archives do not need it. ++/ + +import std.algorithm : endsWith; +import std.conv : to; + +class OcdaZipError : Exception { + this(string msg) { super(msg); } +} + +struct ZipEntry { + string name; + uint method; // 0 stored, 8 deflate + uint compressed; + uint uncompressed; + uint local_offset; +} + +/+ ↓ the entries, from the central directory at the end of the file +/ +ZipEntry[] zipEntries(const(ubyte)[] bytes) { + enum uint eocd_sig = 0x06054b50; + enum uint cd_sig = 0x02014b50; + if (bytes.length < 22) { + throw new OcdaZipError("too short to be a zip archive"); + } + /+ ↓ the end of central directory record is last, and may be followed by a + comment of up to 65535 bytes, so it is searched for backwards +/ + ptrdiff_t eocd = -1; + ptrdiff_t lowest = bytes.length > (22 + 65535) ? bytes.length - (22 + 65535) : 0; + for (ptrdiff_t i = bytes.length - 22; i >= lowest; i--) { + if (u32(bytes, i) == eocd_sig) { eocd = i; break; } + } + if (eocd < 0) { + throw new OcdaZipError("no end of central directory record;" + ~ " it does not look like a zip archive"); + } + uint count = u16(bytes, eocd + 10); + uint offset = u32(bytes, eocd + 16); + ZipEntry[] out_; + size_t p = offset; + foreach (_; 0 .. count) { + if (p + 46 > bytes.length || u32(bytes, p) != cd_sig) { + throw new OcdaZipError("central directory entry " + ~ out_.length.to!string ~ " is malformed"); + } + ZipEntry e; + e.method = u16(bytes, p + 10); + e.compressed = u32(bytes, p + 20); + e.uncompressed = u32(bytes, p + 24); + uint name_len = u16(bytes, p + 28); + uint extra_len = u16(bytes, p + 30); + uint cmnt_len = u16(bytes, p + 32); + e.local_offset = u32(bytes, p + 42); + e.name = cast(string) bytes[p + 46 .. p + 46 + name_len].idup; + out_ ~= e; + p += 46 + name_len + extra_len + cmnt_len; + } + return out_; +} + +/+ ↓ one entry's bytes, from its local header +/ +ubyte[] zipRead(const(ubyte)[] bytes, ZipEntry e) { + enum uint local_sig = 0x04034b50; + size_t p = e.local_offset; + if (p + 30 > bytes.length || u32(bytes, p) != local_sig) { + throw new OcdaZipError(e.name ~ ": local header is malformed"); + } + uint name_len = u16(bytes, p + 26); + uint extra_len = u16(bytes, p + 28); + size_t start = p + 30 + name_len + extra_len; + size_t end = start + e.compressed; + if (end > bytes.length) { + throw new OcdaZipError(e.name ~ ": runs past the end of the archive"); + } + if (e.method != 0) { + throw new OcdaZipError(e.name ~ ": compression method " + ~ e.method.to!string ~ "; this reader takes only stored entries," + ~ " which is what spine writes"); + } + return bytes[start .. end].dup; +} + +/+ ↓ the first entry whose name ends this way, or none. + Used to find media/abstraction/<uid>.<lang>.ssp without knowing the uid. ++/ +ZipEntry[] zipFindSuffix(ZipEntry[] entries, string suffix) { + ZipEntry[] out_; + foreach (e; entries) { + if (e.name.endsWith(suffix)) { out_ ~= e; } + } + return out_; +} + +private uint u16(const(ubyte)[] b, size_t i) { + return cast(uint) b[i] | (cast(uint) b[i + 1] << 8); +} + +private uint u32(const(ubyte)[] b, size_t i) { + return cast(uint) b[i] + | (cast(uint) b[i + 1] << 8) + | (cast(uint) b[i + 2] << 16) + | (cast(uint) b[i + 3] << 24); +} diff --git a/dlang/src/ocda/zstd.d b/dlang/src/ocda/zstd.d new file mode 100644 index 0000000..401f1a3 --- /dev/null +++ b/dlang/src/ocda/zstd.d @@ -0,0 +1,79 @@ +module ocda.zstd; + +/+ ↓ libzstd, declared rather than bound. + . + A .sisupod is one zstd frame wrapping a zip archive, so reading one + needs exactly two things from the library: is this a frame, and what is + in it. Both are declared here and the C library is linked, which is the + same arrangement ocda.sqlite uses: no vendored tree, no generated + header, and the prototypes say what is being called. ++/ + +extern (C) @nogc nothrow private { + size_t ZSTD_decompress(void* dst, size_t dstCapacity, + const(void)* src, size_t compressedSize); + ulong ZSTD_getFrameContentSize(const(void)* src, size_t srcSize); + uint ZSTD_isError(size_t code); + const(char)* ZSTD_getErrorName(size_t code); +} + +/+ ↓ the two values ZSTD_getFrameContentSize returns in place of a size +/ +private enum ulong contentSizeUnknown = ulong.max; +private enum ulong contentSizeError = ulong.max - 1; + +/+ ↓ the magic number of a zstd frame, little endian 0xFD2FB528 +/ +private static immutable ubyte[4] zstd_magic = [0x28, 0xB5, 0x2F, 0xFD]; + +class OcdaZstdError : Exception { + this(string msg) { super(msg); } +} + +/+ ↓ does this begin with a zstd frame? + Asked of the bytes rather than of the filename, because what is inside + decides how a thing is opened. A .sisupod is a frame; a .zip published + before spine wrapped them is not, and both are read. ++/ +bool zstdIsFrame(const(ubyte)[] bytes) { + if (bytes.length < zstd_magic.length) { return false; } + foreach (i, b; zstd_magic) { + if (bytes[i] != b) { return false; } + } + return true; +} + +/+ ↓ one frame in, its contents out. + max_size is a refusal rather than a buffer size: a frame declares how + much it expands to, and a frame that claims more than the caller is + willing to hold is not decompressed at all. A reader of an artefact it + did not make wants that limit. ++/ +ubyte[] zstdDecompress(const(ubyte)[] bytes, size_t max_size) { + if (!zstdIsFrame(bytes)) { + throw new OcdaZstdError("not a zstd frame"); + } + ulong declared = ZSTD_getFrameContentSize(bytes.ptr, bytes.length); + if (declared == contentSizeError) { + throw new OcdaZstdError("zstd frame: could not read its content size"); + } + if (declared == contentSizeUnknown) { + throw new OcdaZstdError("zstd frame declares no content size;" + ~ " this reader takes only frames that do"); + } + if (declared > max_size) { + import std.conv : to; + throw new OcdaZstdError("zstd frame expands to " ~ declared.to!string + ~ " bytes, over the " ~ max_size.to!string ~ " byte limit"); + } + auto out_ = new ubyte[](cast(size_t) declared); + size_t got = ZSTD_decompress(out_.ptr, out_.length, bytes.ptr, bytes.length); + if (ZSTD_isError(got)) { + import std.string : fromStringz; + throw new OcdaZstdError("zstd: " + ~ ZSTD_getErrorName(got).fromStringz.idup); + } + if (got != declared) { + throw new OcdaZstdError("zstd frame expanded to a different size" + ~ " than it declared"); + } + return out_; +} diff --git a/dlang/test/agree.d b/dlang/test/agree.d new file mode 100644 index 0000000..bdf1b2e --- /dev/null +++ b/dlang/test/agree.d @@ -0,0 +1,84 @@ +/++ + Do the two loaders agree? + + The library's whole purpose is that a writer cannot tell which + serialisation it was handed. That is only true if loading a document's + .ssp and loading its .ocda.db give the same model, so this asks. + + Usage: agree <doc>.ssp <doc>.ocda.db [lang] ++/ +import std.conv : to; +import std.stdio; +import ocda.model; +import ocda.load; + +int main(string[] args) { + if (args.length < 3) { + stderr.writeln("usage: agree <doc>.ssp <doc>.ocda.db [lang]"); + return 2; + } + string lang = (args.length > 3) ? args[3] : ""; + OcdaDocument a, b; + try { + a = load(args[1]); + b = load(args[2], lang); + } catch (Exception e) { + stderr.writeln("error: ", e.msg); + return 2; + } + + int differences = 0; + void differ(string what, string from_ssp, string from_db) { + writeln(" differ: ", what); + writeln(" .ssp: ", from_ssp); + writeln(" .ocda.db: ", from_db); + differences++; + } + + if (a.format_version != b.format_version) { + differ("format_version", a.format_version, b.format_version); + } + if (a.source_filename != b.source_filename) { + differ("source_filename", a.source_filename, b.source_filename); + } + + /+ ↓ the sections, object by object. Position within a section is the + identity to compare on: an ocn is not unique and 0 is shared +/ + static immutable sections = ["head", "toc", "body", "endnotes", "glossary", + "bibliography", "bookindex", "blurb", "tail"]; + foreach (name; sections) { + auto sa = a.sectionByName(name); + auto sb = b.sectionByName(name); + if (sa is null || sb is null) { continue; } + if ((*sa).length != (*sb).length) { + differ(name ~ " object count", (*sa).length.to!string, (*sb).length.to!string); + continue; + } + foreach (i, oa; *sa) { + auto ob = (*sb)[i]; + string at = name ~ "[" ~ i.to!string ~ "]"; + if (oa.ocn != ob.ocn) { differ(at ~ " ocn", oa.ocn.to!string, ob.ocn.to!string); } + if (oa.is_a != ob.is_a) { differ(at ~ " is_a", oa.is_a, ob.is_a); } + if (oa.is_of_type != ob.is_of_type) { differ(at ~ " is_of_type", oa.is_of_type, ob.is_of_type); } + if (oa.part != ob.part) { differ(at ~ " part", oa.part, ob.part); } + if (oa.parent_ocn != ob.parent_ocn) { differ(at ~ " parent", oa.parent_ocn.to!string, ob.parent_ocn.to!string); } + if (oa.heading_level != ob.heading_level) { differ(at ~ " level", oa.heading_level, ob.heading_level); } + if (oa.sha256 != ob.sha256) { differ(at ~ " sha256", oa.sha256, ob.sha256); } + if (oa.text != ob.text) { differ(at ~ " text", oa.text, ob.text); } + if (oa.ancestors != ob.ancestors) { + differ(at ~ " ancestors", oa.ancestors.to!string, ob.ancestors.to!string); + } + if (oa.images.length != ob.images.length) { + differ(at ~ " image count", oa.images.length.to!string, ob.images.length.to!string); + } + if (differences > 12) { writeln(" ... stopping after 12"); return 1; } + } + } + + if (differences == 0) { + writeln("agree: the two loaders give the same model"); + return 0; + } + writeln(differences, " difference(s)"); + return 1; +} diff --git a/dlang/test/dump.d b/dlang/test/dump.d new file mode 100644 index 0000000..17aa9bb --- /dev/null +++ b/dlang/test/dump.d @@ -0,0 +1,35 @@ +/++ + Write a loaded abstraction out canonically. + + The oracle for everything that claims to implement this model: load a + .ssp or a .ocda.db and print the model, deterministically, so that two + implementations can be compared with diff. + + Usage: ocda-dump [--unknown] <file>.ssp|<file>.ocda.db [lang] ++/ +import std.stdio; +import ocda.model; +import ocda.load; +import ocda.dump; + +int main(string[] args) { + bool unknown = false; + string[] rest; + foreach (a; args[1 .. $]) { + if (a == "--unknown") { unknown = true; } else { rest ~= a; } + } + if (rest.length < 1) { + stderr.writeln("usage: ocda-dump [--unknown] <file>.ssp|<file>.ocda.db [lang]"); + return 2; + } + string lang = (rest.length > 1) ? rest[1] : ""; + OcdaDocument doc; + try { + doc = load(rest[0], lang); + } catch (Exception e) { + stderr.writeln("error: ", e.msg); + return 2; + } + write(unknown ? dumpUnknown(doc) : dumpDocument(doc)); + return 0; +} diff --git a/gleam/COPYRIGHT b/gleam/COPYRIGHT new file mode 100644 index 0000000..adc5446 --- /dev/null +++ b/gleam/COPYRIGHT @@ -0,0 +1,48 @@ +- Name: sisudoc-ocda-lib + - Description: part of SiSU spine (Doc Reform): documents, structuring, + processing, publishing and search + - [https://www.sisudoc.org] + + - Author: Ralph Amissah + [ralph.amissah@gmail.com] + + - Copyright: (C) 2026 Ralph Amissah, All Rights Reserved. + + - License: AGPL 3 or later, for everything in this repository unless a file or + a directory specifies otherwise.: + + sisudoc-ocda-lib + + part of SiSU spine (Doc Reform): documents, structuring, + processing, publishing and search + + Copyright (C) Ralph Amissah + + This program is free software: you can redistribute it and/or modify it + under the terms of the GNU Affero General Public License (AGPLv3) as + published by the Free Software Foundation, either version 3 of the License, + or (at your option) any later version. + + This program is distributed in the hope that it will be useful, but + WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY + or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public + License for more details. + + You should have received a copy of the GNU Affero General Public License + along with this program. If not, see [https://www.gnu.org/licenses/]. + + If you have Internet connection, the latest version of the AGPL should be + available at: [https://www.gnu.org/licenses/agpl-3.0.html] + + - Spine, Doc Reform (related to SiSU) uses standard: + - docReform markup syntax (based on SiSU markup) + - standard SiSU markup syntax with modified headers and minor + modifications + - docReform object numbering (based on SiSU object citation numbering) + - standard SiSU document object numbering + + - Homepages: + [https://www.sisudoc.org] + + - Copyright for ./nix-gleam/ 2023 Arnar Gauti Ingason + - License for ./nix-gleam/ MIT License diff --git a/gleam/README.md b/gleam/README.md new file mode 100644 index 0000000..d586bd3 --- /dev/null +++ b/gleam/README.md @@ -0,0 +1,102 @@ +# ocda-lib, in Gleam + +The abstraction's model and both its loaders, as a Gleam library, and the +two programs that put it to the question. + +| | | +|-----------------|----------------------------------------------------------------------------------| +| `ocda/model` | the document, the object, the image. Depends on nothing but the standard library | +| `ocda/load_ssp` | text -> model | +| `ocda/load_db` | sqlite -> model | +| `ocda/load` | what a path is, and loading it | +| `ocda/dump` | the model written out canonically, as an oracle | + +## Use + +```gleam +import ocda/load +import ocda/model + +case load.load(path, "en", False) { + Error(why) -> io.println_error(why) + Ok(doc) -> { + // and from here a writer neither knows nor cares which of the two + // files it was handed + list.each(model.section(doc, "body"), fn(obj) { ... }) + } +} +``` + +## The two programs + +```sh +nix build +./result/bin/ocda-lib agree <doc>.ssp <doc>.ocda.db [lang] +./result/bin/ocda-lib dump [--unknown] <file>.ssp|<file>.ocda.db [lang] +``` + +`agree` asks whether the two loaders give the same model. `dump` writes +the whole model out canonically, and is the stronger check of the two, +because it is **byte for byte what the D library's `ocda-dump` writes**. +So the four questions can all be asked with `diff`: + +| | | +|-----------------------------|-----------------------------| +| D: `.ssp` == `.ocda.db` | 36 of 36 document-languages | +| Gleam: `.ssp` == `.ocda.db` | 36 of 36 | +| `.ssp`: D == Gleam | 36 of 36 | +| `.ocda.db`: D == Gleam | 36 of 36 | + +## Deliberately a mirror of the D library + +Field for field, name for name, including where that is not the most +Gleamish choice: + +- `text` is one `String` with the lines joined by newline, not a + `List(String)`. So an object whose only text line is blank is + indistinguishable from an object with no text, *in both languages, in the + same place*. A difference that existed in one and not the other would be + worse than a loss both share, because it would show up in every dump and + mean nothing. +- the properties are typed fields rather than a list of pairs. A writer + wants `obj.parent_ocn`, not a lookup that might miss. + +The point is that the two agree. Making them agree is much cheaper when +they are shaped alike, and the shape is the specification's, not either +language's. + +## What the agreement found + +Four disagreements, none of which a comparison of section counts or of a +handful of fields would have shown. All four are recorded in the D +library's history; the short version: + +1. `.identifier` and `.anchor` are omitted by the text form when they can + be worked out (the identifier is the ocn, the anchor is the segment). + Both loaders now derive them. Over one document that is 1,314 + identifiers. +2. `make` and `doc_has` are one metadata table in the database, + distinguished by a key prefix. Both loaders unprefix them. +3. The JSON columns must be *parsed*, not scanned: the encoder writes `/` + as `\/` and every heading with a slash in it came back wrong. +4. `translation.ocn_aligned` is in the database and cannot be in a `.ssp`, + which is one language. It is a fact about the file, so neither loader + puts it in the document. + +## Build + +```sh +nix build # sandboxed, no network +nix develop -c gleam test +``` + +Shipped as an Erlang shipment rather than an escript, because sqlite on the +BEAM is a NIF and a NIF cannot be loaded from inside an escript archive. +The dependency hash in `flake.nix` pins what `manifest.toml` pins; change +one and the other must change with it. + +## Format + +Specified in **sisudoc-abstraction-format**: `ocda-model.md` for the model, +`ssp-format-specification.md` and `ocda-db-format-specification.md` for the +two serialisations. At format **2.x**. diff --git a/gleam/flake.lock b/gleam/flake.lock new file mode 100644 index 0000000..5f236ab --- /dev/null +++ b/gleam/flake.lock @@ -0,0 +1,27 @@ +{ + "nodes": { + "nixpkgs": { + "locked": { + "lastModified": 1790185690, + "narHash": "sha256-xJ+X4hBtOcAFGBOe5nAMyMUeF9foJBmIOu3NjBqBycU=", + "owner": "NixOS", + "repo": "nixpkgs", + "rev": "4975466d324710c576dc11ad614684e6bd8cad8e", + "type": "github" + }, + "original": { + "owner": "NixOS", + "ref": "nixos-unstable", + "repo": "nixpkgs", + "type": "github" + } + }, + "root": { + "inputs": { + "nixpkgs": "nixpkgs" + } + } + }, + "root": "root", + "version": 7 +} diff --git a/gleam/flake.nix b/gleam/flake.nix new file mode 100644 index 0000000..a5a3407 --- /dev/null +++ b/gleam/flake.nix @@ -0,0 +1,66 @@ +{ + description = "ocda-lib: the object-centric document abstraction as a Gleam library"; + inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable"; + outputs = { self, nixpkgs }: + let + system = "x86_64-linux"; # adjust for aarch64-linux / darwin + pkgs = import nixpkgs { inherit system; }; + + name = "ocda_lib"; + + /* The builder, nix-gleam's, kept here in nix-gleam/ (see the README + there). It fetches each Hex dependency on its own, with the + checksum gleam has already written into manifest.toml, so there is + no hash in this file to keep up to date, and nothing a run of gleam + could make come out differently. + */ + inherit (pkgs.callPackage ./nix-gleam { }) buildGleamApplication; + + /* rebar3 with the port compiler already in it. + esqlite builds a NIF, and its rebar3 configuration asks for the + `pc` plugin, which rebar3 would otherwise fetch from Hex at build + time: impossible in a sandbox, and the build fails with + "missing_package pc". nixpkgs carries the plugin, so the answer is + a rebar3 that has it before it starts. + */ + rebar3-pc = pkgs.beamPackages.rebar3WithPlugins { + plugins = [ pkgs.beamPackages.pc ]; + }; + in { + packages.${system} = { + default = self.packages.${system}.${name}; + /* Shipped as an Erlang shipment rather than as an escript, because + sqlite on the BEAM is a NIF: esqlite compiles its own sqlite into + a shared object, and a NIF cannot be loaded from inside an + escript archive, the runtime needing a real file on disk. The + shipment keeps priv/ on disk, so it loads. + */ + ${name} = buildGleamApplication { + pname = "ocda-lib"; + version = "2.0.0"; + src = ./.; + rebar3Package = rebar3-pc; + nativeBuildInputs = [ pkgs.makeWrapper ]; + preBuild = '' + export HOME=$TMPDIR + ''; + installPhase = '' + runHook preInstall + mkdir -p $out/lib $out/bin + cp -r build/erlang-shipment $out/lib/${name} + makeWrapper $out/lib/${name}/entrypoint.sh $out/bin/ocda-lib \ + --add-flags run \ + --prefix PATH : ${pkgs.lib.makeBinPath [ pkgs.beamPackages.erlang pkgs.coreutils ]} + runHook postInstall + ''; + meta = { + description = "the abstraction as a Gleam library"; + mainProgram = "ocda-lib"; + }; + }; + }; + devShells.${system}.default = pkgs.mkShell { + packages = [ pkgs.gleam pkgs.beamPackages.erlang pkgs.rebar3 pkgs.sqlite ]; + }; + }; +} diff --git a/gleam/gleam.toml b/gleam/gleam.toml new file mode 100644 index 0000000..7b474eb --- /dev/null +++ b/gleam/gleam.toml @@ -0,0 +1,14 @@ +name = "ocda_lib" +version = "2.0.0" +description = "the object-centric document abstraction as a library: the model, and loaders for its two serialisations" +licences = ["AGPL-3.0-or-later"] + +[dependencies] +gleam_stdlib = ">= 1.0.0 and < 2.0.0" +argv = ">= 1.1.0 and < 2.0.0" +simplifile = ">= 2.7.0 and < 3.0.0" +sqlight = ">= 1.0.0 and < 2.0.0" +gleam_json = ">= 3.0.0 and < 4.0.0" + +[dev-dependencies] +gleeunit = ">= 1.0.0 and < 2.0.0" diff --git a/gleam/manifest.toml b/gleam/manifest.toml new file mode 100644 index 0000000..2b0cac1 --- /dev/null +++ b/gleam/manifest.toml @@ -0,0 +1,26 @@ +# Do not manually edit this file, it is managed by Gleam. +# +# This file locks the dependency versions used, to make your build +# deterministic and to prevent unexpected versions from being included +# in your application. +# +# You should check this file into your source control repository. + +packages = [ + { name = "argv", version = "1.1.0", build_tools = ["gleam"], requirements = [], otp_app = "argv", source = "hex", outer_checksum = "3277D100448BDB4A29B6D58C0F36F631CBC349E8BDD09766C6309DF202831140" }, + { name = "esqlite", version = "0.9.0", build_tools = ["rebar3"], requirements = [], otp_app = "esqlite", source = "hex", outer_checksum = "CCF72258A4EE152EC7AD92AA9A03552EB6CA1B06B65C93AD5B6E55C302E05855" }, + { name = "filepath", version = "1.1.2", build_tools = ["gleam"], requirements = ["gleam_stdlib"], otp_app = "filepath", source = "hex", outer_checksum = "B06A9AF0BF10E51401D64B98E4B627F1D2E48C154967DA7AF4D0914780A6D40A" }, + { name = "gleam_json", version = "3.1.0", build_tools = ["gleam"], requirements = ["gleam_stdlib"], otp_app = "gleam_json", source = "hex", outer_checksum = "44FDAA8847BE8FC48CA7A1C089706BD54BADCC4C45B237A992EDDF9F2CDB2836" }, + { name = "gleam_stdlib", version = "1.0.5", build_tools = ["gleam"], requirements = [], otp_app = "gleam_stdlib", source = "hex", outer_checksum = "CEE5B6C076A85B45F60C585F4316C63EC8B7127C119D5738C3958A9C4D50404E" }, + { name = "gleeunit", version = "1.11.0", build_tools = ["gleam"], requirements = ["gleam_stdlib"], otp_app = "gleeunit", source = "hex", outer_checksum = "EC31ABA74256AEA531EDF8169931D775BBB384FED0A8A1BDC4DD9354E3E21826" }, + { name = "simplifile", version = "2.7.0", build_tools = ["gleam"], requirements = ["filepath", "gleam_stdlib"], otp_app = "simplifile", source = "hex", outer_checksum = "A2727627B063E87351934C7F7F008F2D1FDB16F6DE0B8C79F9E46459CFC9C164" }, + { name = "sqlight", version = "1.2.0", build_tools = ["gleam"], requirements = ["esqlite", "gleam_stdlib"], otp_app = "sqlight", source = "hex", outer_checksum = "841768D0A81107EE2DB46B949354F284A68B2E8098315C28A1FE3C07D206C17A" }, +] + +[requirements] +argv = { version = ">= 1.1.0 and < 2.0.0" } +gleam_json = { version = ">= 3.0.0 and < 4.0.0" } +gleam_stdlib = { version = ">= 1.0.0 and < 2.0.0" } +gleeunit = { version = ">= 1.0.0 and < 2.0.0" } +simplifile = { version = ">= 2.7.0 and < 3.0.0" } +sqlight = { version = ">= 1.0.0 and < 2.0.0" } diff --git a/gleam/nix-gleam/LICENSE b/gleam/nix-gleam/LICENSE new file mode 100644 index 0000000..c35ad09 --- /dev/null +++ b/gleam/nix-gleam/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2023 Arnar Gauti Ingason + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/gleam/nix-gleam/README.md b/gleam/nix-gleam/README.md new file mode 100644 index 0000000..39bd1b6 --- /dev/null +++ b/gleam/nix-gleam/README.md @@ -0,0 +1,32 @@ +# nix-gleam, kept here + +`default.nix` is the builder of **nix-gleam**, by Arnar Gauti Ingason, +copied unchanged from + +- https://github.com/arnarg/nix-gleam +- `builder/default.nix` at commit `1f38154` (2026-10-06) + +under the MIT licence, which is in `LICENSE` beside it. + +It is kept here rather than taken as a flake input so that the build does +not depend on a third repository being reachable, and so that what builds +this package is in the package. + +## Why it is used + +There is no Gleam builder in nixpkgs, and a sandboxed build cannot fetch +from Hex. The usual answer, one fixed output derivation that runs +`gleam deps download` and is pinned by a hash of everything it fetched, has +two faults: the hash has to be updated by hand whenever a dependency +changes, and gleam writes `packages.toml` in a different order on each run, +so the same dependencies do not always give the same hash. + +nix-gleam fetches each Hex package on its own with `fetchHex`, using the +checksum gleam has already written for it into `manifest.toml`, and writes +`packages.toml` itself. So there is no hash to maintain, and nothing a run +of gleam could make come out differently. + +## Updating it + +Copy `builder/default.nix` from a newer commit over `default.nix`, update +the commit above, and build. diff --git a/gleam/nix-gleam/default.nix b/gleam/nix-gleam/default.nix new file mode 100644 index 0000000..d074263 --- /dev/null +++ b/gleam/nix-gleam/default.nix @@ -0,0 +1,280 @@ +{ + stdenv, + lib, + git, + xxhash, + fetchHex, + gleam, + beamPackages, + rsync, + nodejs, +}: let + inherit (builtins) fromTOML readFile; +in { + buildGleamApplication = { + src, + nativeBuildInputs ? [], + localPackages ? [], + erlangPackage ? beamPackages.erlang, + rebar3Package ? beamPackages.rebar3, + ... + } @ attrs: let + # gleam.toml contains an application name and version. + gleamToml = lib.importTOML (src + "/gleam.toml"); + + # manifest.toml contains a list of required packages including a sha256 checksum + # that can be used by nix fetchHex fetcher. + manifestToml = lib.importTOML (src + "/manifest.toml"); + + # Specify which target to build for. + buildTarget = attrs.target or gleamToml.target or "erlang"; + + # Generates a packages.toml expected by gleam compiler. + packagesTOML = with lib; + concatStringsSep "\n" ( + ["[packages]"] + ++ (map + (p: "${p.name} = \"${p.version}\"") + manifestToml.packages) + ++ (lib.optional (gitRecordsTOML != "") "") + ++ (lib.optional (gitRecordsTOML != "") gitRecordsTOML) + ); + + # Gleam records a git dependency twice: once in `[packages]`, + # and once in a `[git.<name>]` table with the commit. + # Without the second, Gleam will try to download it. + gitRecordsTOML = with lib; + concatMapStringsSep "\n" ( + p: "[git.${p.name}]\ncommit = \"${p.commit}\"" + ) (filterPackagesBySource "git" manifestToml.packages); + + # Helper function to filter manifest.toml packages + filterPackagesBySource = type: packages: lib.lists.filter (p: p.source == type) packages; + + gitDerivs = + map + (p: { + name = p.name; + derivation = fetchGit { + url = p.repo; + rev = p.commit; + }; + }) + (filterPackagesBySource "git" manifestToml.packages); + + # Fetch all dependencies + depsDerivs = + map + (p: { + name = p.name; + derivation = fetchHex { + inherit (p) version; + pkg = p.name; + sha256 = p.outer_checksum; + }; + }) + (filterPackagesBySource "hex" manifestToml.packages); + + # Find replacement paths for `local` package dependencies + # from `localPackages` list. + localDeps = let + # Build a lookup attrset for local packages. + localDerivs = lib.mergeAttrsList (map ( + p: let + name = (fromTOML (readFile (p + "/gleam.toml"))).name; + in { + "${name}" = p; + } + ) + localPackages); + in + map ( + p: { + inherit (p) name path; + localSrc = + if localDerivs ? "${p.name}" + then localDerivs.${p.name} + else throw "Local dependency \"${p.name}\" not found in `localPackages`."; + # Keep local packages in a writable location during build. + newPath = p.path; + } + ) (filterPackagesBySource "local" manifestToml.packages); + + # Check if elixir is needed in nativeBuildInputs by checking if "mix" is in + # required build_tools. + isElixirProject = with lib; p: any (t: t == "mix") p.build_tools; + needsElixir = with lib; any isElixirProject manifestToml.packages; + + # nativeBuildInputs needed for both targets. + defaultNativeBuildInputs = [gleam beamPackages.hex rsync git xxhash]; + in + # Base common mkDerivation attributes + stdenv.mkDerivation (attrs + // { + pname = attrs.pname or gleamToml.name; + version = attrs.version or gleamToml.version; + + src = lib.cleanSource attrs.src; + + # Here we must copy the dependencies into the right spot and + # create a packages.toml file so the gleam compiler does not + # attempt to pull the dependencies from the internet. + configurePhase = + attrs.configurePhase + or '' + runHook preConfigure + + mkdir -p build/packages + + # Write the packages.toml file + cat <<EOF > build/packages/packages.toml + ${packagesTOML} + EOF + + ${ + # Gleam writes build output into the local dependency's own source + # directory, so it cannot be the read-only store path. + lib.concatStringsSep "\n" ( + lib.forEach localDeps ( + d: '' + mkdir -p "${d.newPath}" + rsync --chmod=Du=rwx,Dg=rx,Do=rx,Fu=rw,Fg=r,Fo=r -r ${d.localSrc}/* "${d.newPath}/" + '' + ) + ) + } + + ${ + lib.concatStringsSep "\n" ( + lib.forEach gitDerivs ( + d: '' + rsync --chmod=Du=rwx,Dg=rx,Do=rx,Fu=rw,Fg=r,Fo=r -r ${d.derivation}/* build/packages/${d.name}/ + '' + ) + ) + } + + # Copy all the dependencies into place + ${lib.concatStringsSep "\n" ( + lib.forEach depsDerivs ( + # gleam outputs files inside the dependency's source directory + # and therefor it needs to have permissive permissions. + d: '' + rsync --chmod=Du=rwx,Dg=rx,Do=rx,Fu=rw,Fg=r,Fo=r -r ${d.derivation}/* build/packages/${d.name}/ + '' + ) + )} + + # To prevent dependency resolution in Gleam 1.15+, local packages + # need to have their fingerprint up-to-date. + ${lib.concatStringsSep "\n" ( + lib.forEach localDeps (d: '' + printf "%u" 0x$(xxhsum -H3 ${d.newPath}/gleam.toml | cut -d' ' -f1 | cut -d '_' -f2) > build/packages/${d.name}.config_fingerprint + '') + )} + + ${ + # Prime local package dependency caches so they do not try to fetch. + # + # Gleam compiles a local dependency inside that dependency's own + # directory, using the package cache found there. + lib.concatStringsSep "\n" ( + lib.forEach localDeps ( + d: '' + mkdir -p "${d.newPath}/build/packages" + cp build/packages/packages.toml "${d.newPath}/build/packages/packages.toml" + rsync --chmod=Du=rwx,Dg=rx,Do=rx,Fu=rw,Fg=r,Fo=r -r build/packages/* "${d.newPath}/build/packages/" + '' + ) + ) + } + + runHook postConfigure + ''; + } + # When the build target is erlang + // lib.optionalAttrs (buildTarget == "erlang") { + nativeBuildInputs = + defaultNativeBuildInputs + ++ [erlangPackage rebar3Package] + ++ (lib.optional needsElixir beamPackages.elixir) + ++ nativeBuildInputs; + + # The gleam compiler has a nice export function for erlang shipment. + buildPhase = + attrs.buildPhase + or '' + runHook preBuild + + export REBAR_CACHE_DIR="$TMP/.rebar-cache" + + gleam export erlang-shipment + + runHook postBuild + ''; + + # Install all built packages into lib and create an entrypoint script + # that starts the application. + installPhase = + attrs.installPhase + or '' + runHook preInstall + + mkdir -p $out/{bin,lib} + + rsync --exclude=entrypoint.sh -r build/erlang-shipment/* $out/lib/ + + cat <<EOF > $out/bin/${gleamToml.name} + #!/usr/bin/env sh + ${erlangPackage}/bin/erl \ + -pa $out/lib/*/ebin \ + -eval "${gleamToml.name}@@main:run(${gleamToml.name})" \ + -noshell \ + -extra "\$@" + EOF + chmod +x $out/bin/${gleamToml.name} + + runHook postInstall + ''; + } + # When the build target is javascript + // lib.optionalAttrs (buildTarget == "javascript") { + nativeBuildInputs = defaultNativeBuildInputs ++ nativeBuildInputs; + + # The gleam compiler doesn't provide an export mechanism for javascript target. + buildPhase = + attrs.buildPhase + or '' + runHook prebuild + + gleam build --target javascript + + runHook postBuild + ''; + + # Install all built packages into lib and create an entrypoint script + # that starts the application. + installPhase = + attrs.installPhase + or '' + runHook preInstall + + mkdir -p $out/{bin,lib} + + rsync --exclude=gleam.lock --exclude=gleam_version -r build/dev/javascript/* $out/lib/ + + cat <<EOF > $out/lib/${gleamToml.name}/main.mjs + import { main } from "./${gleamToml.name}.mjs"; + main(); + EOF + + cat <<EOF > $out/bin/${gleamToml.name} + #!/usr/bin/env sh + ${nodejs}/bin/node $out/lib/${gleamToml.name}/main.mjs "\$@" + EOF + chmod +x $out/bin/${gleamToml.name} + + runHook postInstall + ''; + }); +} diff --git a/gleam/src/COPYRIGHT b/gleam/src/COPYRIGHT new file mode 100644 index 0000000..7e7dd56 --- /dev/null +++ b/gleam/src/COPYRIGHT @@ -0,0 +1,45 @@ +- Name: sisudoc-ocda-lib + - Description: part of SiSU spine (Doc Reform): documents, structuring, + processing, publishing and search + - [https://www.sisudoc.org] + + - Author: Ralph Amissah + [ralph.amissah@gmail.com] + + - Copyright: (C) 2026 Ralph Amissah, All Rights Reserved. + + - License: AGPL 3 or later, for everything in this repository unless a file or + a directory specifies otherwise.: + + sisudoc-ocda-lib + + part of SiSU spine (Doc Reform): documents, structuring, + processing, publishing and search + + Copyright (C) Ralph Amissah + + This program is free software: you can redistribute it and/or modify it + under the terms of the GNU Affero General Public License (AGPLv3) as + published by the Free Software Foundation, either version 3 of the License, + or (at your option) any later version. + + This program is distributed in the hope that it will be useful, but + WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY + or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public + License for more details. + + You should have received a copy of the GNU Affero General Public License + along with this program. If not, see [https://www.gnu.org/licenses/]. + + If you have Internet connection, the latest version of the AGPL should be + available at: [https://www.gnu.org/licenses/agpl-3.0.html] + + - Spine, Doc Reform (related to SiSU) uses standard: + - docReform markup syntax (based on SiSU markup) + - standard SiSU markup syntax with modified headers and minor + modifications + - docReform object numbering (based on SiSU object citation numbering) + - standard SiSU document object numbering + + - Homepages: + [https://www.sisudoc.org] diff --git a/gleam/src/ocda/dump.gleam b/gleam/src/ocda/dump.gleam new file mode 100644 index 0000000..a75da67 --- /dev/null +++ b/gleam/src/ocda/dump.gleam @@ -0,0 +1,247 @@ +//// The model, written out canonically. +//// +//// Not an output format: an *oracle*. Two libraries in two languages +//// implement the same model from the same specification, and the cheap way +//// to ask whether they agree is to have each write the whole loaded +//// document as text and diff the two. Byte for byte identical to the D +//// library's `ocda/dump.d`, which is the whole point of it. +//// +//// The rules, which both implementations follow: +//// +//// - every line is `<key> <value>`, the value running to end of line, +//// and only one free-form value per line; +//// - a field at its default is *omitted*, so the output says what the +//// document has rather than what the type declares; +//// - dictionaries come in sorted key order, because neither language's +//// map has an order worth relying on; +//// - sections come in the fixed order of the format. +//// +//// Anything that reads this to reconstruct a document has misunderstood +//// it. Load the `.ssp` or the `.ocda.db`. + +import gleam/dict.{type Dict} +import gleam/int +import gleam/list +import gleam/string +import ocda/model.{type Document, type Object} + +pub fn document(doc: Document) -> String { + let lines = + [] + |> kv("version", doc.format_version) + |> kv("source_filename", doc.source_filename) + |> entries("source", doc.source) + |> entries("meta", doc.metadata) + |> entries("make", doc.make) + |> entries("doc_has", doc.doc_has) + |> counts(doc) + |> objects(doc) + lines |> list.reverse |> join +} + +/// What the loader met and did not understand, sorted. +/// +/// Separate from the document dump: it describes the loader, not the +/// document, and the `.ocda.db` loader has nothing to report where the +/// `.ssp` loader does. +pub fn unknown(doc: Document) -> String { + [] + |> numbered("property", doc.unknown.property) + |> numbered("block", doc.unknown.block) + |> numbered("line", doc.unknown.line) + |> list.reverse + |> join +} + +fn join(lines: List(String)) -> String { + case lines { + [] -> "" + _ -> string.join(lines, "\n") <> "\n" + } +} + +/// Accumulated newest-first and reversed once, so that nothing appends to +/// the tail of a list that grows to a few hundred thousand lines. +fn kv(acc: List(String), key: String, value: String) -> List(String) { + case value { + "" -> acc + v -> [key <> " " <> v, ..acc] + } +} + +fn kn(acc: List(String), key: String, value: Int) -> List(String) { + case value { + 0 -> acc + n -> [key <> " " <> int.to_string(n), ..acc] + } +} + +fn kb(acc: List(String), key: String, value: Bool) -> List(String) { + case value { + False -> acc + True -> [key <> " 1", ..acc] + } +} + +fn entries( + acc: List(String), + prefix: String, + d: Dict(String, String), +) -> List(String) { + d + |> dict.to_list + |> list.sort(fn(a, b) { string.compare(a.0, b.0) }) + |> list.fold(acc, fn(lines, pair) { + [prefix <> " " <> pair.0 <> " " <> pair.1, ..lines] + }) +} + +fn numbered( + acc: List(String), + prefix: String, + d: Dict(String, Int), +) -> List(String) { + d + |> dict.to_list + |> list.sort(fn(a, b) { string.compare(a.0, b.0) }) + |> list.fold(acc, fn(lines, pair) { + [prefix <> " " <> pair.0 <> " " <> int.to_string(pair.1), ..lines] + }) +} + +fn counts(acc: List(String), doc: Document) -> List(String) { + list.fold(model.section_names, acc, fn(lines, name) { + case model.section(doc, name) { + [] -> lines + objects -> + [ + "section " <> name <> " " <> int.to_string(list.length(objects)), + ..lines + ] + } + }) +} + +fn objects(acc: List(String), doc: Document) -> List(String) { + list.fold(model.section_names, acc, fn(lines, name) { + model.section(doc, name) + |> list.index_fold(lines, fn(l, obj, i) { + object(l, name, i, obj) + }) + }) +} + +fn object( + acc: List(String), + section: String, + index: Int, + obj: Object, +) -> List(String) { + let acc = ["object " <> section <> " " <> int.to_string(index), ..acc] + acc + |> kn("ocn", obj.ocn) + |> kv("is_a", obj.is_a) + |> kv("own_section", obj.own_section) + |> kv("level", obj.heading_level) + |> kv("identifier", obj.identifier) + |> kv("part", obj.part) + |> kv("is_of_type", obj.is_of_type) + |> kv("segment_html_is", obj.segment_html_is) + |> kv("segment_epub", obj.segment_epub) + |> kv("segment_epub_is", obj.segment_epub_is) + |> kn("parent", obj.parent_ocn) + |> kn("last_descendant", obj.last_descendant_ocn) + |> kv("children", ints(obj.children)) + |> kv("ancestors", ints(obj.ancestors)) + |> kb("dummy", obj.dummy_heading) + |> kb("ocn_off", obj.ocn_off) + |> kn("indent_base", obj.indent_base) + |> kn("indent_hang", obj.indent_hang) + |> kb("bullet", obj.bullet) + |> kv("lang", obj.lang) + |> kb("has_links", obj.has_links) + |> kb("has_notes_reg", obj.has_notes_reg) + |> kb("has_notes_star", obj.has_notes_star) + |> kb("has_images", obj.has_images) + |> kv("segment", obj.segment) + |> kv("segment_prev", obj.segment_prev) + |> kv("segment_next", obj.segment_next) + |> kv("anchor", obj.anchor) + |> kv("heading_lev_anchor", obj.heading_lev_anchor) + |> kn("table_cols", obj.table_cols) + |> kv("table_widths", obj.table_widths) + |> kv("table_aligns", obj.table_aligns) + |> kb("table_header", obj.table_header) + |> kv("code_syntax", obj.code_syntax) + |> kb("code_linenumbers", obj.code_linenumbers) + |> kv("heading_ancestors_text", obj.heading_ancestors_text) + |> kv("sha256", obj.sha256) + |> links(obj) + |> anchor_tags(obj) + |> images(obj) + |> text(obj) +} + +fn links(acc: List(String), obj: Object) -> List(String) { + list.fold(obj.stow_link, acc, fn(lines, url) { kv(lines, "stow_link", url) }) +} + +fn anchor_tags(acc: List(String), obj: Object) -> List(String) { + list.fold(obj.anchor_tags, acc, fn(lines, a) { kv(lines, "anchor_tag", a) }) +} + +fn images(acc: List(String), obj: Object) -> List(String) { + list.fold(obj.images, acc, fn(lines, img) { + // the name last, because a filename may hold a space where none of the + // other fields can + let lines = + kv( + lines, + "image", + img.sha256 + <> " " + <> int.to_string(img.bytes) + <> " " + <> int.to_string(img.width) + <> "x" + <> int.to_string(img.height) + <> " " + <> case img.missing { + True -> "1" + False -> "0" + } + <> " " + <> img.name, + ) + img.fields + |> dict.to_list + |> list.sort(fn(a, b) { string.compare(a.0, b.0) }) + |> list.fold(lines, fn(l, pair) { + kv(l, "image_field", pair.0 <> " " <> pair.1) + }) + }) +} + +/// Text, when there is any. +/// +/// An object whose only text line is blank holds "" here, indistinguishable +/// from no text at all, because the model joins the lines into one string. +/// Both libraries lose the same distinction in the same place, which is +/// what makes them comparable; it is recorded rather than hidden. +fn text(acc: List(String), obj: Object) -> List(String) { + case obj.text { + "" -> acc + t -> { + let lines = string.split(t, on: "\n") + let acc = kv(acc, "text", int.to_string(list.length(lines))) + list.fold(lines, acc, fn(l, line) { ["| " <> line, ..l] }) + } + } +} + +fn ints(xs: List(Int)) -> String { + case xs { + [] -> "" + _ -> xs |> list.map(int.to_string) |> string.join(",") + } +} diff --git a/gleam/src/ocda/images.gleam b/gleam/src/ocda/images.gleam new file mode 100644 index 0000000..2f3a56e --- /dev/null +++ b/gleam/src/ocda/images.gleam @@ -0,0 +1,135 @@ +//// An image's bytes, from whichever source the document came from. +//// +//// The model *names* images, with digest, byte size and pixel size; it does +//// not hold them. That is right - an abstraction of a document is not a +//// bundle of its assets - but a writer of any format that shows images needs +//// the bytes, so something has to know where they are. This is that +//// something, and it is in the library rather than in a writer because the +//// answer depends on the *input form* and not on the output. +//// +//// | `.ocda.db` | the `files` table, role 'image', by name | +//// | `.zip` | the archive entry whose path ends in `/image/<name>` | +//// | `.sisupod` | the same, if the frame can be unwrapped, which on the BEAM it cannot | +//// | `.ssp` | beside it: a pod puts the abstraction in `media/abstraction/` and its images in `media/image/` | + +import gleam/bit_array +import gleam/dynamic/decode +import gleam/list +import gleam/result +import gleam/string +import ocda/load +import ocda/zip +import simplifile +import sqlight + +/// The bytes, or an error saying why not. +pub fn image_bytes(source: String, name: String) -> Result(BitArray, String) { + case load.source_of(source) { + load.OcdaDb -> from_db(source, name) + load.Pod -> from_archive(source, name) + load.Ssp -> from_beside(source, name) + load.Unknown -> Error(source <> ": not a source this library reads") + } +} + +/// Can this source give images at all? +/// +/// A writer asks before it decides whether to emit them, because an `<img>` +/// pointing at a file it did not include is worse than no image. +pub fn can_give_images(source: String) -> Bool { + case load.source_of(source) { + load.OcdaDb -> True + load.Pod -> zip_readable(source) + load.Ssp -> image_dir_beside(source) != "" + load.Unknown -> False + } +} + +fn zip_readable(path: String) -> Bool { + case simplifile.read_bits(path) { + Error(_) -> False + Ok(bytes) -> + case zip.entries(bytes) { + Ok(_) -> True + Error(_) -> False + } + } +} + +fn from_db(path: String, name: String) -> Result(BitArray, String) { + case sqlight.open("file:" <> path <> "?mode=ro") { + Error(e) -> Error(path <> ": " <> e.message) + Ok(conn) -> { + let out = + sqlight.query( + "select data from files where role = 'image' and name = ?1", + on: conn, + with: [sqlight.text(name)], + expecting: blob(), + ) + |> result.replace_error(path <> ": could not read " <> name) + let _ = sqlight.close(conn) + case out { + Ok([data, ..]) -> Ok(data) + Ok([]) -> Error(name <> ": not in this database") + Error(e) -> Error(e) + } + } + } +} + +/// A blob column, read as bytes rather than as text: an image read as text +/// stops at the first zero byte, and a png's ninth byte is zero. +fn blob() -> decode.Decoder(BitArray) { + use v <- decode.field(0, decode.bit_array) + decode.success(v) +} + +fn from_archive(path: String, name: String) -> Result(BitArray, String) { + use bytes <- result.try( + simplifile.read_bits(path) + |> result.replace_error(path <> ": cannot read"), + ) + use entries <- result.try(zip.entries(bytes)) + case list.find(entries, fn(e) { string.ends_with(e.name, "/image/" <> name) }) { + Error(_) -> Error(name <> ": not in this archive") + Ok(entry) -> zip.read(bytes, entry) + } +} + +fn from_beside(path: String, name: String) -> Result(BitArray, String) { + case image_dir_beside(path) { + "" -> Error(name <> ": a .ssp outside a pod has no image directory beside it") + dir -> + simplifile.read_bits(dir <> "/" <> name) + |> result.replace_error(dir <> "/" <> name <> ": cannot read") + } +} + +/// `media/abstraction/<x>.ssp` -> `media/image/` +fn image_dir_beside(path: String) -> String { + let parts = string.split(path, on: "/") + let dir = case list.reverse(parts) { + [_file, ..rest] -> string.join(list.reverse(rest), "/") + [] -> "" + } + case string.ends_with(dir, "abstraction") { + False -> "" + True -> { + let parent = case list.reverse(string.split(dir, on: "/")) { + [_, ..rest] -> string.join(list.reverse(rest), "/") + [] -> "" + } + let image_dir = parent <> "/image" + case simplifile.is_directory(image_dir) { + Ok(True) -> image_dir + _ -> "" + } + } + } +} + +/// Unused here, kept so that a caller can size a read before making it. +pub fn byte_size(b: BitArray) -> Int { + bit_array.byte_size(b) +} diff --git a/gleam/src/ocda/load.gleam b/gleam/src/ocda/load.gleam new file mode 100644 index 0000000..2a82ac9 --- /dev/null +++ b/gleam/src/ocda/load.gleam @@ -0,0 +1,79 @@ +//// What a path is, and loading it. +//// +//// One place that decides, so that every reader and every writer accepts +//// the same inputs and a new input form is added once. This mirrors +//// spine's own `ocda/abstraction/load.d`, which names the sources it can +//// enter the pipeline from and tells them apart by suffix. +//// +//// Deliberately *not* here: `.sst` and `.ssm` markup. Reading markup means +//// implementing the parser, which is spine's work and a far larger thing +//// than a reader or a sample writer needs. This library reads +//// abstractions. + +import gleam/list +import gleam/string +import ocda/load_db +import ocda/load_pod +import ocda/load_ssp +import ocda/model.{type Document} + +pub type Source { + /// `.ssp`, the abstraction as text + Ssp + /// `.ocda.db`, the abstraction as sqlite + OcdaDb + /// `.sisupod` or `.zip`, an archive carrying a `.ssp` + Pod + Unknown +} + +/// `.sisupod` is the archive spine writes now; `.zip` is what it wrote +/// before, and every pod already published carries that name, so both are +/// recognised. What is inside decides how it is opened, not the name. +const suffixes = [ + #(".ocda.db", OcdaDb), + #(".ssp", Ssp), + #(".sisupod", Pod), + #(".zip", Pod), + #(".db", OcdaDb), +] + +pub fn source_of(path: String) -> Source { + case list.find(suffixes, fn(s) { string.ends_with(path, s.0) }) { + Ok(#(_, source)) -> source + Error(_) -> Unknown + } +} + +pub fn source_name(source: Source) -> String { + case source { + Ssp -> ".ssp (the abstraction as text)" + OcdaDb -> ".ocda.db (the abstraction as sqlite)" + Pod -> "a pod archive carrying a .ssp" + Unknown -> "not an abstraction this library reads" + } +} + +/// Load whatever the path is into the model. +/// +/// `lang` names which language to take from a `.ocda.db`, which holds +/// every language of its document, and from a pod, which may carry more than +/// one `.ssp`. A `.ssp` is one language by construction and ignores it. +pub fn load( + path: String, + lang: String, + any_version: Bool, +) -> Result(Document, String) { + case source_of(path) { + Ssp -> load_ssp.load_file(path) + OcdaDb -> load_db.load_file(path, lang, any_version) + Pod -> load_pod.load_file(path, lang) + Unknown -> + Error( + path + <> ": " + <> source_name(Unknown) + <> " (expected .ssp, .ocda.db, .sisupod or .zip)", + ) + } +} diff --git a/gleam/src/ocda/load_db.gleam b/gleam/src/ocda/load_db.gleam new file mode 100644 index 0000000..a5f044b --- /dev/null +++ b/gleam/src/ocda/load_db.gleam @@ -0,0 +1,532 @@ +//// Loading a `.ocda.db` into the model. +//// +//// The point of this module is that it returns *the same value* +//// `ocda/load_ssp` returns from the same document's `.ssp`. A writer is +//// then a function from the model to bytes and cannot tell which +//// serialisation it was given, exactly as spine's own writers cannot. +//// +//// Written against `ocda-db-format-specification.md`, whose section 4 +//// gives the column by column correspondence this module implements. + +import gleam/dict +import gleam/dynamic/decode +import gleam/float +import gleam/int +import gleam/json +import gleam/list +import gleam/option +import gleam/result +import gleam/string +import ocda/model.{type Document, Image, Object} +import sqlight + +/// The schema major version this loader knows. +pub const schema_major_read = "2" + +/// One language of the database, as the manifest gives it. +pub type Language { + Language(doc_id: Int, lang: String, source_filename: String, ssp_digest: String) +} + +/// Load one language of a database as the model. `lang` == "" takes the +/// first language the manifest gives. +pub fn load_file( + path: String, + lang: String, + any_version: Bool, +) -> Result(Document, String) { + // read only: a loader has no business writing, and a read-write open + // *creates* an empty database where a path is mistyped, which then reads + // as a document with nothing in it, a failure that looks like an answer + use conn <- with_connection("file:" <> path <> "?mode=ro", path) + + use _ <- result.try(case + has(conn, "documents") && has(conn, "objects") && has(conn, "metadata") + { + True -> Ok(Nil) + False -> + Error( + path + <> ": no documents/objects/metadata tables; it does not look like a .ocda.db", + ) + }) + + let schema_version = + one(conn, "select value from metadata where doc_id is null + and key = 'schema.version'") + use _ <- result.try(case schema_version, any_version { + "", _ -> Ok(Nil) + _, True -> Ok(Nil) + v, False -> + case major_of(v) == schema_major_read { + True -> Ok(Nil) + False -> + Error( + path + <> " is schema v" + <> v + <> ", this loader knows v" + <> schema_major_read + <> ".x", + ) + } + }) + + let languages = languages(conn) + use chosen <- result.try(case languages, lang { + [], _ -> Error(path <> ": no documents rows") + [first, ..], "" -> Ok(first) + _, wanted -> + case list.find(languages, fn(l) { l.lang == wanted }) { + Ok(found) -> Ok(found) + Error(_) -> + Error( + path + <> ": no language '" + <> wanted + <> "'; it holds: " + <> string.join(list.map(languages, fn(l) { l.lang }), " "), + ) + } + }) + + let source = + dict.new() + |> dict.insert("language", chosen.lang) + |> dict.insert( + "languages", + string.join(list.map(languages, fn(l) { l.lang }), " "), + ) + let digest = meta_value(conn, chosen.doc_id, "source.digest") + let source = case digest { + "" -> source + d -> dict.insert(source, "digest", d) + } + + let document = + model.Document( + ..model.new_document(), + format_version: schema_version, + source_filename: chosen.source_filename, + source: source, + ) + let document = metadata(conn, chosen.doc_id, document) + let document = objects(conn, chosen.doc_id, document) + Ok(document) +} + +fn with_connection( + uri: String, + path: String, + then: fn(sqlight.Connection) -> Result(a, String), +) -> Result(a, String) { + case sqlight.open(uri) { + Error(e) -> Error(path <> ": " <> e.message) + Ok(conn) -> { + let outcome = then(conn) + let _ = sqlight.close(conn) + outcome + } + } +} + +/// Every language the database holds, in manifest order. +pub fn languages(conn: sqlight.Connection) -> List(Language) { + let d = { + use id <- decode.field(0, decode.int) + use lang <- decode.field(1, decode.string) + use source <- decode.field(2, decode.string) + use digest <- decode.field(3, decode.string) + decode.success(Language(id, lang, source, digest)) + } + sqlight.query( + "select id, lang, coalesce(source_filename,''), coalesce(ssp_digest,'') + from documents order by id", + on: conn, + with: [], + expecting: d, + ) + |> result.unwrap([]) +} + +/// The three preamble blocks of the text form, out of the one metadata +/// table the database keeps them in. +fn metadata( + conn: sqlight.Connection, + doc_id: Int, + document: Document, +) -> Document { + let d = { + use key <- decode.field(0, decode.string) + use value <- decode.field(1, decode.string) + decode.success(#(key, value)) + } + let rows = + sqlight.query( + "select key, value from metadata where doc_id = " + <> int.to_string(doc_id) + <> " order by key", + on: conn, + with: [], + expecting: d, + ) + |> result.unwrap([]) + + list.fold(rows, document, fn(doc, row) { + let #(key, raw) = row + let value = string.trim(raw) + case key { + // source.* describes the artefact rather than the document, and is + // @source in the text form + "source." <> _ -> doc + // translation.ocn_aligned says whether the languages of this document + // share one numbering. A .ocda.db can know it because it holds every + // language; a .ssp is one language and cannot. It is a fact about the + // file, not the document, and carrying it here would let a writer + // tell which serialisation it was handed. + "translation." <> _ -> doc + // the database tells the three blocks apart by a key prefix. + // Unprefixing them is what makes the two loaders give one model: a + // writer asks make["footer"], never metadata["make.footer"]. + "make." <> rest -> + model.Document(..doc, make: dict.insert(doc.make, rest, value)) + "doc_has." <> rest -> + model.Document(..doc, doc_has: dict.insert(doc.doc_has, rest, value)) + _ -> + model.Document(..doc, metadata: dict.insert(doc.metadata, key, value)) + } + }) +} + +/// The objects, in document order within each section, the sections in the +/// order the format gives rather than the alphabetical one sqlite would. +fn objects( + conn: sqlight.Connection, + doc_id: Int, + document: Document, +) -> Document { + let d = { + use section <- decode.field(0, decode.string) + use ocn <- decode.field(2, decode.int) + use is_a <- decode.field(3, decode.string) + use part <- decode.field(4, decode.string) + use is_of_section <- decode.field(5, decode.string) + use is_of_type <- decode.field(6, decode.string) + use heading_level <- decode.field(7, decode.int) + use identifier <- decode.field(8, decode.string) + use parent <- decode.field(9, decode.int) + use last_descendant <- decode.field(10, decode.int) + use children <- decode.field(11, decode.string) + use ancestors <- decode.field(12, decode.string) + use dummy <- decode.field(13, decode.int) + use ocn_off <- decode.field(14, decode.int) + use indent_base <- decode.field(15, decode.int) + use indent_hang <- decode.field(16, decode.int) + use bullet <- decode.field(17, decode.int) + use lang <- decode.field(18, decode.string) + use has_links <- decode.field(19, decode.int) + use has_notes_reg <- decode.field(20, decode.int) + use has_notes_star <- decode.field(21, decode.int) + use has_images <- decode.field(22, decode.int) + use segment <- decode.field(23, decode.string) + use segment_prev <- decode.field(24, decode.string) + use segment_next <- decode.field(25, decode.string) + use anchor <- decode.field(26, decode.string) + use segment_html_is <- decode.field(27, decode.string) + use table_cols <- decode.field(28, decode.int) + use table_widths <- decode.field(29, decode.string) + use table_aligns <- decode.field(30, decode.string) + use table_header <- decode.field(31, decode.int) + use code_linenumbers <- decode.field(32, decode.int) + use hat <- decode.field(33, decode.string) + use sha256 <- decode.field(34, decode.string) + use text <- decode.field(35, decode.string) + use id <- decode.field(36, decode.int) + use hla <- decode.field(37, decode.string) + use seg_epub <- decode.field(38, decode.string) + use seg_epub_is <- decode.field(39, decode.string) + decode.success(#( + id, + Object( + ocn: ocn, + is_a: is_a, + section: section, + heading_level: level_name(heading_level), + identifier: identifier, + part: part, + is_of_type: is_of_type, + own_section: case is_of_section == section { + True -> "" + False -> is_of_section + }, + segment_html_is: segment_html_is, + table_aligns: json_words(table_aligns), + stow_link: [], + anchor_tags: [], + parent_ocn: parent, + last_descendant_ocn: last_descendant, + children: json_ints(children), + ancestors: json_ints(ancestors), + dummy_heading: dummy != 0, + ocn_off: ocn_off != 0, + indent_base: indent_base, + indent_hang: indent_hang, + bullet: bullet != 0, + lang: lang, + has_links: has_links != 0, + has_notes_reg: has_notes_reg != 0, + has_notes_star: has_notes_star != 0, + has_images: has_images != 0, + segment: segment, + segment_prev: segment_prev, + segment_next: segment_next, + anchor: anchor, + heading_lev_anchor: hla, + segment_epub: seg_epub, + segment_epub_is: seg_epub_is, + table_cols: table_cols, + table_widths: json_words(table_widths), + table_header: table_header != 0, + code_syntax: "", + code_linenumbers: code_linenumbers != 0, + heading_ancestors_text: json_joined(hat), + sha256: sha256, + images: [], + text: text, + ), + )) + } + let rows = + sqlight.query(object_sql(doc_id), on: conn, with: [], expecting: d) + |> result.unwrap([]) + + // Built newest-first per section and reversed once at the end. Appending + // to the tail of each section instead would be quadratic, and the largest + // document in the reference collection has some thousands of objects in + // one section. + let sections = + list.fold(rows, dict.new(), fn(acc, row) { + let #(id, object) = row + // the four repeatable properties of the text form are child tables + // here, one query each + let object = + Object( + ..object, + images: images(conn, id), + stow_link: links(conn, id), + anchor_tags: anchors(conn, id), + ) + case model.is_section(object.section) { + False -> acc + True -> + dict.upsert(acc, object.section, fn(existing) { + case existing { + option.Some(previous) -> [object, ..previous] + option.None -> [object] + } + }) + } + }) + |> dict.map_values(fn(_, objects) { list.reverse(objects) }) + model.Document(..document, sections: sections) +} + +fn object_sql(doc_id: Int) -> String { + "select section, seq, ocn, is_a, coalesce(is_of_part,''), + coalesce(is_of_section,''), coalesce(is_of_type,''), + coalesce(heading_level,-1), coalesce(identifier,''), + coalesce(parent_ocn,0), coalesce(last_descendant_ocn,0), + coalesce(children,''), coalesce(ancestors,''), + coalesce(dummy_heading,0), coalesce(object_number_off,0), + coalesce(indent_base,0), coalesce(indent_hang,0), + coalesce(bullet,0), coalesce(lang,''), + coalesce(has_links,0), coalesce(has_notes_reg,0), + coalesce(has_notes_star,0), coalesce(has_images,0), + coalesce(segment,''), coalesce(segment_prev,''), + coalesce(segment_next,''), coalesce(anchor,''), + coalesce(segment_html_is,''), + coalesce(table_cols,0), coalesce(table_widths,''), + coalesce(table_aligns,''), coalesce(table_header,0), + coalesce(code_linenumbers,0), + coalesce(heading_ancestors_text,''), coalesce(sha256,''), + coalesce(text,''), id, coalesce(heading_lev_anchor,''), + coalesce(segment_epub,''), coalesce(segment_epub_is,'') + from objects where doc_id = " <> int.to_string(doc_id) <> " + order by case section + when 'head' then 1 when 'toc' then 2 when 'body' then 3 + when 'endnotes' then 4 when 'glossary' then 5 + when 'bibliography' then 6 when 'bookindex' then 7 + when 'blurb' then 8 when 'tail' then 9 else 10 end, seq" +} + +fn images(conn: sqlight.Connection, object_id: Int) -> List(model.Image) { + let d = { + use name <- decode.field(0, decode.string) + use sha256 <- decode.field(1, decode.string) + use bytes <- decode.field(2, decode.int) + use width <- decode.field(3, decode.int) + use height <- decode.field(4, decode.int) + use missing <- decode.field(5, decode.int) + decode.success(Image( + name: name, + sha256: sha256, + bytes: bytes, + width: width, + height: height, + missing: missing != 0, + fields: dict.new(), + )) + } + sqlight.query( + "select name, coalesce(sha256,''), coalesce(bytes,0), + coalesce(width,0), coalesce(height,0), coalesce(missing,0) + from object_images where object_id = " + <> int.to_string(object_id) + <> " order by seq", + on: conn, + with: [], + expecting: d, + ) + |> result.unwrap([]) +} + +fn links(conn: sqlight.Connection, object_id: Int) -> List(String) { + let d = { + use url <- decode.field(0, decode.string) + decode.success(url) + } + sqlight.query( + "select url from object_links where object_id = " + <> int.to_string(object_id) + <> " order by seq", + on: conn, + with: [], + expecting: d, + ) + |> result.unwrap([]) +} + +fn anchors(conn: sqlight.Connection, object_id: Int) -> List(String) { + let d = { + use anchor <- decode.field(0, decode.string) + decode.success(anchor) + } + sqlight.query( + "select anchor from object_anchors where object_id = " + <> int.to_string(object_id) + <> " order by seq", + on: conn, + with: [], + expecting: d, + ) + |> result.unwrap([]) +} + +fn one(conn: sqlight.Connection, sql: String) -> String { + let d = { + use v <- decode.field(0, decode.string) + decode.success(v) + } + case sqlight.query(sql, on: conn, with: [], expecting: d) { + Ok([v, ..]) -> v + _ -> "" + } +} + +fn has(conn: sqlight.Connection, name: String) -> Bool { + one( + conn, + "select cast(count(*) as text) from sqlite_master where name = '" + <> name + <> "'", + ) + != "0" +} + +fn meta_value(conn: sqlight.Connection, doc_id: Int, key: String) -> String { + one( + conn, + "select value from metadata where doc_id = " + <> int.to_string(doc_id) + <> " and key = '" + <> key + <> "'", + ) +} + +fn major_of(version: String) -> String { + case string.split_once(version, ".") { + Ok(#(major, _)) -> major + Error(_) -> version + } +} + +/// The eight heading levels and their indices. +fn level_name(index: Int) -> String { + case index { + 0 -> "A" + 1 -> "B" + 2 -> "C" + 3 -> "D" + 4 -> "1" + 5 -> "2" + 6 -> "3" + 7 -> "4" + _ -> "" + } +} + +/// The JSON columns, read as JSON. +/// +/// Not as a character scan that strips brackets and quotes: that is wrong +/// the moment a value holds an escape, and spine's encoder writes "/" as +/// "\/", so every heading containing a slash comes back with a backslash in +/// it. The D library had exactly that bug and all 36 document-languages +/// differed on heading_ancestors_text because of it. +fn json_strings(value: String) -> List(String) { + case value { + "" -> [] + v -> + case json.parse(v, decode.list(of: loose_string())) { + Ok(items) -> items + Error(_) -> [] + } + } +} + +/// An element that may be a string or a number, which is what these columns +/// hold: children and ancestors are int arrays, table_aligns and +/// heading_ancestors_text string arrays, and table_widths an int array +/// except where the columns divide evenly, when it is [33.3333,...]. One +/// table in the reference collection does that, and without the float +/// branch the widths came back as three empty strings: the fallback turned +/// a value it could not read into a value that looked read. +fn loose_string() -> decode.Decoder(String) { + decode.one_of(decode.string, [ + decode.int |> decode.map(int.to_string), + decode.float |> decode.map(float.to_string), + decode.success(""), + ]) +} + +fn json_ints(value: String) -> List(Int) { + value + |> json_strings + |> list.filter(fn(s) { s != "" }) + |> list.map(fn(s) { s |> int.parse |> result.unwrap(0) }) +} + +/// Space separated in the text form: table_widths, table_aligns. +fn json_words(value: String) -> String { + value |> json_strings |> string.join(" ") +} + +/// heading_ancestors_text is eight slots, joined with "|" in the text form +/// and a JSON array here. The empty slots are significant: they say which +/// levels the heading sits under. +fn json_joined(value: String) -> String { + case json_strings(value) { + [] -> "" + parts -> string.join(parts, "|") + } +} diff --git a/gleam/src/ocda/load_pod.gleam b/gleam/src/ocda/load_pod.gleam new file mode 100644 index 0000000..35483c6 --- /dev/null +++ b/gleam/src/ocda/load_pod.gleam @@ -0,0 +1,138 @@ +//// The abstraction out of a pod archive. +//// +//// A pod built with `--pod2` carries the abstraction inside it, at +//// `media/abstraction/<uid>.<lang>.ssp`, and that is what is read. So this is +//// a third input form and *not* a third model: the `.ssp` inside is parsed by +//// `ocda/load_ssp` and comes back as the same value. A writer over the +//// library gains pods without being touched, which is the whole reason the +//// dispatch is in one place. +//// +//// ## The one thing this cannot do +//// +//// A `.sisupod` is **one zstd frame wrapping** the zip. This reads the zip +//// and **not the frame**, because there is no zstd on the BEAM without a NIF, +//// and the one that exists, `ezstd`, clones zstd from github in its build, so +//// it cannot be built in a sandbox without patching it. The D library has +//// both, through a declared `libzstd`. +//// +//// So: a `.zip` pod is read, a `.sisupod` is refused, and the refusal says +//// why rather than pretending the file is malformed. It is the only place the +//// two libraries differ in what they accept, and it is a packaging limit +//// rather than anything about the format. + +import gleam/list +import gleam/result +import gleam/string +import ocda/load_ssp +import ocda/model.{type Document} +import ocda/zip +import simplifile + +/// The magic number of a zstd frame, little endian 0xFD2FB528. +const zstd_magic = <<0x28, 0xB5, 0x2F, 0xFD>> + +/// Which languages of the abstraction the pod carries, sorted. +pub fn languages(path: String) -> Result(List(String), String) { + use bytes <- result.try(archive_bytes(path)) + use entries <- result.try(zip.entries(bytes)) + zip.find_suffix(entries, ".ssp") + |> list.filter_map(fn(e) { lang_of(e.name) }) + |> list.sort(string.compare) + |> Ok +} + +/// Load one language of the abstraction a pod carries. +/// +/// `lang` == "" takes the only one there is, and where there is a choice it +/// says so rather than choosing, as the `.ocda.db` loader does. +pub fn load_file(path: String, lang: String) -> Result(Document, String) { + use bytes <- result.try(archive_bytes(path)) + use entries <- result.try(zip.entries(bytes)) + let ssps = zip.find_suffix(entries, ".ssp") + use chosen <- result.try(case ssps, lang { + [], _ -> + Error( + path + <> ": carries no .ssp. A pod carries the abstraction only when spine" + <> " was asked for it: spine --pod2 puts it in, spine --source does not.", + ) + [only], "" -> Ok(only) + _, "" -> + Error( + path + <> ": holds more than one language (" + <> string.join(langs(ssps), " ") + <> "); name the one wanted", + ) + _, wanted -> + case list.find(ssps, fn(e) { lang_of(e.name) == Ok(wanted) }) { + Ok(found) -> Ok(found) + Error(_) -> + Error( + path + <> ": no language '" + <> wanted + <> "'; it holds: " + <> string.join(langs(ssps), " "), + ) + } + }) + use content <- result.try(zip.read(bytes, chosen)) + case bit_array_to_string(content) { + Ok(text) -> Ok(load_ssp.parse(text)) + Error(_) -> Error(chosen.name <> ": not valid UTF-8") + } +} + +fn langs(entries: List(zip.Entry)) -> List(String) { + list.filter_map(entries, fn(e) { lang_of(e.name) }) +} + +/// `<uid>.<lang>.ssp` -> `<lang>` +fn lang_of(name: String) -> Result(String, Nil) { + case string.ends_with(name, ".ssp") { + False -> Error(Nil) + True -> { + let stem = string.drop_end(name, 4) + case string.split(stem, on: ".") |> list.last { + Ok(lang) if lang != stem -> Ok(lang) + _ -> Error(Nil) + } + } + } +} + +/// The bytes decide, not the name: a file misnamed either way is treated by +/// what it holds. +fn archive_bytes(path: String) -> Result(BitArray, String) { + case simplifile.read_bits(path) { + Error(e) -> Error(path <> ": " <> simplifile.describe_error(e)) + Ok(bytes) -> + case is_zstd_frame(bytes) { + False -> Ok(bytes) + True -> + Error( + path + <> " is a zstd frame wrapping the archive, and this library has no" + <> " zstd: the only one on the BEAM is a NIF that fetches zstd in" + <> " its own build, so it cannot be built sandboxed. Unwrap it" + <> " first (zstd -d) and read the archive, or use the D library," + <> " which reads both.", + ) + } + } +} + +fn is_zstd_frame(bytes: BitArray) -> Bool { + case bytes { + <<head:bytes-size(4), _:bits>> -> head == zstd_magic + _ -> False + } +} + +@external(erlang, "unicode", "characters_to_binary") +fn bit_array_to_string_ffi(b: BitArray) -> String + +fn bit_array_to_string(b: BitArray) -> Result(String, Nil) { + Ok(bit_array_to_string_ffi(b)) +} diff --git a/gleam/src/ocda/load_ssp.gleam b/gleam/src/ocda/load_ssp.gleam new file mode 100644 index 0000000..b48e1c5 --- /dev/null +++ b/gleam/src/ocda/load_ssp.gleam @@ -0,0 +1,478 @@ +//// Loading a `.ssp` into the model. +//// +//// The format is line oriented and every line is one of eight shapes told +//// apart by its first character, so the loader is a fold over the lines +//// carrying the state the shapes need. +//// +//// Written against `ssp-format-specification.md`. Where it and the D +//// library differ the D library is the bug, or this is: they are meant to +//// give the same model, and `ocda-dump` says whether they do. + +import gleam/dict.{type Dict} +import gleam/int +import gleam/list +import gleam/option +import gleam/result +import gleam/string +import ocda/model.{type Document, type Image, type Object, Image, Object} +import simplifile + +pub fn load_file(path: String) -> Result(Document, String) { + case simplifile.read(path) { + Ok(content) -> Ok(parse(content)) + Error(e) -> Error(path <> ": " <> simplifile.describe_error(e)) + } +} + +type State { + State( + document: Document, + /// the block currently open, "" when none + block: String, + /// the section currently open, "" when in the preamble + section: String, + /// objects of the current section, newest first + objects: List(Object), + /// the object being read, if any + current: Result(Object, Nil), + /// text lines of the object being read, newest first + text: List(String), + /// whether a text line has been seen for the object being read. + /// Not "is the text non-empty": an object whose first text line is + /// blank, which the reference collection has seven of, would then lose + /// it. + seen_text: Bool, + ) +} + +pub fn parse(content: String) -> Document { + content + |> string.split(on: "\n") + |> list.fold(from: initial(), with: line) + |> finish +} + +fn initial() -> State { + State( + document: model.new_document(), + block: "", + section: "", + objects: [], + current: Error(Nil), + text: [], + seen_text: False, + ) +} + +fn line(state: State, raw: String) -> State { + case raw { + "% SiSU Document Abstraction v" <> version -> + case state.document.format_version { + "" -> + State( + ..state, + document: model.Document( + ..state.document, + format_version: string.trim(version), + ), + ) + _ -> state + } + "% Source: " <> filename -> + case state.document.source_filename { + "" -> + State( + ..state, + document: model.Document( + ..state.document, + source_filename: string.trim(filename), + ), + ) + _ -> state + } + "%" <> _ -> state + "@" <> rest -> open_block(state, rest) + "}" <> _ -> close_block(state) + "[" <> rest -> declaration(state, rest) + "." <> rest -> property(state, rest) + "|" <> rest -> text_line(state, rest) + _ -> + case string.trim(raw) { + // a blank line ends the object being read + "" -> commit(state) + _ -> + case model.is_preamble_block(state.block) { + True -> preamble_pair(state, raw) + False -> unknown_line(state, raw) + } + } + } +} + +fn open_block(state: State, rest: String) -> State { + let name = case string.split_once(rest, "{") { + Ok(#(name, _)) -> string.trim(name) + Error(_) -> string.trim(rest) + } + case model.is_preamble_block(name) { + True -> State(..flush(state), block: name) + False -> { + let state = flush(state) + let document = case model.is_section(name) { + True -> state.document + // a block that is neither preamble nor one of the nine sections: + // counted, so that a reader can say the file held something it did + // not expect + False -> + model.Document( + ..state.document, + unknown: model.Unknown( + ..state.document.unknown, + block: bump(state.document.unknown.block, name), + ), + ) + } + State(..state, block: name, section: name, document: document) + } + } +} + +fn close_block(state: State) -> State { + State(..flush(state), block: "") +} + +/// Commit the object being read, then put the section's objects into the +/// document and close the section. +fn flush(state: State) -> State { + let state = commit(state) + case state.section, state.objects { + "", _ -> state + _, [] -> State(..state, section: "", objects: []) + name, objects -> + State( + ..state, + section: "", + objects: [], + document: model.Document( + ..state.document, + sections: add_section( + state.document.sections, + name, + list.reverse(objects), + ), + ), + ) + } +} + +fn preamble_pair(state: State, raw: String) -> State { + case string.split_once(string.trim_start(raw), ":") { + Error(_) -> state + Ok(#(key, value)) -> { + let key = string.trim(key) + let value = string.trim(value) + let document = state.document + let document = case state.block { + "meta" -> + model.Document( + ..document, + metadata: dict.insert(document.metadata, key, value), + ) + "make" -> + model.Document(..document, make: dict.insert(document.make, key, value)) + "source" -> + model.Document( + ..document, + source: dict.insert(document.source, key, value), + ) + _ -> + model.Document( + ..document, + doc_has: dict.insert(document.doc_has, key, value), + ) + } + State(..state, document: document) + } + } +} + +/// `[<ocn>] <kind>`, and for a heading `[<ocn>] heading :<level> +/// [<identifier>]`. +fn declaration(state: State, rest: String) -> State { + let state = commit(state) + case string.split_once(rest, "]") { + Error(_) -> state + Ok(#(ocn_text, tail)) -> { + let ocn = parse_int(string.trim(ocn_text)) + let object = + Object(..model.new_object(), ocn: ocn, section: state.section) + let object = case string.split(string.trim(tail), on: " ") { + ["heading", level, ..identifier] -> + Object( + ..object, + is_a: "heading", + heading_level: string.drop_start(level, 1), + identifier: string.join(identifier, " "), + ) + [""] -> object + [kind, ..] -> Object(..object, is_a: kind) + [] -> object + } + State(..state, current: Ok(object), text: [], seen_text: False) + } + } +} + +fn property(state: State, rest: String) -> State { + case state.current, string.split_once(rest, ":") { + Ok(object), Ok(#(name, raw_value)) -> { + let name = string.trim(name) + let value = string.trim(raw_value) + let #(object, known) = apply_property(object, name, value) + let document = case known { + True -> state.document + False -> + model.Document( + ..state.document, + unknown: model.Unknown( + ..state.document.unknown, + property: bump(state.document.unknown.property, name), + ), + ) + } + State(..state, current: Ok(object), document: document) + } + _, _ -> state + } +} + +/// The properties this library reads, and what each sets. +/// +/// A property it has no use for is counted rather than passed over: +/// ignoring one is conforming within a major version, doing so in silence +/// is what stops a reader being a check on the specification. +fn apply_property( + object: Object, + name: String, + value: String, +) -> #(Object, Bool) { + case name { + "part" -> #(Object(..object, part: value), True) + "is_of_type" -> #(Object(..object, is_of_type: value), True) + "section" -> #(Object(..object, own_section: value), True) + "identifier" -> #(Object(..object, identifier: value), True) + "segment_html_is" -> #(Object(..object, segment_html_is: value), True) + "segment_epub" -> #(Object(..object, segment_epub: value), True) + "segment_epub_is" -> #(Object(..object, segment_epub_is: value), True) + "table_aligns" -> #(Object(..object, table_aligns: value), True) + "stow_link" -> #( + Object(..object, stow_link: list.append(object.stow_link, [value])), + True, + ) + "anchor_tag" -> #( + Object(..object, anchor_tags: list.append(object.anchor_tags, [value])), + True, + ) + "parent" -> #(Object(..object, parent_ocn: parse_int(value)), True) + "last_descendant" -> #( + Object(..object, last_descendant_ocn: parse_int(value)), + True, + ) + "children" -> #(Object(..object, children: numbers(value)), True) + "ancestors" -> #(Object(..object, ancestors: numbers(value)), True) + "dummy" -> #(Object(..object, dummy_heading: value == "true"), True) + "ocn_off" -> #(Object(..object, ocn_off: value == "true"), True) + "indent" -> { + let parts = string.split(value, on: " ") + let base = case parts { + [first, ..] -> parse_int(first) + [] -> 0 + } + let hang = case parts { + [_, second, ..] -> parse_int(second) + _ -> 0 + } + #(Object(..object, indent_base: base, indent_hang: hang), True) + } + "bullet" -> #(Object(..object, bullet: value == "true"), True) + "lang" -> #(Object(..object, lang: value), True) + "has" -> #(has_flags(object, value), True) + "segment" -> #(Object(..object, segment: value), True) + "segment_prev" -> #(Object(..object, segment_prev: value), True) + "segment_next" -> #(Object(..object, segment_next: value), True) + "anchor" -> #(Object(..object, anchor: value), True) + "heading_lev_anchor" -> #( + Object(..object, heading_lev_anchor: value), + True, + ) + "table_cols" -> #(Object(..object, table_cols: parse_int(value)), True) + "table_widths" -> #(Object(..object, table_widths: value), True) + "table_header" -> #(Object(..object, table_header: value == "true"), True) + "code_syntax" -> #(Object(..object, code_syntax: value), True) + "code_linenumbers" -> #( + Object(..object, code_linenumbers: value == "true"), + True, + ) + "heading_ancestors_text" -> #( + Object(..object, heading_ancestors_text: value), + True, + ) + "sha256" -> #(Object(..object, sha256: value), True) + "image" -> #( + Object(..object, images: list.append(object.images, [image(value)])), + True, + ) + _ -> #(object, False) + } +} + +fn has_flags(object: Object, value: String) -> Object { + value + |> string.split(on: " ") + |> list.fold(from: object, with: fn(obj, flag) { + case flag { + "links" -> Object(..obj, has_links: True) + "notes_reg" -> Object(..obj, has_notes_reg: True) + "notes_star" -> Object(..obj, has_notes_star: True) + "images" -> Object(..obj, has_images: True) + _ -> obj + } + }) +} + +/// `<filename> sha256:<HEX> bytes:<n> [px:<w>x<h>] [missing:true] [...]` +/// +/// The filename comes first, the rest are order-independent, and a field +/// this library does not recognise is kept rather than discarded. +fn image(value: String) -> Image { + case string.split(value, on: " ") { + [] -> model.new_image() + [name, ..fields] -> + list.fold(fields, model.new_image(), fn(img, field) { + case string.split_once(field, ":") { + Error(_) -> img + Ok(#(key, val)) -> + case key { + "sha256" -> Image(..img, sha256: val) + "bytes" -> Image(..img, bytes: parse_int(val)) + "px" -> + case string.split_once(val, "x") { + Ok(#(w, h)) -> + Image(..img, width: parse_int(w), height: parse_int(h)) + Error(_) -> img + } + "missing" -> Image(..img, missing: val == "true") + _ -> Image(..img, fields: dict.insert(img.fields, key, val)) + } + } + }) + |> fn(img) { Image(..img, name: name) } + } +} + +/// `| <text>`, and `| ` alone for a blank line. Never a bare `|`: the +/// emitter writes the space. +fn text_line(state: State, rest: String) -> State { + case state.current { + Error(_) -> state + Ok(_) -> { + let content = string.drop_start(rest, 1) + State(..state, text: [content, ..state.text], seen_text: True) + } + } +} + +fn unknown_line(state: State, raw: String) -> State { + let first = string.slice(raw, 0, 1) + State( + ..state, + document: model.Document( + ..state.document, + unknown: model.Unknown( + ..state.document.unknown, + line: bump(state.document.unknown.line, first), + ), + ), + ) +} + +fn commit(state: State) -> State { + case state.current { + Error(_) -> state + Ok(object) -> { + let text = case state.seen_text { + True -> string.join(list.reverse(state.text), "\n") + False -> "" + } + let object = derive_omitted(Object(..object, text: text)) + let objects = case model.is_section(state.section) { + True -> [object, ..state.objects] + False -> state.objects + } + State( + ..state, + current: Error(Nil), + text: [], + seen_text: False, + objects: objects, + ) + } + } +} + +/// Two values the text form leaves out because they can be worked out. +/// +/// `.identifier` is absent when it is simply the ocn, `.anchor` when it is +/// the segment name. The `.ocda.db`, being columns rather than lines, +/// stores both unconditionally. So a loader that takes the text form +/// literally gives a different model from the one the database gives, and +/// over one document that is 1,314 identifiers and 69 anchors. +fn derive_omitted(object: Object) -> Object { + let object = case object.identifier, object.ocn { + "", 0 -> object + "", ocn -> Object(..object, identifier: int.to_string(ocn)) + _, _ -> object + } + case object.anchor, object.segment { + "", "" -> object + "", segment -> Object(..object, anchor: segment) + _, _ -> object + } +} + +fn finish(state: State) -> Document { + flush(state).document +} + +fn add_section( + sections: Dict(String, List(Object)), + name: String, + objects: List(Object), +) -> Dict(String, List(Object)) { + dict.upsert(sections, name, fn(existing) { + case existing { + option.Some(previous) -> list.append(previous, objects) + option.None -> objects + } + }) +} + +fn bump(counts: Dict(String, Int), key: String) -> Dict(String, Int) { + dict.upsert(counts, key, fn(existing) { + case existing { + option.Some(n) -> n + 1 + option.None -> 1 + } + }) +} + +fn parse_int(value: String) -> Int { + value |> string.trim |> int.parse |> result.unwrap(0) +} + +fn numbers(value: String) -> List(Int) { + value + |> string.split(on: " ") + |> list.filter(fn(s) { s != "" }) + |> list.map(parse_int) +} diff --git a/gleam/src/ocda/model.gleam b/gleam/src/ocda/model.gleam new file mode 100644 index 0000000..050a97b --- /dev/null +++ b/gleam/src/ocda/model.gleam @@ -0,0 +1,278 @@ +//// The object-centric document abstraction, as Gleam types. +//// +//// A deliberate mirror of the D library's `ocda.model`, field for field. +//// The two are meant to agree, and the cheapest way to make that testable +//// is for them to hold the same things under the same names: the dump of a +//// loaded document is then identical by construction rather than by +//// negotiation. +//// +//// Written against specs/sisudoc-abstraction-format: `ocda-model.md` for +//// the model, the two format specifications for what each serialisation +//// says about it. + +import gleam/dict.{type Dict} +import gleam/list +import gleam/string + +/// The format major version this library knows. +/// +/// The `.ssp` and the `.ocda.db` carry one version between them, declared +/// in band. The minor part rises when a property is added and a reader of +/// the newer format still reads the older, so only the major is checked. +pub const format_major_read = "2" + +/// The nine sections, in the order the format gives them, which is not the +/// order a sorted key list or a `group by` would give. +pub const section_names = [ + "head", "toc", "body", "endnotes", "glossary", "bibliography", "bookindex", + "blurb", "tail", +] + +/// The four blocks that are document preamble rather than sections of +/// objects. +pub const preamble_blocks = ["source", "meta", "make", "doc_has"] + +/// A repeatable `.image` record: the file as the abstraction found it. +pub type Image { + Image( + /// filename as the object refers to it + name: String, + /// hex, uppercase + sha256: String, + bytes: Int, + /// the file's own pixel dimensions, not any display width + width: Int, + height: Int, + /// absent or unreadable, so no digest and no size + missing: Bool, + /// any further key:value fields, kept rather than dropped + fields: Dict(String, String), + ) +} + +pub fn new_image() -> Image { + Image( + name: "", + sha256: "", + bytes: 0, + width: 0, + height: 0, + missing: False, + fields: dict.new(), + ) +} + +/// One object of the abstraction. +/// +/// `text` is one string with the lines joined by newline, as in the D +/// library and for the same reason: an object whose only text line is +/// blank is then indistinguishable from an object with no text, in both +/// libraries, in the same place. A difference that exists in one language +/// and not the other would be worse than a loss both share. +pub type Object { + Object( + ocn: Int, + /// one of the sixteen kinds + is_a: String, + /// the section this object was found in + section: String, + /// headings: A B C D 1 2 3 4 + heading_level: String, + identifier: String, + part: String, + /// how the text behaves: para, block, comment + is_of_type: String, + /// the object's own section, when it differs from the block it sits in + own_section: String, + segment_html_is: String, + segment_epub: String, + segment_epub_is: String, + table_aligns: String, + stow_link: List(String), + anchor_tags: List(String), + parent_ocn: Int, + last_descendant_ocn: Int, + children: List(Int), + ancestors: List(Int), + dummy_heading: Bool, + ocn_off: Bool, + indent_base: Int, + indent_hang: Int, + bullet: Bool, + lang: String, + has_links: Bool, + has_notes_reg: Bool, + has_notes_star: Bool, + has_images: Bool, + segment: String, + segment_prev: String, + segment_next: String, + anchor: String, + heading_lev_anchor: String, + table_cols: Int, + table_widths: String, + table_header: Bool, + code_syntax: String, + code_linenumbers: Bool, + heading_ancestors_text: String, + sha256: String, + images: List(Image), + text: String, + ) +} + +pub fn new_object() -> Object { + Object( + ocn: 0, + is_a: "", + section: "", + heading_level: "", + identifier: "", + part: "", + is_of_type: "", + own_section: "", + segment_html_is: "", + segment_epub: "", + segment_epub_is: "", + table_aligns: "", + stow_link: [], + anchor_tags: [], + parent_ocn: 0, + last_descendant_ocn: 0, + children: [], + ancestors: [], + dummy_heading: False, + ocn_off: False, + indent_base: 0, + indent_hang: 0, + bullet: False, + lang: "", + has_links: False, + has_notes_reg: False, + has_notes_star: False, + has_images: False, + segment: "", + segment_prev: "", + segment_next: "", + anchor: "", + heading_lev_anchor: "", + table_cols: 0, + table_widths: "", + table_header: False, + code_syntax: "", + code_linenumbers: False, + heading_ancestors_text: "", + sha256: "", + images: [], + text: "", + ) +} + +/// Whether this object is rendered at all. +/// +/// `is_of_type` "comment" is markup commentary: carried by the abstraction +/// so that nothing is lost, and not put into output. An output that +/// renders it leaks the author's notes into the document, which is what +/// spine itself does not do. +pub fn renders(object: Object) -> Bool { + object.is_of_type != "comment" +} + +/// The section this object belongs to: the block it sits in, unless it +/// says otherwise. +pub fn section_of(object: Object) -> String { + case object.own_section { + "" -> object.section + own -> own + } +} + +/// What the loader met and did not understand. +/// +/// A loader that silently drops what it does not know cannot say whether +/// it understood the document, and so cannot be used to check a +/// specification against an implementation: silence and understanding look +/// alike. +pub type Unknown { + Unknown( + /// `.name` -> times seen + property: Dict(String, Int), + /// `@name` -> times seen + block: Dict(String, Int), + /// first character -> times seen, for lines of no known shape + line: Dict(String, Int), + ) +} + +pub fn new_unknown() -> Unknown { + Unknown(property: dict.new(), block: dict.new(), line: dict.new()) +} + +pub fn any_unknown(unknown: Unknown) -> Bool { + dict.size(unknown.property) > 0 + || dict.size(unknown.block) > 0 + || dict.size(unknown.line) > 0 +} + +pub type Document { + Document( + /// as declared on line 1 of a `.ssp` or in `schema.version`, "" when + /// absent + format_version: String, + source_filename: String, + /// language, languages, digest + source: Dict(String, String), + unknown: Unknown, + metadata: Dict(String, String), + make: Dict(String, String), + doc_has: Dict(String, String), + /// section name -> objects, in document order + sections: Dict(String, List(Object)), + ) +} + +pub fn new_document() -> Document { + Document( + format_version: "", + source_filename: "", + source: dict.new(), + unknown: new_unknown(), + metadata: dict.new(), + make: dict.new(), + doc_has: dict.new(), + sections: dict.new(), + ) +} + +/// The major part of the declared version, "" when none was declared. +pub fn version_major(document: Document) -> String { + case string.split_once(document.format_version, ".") { + Ok(#(major, _)) -> major + Error(_) -> document.format_version + } +} + +pub fn meta(document: Document, key: String) -> String { + case dict.get(document.metadata, key) { + Ok(value) -> value + Error(_) -> "" + } +} + +/// The objects of a named section, empty when the document has none. +pub fn section(document: Document, name: String) -> List(Object) { + case dict.get(document.sections, name) { + Ok(objects) -> objects + Error(_) -> [] + } +} + +/// Is this one of the nine sections of the format? +pub fn is_section(name: String) -> Bool { + list.contains(section_names, name) +} + +/// Is this one of the four preamble blocks? +pub fn is_preamble_block(name: String) -> Bool { + list.contains(preamble_blocks, name) +} diff --git a/gleam/src/ocda/zip.gleam b/gleam/src/ocda/zip.gleam new file mode 100644 index 0000000..5ce6e09 --- /dev/null +++ b/gleam/src/ocda/zip.gleam @@ -0,0 +1,170 @@ +//// Enough of the zip format to find one file in a pod and read it. +//// +//// Not a zip library. A pod archive holds a handful of entries, spine writes +//// them **stored** (uncompressed) so that the outer zstd frame does the +//// compressing once over the whole thing, and what is wanted from it here is +//// one named entry. +//// +//// Deflate is *refused rather than mishandled*. Erlang's `zlib` could +//// inflate it, but a library that says it read an entry it in fact skipped is +//// worse than one that says it cannot, and spine's own archives do not need +//// it. + +import gleam/bit_array +import gleam/int +import gleam/list +import gleam/result +import gleam/string + +pub type Entry { + Entry( + name: String, + /// 0 stored, 8 deflate + method: Int, + compressed: Int, + uncompressed: Int, + local_offset: Int, + ) +} + +const eocd_sig = 0x06054b50 + +const cd_sig = 0x02014b50 + +const local_sig = 0x04034b50 + +/// The entries, from the central directory at the end of the file. +pub fn entries(bytes: BitArray) -> Result(List(Entry), String) { + let size = bit_array.byte_size(bytes) + use _ <- result.try(case size >= 22 { + True -> Ok(Nil) + False -> Error("too short to be a zip archive") + }) + use eocd <- result.try(find_eocd(bytes, size - 22, lowest(size))) + use count <- result.try(u16(bytes, eocd + 10)) + use offset <- result.try(u32(bytes, eocd + 16)) + read_central(bytes, offset, count, []) +} + +/// The end of central directory record is last, and may be followed by a +/// comment of up to 65535 bytes, so it is searched for backwards. +fn find_eocd(bytes: BitArray, at: Int, lowest: Int) -> Result(Int, String) { + case at < lowest { + True -> + Error( + "no end of central directory record; it does not look like a zip archive", + ) + False -> + case u32(bytes, at) { + Ok(sig) if sig == eocd_sig -> Ok(at) + _ -> find_eocd(bytes, at - 1, lowest) + } + } +} + +fn lowest(size: Int) -> Int { + case size > 22 + 65_535 { + True -> size - 22 - 65_535 + False -> 0 + } +} + +fn read_central( + bytes: BitArray, + at: Int, + remaining: Int, + acc: List(Entry), +) -> Result(List(Entry), String) { + case remaining { + 0 -> Ok(list.reverse(acc)) + _ -> { + use sig <- result.try(u32(bytes, at)) + use _ <- result.try(case sig == cd_sig { + True -> Ok(Nil) + False -> + Error( + "central directory entry " + <> int.to_string(list.length(acc)) + <> " is malformed", + ) + }) + use method <- result.try(u16(bytes, at + 10)) + use compressed <- result.try(u32(bytes, at + 20)) + use uncompressed <- result.try(u32(bytes, at + 24)) + use name_len <- result.try(u16(bytes, at + 28)) + use extra_len <- result.try(u16(bytes, at + 30)) + use cmnt_len <- result.try(u16(bytes, at + 32)) + use local_offset <- result.try(u32(bytes, at + 42)) + use name <- result.try(text(bytes, at + 46, name_len)) + let entry = + Entry( + name: name, + method: method, + compressed: compressed, + uncompressed: uncompressed, + local_offset: local_offset, + ) + read_central( + bytes, + at + 46 + name_len + extra_len + cmnt_len, + remaining - 1, + [entry, ..acc], + ) + } + } +} + +/// One entry's bytes, from its local header. +pub fn read(bytes: BitArray, entry: Entry) -> Result(BitArray, String) { + use sig <- result.try(u32(bytes, entry.local_offset)) + use _ <- result.try(case sig == local_sig { + True -> Ok(Nil) + False -> Error(entry.name <> ": local header is malformed") + }) + use name_len <- result.try(u16(bytes, entry.local_offset + 26)) + use extra_len <- result.try(u16(bytes, entry.local_offset + 28)) + use _ <- result.try(case entry.method { + 0 -> Ok(Nil) + m -> + Error( + entry.name + <> ": compression method " + <> int.to_string(m) + <> "; this reader takes only stored entries, which is what spine writes", + ) + }) + let start = entry.local_offset + 30 + name_len + extra_len + case bit_array.slice(bytes, start, entry.compressed) { + Ok(slice) -> Ok(slice) + Error(_) -> Error(entry.name <> ": runs past the end of the archive") + } +} + +/// The entries whose names end this way, in archive order. Used to find +/// media/abstraction/<uid>.<lang>.ssp without knowing the uid. +pub fn find_suffix(entries: List(Entry), suffix: String) -> List(Entry) { + list.filter(entries, fn(e) { string.ends_with(e.name, suffix) }) +} + +fn u16(bytes: BitArray, at: Int) -> Result(Int, String) { + case bit_array.slice(bytes, at, 2) { + Ok(<<v:little-size(16)>>) -> Ok(v) + _ -> Error("read past the end of the archive at " <> int.to_string(at)) + } +} + +fn u32(bytes: BitArray, at: Int) -> Result(Int, String) { + case bit_array.slice(bytes, at, 4) { + Ok(<<v:little-size(32)>>) -> Ok(v) + _ -> Error("read past the end of the archive at " <> int.to_string(at)) + } +} + +fn text(bytes: BitArray, at: Int, len: Int) -> Result(String, String) { + case bit_array.slice(bytes, at, len) { + Ok(slice) -> + bit_array.to_string(slice) + |> result.replace_error("an entry name is not valid UTF-8") + Error(_) -> Error("an entry name runs past the end of the archive") + } +} diff --git a/gleam/src/ocda_lib.gleam b/gleam/src/ocda_lib.gleam new file mode 100644 index 0000000..1e8a5ae --- /dev/null +++ b/gleam/src/ocda_lib.gleam @@ -0,0 +1,136 @@ +//// ocda-lib, in Gleam: the abstraction as a library, put to the question. +//// +//// A library has no binary, so this is the two programs that test its +//// claims. +//// +//// ocda-lib dump [--unknown] <file>.ssp|<file>.ocda.db [lang] +//// ocda-lib agree <doc>.ssp <doc>.ocda.db [lang] +//// +//// `dump` writes the whole loaded model out canonically, byte for byte as +//// the D library's `ocda-dump` writes it, so that the two implementations +//// can be diffed. `agree` asks the same question the D library's +//// `ocda-agree` asks, within this language: does loading a document's +//// `.ssp` give what loading its `.ocda.db` gives? + +import argv +import gleam/int +import gleam/io +import gleam/list +import gleam/string +import ocda/dump +import ocda/load +import ocda/model.{type Document} + +pub fn main() { + case argv.load().arguments { + ["dump", "--unknown", path] -> dump_file(path, "", True) + ["dump", "--unknown", path, lang] -> dump_file(path, lang, True) + ["dump", path] -> dump_file(path, "", False) + ["dump", path, lang] -> dump_file(path, lang, False) + ["agree", ssp, db] -> agree(ssp, db, "") + ["agree", ssp, db, lang] -> agree(ssp, db, lang) + _ -> usage() + } +} + +fn usage() -> Nil { + io.println_error( + "usage: ocda-lib dump [--unknown] <file>.ssp|<file>.ocda.db [lang] + ocda-lib agree <doc>.ssp <doc>.ocda.db [lang]", + ) + halt(2) +} + +fn dump_file(path: String, lang: String, unknown: Bool) -> Nil { + case load.load(path, lang, False) { + Error(message) -> fail(message) + Ok(doc) -> + case unknown { + True -> io.print(dump.unknown(doc)) + False -> io.print(dump.document(doc)) + } + } +} + +/// The library's whole purpose is that a writer cannot tell which +/// serialisation it was handed. That is only true if the two loaders give +/// the same model, so this asks, and says where they part company rather +/// than only that they did. +fn agree(ssp_path: String, db_path: String, lang: String) -> Nil { + case load.load(ssp_path, "", False), load.load(db_path, lang, False) { + Error(message), _ -> fail(message) + _, Error(message) -> fail(message) + Ok(from_ssp), Ok(from_db) -> { + let differences = compare(from_ssp, from_db) + case differences { + [] -> io.println("agree: the two loaders give the same model") + _ -> { + list.each(list.take(differences, 12), io.println) + case list.length(differences) > 12 { + True -> io.println(" ... stopping after 12") + False -> Nil + } + io.println(int.to_string(list.length(differences)) <> " difference(s)") + halt(1) + } + } + } + } +} + +/// Compared through the dump rather than field by field, which is both +/// shorter and stricter: the dump carries every field, so nothing can +/// differ without showing here. +fn compare(from_ssp: Document, from_db: Document) -> List(String) { + let a = dump.document(from_ssp) + let b = dump.document(from_db) + case a == b { + True -> [] + False -> diff_lines(a, b) + } +} + +fn diff_lines(a: String, b: String) -> List(String) { + let la = split_lines(a) + let lb = split_lines(b) + case list.length(la) == list.length(lb) { + False -> [ + " differ: line count", + " .ssp: " <> int.to_string(list.length(la)), + " .ocda.db: " <> int.to_string(list.length(lb)), + ..pairwise(la, lb) + ] + True -> pairwise(la, lb) + } +} + +fn pairwise(la: List(String), lb: List(String)) -> List(String) { + list.zip(la, lb) + |> list.index_map(fn(pair, i) { #(i, pair.0, pair.1) }) + |> list.filter(fn(t) { t.1 != t.2 }) + |> list.map(fn(t) { + " differ: line " + <> int.to_string(t.0 + 1) + <> "\n .ssp: " + <> t.1 + <> "\n .ocda.db: " + <> t.2 + }) +} + +fn split_lines(text: String) -> List(String) { + case text { + "" -> [] + _ -> string.split(text, on: "\n") + } +} + +fn fail(message: String) -> Nil { + io.println_error("error: " <> message) + halt(2) +} + +/// Gleam has no standard exit, and the exit status is what a sweep script +/// reads. +@external(erlang, "erlang", "halt") +fn halt(code: Int) -> Nil diff --git a/gleam/test/ocda_lib_test.gleam b/gleam/test/ocda_lib_test.gleam new file mode 100644 index 0000000..02fe463 --- /dev/null +++ b/gleam/test/ocda_lib_test.gleam @@ -0,0 +1,148 @@ +import gleam/dict +import gleam/list +import simplifile +import gleam/string +import gleeunit +import gleeunit/should +import ocda/dump +import ocda/load +import ocda/load_ssp +import ocda/model + +pub fn main() { + gleeunit.main() +} + +const minimal = "% SiSU Document Abstraction v2.0 +% Source: example.sst + +@source { + language: en + languages: en + digest: ABC +} + +@meta { + title.main: An Example +} + +@body { +[1] heading :A the_title +.part: frontmatter +.sha256: DEAD +| An Example + +[2] para +.part: body +.parent: 1 +| First line +| +| Third line, after a blank one + +} +" + +pub fn version_test() { + let doc = load_ssp.parse(minimal) + doc.format_version |> should.equal("2.0") + model.version_major(doc) |> should.equal("2") + doc.source_filename |> should.equal("example.sst") +} + +pub fn preamble_test() { + let doc = load_ssp.parse(minimal) + model.meta(doc, "title.main") |> should.equal("An Example") +} + +pub fn objects_test() { + let doc = load_ssp.parse(minimal) + let body = model.section(doc, "body") + body |> list.length |> should.equal(2) +} + +/// The blank line in the middle of an object's text is kept, and so is the +/// count: three lines, not two. +pub fn blank_text_line_test() { + let doc = load_ssp.parse(minimal) + case model.section(doc, "body") { + [_, second] -> + second.text + |> should.equal("First line\n\nThird line, after a blank one") + _ -> should.fail() + } +} + +/// A heading's identifier comes from its declaration line; an object whose +/// identifier is simply its ocn has it derived, because the text form does +/// not write what it can work out. +pub fn derived_identifier_test() { + let doc = load_ssp.parse(minimal) + case model.section(doc, "body") { + [first, second] -> { + first.identifier |> should.equal("the_title") + second.identifier |> should.equal("2") + } + _ -> should.fail() + } +} + +pub fn source_of_test() { + load.source_of("a/b.ssp") |> should.equal(load.Ssp) + load.source_of("a/b.ocda.db") |> should.equal(load.OcdaDb) + load.source_of("a/b.sst") |> should.equal(load.Unknown) +} + +/// The dump omits a field at its default, so a minimal document gives a +/// short dump and every line in it is a fact. +pub fn dump_test() { + let text = dump.document(load_ssp.parse(minimal)) + text |> string.contains(contain: "version 2.0") |> should.be_true + text |> string.contains(contain: "meta title.main An Example") |> should.be_true + text |> string.contains(contain: "object body 0") |> should.be_true + text |> string.contains(contain: "level A") |> should.be_true + // a false boolean is not written at all + text |> string.contains(contain: "bullet") |> should.be_false +} + + +/// A property is read, which the fixture above had to be corrected to show: +/// a property line is `.name: value` at the start of the line, and an +/// indented one is a line of no known shape. The fixture indented them, so +/// every property in it was being counted as malformed and the tests around +/// it passed for the wrong reason. +pub fn property_test() { + let doc = load_ssp.parse(minimal) + case model.section(doc, "body") { + [first, second] -> { + first.part |> should.equal("frontmatter") + first.sha256 |> should.equal("DEAD") + second.parent_ocn |> should.equal(1) + } + _ -> should.fail() + } + // and nothing in the fixture is malformed any more + dict_size(doc.unknown.line) |> should.equal(0) +} + +@external(erlang, "maps", "size") +fn dict_size(d: dict.Dict(a, b)) -> Int + +/// What a path is, including the two archive forms. +pub fn pod_source_test() { + load.source_of("a/b.sisupod") |> should.equal(load.Pod) + load.source_of("a/b.zip") |> should.equal(load.Pod) +} + +/// A zstd-wrapped archive is refused, and the refusal says why rather than +/// claiming the file is malformed. This is the one place the D and Gleam +/// libraries differ in what they accept, and it is a packaging limit: the +/// only zstd on the BEAM is a NIF that fetches zstd in its own build. +pub fn zstd_refusal_test() { + let path = "test/fixture-zstd-frame.sisupod" + let assert Ok(_) = simplifile.write_bits(path, <<0x28, 0xB5, 0x2F, 0xFD, 0>>) + let assert Error(message) = load.load(path, "", False) + message |> string.contains(contain: "zstd") |> should.be_true + message |> string.contains(contain: "zstd -d") |> should.be_true + let _ = simplifile.delete(path) + Nil +} diff --git a/org/COPYRIGHT b/org/COPYRIGHT new file mode 100644 index 0000000..23b22f5 --- /dev/null +++ b/org/COPYRIGHT @@ -0,0 +1,19 @@ +- Name: sisudoc-ocda-lib + - Description: part of SiSU spine (Doc Reform): documents, structuring, + processing, publishing and search + - [https://www.sisudoc.org] + + - Author: Ralph Amissah + [ralph.amissah@gmail.com] + + - Copyright: (C) 2026 Ralph Amissah, All Rights Reserved. + + - Spine, Doc Reform (related to SiSU) uses standard: + - docReform markup syntax (based on SiSU markup) + - standard SiSU markup syntax with modified headers and minor + modifications + - docReform object numbering (based on SiSU object citation numbering) + - standard SiSU document object numbering + + - Homepages: + [https://www.sisudoc.org] diff --git a/org/config-lib.org b/org/config-lib.org new file mode 100644 index 0000000..7b7ba18 --- /dev/null +++ b/org/config-lib.org @@ -0,0 +1,279 @@ +-*- mode: org -*- +#+TITLE: configuration: .gitignore and copyright +#+DESCRIPTION: setup only; the code, READMEs and build files are kept as files +#+FILETAGS: :spine:build:tools: +#+AUTHOR: Ralph Amissah +#+EMAIL: [[mailto:ralph.amissah@gmail.com][ralph.amissah@gmail.com]] +#+COPYRIGHT: Copyright (C) 2026 Ralph Amissah +#+LANGUAGE: en +#+STARTUP: show3levels hideblocks hidestars noindent entitiespretty +#+PROPERTY: header-args+ :eval never-export :exports code +#+PROPERTY: header-args+ :noweb yes :padline no +#+PROPERTY: header-args+ :results silent :cache no +#+PROPERTY: header-args+ :mkdirp yes +#+OPTIONS: H:3 num:nil toc:t \n:t ::t |:t ^:nil -:t f:t *:t + +* ./ +** git .gitignore :gitignore: + +- git ls-files --others --exclude-from=.git/info/exclude +- git ls-files --others --ignored --exclude-standard --directory +- git check-ignore -v flake.lock +- git clean -ix +- find . -name "*_" | xargs rm -rf +- find . -name "*.o" | xargs rm -rf + +#+HEADER: :tangle "../.gitignore" +#+BEGIN_SRC gitignore +# Deny everything, then whitelist what belongs in the repository. +# +# The practice is deliberate: a repository should hold what was decided to +# put in it, not whatever happened to be in the directory when someone ran +# "git add .". Build output, editor droppings, downloaded dependencies and +# exported binaries are numerous, are regenerated, and differ between +# machines; source is few and is chosen. So the default is no, and every +# yes below is a decision that can be read. +# +# To see what is being excluded, and why: +# git status --ignored +# git check-ignore -v <path> +# git ls-files --others --ignored --exclude-standard --directory + +* + +# ── the repository itself ──────────────────────────────────────────────── +!.gitignore +!README.md +!LICENSE +!**/LICENSE + +# ── directories have to be walked into before a file inside can match ──── +!gleam/ +!dlang/ +!org/ +!gleam/**/ +!dlang/**/ +!org/**/ +# but not these, whatever they contain +build/ +.build/ +.direnv/ +.gleam/ +bin/ +result +result-* + +# ── documentation and notes, in any language directory ─────────────────── +!**/COPYRIGHT +!**/*.md +!**/*.org + +# ── nix ────────────────────────────────────────────────────────────────── +!**/*.nix +!**/flake.lock +!**/.envrc + +# ── dlang ──────────────────────────────────────────────────────────────── +!**/*.d +!**/dub.json +!**/dub.sdl +!**/dub.selections.json +*.o +*.a +*.lst + +# ── gleam ──────────────────────────────────────────────────────────────── +!**/*.gleam +# Erlang, for what Gleam reaches through FFI (a sha256, from OTP crypto) +!**/src/*.erl +# gleam.toml is the manifest, manifest.toml is the lockfile: both belong +!**/*.toml +*.beam +*.ez +erl_crash.dump + +# ── ruby, when it arrives ──────────────────────────────────────────────── +!**/*.rb +!**/Gemfile +!**/Gemfile.lock +!**/*.gemspec + +# ── continuous integration ─────────────────────────────────────────────── +!**/*.yml +!**/*.yaml +#+END_SRC + +* COPYRIGHT & LICENSE +** notices +*** project (project root) ./ + +#+HEADER: :tangle "../COPYRIGHT" +#+HEADER: :noweb yes +#+BEGIN_SRC text +- Name: <<project_name>> + <<sisudoc_spine_copyright>> + + <<sisudoc_spine_license_agpl3>> + + <<sisudoc_spine_summary>> +#+END_SRC + +#+HEADER: :tangle "../LICENSE" +#+HEADER: :noweb yes +#+BEGIN_SRC text +<<sisudoc_spine_license_agpl3_the_license>> +#+END_SRC + +*** code org ./org + +- .org literate programming: + - copyright asserted + - omitted + - license + - dependency information + +#+HEADER: :tangle "../org/COPYRIGHT" +#+HEADER: :noweb yes +#+BEGIN_SRC text +- Name: <<project_name>> + <<sisudoc_spine_copyright>> + + <<sisudoc_spine_summary>> +#+END_SRC + + +#+HEADER: :noweb yes +#+BEGIN_SRC text +- Name: <<project_name>> + <<sisudoc_spine_copyright>> + + <<sisudoc_spine_license_agpl3>> + + <<sisudoc_spine_summary>> +#+END_SRC + +*** code source +**** gleam + +#+HEADER: :tangle "../gleam/COPYRIGHT" +#+HEADER: :noweb yes +#+BEGIN_SRC org +- Name: <<project_name>> + <<sisudoc_spine_copyright>> + + <<sisudoc_spine_license_agpl3>> + + <<sisudoc_spine_summary>> + + - Copyright for ./nix-gleam/ 2023 Arnar Gauti Ingason + - License for ./nix-gleam/ MIT License +#+END_SRC + +***** gleam src + +#+HEADER: :tangle "../gleam/src/COPYRIGHT" +#+HEADER: :noweb yes +#+BEGIN_SRC org +- Name: <<project_name>> + <<sisudoc_spine_copyright>> + + <<sisudoc_spine_license_agpl3>> + + <<sisudoc_spine_summary>> +#+END_SRC + +**** dlang +***** dlang + +#+HEADER: :tangle "../dlang/COPYRIGHT" +#+HEADER: :noweb yes +#+BEGIN_SRC org +- Name: <<project_name>> + <<sisudoc_spine_copyright>> + + <<sisudoc_spine_license_agpl3>> + + <<sisudoc_spine_summary>> +#+END_SRC + +***** dlang src + +#+HEADER: :tangle "../dlang/src/COPYRIGHT" +#+HEADER: :noweb yes +#+BEGIN_SRC org +- Name: <<project_name>> + <<sisudoc_spine_copyright>> + + <<sisudoc_spine_license_agpl3>> + + <<sisudoc_spine_summary>> +#+END_SRC + +* org includes +** project name + +#+NAME: project_name +#+HEADER: :noweb yes +#+BEGIN_SRC emacs-lisp +<<./sisudoc_spine_version_info_and_doc_header_including_copyright_and_license.org:project_name()>> +#+END_SRC + +** project version + +#+NAME: spine_version +#+HEADER: :noweb yes +#+BEGIN_SRC emacs-lisp +<<./sisudoc_spine_version_info_and_doc_header_including_copyright_and_license.org:spine_project_version()>> +#+END_SRC + +** ocda ssp format version + +#+NAME: ocda_ssp_format_version +#+HEADER: :noweb yes +#+BEGIN_SRC emacs-lisp +<<./sisudoc_spine_version_info_and_doc_header_including_copyright_and_license.org:ocda_ssp_format_version()>> +#+END_SRC + +** copyright + +#+NAME: sisudoc_spine_copyright +#+BEGIN_SRC emacs-lisp +<<./sisudoc_spine_version_info_and_doc_header_including_copyright_and_license.org:sisudoc_spine_copyright()>> +#+END_SRC + +** license agpl3 + +#+NAME: sisudoc_spine_license_agpl3 +#+BEGIN_SRC emacs-lisp +<<./sisudoc_spine_version_info_and_doc_header_including_copyright_and_license.org:sisudoc_spine_license_agpl3()>> +#+END_SRC + +#+NAME: sisudoc_spine_license_agpl3_the_license +#+BEGIN_SRC emacs-lisp +<<./sisudoc_spine_version_info_and_doc_header_including_copyright_and_license.org:sisudoc_spine_license_agpl3_the_license()>> +#+END_SRC + +** spine_summary + +#+NAME: sisudoc_spine_summary +#+BEGIN_SRC emacs-lisp +<<./sisudoc_spine_version_info_and_doc_header_including_copyright_and_license.org:sisudoc_spine_summary()>> +#+END_SRC + +** year + +#+NAME: year +#+HEADER: :noweb yes +#+BEGIN_SRC emacs-lisp +<<./sisudoc_spine_version_info_and_doc_header_including_copyright_and_license.org:year()>> +#+END_SRC + +* nixpkgs source + +#+NAME: nixpkgs_unstable +#+HEADER: :noweb yes +#+BEGIN_SRC emacs-lisp +<<./sisudoc_spine_version_info_and_doc_header_including_copyright_and_license.org:nixpkgs_unstable()>> +#+END_SRC + +* __END__ diff --git a/org/sisudoc_spine_version_info_and_doc_header_including_copyright_and_license.org b/org/sisudoc_spine_version_info_and_doc_header_including_copyright_and_license.org new file mode 100644 index 0000000..7b6fd66 --- /dev/null +++ b/org/sisudoc_spine_version_info_and_doc_header_including_copyright_and_license.org @@ -0,0 +1,972 @@ +-*- mode: org -*- +#+TITLE: sisudoc spine (doc_reform) object-centric document abstraction +#+DESCRIPTION: documents - structuring, publishing in multiple formats & search +#+FILETAGS: :spine:abstraction: +#+AUTHOR: Ralph Amissah +#+EMAIL: [[mailto:ralph.amissah@gmail.com][ralph.amissah@gmail.com]] +#+COPYRIGHT: Copyright (C) 2026 Ralph Amissah +#+LANGUAGE: en +#+STARTUP: content hideblocks hidestars noindent entitiespretty +#+PROPERTY: header-args+ :eval never-export :exports code +#+PROPERTY: header-args+ :noweb yes :padline no +#+PROPERTY: header-args+ :results silent :cache no +#+PROPERTY: header-args+ :mkdirp yes +#+OPTIONS: H:3 num:nil toc:t \n:t ::t |:t ^:nil -:t f:t *:t + +- [[./doc-reform.org][doc-reform.org]] [[./][org/]] + +* spine doc header including copyright & license + +#+NAME: spine_doc_header_including_copyright_and_license +#+HEADER: :noweb yes +#+BEGIN_SRC org +/+ +- Name: <<project_name>> + - Description: <<project_description>> + - static content generator + + - Author: Ralph Amissah + [ralph.amissah@gmail.com] + + - Copyright: (C) <<project_year_started>> Ralph Amissah, All Rights Reserved. + + - License: AGPL 3 or later, for everything in this repository unless a file or + a directory specifies otherwise.: + + Spine (SiSU), a framework for document structuring, processing, publishing + and search + + Copyright (C) Ralph Amissah + + This program is free software: you can redistribute it and/or modify it + under the terms of the GNU Affero General Public License (AGPLv3) as + published by the Free Software Foundation (FSF), either version 3 of the + License, or (at your option) any later version. + + This program is distributed in the hope that it will be useful, but WITHOUT + ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License + for more details. + + You should have received a copy of the GNU Affero General Public License + along with this program. If not, see [https://www.gnu.org/licenses/]. + + If you have Internet connection, the latest version of the AGPL should be + available at: [https://www.gnu.org/licenses/agpl-3.0.html] + + - Spine (by Doc Reform, related to SiSU) uses standard: + - docReform markup syntax + - standard SiSU markup syntax with modified headers and minor modifications + - docReform object numbering + - standard SiSU object citation numbering & system + + - Homepages: + [https://www.sisudoc.org] + [https://www.doc-reform.org] + + - Git + [https://git.sisudoc.org/] + ++/ +#+END_SRC + +** incorporate +*** copyright + +#+NAME: sisudoc_spine_copyright +#+BEGIN_SRC org +- Description: part of SiSU spine (Doc Reform): documents, structuring, + processing, publishing and search + - [https://www.sisudoc.org] + +- Author: Ralph Amissah + [ralph.amissah@gmail.com] + +- Copyright: (C) <<project_year_started>> Ralph Amissah, All Rights Reserved. +#+END_SRC + +*** license +**** AGPLv3 notice + +#+NAME: sisudoc_spine_license_agpl3 +#+HEADER: :noweb yes +#+BEGIN_SRC org +- License: AGPL 3 or later, for everything in this repository unless a file or + a directory specifies otherwise.: + + <<project_name>> + + <<project_description>> + + Copyright (C) Ralph Amissah + + This program is free software: you can redistribute it and/or modify it + under the terms of the GNU Affero General Public License (AGPLv3) as + published by the Free Software Foundation, either version 3 of the License, + or (at your option) any later version. + + This program is distributed in the hope that it will be useful, but + WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY + or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public + License for more details. + + You should have received a copy of the GNU Affero General Public License + along with this program. If not, see [https://www.gnu.org/licenses/]. + + If you have Internet connection, the latest version of the AGPL should be + available at: [https://www.gnu.org/licenses/agpl-3.0.html] +#+END_SRC + +**** the AGPLv3 License + +#+NAME: sisudoc_spine_license_agpl3_the_license +#+BEGIN_SRC org +# GNU AFFERO GENERAL PUBLIC LICENSE + +Version 3, 19 November 2007 + +Copyright (C) 2007 Free Software Foundation, Inc. +<https://fsf.org/> + +Everyone is permitted to copy and distribute verbatim copies of this +license document, but changing it is not allowed. + +## Preamble + +The GNU Affero General Public License is a free, copyleft license for +software and other kinds of works, specifically designed to ensure +cooperation with the community in the case of network server software. + +The licenses for most software and other practical works are designed +to take away your freedom to share and change the works. By contrast, +our General Public Licenses are intended to guarantee your freedom to +share and change all versions of a program--to make sure it remains +free software for all its users. + +When we speak of free software, we are referring to freedom, not +price. Our General Public Licenses are designed to make sure that you +have the freedom to distribute copies of free software (and charge for +them if you wish), that you receive source code or can get it if you +want it, that you can change the software or use pieces of it in new +free programs, and that you know you can do these things. + +Developers that use our General Public Licenses protect your rights +with two steps: (1) assert copyright on the software, and (2) offer +you this License which gives you legal permission to copy, distribute +and/or modify the software. + +A secondary benefit of defending all users' freedom is that +improvements made in alternate versions of the program, if they +receive widespread use, become available for other developers to +incorporate. Many developers of free software are heartened and +encouraged by the resulting cooperation. However, in the case of +software used on network servers, this result may fail to come about. +The GNU General Public License permits making a modified version and +letting the public access it on a server without ever releasing its +source code to the public. + +The GNU Affero General Public License is designed specifically to +ensure that, in such cases, the modified source code becomes available +to the community. It requires the operator of a network server to +provide the source code of the modified version running there to the +users of that server. Therefore, public use of a modified version, on +a publicly accessible server, gives the public access to the source +code of the modified version. + +An older license, called the Affero General Public License and +published by Affero, was designed to accomplish similar goals. This is +a different license, not a version of the Affero GPL, but Affero has +released a new version of the Affero GPL which permits relicensing +under this license. + +The precise terms and conditions for copying, distribution and +modification follow. + +## TERMS AND CONDITIONS + +### 0. Definitions. + +"This License" refers to version 3 of the GNU Affero General Public +License. + +"Copyright" also means copyright-like laws that apply to other kinds +of works, such as semiconductor masks. + +"The Program" refers to any copyrightable work licensed under this +License. Each licensee is addressed as "you". "Licensees" and +"recipients" may be individuals or organizations. + +To "modify" a work means to copy from or adapt all or part of the work +in a fashion requiring copyright permission, other than the making of +an exact copy. The resulting work is called a "modified version" of +the earlier work or a work "based on" the earlier work. + +A "covered work" means either the unmodified Program or a work based +on the Program. + +To "propagate" a work means to do anything with it that, without +permission, would make you directly or secondarily liable for +infringement under applicable copyright law, except executing it on a +computer or modifying a private copy. Propagation includes copying, +distribution (with or without modification), making available to the +public, and in some countries other activities as well. + +To "convey" a work means any kind of propagation that enables other +parties to make or receive copies. Mere interaction with a user +through a computer network, with no transfer of a copy, is not +conveying. + +An interactive user interface displays "Appropriate Legal Notices" to +the extent that it includes a convenient and prominently visible +feature that (1) displays an appropriate copyright notice, and (2) +tells the user that there is no warranty for the work (except to the +extent that warranties are provided), that licensees may convey the +work under this License, and how to view a copy of this License. If +the interface presents a list of user commands or options, such as a +menu, a prominent item in the list meets this criterion. + +### 1. Source Code. + +The "source code" for a work means the preferred form of the work for +making modifications to it. "Object code" means any non-source form of +a work. + +A "Standard Interface" means an interface that either is an official +standard defined by a recognized standards body, or, in the case of +interfaces specified for a particular programming language, one that +is widely used among developers working in that language. + +The "System Libraries" of an executable work include anything, other +than the work as a whole, that (a) is included in the normal form of +packaging a Major Component, but which is not part of that Major +Component, and (b) serves only to enable use of the work with that +Major Component, or to implement a Standard Interface for which an +implementation is available to the public in source code form. A +"Major Component", in this context, means a major essential component +(kernel, window system, and so on) of the specific operating system +(if any) on which the executable work runs, or a compiler used to +produce the work, or an object code interpreter used to run it. + +The "Corresponding Source" for a work in object code form means all +the source code needed to generate, install, and (for an executable +work) run the object code and to modify the work, including scripts to +control those activities. However, it does not include the work's +System Libraries, or general-purpose tools or generally available free +programs which are used unmodified in performing those activities but +which are not part of the work. For example, Corresponding Source +includes interface definition files associated with source files for +the work, and the source code for shared libraries and dynamically +linked subprograms that the work is specifically designed to require, +such as by intimate data communication or control flow between those +subprograms and other parts of the work. + +The Corresponding Source need not include anything that users can +regenerate automatically from other parts of the Corresponding Source. + +The Corresponding Source for a work in source code form is that same +work. + +### 2. Basic Permissions. + +All rights granted under this License are granted for the term of +copyright on the Program, and are irrevocable provided the stated +conditions are met. This License explicitly affirms your unlimited +permission to run the unmodified Program. The output from running a +covered work is covered by this License only if the output, given its +content, constitutes a covered work. This License acknowledges your +rights of fair use or other equivalent, as provided by copyright law. + +You may make, run and propagate covered works that you do not convey, +without conditions so long as your license otherwise remains in force. +You may convey covered works to others for the sole purpose of having +them make modifications exclusively for you, or provide you with +facilities for running those works, provided that you comply with the +terms of this License in conveying all material for which you do not +control copyright. Those thus making or running the covered works for +you must do so exclusively on your behalf, under your direction and +control, on terms that prohibit them from making any copies of your +copyrighted material outside their relationship with you. + +Conveying under any other circumstances is permitted solely under the +conditions stated below. Sublicensing is not allowed; section 10 makes +it unnecessary. + +### 3. Protecting Users' Legal Rights From Anti-Circumvention Law. + +No covered work shall be deemed part of an effective technological +measure under any applicable law fulfilling obligations under article +11 of the WIPO copyright treaty adopted on 20 December 1996, or +similar laws prohibiting or restricting circumvention of such +measures. + +When you convey a covered work, you waive any legal power to forbid +circumvention of technological measures to the extent such +circumvention is effected by exercising rights under this License with +respect to the covered work, and you disclaim any intention to limit +operation or modification of the work as a means of enforcing, against +the work's users, your or third parties' legal rights to forbid +circumvention of technological measures. + +### 4. Conveying Verbatim Copies. + +You may convey verbatim copies of the Program's source code as you +receive it, in any medium, provided that you conspicuously and +appropriately publish on each copy an appropriate copyright notice; +keep intact all notices stating that this License and any +non-permissive terms added in accord with section 7 apply to the code; +keep intact all notices of the absence of any warranty; and give all +recipients a copy of this License along with the Program. + +You may charge any price or no price for each copy that you convey, +and you may offer support or warranty protection for a fee. + +### 5. Conveying Modified Source Versions. + +You may convey a work based on the Program, or the modifications to +produce it from the Program, in the form of source code under the +terms of section 4, provided that you also meet all of these +conditions: + +- a) The work must carry prominent notices stating that you modified + it, and giving a relevant date. +- b) The work must carry prominent notices stating that it is + released under this License and any conditions added under + section 7. This requirement modifies the requirement in section 4 + to "keep intact all notices". +- c) You must license the entire work, as a whole, under this + License to anyone who comes into possession of a copy. This + License will therefore apply, along with any applicable section 7 + additional terms, to the whole of the work, and all its parts, + regardless of how they are packaged. This License gives no + permission to license the work in any other way, but it does not + invalidate such permission if you have separately received it. +- d) If the work has interactive user interfaces, each must display + Appropriate Legal Notices; however, if the Program has interactive + interfaces that do not display Appropriate Legal Notices, your + work need not make them do so. + +A compilation of a covered work with other separate and independent +works, which are not by their nature extensions of the covered work, +and which are not combined with it such as to form a larger program, +in or on a volume of a storage or distribution medium, is called an +"aggregate" if the compilation and its resulting copyright are not +used to limit the access or legal rights of the compilation's users +beyond what the individual works permit. Inclusion of a covered work +in an aggregate does not cause this License to apply to the other +parts of the aggregate. + +### 6. Conveying Non-Source Forms. + +You may convey a covered work in object code form under the terms of +sections 4 and 5, provided that you also convey the machine-readable +Corresponding Source under the terms of this License, in one of these +ways: + +- a) Convey the object code in, or embodied in, a physical product + (including a physical distribution medium), accompanied by the + Corresponding Source fixed on a durable physical medium + customarily used for software interchange. +- b) Convey the object code in, or embodied in, a physical product + (including a physical distribution medium), accompanied by a + written offer, valid for at least three years and valid for as + long as you offer spare parts or customer support for that product + model, to give anyone who possesses the object code either (1) a + copy of the Corresponding Source for all the software in the + product that is covered by this License, on a durable physical + medium customarily used for software interchange, for a price no + more than your reasonable cost of physically performing this + conveying of source, or (2) access to copy the Corresponding + Source from a network server at no charge. +- c) Convey individual copies of the object code with a copy of the + written offer to provide the Corresponding Source. This + alternative is allowed only occasionally and noncommercially, and + only if you received the object code with such an offer, in accord + with subsection 6b. +- d) Convey the object code by offering access from a designated + place (gratis or for a charge), and offer equivalent access to the + Corresponding Source in the same way through the same place at no + further charge. You need not require recipients to copy the + Corresponding Source along with the object code. If the place to + copy the object code is a network server, the Corresponding Source + may be on a different server (operated by you or a third party) + that supports equivalent copying facilities, provided you maintain + clear directions next to the object code saying where to find the + Corresponding Source. Regardless of what server hosts the + Corresponding Source, you remain obligated to ensure that it is + available for as long as needed to satisfy these requirements. +- e) Convey the object code using peer-to-peer transmission, + provided you inform other peers where the object code and + Corresponding Source of the work are being offered to the general + public at no charge under subsection 6d. + +A separable portion of the object code, whose source code is excluded +from the Corresponding Source as a System Library, need not be +included in conveying the object code work. + +A "User Product" is either (1) a "consumer product", which means any +tangible personal property which is normally used for personal, +family, or household purposes, or (2) anything designed or sold for +incorporation into a dwelling. In determining whether a product is a +consumer product, doubtful cases shall be resolved in favor of +coverage. For a particular product received by a particular user, +"normally used" refers to a typical or common use of that class of +product, regardless of the status of the particular user or of the way +in which the particular user actually uses, or expects or is expected +to use, the product. A product is a consumer product regardless of +whether the product has substantial commercial, industrial or +non-consumer uses, unless such uses represent the only significant +mode of use of the product. + +"Installation Information" for a User Product means any methods, +procedures, authorization keys, or other information required to +install and execute modified versions of a covered work in that User +Product from a modified version of its Corresponding Source. The +information must suffice to ensure that the continued functioning of +the modified object code is in no case prevented or interfered with +solely because modification has been made. + +If you convey an object code work under this section in, or with, or +specifically for use in, a User Product, and the conveying occurs as +part of a transaction in which the right of possession and use of the +User Product is transferred to the recipient in perpetuity or for a +fixed term (regardless of how the transaction is characterized), the +Corresponding Source conveyed under this section must be accompanied +by the Installation Information. But this requirement does not apply +if neither you nor any third party retains the ability to install +modified object code on the User Product (for example, the work has +been installed in ROM). + +The requirement to provide Installation Information does not include a +requirement to continue to provide support service, warranty, or +updates for a work that has been modified or installed by the +recipient, or for the User Product in which it has been modified or +installed. Access to a network may be denied when the modification +itself materially and adversely affects the operation of the network +or violates the rules and protocols for communication across the +network. + +Corresponding Source conveyed, and Installation Information provided, +in accord with this section must be in a format that is publicly +documented (and with an implementation available to the public in +source code form), and must require no special password or key for +unpacking, reading or copying. + +### 7. Additional Terms. + +"Additional permissions" are terms that supplement the terms of this +License by making exceptions from one or more of its conditions. +Additional permissions that are applicable to the entire Program shall +be treated as though they were included in this License, to the extent +that they are valid under applicable law. If additional permissions +apply only to part of the Program, that part may be used separately +under those permissions, but the entire Program remains governed by +this License without regard to the additional permissions. + +When you convey a copy of a covered work, you may at your option +remove any additional permissions from that copy, or from any part of +it. (Additional permissions may be written to require their own +removal in certain cases when you modify the work.) You may place +additional permissions on material, added by you to a covered work, +for which you have or can give appropriate copyright permission. + +Notwithstanding any other provision of this License, for material you +add to a covered work, you may (if authorized by the copyright holders +of that material) supplement the terms of this License with terms: + +- a) Disclaiming warranty or limiting liability differently from the + terms of sections 15 and 16 of this License; or +- b) Requiring preservation of specified reasonable legal notices or + author attributions in that material or in the Appropriate Legal + Notices displayed by works containing it; or +- c) Prohibiting misrepresentation of the origin of that material, + or requiring that modified versions of such material be marked in + reasonable ways as different from the original version; or +- d) Limiting the use for publicity purposes of names of licensors + or authors of the material; or +- e) Declining to grant rights under trademark law for use of some + trade names, trademarks, or service marks; or +- f) Requiring indemnification of licensors and authors of that + material by anyone who conveys the material (or modified versions + of it) with contractual assumptions of liability to the recipient, + for any liability that these contractual assumptions directly + impose on those licensors and authors. + +All other non-permissive additional terms are considered "further +restrictions" within the meaning of section 10. If the Program as you +received it, or any part of it, contains a notice stating that it is +governed by this License along with a term that is a further +restriction, you may remove that term. If a license document contains +a further restriction but permits relicensing or conveying under this +License, you may add to a covered work material governed by the terms +of that license document, provided that the further restriction does +not survive such relicensing or conveying. + +If you add terms to a covered work in accord with this section, you +must place, in the relevant source files, a statement of the +additional terms that apply to those files, or a notice indicating +where to find the applicable terms. + +Additional terms, permissive or non-permissive, may be stated in the +form of a separately written license, or stated as exceptions; the +above requirements apply either way. + +### 8. Termination. + +You may not propagate or modify a covered work except as expressly +provided under this License. Any attempt otherwise to propagate or +modify it is void, and will automatically terminate your rights under +this License (including any patent licenses granted under the third +paragraph of section 11). + +However, if you cease all violation of this License, then your license +from a particular copyright holder is reinstated (a) provisionally, +unless and until the copyright holder explicitly and finally +terminates your license, and (b) permanently, if the copyright holder +fails to notify you of the violation by some reasonable means prior to +60 days after the cessation. + +Moreover, your license from a particular copyright holder is +reinstated permanently if the copyright holder notifies you of the +violation by some reasonable means, this is the first time you have +received notice of violation of this License (for any work) from that +copyright holder, and you cure the violation prior to 30 days after +your receipt of the notice. + +Termination of your rights under this section does not terminate the +licenses of parties who have received copies or rights from you under +this License. If your rights have been terminated and not permanently +reinstated, you do not qualify to receive new licenses for the same +material under section 10. + +### 9. Acceptance Not Required for Having Copies. + +You are not required to accept this License in order to receive or run +a copy of the Program. Ancillary propagation of a covered work +occurring solely as a consequence of using peer-to-peer transmission +to receive a copy likewise does not require acceptance. However, +nothing other than this License grants you permission to propagate or +modify any covered work. These actions infringe copyright if you do +not accept this License. Therefore, by modifying or propagating a +covered work, you indicate your acceptance of this License to do so. + +### 10. Automatic Licensing of Downstream Recipients. + +Each time you convey a covered work, the recipient automatically +receives a license from the original licensors, to run, modify and +propagate that work, subject to this License. You are not responsible +for enforcing compliance by third parties with this License. + +An "entity transaction" is a transaction transferring control of an +organization, or substantially all assets of one, or subdividing an +organization, or merging organizations. If propagation of a covered +work results from an entity transaction, each party to that +transaction who receives a copy of the work also receives whatever +licenses to the work the party's predecessor in interest had or could +give under the previous paragraph, plus a right to possession of the +Corresponding Source of the work from the predecessor in interest, if +the predecessor has it or can get it with reasonable efforts. + +You may not impose any further restrictions on the exercise of the +rights granted or affirmed under this License. For example, you may +not impose a license fee, royalty, or other charge for exercise of +rights granted under this License, and you may not initiate litigation +(including a cross-claim or counterclaim in a lawsuit) alleging that +any patent claim is infringed by making, using, selling, offering for +sale, or importing the Program or any portion of it. + +### 11. Patents. + +A "contributor" is a copyright holder who authorizes use under this +License of the Program or a work on which the Program is based. The +work thus licensed is called the contributor's "contributor version". + +A contributor's "essential patent claims" are all patent claims owned +or controlled by the contributor, whether already acquired or +hereafter acquired, that would be infringed by some manner, permitted +by this License, of making, using, or selling its contributor version, +but do not include claims that would be infringed only as a +consequence of further modification of the contributor version. For +purposes of this definition, "control" includes the right to grant +patent sublicenses in a manner consistent with the requirements of +this License. + +Each contributor grants you a non-exclusive, worldwide, royalty-free +patent license under the contributor's essential patent claims, to +make, use, sell, offer for sale, import and otherwise run, modify and +propagate the contents of its contributor version. + +In the following three paragraphs, a "patent license" is any express +agreement or commitment, however denominated, not to enforce a patent +(such as an express permission to practice a patent or covenant not to +sue for patent infringement). To "grant" such a patent license to a +party means to make such an agreement or commitment not to enforce a +patent against the party. + +If you convey a covered work, knowingly relying on a patent license, +and the Corresponding Source of the work is not available for anyone +to copy, free of charge and under the terms of this License, through a +publicly available network server or other readily accessible means, +then you must either (1) cause the Corresponding Source to be so +available, or (2) arrange to deprive yourself of the benefit of the +patent license for this particular work, or (3) arrange, in a manner +consistent with the requirements of this License, to extend the patent +license to downstream recipients. "Knowingly relying" means you have +actual knowledge that, but for the patent license, your conveying the +covered work in a country, or your recipient's use of the covered work +in a country, would infringe one or more identifiable patents in that +country that you have reason to believe are valid. + +If, pursuant to or in connection with a single transaction or +arrangement, you convey, or propagate by procuring conveyance of, a +covered work, and grant a patent license to some of the parties +receiving the covered work authorizing them to use, propagate, modify +or convey a specific copy of the covered work, then the patent license +you grant is automatically extended to all recipients of the covered +work and works based on it. + +A patent license is "discriminatory" if it does not include within the +scope of its coverage, prohibits the exercise of, or is conditioned on +the non-exercise of one or more of the rights that are specifically +granted under this License. You may not convey a covered work if you +are a party to an arrangement with a third party that is in the +business of distributing software, under which you make payment to the +third party based on the extent of your activity of conveying the +work, and under which the third party grants, to any of the parties +who would receive the covered work from you, a discriminatory patent +license (a) in connection with copies of the covered work conveyed by +you (or copies made from those copies), or (b) primarily for and in +connection with specific products or compilations that contain the +covered work, unless you entered into that arrangement, or that patent +license was granted, prior to 28 March 2007. + +Nothing in this License shall be construed as excluding or limiting +any implied license or other defenses to infringement that may +otherwise be available to you under applicable patent law. + +### 12. No Surrender of Others' Freedom. + +If conditions are imposed on you (whether by court order, agreement or +otherwise) that contradict the conditions of this License, they do not +excuse you from the conditions of this License. If you cannot convey a +covered work so as to satisfy simultaneously your obligations under +this License and any other pertinent obligations, then as a +consequence you may not convey it at all. For example, if you agree to +terms that obligate you to collect a royalty for further conveying +from those to whom you convey the Program, the only way you could +satisfy both those terms and this License would be to refrain entirely +from conveying the Program. + +### 13. Remote Network Interaction; Use with the GNU General Public License. + +Notwithstanding any other provision of this License, if you modify the +Program, your modified version must prominently offer all users +interacting with it remotely through a computer network (if your +version supports such interaction) an opportunity to receive the +Corresponding Source of your version by providing access to the +Corresponding Source from a network server at no charge, through some +standard or customary means of facilitating copying of software. This +Corresponding Source shall include the Corresponding Source for any +work covered by version 3 of the GNU General Public License that is +incorporated pursuant to the following paragraph. + +Notwithstanding any other provision of this License, you have +permission to link or combine any covered work with a work licensed +under version 3 of the GNU General Public License into a single +combined work, and to convey the resulting work. The terms of this +License will continue to apply to the part which is the covered work, +but the work with which it is combined will remain governed by version +3 of the GNU General Public License. + +### 14. Revised Versions of this License. + +The Free Software Foundation may publish revised and/or new versions +of the GNU Affero General Public License from time to time. Such new +versions will be similar in spirit to the present version, but may +differ in detail to address new problems or concerns. + +Each version is given a distinguishing version number. If the Program +specifies that a certain numbered version of the GNU Affero General +Public License "or any later version" applies to it, you have the +option of following the terms and conditions either of that numbered +version or of any later version published by the Free Software +Foundation. If the Program does not specify a version number of the +GNU Affero General Public License, you may choose any version ever +published by the Free Software Foundation. + +If the Program specifies that a proxy can decide which future versions +of the GNU Affero General Public License can be used, that proxy's +public statement of acceptance of a version permanently authorizes you +to choose that version for the Program. + +Later license versions may give you additional or different +permissions. However, no additional obligations are imposed on any +author or copyright holder as a result of your choosing to follow a +later version. + +### 15. Disclaimer of Warranty. + +THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY +APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT +HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT +WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND +PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE +DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR +CORRECTION. + +### 16. Limitation of Liability. + +IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING +WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR +CONVEYS THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, +INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES +ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT +NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR +LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM +TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER +PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. + +### 17. Interpretation of Sections 15 and 16. + +If the disclaimer of warranty and limitation of liability provided +above cannot be given local legal effect according to their terms, +reviewing courts shall apply local law that most closely approximates +an absolute waiver of all civil liability in connection with the +Program, unless a warranty or assumption of liability accompanies a +copy of the Program in return for a fee. + +END OF TERMS AND CONDITIONS + +## How to Apply These Terms to Your New Programs + +If you develop a new program, and you want it to be of the greatest +possible use to the public, the best way to achieve this is to make it +free software which everyone can redistribute and change under these +terms. + +To do so, attach the following notices to the program. It is safest to +attach them to the start of each source file to most effectively state +the exclusion of warranty; and each file should have at least the +"copyright" line and a pointer to where the full notice is found. + + <one line to give the program's name and a brief idea of what it does.> + Copyright (C) <year> <name of author> + + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU Affero General Public License as + published by the Free Software Foundation, either version 3 of the + License, or (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU Affero General Public License for more details. + + You should have received a copy of the GNU Affero General Public License + along with this program. If not, see <https://www.gnu.org/licenses/>. + +Also add information on how to contact you by electronic and paper +mail. + +If your software can interact with users remotely through a computer +network, you should also make sure that it provides a way for users to +get its source. For example, if your program is a web application, its +interface could display a "Source" link that leads users to an archive +of the code. There are many ways you could offer source, and different +solutions will be better for different programs; see section 13 for +the specific requirements. + +You should also get your employer (if you work as a programmer) or +school, if any, to sign a "copyright disclaimer" for the program, if +necessary. For more information on this, and how to apply and follow +the GNU AGPL, see <https://www.gnu.org/licenses/>. +#+END_SRC + +*** spine_summary + +#+NAME: sisudoc_spine_summary +#+BEGIN_SRC org +- Spine, Doc Reform (related to SiSU) uses standard: + - docReform markup syntax (based on SiSU markup) + - standard SiSU markup syntax with modified headers and minor + modifications + - docReform object numbering (based on SiSU object citation numbering) + - standard SiSU document object numbering + +- Homepages: + [https://www.sisudoc.org] +#+END_SRC + +#+BEGIN_SRC org + - Homepages: + [https://www.sisudoc.org] + + - Git + [] +#+END_SRC + +*** markup_samples + +#+NAME: sisudoc_spine_markup_samples +#+BEGIN_SRC org +- Spine, Doc Reform (SiSU) markup samples + Individual document content Copyright (Author) [as stated in document header] + Individual document content License (Author) [as stated in document header] +#+END_SRC + +* project variables +** project name + +#+NAME: project_name +#+BEGIN_SRC org +sisudoc-ocda-lib +#+END_SRC + +#+BEGIN_SRC org +SisuDoc Spine, Doc Reform [a part of] +#+END_SRC + +** project description + +#+NAME: project_description +#+BEGIN_SRC org +part of SiSU spine (Doc Reform): documents, structuring, +processing, publishing and search +#+END_SRC + +#+BEGIN_SRC org +Spine, Doc Reform (SiSU), a framework for document structuring, publishing and search +#+END_SRC + +#+BEGIN_SRC org +documents, structuring, processing, publishing, search +#+END_SRC + +** project start year + +#+NAME: project_year_started +#+BEGIN_SRC org +2026 +#+END_SRC + +** project paths + +#+NAME: project_code_paths +#+BEGIN_SRC org +dlang/src/* gleam/src/* +#+END_SRC + +#+BEGIN_SRC org +src/* src/sisudoc/* +#+END_SRC + +* spine version info SET VERSION +** version in full + +#+NAME: spine_project_version +#+BEGIN_SRC org +<<spine_project_version_part_major>>.<<spine_project_version_part_minor>>.<<spine_project_version_part_patch>> +#+END_SRC + +** version parts (major, minor, patch) + +#+NAME: spine_project_version_major_minor_patch +#+BEGIN_SRC org +<<spine_project_version_part_major>>, <<spine_project_version_part_minor>>, <<spine_project_version_part_patch>> +#+END_SRC + +*** major + +#+NAME: spine_project_version_part_major +#+BEGIN_SRC org +0 +#+END_SRC + +*** minor + +#+NAME: spine_project_version_part_minor +#+BEGIN_SRC org +25 +#+END_SRC + +*** patch + +#+NAME: spine_project_version_part_patch +#+BEGIN_SRC org +2 +#+END_SRC + +* year + +#+NAME: year +#+BEGIN_SRC org +2026 +#+END_SRC + +* sisudoc spine git source SET +** sisudoc-spine src + +#+NAME: sisudoc_spine_nix_src +#+BEGIN_SRC org +git://git.sisudoc.org/software/sisudoc-spine +#+END_SRC + +#+BEGIN_SRC org +git+file:///home/ralph/grotto/repo/git.repo/projects/doc-reform/code/software/sisudoc-spine +#+END_SRC + +** dlang overlay src + +#+NAME: sisudoc_dlang_nix_overlay_src +#+BEGIN_SRC org +git://git.sisudoc.org/package/dlang-nix-overlays-and-flake +#+END_SRC + +#+BEGIN_SRC org +git+file:///home/ralph/grotto/repo/git.repo/projects/doc-reform/packages/nix-flakes/dlang/dlang-nix-flakes +#+END_SRC + +* ocda ssp version format SET +** ocda ssp version in full + +#+NAME: ocda_ssp_format_version +#+BEGIN_SRC org +<<ocda_ssp_format_version_part_major>>.<<ocda_ssp_format_version_part_minor>> +#+END_SRC + +** parts +*** major + +#+NAME: ocda_ssp_format_version_part_major +#+BEGIN_SRC org +2 +#+END_SRC + +*** minor + +#+NAME: ocda_ssp_format_version_part_minor +#+BEGIN_SRC org +1 +#+END_SRC + +* sqlite collection search db schema version + +#+NAME: sqlite_collection_search_db_schema_version +#+BEGIN_SRC org +1 +#+END_SRC + +* nipkgs source + +- github:NixOS/nixpkgs/nixpkgs-unstable +- https://channels.nixos.org/nixpkgs-unstable/nixexprs.tar.zst + +#+NAME: nixpkgs_unstable +#+BEGIN_SRC org +https://channels.nixos.org/nixpkgs-unstable/nixexprs.tar.zst +#+END_SRC + +#+BEGIN_SRC org +github:NixOS/nixpkgs/nixpkgs-unstable +#+END_SRC |
