diff options
46 files changed, 7136 insertions, 0 deletions
diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..12a3507 --- /dev/null +++ b/.gitignore @@ -0,0 +1,76 @@ +# Deny everything, then whitelist what belongs in the repository. +# +# The practice is deliberate: a repository should hold what was decided to +# put in it, not whatever happened to be in the directory when someone ran +# "git add .". Build output, editor droppings, downloaded dependencies and +# exported binaries are numerous, are regenerated, and differ between +# machines; source is few and is chosen. So the default is no, and every +# yes below is a decision that can be read. +# +# To see what is being excluded, and why: +# git status --ignored +# git check-ignore -v <path> +# git ls-files --others --ignored --exclude-standard --directory + +* + +# ── the repository itself ──────────────────────────────────────────────── +!.gitignore +!README.md +!LICENSE +!**/LICENSE + +# ── directories have to be walked into before a file inside can match ──── +!gleam/ +!dlang/ +!org/ +!gleam/**/ +!dlang/**/ +!org/**/ +# but not these, whatever they contain +build/ +.build/ +.direnv/ +.gleam/ +bin/ +result +result-* + +# ── documentation and notes, in any language directory ─────────────────── +!**/COPYRIGHT +!**/*.md +!**/*.org + +# ── nix ────────────────────────────────────────────────────────────────── +!**/*.nix +!**/flake.lock +!**/.envrc + +# ── dlang ──────────────────────────────────────────────────────────────── +!**/*.d +!**/dub.json +!**/dub.sdl +!**/dub.selections.json +*.o +*.a +*.lst + +# ── gleam ──────────────────────────────────────────────────────────────── +!**/*.gleam +# Erlang, for what Gleam reaches through FFI (a sha256, from OTP crypto) +!**/src/*.erl +# gleam.toml is the manifest, manifest.toml is the lockfile: both belong +!**/*.toml +*.beam +*.ez +erl_crash.dump + +# ── ruby, when it arrives ──────────────────────────────────────────────── +!**/*.rb +!**/Gemfile +!**/Gemfile.lock +!**/*.gemspec + +# ── continuous integration ─────────────────────────────────────────────── +!**/*.yml +!**/*.yaml diff --git a/COPYRIGHT b/COPYRIGHT new file mode 100644 index 0000000..7e7dd56 --- /dev/null +++ b/COPYRIGHT @@ -0,0 +1,45 @@ +- Name: sisudoc-ocda-lib + - Description: part of SiSU spine (Doc Reform): documents, structuring, + processing, publishing and search + - [https://www.sisudoc.org] + + - Author: Ralph Amissah + [ralph.amissah@gmail.com] + + - Copyright: (C) 2026 Ralph Amissah, All Rights Reserved. + + - License: AGPL 3 or later, for everything in this repository unless a file or + a directory specifies otherwise.: + + sisudoc-ocda-lib + + part of SiSU spine (Doc Reform): documents, structuring, + processing, publishing and search + + Copyright (C) Ralph Amissah + + This program is free software: you can redistribute it and/or modify it + under the terms of the GNU Affero General Public License (AGPLv3) as + published by the Free Software Foundation, either version 3 of the License, + or (at your option) any later version. + + This program is distributed in the hope that it will be useful, but + WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY + or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public + License for more details. + + You should have received a copy of the GNU Affero General Public License + along with this program. If not, see [https://www.gnu.org/licenses/]. + + If you have Internet connection, the latest version of the AGPL should be + available at: [https://www.gnu.org/licenses/agpl-3.0.html] + + - Spine, Doc Reform (related to SiSU) uses standard: + - docReform markup syntax (based on SiSU markup) + - standard SiSU markup syntax with modified headers and minor + modifications + - docReform object numbering (based on SiSU object citation numbering) + - standard SiSU document object numbering + + - Homepages: + [https://www.sisudoc.org] @@ -0,0 +1,660 @@ +# GNU AFFERO GENERAL PUBLIC LICENSE + +Version 3, 19 November 2007 + +Copyright (C) 2007 Free Software Foundation, Inc. +<https://fsf.org/> + +Everyone is permitted to copy and distribute verbatim copies of this +license document, but changing it is not allowed. + +## Preamble + +The GNU Affero General Public License is a free, copyleft license for +software and other kinds of works, specifically designed to ensure +cooperation with the community in the case of network server software. + +The licenses for most software and other practical works are designed +to take away your freedom to share and change the works. By contrast, +our General Public Licenses are intended to guarantee your freedom to +share and change all versions of a program--to make sure it remains +free software for all its users. + +When we speak of free software, we are referring to freedom, not +price. Our General Public Licenses are designed to make sure that you +have the freedom to distribute copies of free software (and charge for +them if you wish), that you receive source code or can get it if you +want it, that you can change the software or use pieces of it in new +free programs, and that you know you can do these things. + +Developers that use our General Public Licenses protect your rights +with two steps: (1) assert copyright on the software, and (2) offer +you this License which gives you legal permission to copy, distribute +and/or modify the software. + +A secondary benefit of defending all users' freedom is that +improvements made in alternate versions of the program, if they +receive widespread use, become available for other developers to +incorporate. Many developers of free software are heartened and +encouraged by the resulting cooperation. However, in the case of +software used on network servers, this result may fail to come about. +The GNU General Public License permits making a modified version and +letting the public access it on a server without ever releasing its +source code to the public. + +The GNU Affero General Public License is designed specifically to +ensure that, in such cases, the modified source code becomes available +to the community. It requires the operator of a network server to +provide the source code of the modified version running there to the +users of that server. Therefore, public use of a modified version, on +a publicly accessible server, gives the public access to the source +code of the modified version. + +An older license, called the Affero General Public License and +published by Affero, was designed to accomplish similar goals. This is +a different license, not a version of the Affero GPL, but Affero has +released a new version of the Affero GPL which permits relicensing +under this license. + +The precise terms and conditions for copying, distribution and +modification follow. + +## TERMS AND CONDITIONS + +### 0. Definitions. + +"This License" refers to version 3 of the GNU Affero General Public +License. + +"Copyright" also means copyright-like laws that apply to other kinds +of works, such as semiconductor masks. + +"The Program" refers to any copyrightable work licensed under this +License. Each licensee is addressed as "you". "Licensees" and +"recipients" may be individuals or organizations. + +To "modify" a work means to copy from or adapt all or part of the work +in a fashion requiring copyright permission, other than the making of +an exact copy. The resulting work is called a "modified version" of +the earlier work or a work "based on" the earlier work. + +A "covered work" means either the unmodified Program or a work based +on the Program. + +To "propagate" a work means to do anything with it that, without +permission, would make you directly or secondarily liable for +infringement under applicable copyright law, except executing it on a +computer or modifying a private copy. Propagation includes copying, +distribution (with or without modification), making available to the +public, and in some countries other activities as well. + +To "convey" a work means any kind of propagation that enables other +parties to make or receive copies. Mere interaction with a user +through a computer network, with no transfer of a copy, is not +conveying. + +An interactive user interface displays "Appropriate Legal Notices" to +the extent that it includes a convenient and prominently visible +feature that (1) displays an appropriate copyright notice, and (2) +tells the user that there is no warranty for the work (except to the +extent that warranties are provided), that licensees may convey the +work under this License, and how to view a copy of this License. If +the interface presents a list of user commands or options, such as a +menu, a prominent item in the list meets this criterion. + +### 1. Source Code. + +The "source code" for a work means the preferred form of the work for +making modifications to it. "Object code" means any non-source form of +a work. + +A "Standard Interface" means an interface that either is an official +standard defined by a recognized standards body, or, in the case of +interfaces specified for a particular programming language, one that +is widely used among developers working in that language. + +The "System Libraries" of an executable work include anything, other +than the work as a whole, that (a) is included in the normal form of +packaging a Major Component, but which is not part of that Major +Component, and (b) serves only to enable use of the work with that +Major Component, or to implement a Standard Interface for which an +implementation is available to the public in source code form. A +"Major Component", in this context, means a major essential component +(kernel, window system, and so on) of the specific operating system +(if any) on which the executable work runs, or a compiler used to +produce the work, or an object code interpreter used to run it. + +The "Corresponding Source" for a work in object code form means all +the source code needed to generate, install, and (for an executable +work) run the object code and to modify the work, including scripts to +control those activities. However, it does not include the work's +System Libraries, or general-purpose tools or generally available free +programs which are used unmodified in performing those activities but +which are not part of the work. For example, Corresponding Source +includes interface definition files associated with source files for +the work, and the source code for shared libraries and dynamically +linked subprograms that the work is specifically designed to require, +such as by intimate data communication or control flow between those +subprograms and other parts of the work. + +The Corresponding Source need not include anything that users can +regenerate automatically from other parts of the Corresponding Source. + +The Corresponding Source for a work in source code form is that same +work. + +### 2. Basic Permissions. + +All rights granted under this License are granted for the term of +copyright on the Program, and are irrevocable provided the stated +conditions are met. This License explicitly affirms your unlimited +permission to run the unmodified Program. The output from running a +covered work is covered by this License only if the output, given its +content, constitutes a covered work. This License acknowledges your +rights of fair use or other equivalent, as provided by copyright law. + +You may make, run and propagate covered works that you do not convey, +without conditions so long as your license otherwise remains in force. +You may convey covered works to others for the sole purpose of having +them make modifications exclusively for you, or provide you with +facilities for running those works, provided that you comply with the +terms of this License in conveying all material for which you do not +control copyright. Those thus making or running the covered works for +you must do so exclusively on your behalf, under your direction and +control, on terms that prohibit them from making any copies of your +copyrighted material outside their relationship with you. + +Conveying under any other circumstances is permitted solely under the +conditions stated below. Sublicensing is not allowed; section 10 makes +it unnecessary. + +### 3. Protecting Users' Legal Rights From Anti-Circumvention Law. + +No covered work shall be deemed part of an effective technological +measure under any applicable law fulfilling obligations under article +11 of the WIPO copyright treaty adopted on 20 December 1996, or +similar laws prohibiting or restricting circumvention of such +measures. + +When you convey a covered work, you waive any legal power to forbid +circumvention of technological measures to the extent such +circumvention is effected by exercising rights under this License with +respect to the covered work, and you disclaim any intention to limit +operation or modification of the work as a means of enforcing, against +the work's users, your or third parties' legal rights to forbid +circumvention of technological measures. + +### 4. Conveying Verbatim Copies. + +You may convey verbatim copies of the Program's source code as you +receive it, in any medium, provided that you conspicuously and +appropriately publish on each copy an appropriate copyright notice; +keep intact all notices stating that this License and any +non-permissive terms added in accord with section 7 apply to the code; +keep intact all notices of the absence of any warranty; and give all +recipients a copy of this License along with the Program. + +You may charge any price or no price for each copy that you convey, +and you may offer support or warranty protection for a fee. + +### 5. Conveying Modified Source Versions. + +You may convey a work based on the Program, or the modifications to +produce it from the Program, in the form of source code under the +terms of section 4, provided that you also meet all of these +conditions: + +- a) The work must carry prominent notices stating that you modified + it, and giving a relevant date. +- b) The work must carry prominent notices stating that it is + released under this License and any conditions added under + section 7. This requirement modifies the requirement in section 4 + to "keep intact all notices". +- c) You must license the entire work, as a whole, under this + License to anyone who comes into possession of a copy. This + License will therefore apply, along with any applicable section 7 + additional terms, to the whole of the work, and all its parts, + regardless of how they are packaged. This License gives no + permission to license the work in any other way, but it does not + invalidate such permission if you have separately received it. +- d) If the work has interactive user interfaces, each must display + Appropriate Legal Notices; however, if the Program has interactive + interfaces that do not display Appropriate Legal Notices, your + work need not make them do so. + +A compilation of a covered work with other separate and independent +works, which are not by their nature extensions of the covered work, +and which are not combined with it such as to form a larger program, +in or on a volume of a storage or distribution medium, is called an +"aggregate" if the compilation and its resulting copyright are not +used to limit the access or legal rights of the compilation's users +beyond what the individual works permit. Inclusion of a covered work +in an aggregate does not cause this License to apply to the other +parts of the aggregate. + +### 6. Conveying Non-Source Forms. + +You may convey a covered work in object code form under the terms of +sections 4 and 5, provided that you also convey the machine-readable +Corresponding Source under the terms of this License, in one of these +ways: + +- a) Convey the object code in, or embodied in, a physical product + (including a physical distribution medium), accompanied by the + Corresponding Source fixed on a durable physical medium + customarily used for software interchange. +- b) Convey the object code in, or embodied in, a physical product + (including a physical distribution medium), accompanied by a + written offer, valid for at least three years and valid for as + long as you offer spare parts or customer support for that product + model, to give anyone who possesses the object code either (1) a + copy of the Corresponding Source for all the software in the + product that is covered by this License, on a durable physical + medium customarily used for software interchange, for a price no + more than your reasonable cost of physically performing this + conveying of source, or (2) access to copy the Corresponding + Source from a network server at no charge. +- c) Convey individual copies of the object code with a copy of the + written offer to provide the Corresponding Source. This + alternative is allowed only occasionally and noncommercially, and + only if you received the object code with such an offer, in accord + with subsection 6b. +- d) Convey the object code by offering access from a designated + place (gratis or for a charge), and offer equivalent access to the + Corresponding Source in the same way through the same place at no + further charge. You need not require recipients to copy the + Corresponding Source along with the object code. If the place to + copy the object code is a network server, the Corresponding Source + may be on a different server (operated by you or a third party) + that supports equivalent copying facilities, provided you maintain + clear directions next to the object code saying where to find the + Corresponding Source. Regardless of what server hosts the + Corresponding Source, you remain obligated to ensure that it is + available for as long as needed to satisfy these requirements. +- e) Convey the object code using peer-to-peer transmission, + provided you inform other peers where the object code and + Corresponding Source of the work are being offered to the general + public at no charge under subsection 6d. + +A separable portion of the object code, whose source code is excluded +from the Corresponding Source as a System Library, need not be +included in conveying the object code work. + +A "User Product" is either (1) a "consumer product", which means any +tangible personal property which is normally used for personal, +family, or household purposes, or (2) anything designed or sold for +incorporation into a dwelling. In determining whether a product is a +consumer product, doubtful cases shall be resolved in favor of +coverage. For a particular product received by a particular user, +"normally used" refers to a typical or common use of that class of +product, regardless of the status of the particular user or of the way +in which the particular user actually uses, or expects or is expected +to use, the product. A product is a consumer product regardless of +whether the product has substantial commercial, industrial or +non-consumer uses, unless such uses represent the only significant +mode of use of the product. + +"Installation Information" for a User Product means any methods, +procedures, authorization keys, or other information required to +install and execute modified versions of a covered work in that User +Product from a modified version of its Corresponding Source. The +information must suffice to ensure that the continued functioning of +the modified object code is in no case prevented or interfered with +solely because modification has been made. + +If you convey an object code work under this section in, or with, or +specifically for use in, a User Product, and the conveying occurs as +part of a transaction in which the right of possession and use of the +User Product is transferred to the recipient in perpetuity or for a +fixed term (regardless of how the transaction is characterized), the +Corresponding Source conveyed under this section must be accompanied +by the Installation Information. But this requirement does not apply +if neither you nor any third party retains the ability to install +modified object code on the User Product (for example, the work has +been installed in ROM). + +The requirement to provide Installation Information does not include a +requirement to continue to provide support service, warranty, or +updates for a work that has been modified or installed by the +recipient, or for the User Product in which it has been modified or +installed. Access to a network may be denied when the modification +itself materially and adversely affects the operation of the network +or violates the rules and protocols for communication across the +network. + +Corresponding Source conveyed, and Installation Information provided, +in accord with this section must be in a format that is publicly +documented (and with an implementation available to the public in +source code form), and must require no special password or key for +unpacking, reading or copying. + +### 7. Additional Terms. + +"Additional permissions" are terms that supplement the terms of this +License by making exceptions from one or more of its conditions. +Additional permissions that are applicable to the entire Program shall +be treated as though they were included in this License, to the extent +that they are valid under applicable law. If additional permissions +apply only to part of the Program, that part may be used separately +under those permissions, but the entire Program remains governed by +this License without regard to the additional permissions. + +When you convey a copy of a covered work, you may at your option +remove any additional permissions from that copy, or from any part of +it. (Additional permissions may be written to require their own +removal in certain cases when you modify the work.) You may place +additional permissions on material, added by you to a covered work, +for which you have or can give appropriate copyright permission. + +Notwithstanding any other provision of this License, for material you +add to a covered work, you may (if authorized by the copyright holders +of that material) supplement the terms of this License with terms: + +- a) Disclaiming warranty or limiting liability differently from the + terms of sections 15 and 16 of this License; or +- b) Requiring preservation of specified reasonable legal notices or + author attributions in that material or in the Appropriate Legal + Notices displayed by works containing it; or +- c) Prohibiting misrepresentation of the origin of that material, + or requiring that modified versions of such material be marked in + reasonable ways as different from the original version; or +- d) Limiting the use for publicity purposes of names of licensors + or authors of the material; or +- e) Declining to grant rights under trademark law for use of some + trade names, trademarks, or service marks; or +- f) Requiring indemnification of licensors and authors of that + material by anyone who conveys the material (or modified versions + of it) with contractual assumptions of liability to the recipient, + for any liability that these contractual assumptions directly + impose on those licensors and authors. + +All other non-permissive additional terms are considered "further +restrictions" within the meaning of section 10. If the Program as you +received it, or any part of it, contains a notice stating that it is +governed by this License along with a term that is a further +restriction, you may remove that term. If a license document contains +a further restriction but permits relicensing or conveying under this +License, you may add to a covered work material governed by the terms +of that license document, provided that the further restriction does +not survive such relicensing or conveying. + +If you add terms to a covered work in accord with this section, you +must place, in the relevant source files, a statement of the +additional terms that apply to those files, or a notice indicating +where to find the applicable terms. + +Additional terms, permissive or non-permissive, may be stated in the +form of a separately written license, or stated as exceptions; the +above requirements apply either way. + +### 8. Termination. + +You may not propagate or modify a covered work except as expressly +provided under this License. Any attempt otherwise to propagate or +modify it is void, and will automatically terminate your rights under +this License (including any patent licenses granted under the third +paragraph of section 11). + +However, if you cease all violation of this License, then your license +from a particular copyright holder is reinstated (a) provisionally, +unless and until the copyright holder explicitly and finally +terminates your license, and (b) permanently, if the copyright holder +fails to notify you of the violation by some reasonable means prior to +60 days after the cessation. + +Moreover, your license from a particular copyright holder is +reinstated permanently if the copyright holder notifies you of the +violation by some reasonable means, this is the first time you have +received notice of violation of this License (for any work) from that +copyright holder, and you cure the violation prior to 30 days after +your receipt of the notice. + +Termination of your rights under this section does not terminate the +licenses of parties who have received copies or rights from you under +this License. If your rights have been terminated and not permanently +reinstated, you do not qualify to receive new licenses for the same +material under section 10. + +### 9. Acceptance Not Required for Having Copies. + +You are not required to accept this License in order to receive or run +a copy of the Program. Ancillary propagation of a covered work +occurring solely as a consequence of using peer-to-peer transmission +to receive a copy likewise does not require acceptance. However, +nothing other than this License grants you permission to propagate or +modify any covered work. These actions infringe copyright if you do +not accept this License. Therefore, by modifying or propagating a +covered work, you indicate your acceptance of this License to do so. + +### 10. Automatic Licensing of Downstream Recipients. + +Each time you convey a covered work, the recipient automatically +receives a license from the original licensors, to run, modify and +propagate that work, subject to this License. You are not responsible +for enforcing compliance by third parties with this License. + +An "entity transaction" is a transaction transferring control of an +organization, or substantially all assets of one, or subdividing an +organization, or merging organizations. If propagation of a covered +work results from an entity transaction, each party to that +transaction who receives a copy of the work also receives whatever +licenses to the work the party's predecessor in interest had or could +give under the previous paragraph, plus a right to possession of the +Corresponding Source of the work from the predecessor in interest, if +the predecessor has it or can get it with reasonable efforts. + +You may not impose any further restrictions on the exercise of the +rights granted or affirmed under this License. For example, you may +not impose a license fee, royalty, or other charge for exercise of +rights granted under this License, and you may not initiate litigation +(including a cross-claim or counterclaim in a lawsuit) alleging that +any patent claim is infringed by making, using, selling, offering for +sale, or importing the Program or any portion of it. + +### 11. Patents. + +A "contributor" is a copyright holder who authorizes use under this +License of the Program or a work on which the Program is based. The +work thus licensed is called the contributor's "contributor version". + +A contributor's "essential patent claims" are all patent claims owned +or controlled by the contributor, whether already acquired or +hereafter acquired, that would be infringed by some manner, permitted +by this License, of making, using, or selling its contributor version, +but do not include claims that would be infringed only as a +consequence of further modification of the contributor version. For +purposes of this definition, "control" includes the right to grant +patent sublicenses in a manner consistent with the requirements of +this License. + +Each contributor grants you a non-exclusive, worldwide, royalty-free +patent license under the contributor's essential patent claims, to +make, use, sell, offer for sale, import and otherwise run, modify and +propagate the contents of its contributor version. + +In the following three paragraphs, a "patent license" is any express +agreement or commitment, however denominated, not to enforce a patent +(such as an express permission to practice a patent or covenant not to +sue for patent infringement). To "grant" such a patent license to a +party means to make such an agreement or commitment not to enforce a +patent against the party. + +If you convey a covered work, knowingly relying on a patent license, +and the Corresponding Source of the work is not available for anyone +to copy, free of charge and under the terms of this License, through a +publicly available network server or other readily accessible means, +then you must either (1) cause the Corresponding Source to be so +available, or (2) arrange to deprive yourself of the benefit of the +patent license for this particular work, or (3) arrange, in a manner +consistent with the requirements of this License, to extend the patent +license to downstream recipients. "Knowingly relying" means you have +actual knowledge that, but for the patent license, your conveying the +covered work in a country, or your recipient's use of the covered work +in a country, would infringe one or more identifiable patents in that +country that you have reason to believe are valid. + +If, pursuant to or in connection with a single transaction or +arrangement, you convey, or propagate by procuring conveyance of, a +covered work, and grant a patent license to some of the parties +receiving the covered work authorizing them to use, propagate, modify +or convey a specific copy of the covered work, then the patent license +you grant is automatically extended to all recipients of the covered +work and works based on it. + +A patent license is "discriminatory" if it does not include within the +scope of its coverage, prohibits the exercise of, or is conditioned on +the non-exercise of one or more of the rights that are specifically +granted under this License. You may not convey a covered work if you +are a party to an arrangement with a third party that is in the +business of distributing software, under which you make payment to the +third party based on the extent of your activity of conveying the +work, and under which the third party grants, to any of the parties +who would receive the covered work from you, a discriminatory patent +license (a) in connection with copies of the covered work conveyed by +you (or copies made from those copies), or (b) primarily for and in +connection with specific products or compilations that contain the +covered work, unless you entered into that arrangement, or that patent +license was granted, prior to 28 March 2007. + +Nothing in this License shall be construed as excluding or limiting +any implied license or other defenses to infringement that may +otherwise be available to you under applicable patent law. + +### 12. No Surrender of Others' Freedom. + +If conditions are imposed on you (whether by court order, agreement or +otherwise) that contradict the conditions of this License, they do not +excuse you from the conditions of this License. If you cannot convey a +covered work so as to satisfy simultaneously your obligations under +this License and any other pertinent obligations, then as a +consequence you may not convey it at all. For example, if you agree to +terms that obligate you to collect a royalty for further conveying +from those to whom you convey the Program, the only way you could +satisfy both those terms and this License would be to refrain entirely +from conveying the Program. + +### 13. Remote Network Interaction; Use with the GNU General Public License. + +Notwithstanding any other provision of this License, if you modify the +Program, your modified version must prominently offer all users +interacting with it remotely through a computer network (if your +version supports such interaction) an opportunity to receive the +Corresponding Source of your version by providing access to the +Corresponding Source from a network server at no charge, through some +standard or customary means of facilitating copying of software. This +Corresponding Source shall include the Corresponding Source for any +work covered by version 3 of the GNU General Public License that is +incorporated pursuant to the following paragraph. + +Notwithstanding any other provision of this License, you have +permission to link or combine any covered work with a work licensed +under version 3 of the GNU General Public License into a single +combined work, and to convey the resulting work. The terms of this +License will continue to apply to the part which is the covered work, +but the work with which it is combined will remain governed by version +3 of the GNU General Public License. + +### 14. Revised Versions of this License. + +The Free Software Foundation may publish revised and/or new versions +of the GNU Affero General Public License from time to time. Such new +versions will be similar in spirit to the present version, but may +differ in detail to address new problems or concerns. + +Each version is given a distinguishing version number. If the Program +specifies that a certain numbered version of the GNU Affero General +Public License "or any later version" applies to it, you have the +option of following the terms and conditions either of that numbered +version or of any later version published by the Free Software +Foundation. If the Program does not specify a version number of the +GNU Affero General Public License, you may choose any version ever +published by the Free Software Foundation. + +If the Program specifies that a proxy can decide which future versions +of the GNU Affero General Public License can be used, that proxy's +public statement of acceptance of a version permanently authorizes you +to choose that version for the Program. + +Later license versions may give you additional or different +permissions. However, no additional obligations are imposed on any +author or copyright holder as a result of your choosing to follow a +later version. + +### 15. Disclaimer of Warranty. + +THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY +APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT +HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT +WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND +PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE +DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR +CORRECTION. + +### 16. Limitation of Liability. + +IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING +WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR +CONVEYS THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, +INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES +ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT +NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR +LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM +TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER +PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. + +### 17. Interpretation of Sections 15 and 16. + +If the disclaimer of warranty and limitation of liability provided +above cannot be given local legal effect according to their terms, +reviewing courts shall apply local law that most closely approximates +an absolute waiver of all civil liability in connection with the +Program, unless a warranty or assumption of liability accompanies a +copy of the Program in return for a fee. + +END OF TERMS AND CONDITIONS + +## How to Apply These Terms to Your New Programs + +If you develop a new program, and you want it to be of the greatest +possible use to the public, the best way to achieve this is to make it +free software which everyone can redistribute and change under these +terms. + +To do so, attach the following notices to the program. It is safest to +attach them to the start of each source file to most effectively state +the exclusion of warranty; and each file should have at least the +"copyright" line and a pointer to where the full notice is found. + + <one line to give the program's name and a brief idea of what it does.> + Copyright (C) <year> <name of author> + + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU Affero General Public License as + published by the Free Software Foundation, either version 3 of the + License, or (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU Affero General Public License for more details. + + You should have received a copy of the GNU Affero General Public License + along with this program. If not, see <https://www.gnu.org/licenses/>. + +Also add information on how to contact you by electronic and paper +mail. + +If your software can interact with users remotely through a computer +network, you should also make sure that it provides a way for users to +get its source. For example, if your program is a web application, its +interface could display a "Source" link that leads users to an archive +of the code. There are many ways you could offer source, and different +solutions will be better for different programs; see section 13 for +the specific requirements. + +You should also get your employer (if you work as a programmer) or +school, if any, to sign a "copyright disclaimer" for the program, if +necessary. For more information on this, and how to apply and follow +the GNU AGPL, see <https://www.gnu.org/licenses/>. diff --git a/README.md b/README.md new file mode 100644 index 0000000..5e328c0 --- /dev/null +++ b/README.md @@ -0,0 +1,150 @@ +# sisudoc-ocda-lib + +The **object-centric document abstraction as a library**: the model, and +loaders for both of the forms it is written in. + +Written by Claude (Anthropic), working in Claude Code from the specification, +at the direction of and reviewed by Ralph Amissah. + +**One reader per language, in a directory named for the language.** + +| | | +|--------------------|---------| +| [`dlang/`](dlang/) | D | +| [`gleam/`](gleam/) | Gleam | +| `ruby/` | to come | + +## What it is for + +The readers in `sisudoc-ssp-reader` and `sisudoc-ocda-db-reader` each read +one serialisation and print. A **writer** needs neither: it needs the document. + +``` + .ssp ─┐ + ├─> the model ─> a writer + .ocda.db ─┘ +``` + +So this library holds the model and both loaders, and **a writer cannot +tell which serialisation it was handed** - exactly as spine's own writers +cannot, since spine 0.25.0. + +That is the whole of the idea. The readers become thin programs over this, +and the writers are functions from the model to bytes. + +## The checks that matter + +The claim above is only true if the loaders agree, so the library asks. Two +programs, in each language: + +```sh +ocda-agree <doc>.ssp <doc>.ocda.db [lang] # do the two loaders agree? +ocda-dump <file>.ssp|<file>.ocda.db [lang] # the whole model, canonically +``` + +`agree` compares two models within one language. **`dump` is the stronger +of the two**, and the reason it exists: it writes the entire loaded model +out in a fully determined form, and the D and the Gleam library write it +*byte for byte alike*. So four questions can be asked, all of them with +`diff`: + +| | | +|-----------------------------|---------------------------------| +| D: `.ssp` == `.ocda.db` | **36 of 36** document-languages | +| Gleam: `.ssp` == `.ocda.db` | **36 of 36** | +| `.ssp`: D == Gleam | **36 of 36** | +| `.ocda.db`: D == Gleam | **36 of 36** | + +None of those held at first. + +## What the checks found + +### the ten-field check: a blank first text line + +An object whose **first text line is blank** lost it. The loader decided +whether to start a new line by asking whether the accumulated text was +empty, which is not the same question as whether a line has been seen. Of +the sample collection **seven documents have verse opening on a blank +line**, and in every one of them the blank was dropped. Eight of +thirty-six documents disagreed. + +Nothing else had caught it. The two `.ssp` readers had been compared on +*section counts*, which this does not change. It took loading the same +document two ways and comparing the objects. + +Fixed here and in `sisudoc-ssp-reader/dlang`, which carried the same code. + +### the whole-model dump: four more + +The ten-field check was enough for that and not enough for anything +subtler. Dumping the whole model found four further disagreements between +the two loaders: + +1. **`.identifier` and `.anchor`.** The `.ssp` emits a property only when it + carries information, so the identifier is absent when it is simply the + ocn and the anchor is absent when it is the segment name; the database + stores both unconditionally. Over one document that is **1,314 + identifiers and 69 anchors**. The loaders now derive them, which is what + the model means. +2. **`make` and `doc_has`.** The database has one metadata table where the + text form has three blocks, and tells them apart by a key prefix. The + loaders unprefix them, so a writer asks `make["footer"]` whichever + serialisation it was handed. +3. **The JSON columns were scanned rather than parsed.** Phobos writes `/` + as `\/`, so every heading containing a slash came back with a backslash + in it: **all 36 document-languages** differed on + `heading_ancestors_text` and nothing else had shown it. +4. **`translation.ocn_aligned`** is in the database and cannot be in a + `.ssp`, which is one language and cannot know whether the languages + share a numbering. It is a fact about the *file*, so the loaders leave + it out rather than letting a writer tell which serialisation it was + given. + +The first three are defects. The fourth is a real asymmetry between the two +serialisations, and naming it is the useful part. + +## What it reads, and what it does not + +| | | | +|-----------------------|------------------------------------------------------------------|-------------------| +| `.ssp` | yes | | +| `.ocda.db` | yes, any language of it | | +| `.sisupod` | yes, any language it carries | D only, see below | +| `.zip` pod | yes, any language it carries | | +| `.sst`, `.ssm` markup | **no, and not planned**: that is a parser, which is spine's work | | +| a url | no | | + +A pod carries the abstraction only when spine was asked for it: `--pod2` +puts it in, `--source` does not. A pod without one is refused with a message +saying which flag makes a pod this library can read, because that is a fact +about the pod rather than about the library. + +**Over the reference collection, a `.sisupod` gives the same model as the +`.ssp` beside it: 36 of 36 document-languages, byte identical dumps.** A +writer over the library therefore gained a third input form without being +touched, which is the point of the dispatch being in one place. + +### the one thing the Gleam library cannot do + +A `.sisupod` is **one zstd frame wrapping** the zip. The D library reads both +through a declared `libzstd`. The Gleam library reads the **zip and not the +frame**, because the only zstd on the BEAM is a NIF that clones zstd from +github in its own build and so cannot be built in a sandbox without patching +it. + +So a `.zip` pod is read in both languages, a `.sisupod` in D only, and the +Gleam refusal says why and says `zstd -d` rather than claiming the file is +malformed. It is a packaging limit, not anything about the format, and it is +the only place the two libraries differ in what they accept. + +`ocda/load` is the one place in each language that decides what a path is, +so a new input form is added once and every reader and writer gains it. + +## The format + +Specified in **sisudoc-abstraction-format**: `ocda-model.md` for the model +this library implements, and the two format specifications for the loaders. +Section 4 of `ocda-db-format-specification.md` gives the column by column +correspondence the `load_db` of each language follows. + +At format **2.x**. diff --git a/dlang/COPYRIGHT b/dlang/COPYRIGHT new file mode 100644 index 0000000..7e7dd56 --- /dev/null +++ b/dlang/COPYRIGHT @@ -0,0 +1,45 @@ +- Name: sisudoc-ocda-lib + - Description: part of SiSU spine (Doc Reform): documents, structuring, + processing, publishing and search + - [https://www.sisudoc.org] + + - Author: Ralph Amissah + [ralph.amissah@gmail.com] + + - Copyright: (C) 2026 Ralph Amissah, All Rights Reserved. + + - License: AGPL 3 or later, for everything in this repository unless a file or + a directory specifies otherwise.: + + sisudoc-ocda-lib + + part of SiSU spine (Doc Reform): documents, structuring, + processing, publishing and search + + Copyright (C) Ralph Amissah + + This program is free software: you can redistribute it and/or modify it + under the terms of the GNU Affero General Public License (AGPLv3) as + published by the Free Software Foundation, either version 3 of the License, + or (at your option) any later version. + + This program is distributed in the hope that it will be useful, but + WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY + or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public + License for more details. + + You should have received a copy of the GNU Affero General Public License + along with this program. If not, see [https://www.gnu.org/licenses/]. + + If you have Internet connection, the latest version of the AGPL should be + available at: [https://www.gnu.org/licenses/agpl-3.0.html] + + - Spine, Doc Reform (related to SiSU) uses standard: + - docReform markup syntax (based on SiSU markup) + - standard SiSU markup syntax with modified headers and minor + modifications + - docReform object numbering (based on SiSU object citation numbering) + - standard SiSU document object numbering + + - Homepages: + [https://www.sisudoc.org] diff --git a/dlang/README.md b/dlang/README.md new file mode 100644 index 0000000..124e4d6 --- /dev/null +++ b/dlang/README.md @@ -0,0 +1,62 @@ +# ocda-lib, in D + +The abstraction's model and both its loaders, as a dub library. + +## Modules + +| | | +|-----------------|---------------------------------------------------------------| +| `ocda.model` | the document, the object, the eight sections. Imports nothing | +| `ocda.load_ssp` | text -> model | +| `ocda.load_db` | sqlite -> model | +| `ocda.load` | what a path is, and loading it | +| `ocda.dump` | the model written out canonically, as an oracle | +| `ocda.sqlite` | the sqlite3 C API, declared rather than depended upon | + +`ocda.model` importing nothing is the property that matters: a writer +depends on the model and on neither loader. + +## Use + +```json +"dependencies": { "ocda-lib": { "path": "../../sisudoc-ocda-lib/dlang" } } +``` + +```d +import ocda.load; +auto doc = load("document.ocda.db", "de"); // or "document.ssp" +foreach (obj; doc.body) { ... } +``` + +Link `sqlite3`; the library declares the dozen functions it needs rather +than taking a binding from the registry, so there is nothing to fetch. + +## The two checks + +```sh +nix build .#ocda-agree && ./result/bin/ocda-agree <doc>.ssp <doc>.ocda.db [lang] +nix build .#ocda-dump && ./result/bin/ocda-dump [--unknown] <file> [lang] +``` + +`agree` loads both and compares ten fields of each object, naming the field +where they part company. `agree` found the blank-first-text-line defect and +was not enough for anything subtler. + +**`ocda-dump` is the stronger check**: the whole loaded model, every field, +written out in a form fully determined by the model, so that two +implementations can be compared with `diff`. It is an oracle, not an output +format; anything reading it to reconstruct a document has misunderstood it. +It agrees byte for byte with the Gleam library's `ocda-lib dump`, which is +the point of having it. + +It found four disagreements `agree` could not see. They are in the +repository README and in the history. + +## Build + +```sh +nix build .#ocda-agree # the agreement check +nix build .#ocda-dump # the canonical dump +nix develop # ldc, dub, dtools, sqlite +dub build # the library +``` diff --git a/dlang/dub.json b/dlang/dub.json new file mode 100644 index 0000000..4f59aec --- /dev/null +++ b/dlang/dub.json @@ -0,0 +1,13 @@ +{ + "authors": [ "Ralph Amissah" ], + "copyright": "Copyright (C) 2026 Ralph Amissah", + "name": "ocda-lib", + "version": "2.0.0", + "description": "the object-centric document abstraction as a library: the model, and loaders for its two serialisations", + "homepage": "https://sisudoc.org", + "license": "AGPL-3.0+", + "targetType": "library", + "sourcePaths": [ "./src" ], + "importPaths": [ "./src" ], + "libs": [ "sqlite3", "zstd" ] +} diff --git a/dlang/dub.selections.json b/dlang/dub.selections.json new file mode 100644 index 0000000..84a60aa --- /dev/null +++ b/dlang/dub.selections.json @@ -0,0 +1 @@ +{ "fileVersion": 1, "versions": {} } diff --git a/dlang/flake.lock b/dlang/flake.lock new file mode 100644 index 0000000..5f236ab --- /dev/null +++ b/dlang/flake.lock @@ -0,0 +1,27 @@ +{ + "nodes": { + "nixpkgs": { + "locked": { + "lastModified": 1790185690, + "narHash": "sha256-xJ+X4hBtOcAFGBOe5nAMyMUeF9foJBmIOu3NjBqBycU=", + "owner": "NixOS", + "repo": "nixpkgs", + "rev": "4975466d324710c576dc11ad614684e6bd8cad8e", + "type": "github" + }, + "original": { + "owner": "NixOS", + "ref": "nixos-unstable", + "repo": "nixpkgs", + "type": "github" + } + }, + "root": { + "inputs": { + "nixpkgs": "nixpkgs" + } + } + }, + "root": "root", + "version": 7 +} diff --git a/dlang/flake.nix b/dlang/flake.nix new file mode 100644 index 0000000..3aa9a60 --- /dev/null +++ b/dlang/flake.nix @@ -0,0 +1,63 @@ +{ + description = "ocda-lib: the object-centric document abstraction as a D library"; + + inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable"; + + outputs = { self, nixpkgs }: + let + system = "x86_64-linux"; # adjust for aarch64-linux / darwin + pkgs = import nixpkgs { inherit system; }; + + /* A library, so there is nothing to install as a binary. What is + built here are the two programs that put the library to the + question: the agreement check, which loads a document's .ssp and + its .ocda.db and compares the two models, and the dump, which + writes a loaded model out canonically so that an implementation in + another language can be diffed against this one. + */ + mkTool = { tool, binary, what, compiler, compilerBin }: + pkgs.stdenv.mkDerivation { + pname = "ocda-lib-${tool}"; + version = "2.0.0"; + src = self; + nativeBuildInputs = [ compiler ]; + buildInputs = [ pkgs.sqlite pkgs.zstd ]; + buildPhase = '' + runHook preBuild + export HOME=$TMPDIR + ${compilerBin} -I=src -of=${tool} src/ocda/*.d test/${tool}.d -L-lsqlite3 -L-lzstd + runHook postBuild + ''; + installPhase = '' + runHook preInstall + mkdir -p $out/bin + install -m755 ${tool} $out/bin/${binary} + runHook postInstall + ''; + meta = { + description = what; + mainProgram = binary; + }; + }; + + ldc = { compiler = pkgs.ldc; compilerBin = "ldmd2"; }; + in { + packages.${system} = { + default = self.packages.${system}.ocda-agree; + + ocda-agree = mkTool (ldc // { + tool = "agree"; binary = "ocda-agree"; + what = "checks that the .ssp and .ocda.db loaders give one model"; + }); + + ocda-dump = mkTool (ldc // { + tool = "dump"; binary = "ocda-dump"; + what = "writes a loaded abstraction out canonically, as an oracle"; + }); + }; + + devShells.${system}.default = pkgs.mkShell { + packages = [ pkgs.ldc pkgs.dub pkgs.dtools pkgs.sqlite pkgs.zstd ]; + }; + }; +} diff --git a/dlang/src/COPYRIGHT b/dlang/src/COPYRIGHT new file mode 100644 index 0000000..7e7dd56 --- /dev/null +++ b/dlang/src/COPYRIGHT @@ -0,0 +1,45 @@ +- Name: sisudoc-ocda-lib + - Description: part of SiSU spine (Doc Reform): documents, structuring, + processing, publishing and search + - [https://www.sisudoc.org] + + - Author: Ralph Amissah + [ralph.amissah@gmail.com] + + - Copyright: (C) 2026 Ralph Amissah, All Rights Reserved. + + - License: AGPL 3 or later, for everything in this repository unless a file or + a directory specifies otherwise.: + + sisudoc-ocda-lib + + part of SiSU spine (Doc Reform): documents, structuring, + processing, publishing and search + + Copyright (C) Ralph Amissah + + This program is free software: you can redistribute it and/or modify it + under the terms of the GNU Affero General Public License (AGPLv3) as + published by the Free Software Foundation, either version 3 of the License, + or (at your option) any later version. + + This program is distributed in the hope that it will be useful, but + WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY + or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public + License for more details. + + You should have received a copy of the GNU Affero General Public License + along with this program. If not, see [https://www.gnu.org/licenses/]. + + If you have Internet connection, the latest version of the AGPL should be + available at: [https://www.gnu.org/licenses/agpl-3.0.html] + + - Spine, Doc Reform (related to SiSU) uses standard: + - docReform markup syntax (based on SiSU markup) + - standard SiSU markup syntax with modified headers and minor + modifications + - docReform object numbering (based on SiSU object citation numbering) + - standard SiSU document object numbering + + - Homepages: + [https://www.sisudoc.org] diff --git a/dlang/src/ocda/dump.d b/dlang/src/ocda/dump.d new file mode 100644 index 0000000..7a4f11a --- /dev/null +++ b/dlang/src/ocda/dump.d @@ -0,0 +1,164 @@ +module ocda.dump; + +/+ ↓ the model, written out canonically. + . + Not an output format: an *oracle*. Two libraries in two languages + implement the same model from the same specification, and the cheap + way to ask whether they agree is to have each write the whole loaded + document as text and diff the two. That is only useful if the text is + fully determined by the model, so: + . + - every line is "<key> <value>", the value running to end of line, + and only one free-form value per line; + - a field at its default is *omitted*, so the output says what the + document has rather than what the struct declares; + - dictionaries are emitted in sorted key order, because neither + language's hash map has an order worth relying on; + - sections come in the fixed order of the format, not + alphabetically. + . + Anything that reads this to reconstruct a document has misunderstood + it. Load the .ssp or the .ocda.db. ++/ + +import std.algorithm : sort, splitter; +import std.array : appender, array, join; +import std.conv : to; +import ocda.model; + +/+ ↓ the nine sections, in format order +/ +static immutable string[] dump_sections = ["head", "toc", "body", "endnotes", + "glossary", "bibliography", "bookindex", "blurb", "tail"]; + +string dumpDocument(ref OcdaDocument doc) { + auto out_ = appender!string; + + void kv(string key, string value) { + if (value.length == 0) { return; } + out_ ~= key; out_ ~= " "; out_ ~= value; out_ ~= "\n"; + } + void kn(string key, long value) { + if (value == 0) { return; } + kv(key, value.to!string); + } + void kb(string key, bool value) { + if (!value) { return; } + kv(key, "1"); + } + void dict(string prefix, string[string] d) { + foreach (key; d.keys.dup.sort) { + out_ ~= prefix; out_ ~= " "; out_ ~= key; out_ ~= " "; + out_ ~= d[key]; out_ ~= "\n"; + } + } + + kv("version", doc.format_version); + kv("source_filename", doc.source_filename); + dict("source", doc.source); + dict("meta", doc.metadata); + dict("make", doc.make); + dict("doc_has", doc.doc_has); + + foreach (name; dump_sections) { + auto sec = doc.sectionByName(name); + if (sec is null || (*sec).length == 0) { continue; } + kv("section", name ~ " " ~ (*sec).length.to!string); + } + + foreach (name; dump_sections) { + auto sec = doc.sectionByName(name); + if (sec is null) { continue; } + foreach (i, obj; *sec) { + out_ ~= "object "; out_ ~= name; out_ ~= " "; + out_ ~= i.to!string; out_ ~= "\n"; + kn("ocn", obj.ocn); + kv("is_a", obj.is_a); + kv("own_section", obj.own_section); + kv("level", obj.heading_level); + kv("identifier", obj.identifier); + kv("part", obj.part); + kv("is_of_type", obj.is_of_type); + kv("segment_html_is", obj.segment_html_is); + kv("segment_epub", obj.segment_epub); + kv("segment_epub_is", obj.segment_epub_is); + kn("parent", obj.parent_ocn); + kn("last_descendant", obj.last_descendant_ocn); + kv("children", ints(obj.children)); + kv("ancestors", ints(obj.ancestors)); + kb("dummy", obj.dummy_heading); + kb("ocn_off", obj.ocn_off); + kn("indent_base", obj.indent_base); + kn("indent_hang", obj.indent_hang); + kb("bullet", obj.bullet); + kv("lang", obj.lang); + kb("has_links", obj.has_links); + kb("has_notes_reg", obj.has_notes_reg); + kb("has_notes_star", obj.has_notes_star); + kb("has_images", obj.has_images); + kv("segment", obj.segment); + kv("segment_prev", obj.segment_prev); + kv("segment_next", obj.segment_next); + kv("anchor", obj.anchor); + kv("heading_lev_anchor", obj.heading_lev_anchor); + kn("table_cols", obj.table_cols); + kv("table_widths", obj.table_widths); + kv("table_aligns", obj.table_aligns); + kb("table_header", obj.table_header); + kv("code_syntax", obj.code_syntax); + kb("code_linenumbers", obj.code_linenumbers); + kv("heading_ancestors_text", obj.heading_ancestors_text); + kv("sha256", obj.sha256); + foreach (url; obj.stow_link) { kv("stow_link", url); } + foreach (a; obj.anchor_tags) { kv("anchor_tag", a); } + foreach (img; obj.images) { + /+ ↓ the name last, because a filename may hold a space where + none of the other fields can +/ + kv("image", img.sha256 ~ " " ~ img.bytes.to!string ~ " " + ~ img.width.to!string ~ "x" ~ img.height.to!string ~ " " + ~ (img.missing ? "1" : "0") ~ " " ~ img.name); + foreach (key; img.fields.keys.dup.sort) { + kv("image_field", key ~ " " ~ img.fields[key]); + } + } + /+ ↓ text, when there is any. An object whose only text line is + blank holds "" here, indistinguishable from no text at all, + because the model joins the lines into one string. Both + libraries lose the same distinction in the same place, which is + what makes them comparable; it is recorded rather than hidden. + +/ + if (obj.text.length > 0) { + auto lines = obj.text.splitter("\n").array; + kv("text", lines.length.to!string); + foreach (line; lines) { + out_ ~= "| "; out_ ~= line; out_ ~= "\n"; + } + } + } + } + return out_.data; +} + +/+ ↓ what the reader met and did not understand, sorted. Separate from the + document dump: it describes the reader, not the document, and the + .ocda.db loader has nothing to report where the .ssp loader does. ++/ +string dumpUnknown(ref OcdaDocument doc) { + auto out_ = appender!string; + void counts(string prefix, size_t[string] d) { + foreach (key; d.keys.dup.sort) { + out_ ~= prefix; out_ ~= " "; out_ ~= key; out_ ~= " "; + out_ ~= d[key].to!string; out_ ~= "\n"; + } + } + counts("property", doc.unknown.property); + counts("block", doc.unknown.block); + counts("line", doc.unknown.line); + return out_.data; +} + +private string ints(int[] xs) { + if (xs.length == 0) { return ""; } + string[] parts; + foreach (x; xs) { parts ~= x.to!string; } + return parts.join(","); +} diff --git a/dlang/src/ocda/images.d b/dlang/src/ocda/images.d new file mode 100644 index 0000000..b3a9652 --- /dev/null +++ b/dlang/src/ocda/images.d @@ -0,0 +1,100 @@ +module ocda.images; + +/+ ↓ an image's bytes, from whichever source the document came from. + . + The model *names* images, with digest, byte size and pixel size; it does + not hold them. That is right - an abstraction of a document is not a + bundle of its assets - but a writer of any format that shows images + needs the bytes, so something has to know where they are. This is that + something, and it is in the library rather than in a writer because the + answer depends on the *input form* and not on the output. + . + Three answers, and one refusal: + . + .ocda.db the files table, role 'image', by name + .sisupod the archive entry whose path ends in /image/<name> + .zip the same + .ssp beside it: a pod puts the abstraction in + media/abstraction/ and its images in media/image/, so the + path is documented rather than guessed + . + A .ssp that is not in a pod has nowhere to look, and that is reported as + absent rather than as an error: a reader that cannot find an image + should say so and carry on, because the document is still readable. ++/ + +import std.algorithm : endsWith; +import std.array : array; +import std.file : exists, isFile, read; +import std.path : buildPath, dirName; +import ocda.load; +import ocda.sqlite; +import ocda.zip; +import ocda.zstd; + +/+ ↓ the bytes, or an empty slice when the source has no such image +/ +ubyte[] imageBytes(string source, string name) { + final switch (sourceOf(source)) { + case Source.ocda_db: return fromDb(source, name); + case Source.pod: return fromArchive(source, name); + case Source.ssp: return fromBeside(source, name); + case Source.unknown: return []; + } +} + +/+ ↓ can this source give images at all? + A writer asks before it decides whether to emit them, because an <img> + pointing at a file it did not include is worse than no image. ++/ +bool canGiveImages(string source) { + final switch (sourceOf(source)) { + case Source.ocda_db: return true; + case Source.pod: return true; + case Source.ssp: return imageDirBeside(source).length > 0; + case Source.unknown: return false; + } +} + +private ubyte[] fromDb(string path, string name) { + auto db = Db.open(path); + scope(exit) { db.close(); } + if (!db.has("files")) { return []; } + /+ ↓ the name is from the abstraction, not from a caller, and the quote + doubling is what keeps it a value rather than syntax +/ + return db.blob("SELECT data FROM files WHERE role = 'image' AND name = '" + ~ sqlQuote(name) ~ "'"); +} + +private ubyte[] fromArchive(string path, string name) { + auto bytes = cast(ubyte[]) read(path); + if (zstdIsFrame(bytes)) { bytes = zstdDecompress(bytes, 512 * 1024 * 1024); } + auto entries = zipEntries(bytes); + foreach (e; entries) { + if (e.name.endsWith("/image/" ~ name)) { return zipRead(bytes, e); } + } + return []; +} + +private ubyte[] fromBeside(string path, string name) { + string dir = imageDirBeside(path); + if (dir.length == 0) { return []; } + string file = buildPath(dir, name); + return file.exists && file.isFile ? cast(ubyte[]) read(file) : []; +} + +/+ ↓ media/abstraction/<x>.ssp -> media/image/ +/ +private string imageDirBeside(string path) { + string dir = path.dirName; + if (!dir.endsWith("abstraction")) { return ""; } + string image_dir = buildPath(dir.dirName, "image"); + return image_dir.exists ? image_dir : ""; +} + +/+ ↓ a single quote doubled is how sql says a single quote +/ +private string sqlQuote(string s) { + string out_; + foreach (c; s) { + out_ ~= (c == '\'') ? "''" : [c]; + } + return out_; +} diff --git a/dlang/src/ocda/load.d b/dlang/src/ocda/load.d new file mode 100644 index 0000000..89c8dcc --- /dev/null +++ b/dlang/src/ocda/load.d @@ -0,0 +1,69 @@ +module ocda.load; + +/+ ↓ what a path is, and loading it. + . + One place that decides, so that every reader and every writer accepts + the same inputs and a new input type is added once. This mirrors + spine's own ocda/abstraction/load.d, which names the sources it can + enter the pipeline from and tells them apart by suffix. + . + Deliberately *not* here: .sst and .ssm markup. Reading markup means + implementing the parser, which is spine's work and a far larger thing + than a reader or a sample writer needs. This library reads + abstractions. ++/ + +import std.algorithm : endsWith; +import ocda.model; +import ocda.load_ssp; +import ocda.load_db; +import ocda.load_pod; + +enum Source { + ssp, // .ssp, the abstraction as text + ocda_db, // .ocda.db, the abstraction as sqlite + pod, // .sisupod or .zip, an archive carrying a .ssp + unknown, +} + +Source sourceOf(string path) { + if (path.endsWith(".ocda.db")) { return Source.ocda_db; } + if (path.endsWith(".ssp")) { return Source.ssp; } + /+ ↓ .sisupod is the archive spine writes now; .zip is what it wrote + before, and every pod already published carries that name, so both + are read. What is inside decides how it is opened, not the name. + +/ + if (path.endsWith(".sisupod")) { return Source.pod; } + if (path.endsWith(".zip")) { return Source.pod; } + if (path.endsWith(".db")) { return Source.ocda_db; } + return Source.unknown; +} + +string sourceName(Source s) { + final switch (s) { + case Source.ssp: return ".ssp (the abstraction as text)"; + case Source.ocda_db: return ".ocda.db (the abstraction as sqlite)"; + case Source.pod: return "a pod archive carrying a .ssp"; + case Source.unknown: return "not an abstraction this library reads"; + } +} + +class OcdaLoadError : Exception { + this(string msg) { super(msg); } +} + +/+ ↓ load whatever the path is into the model. + lang names which language to take from a .ocda.db, which holds + every language of its document, and from a pod, which may carry more + than one .ssp. A .ssp is one language by construction and ignores it. ++/ +OcdaDocument load(string path, string lang = "", bool any_version = false) { + final switch (sourceOf(path)) { + case Source.ssp: return parseSspFile(path); + case Source.ocda_db: return loadDbFile(path, lang, any_version); + case Source.pod: return loadPodFile(path, lang); + case Source.unknown: + throw new OcdaLoadError(path ~ ": " ~ sourceName(Source.unknown) + ~ " (expected .ssp, .ocda.db, .sisupod or .zip)"); + } +} diff --git a/dlang/src/ocda/load_db.d b/dlang/src/ocda/load_db.d new file mode 100644 index 0000000..dec0626 --- /dev/null +++ b/dlang/src/ocda/load_db.d @@ -0,0 +1,310 @@ +module ocda.load_db; + +/+ ↓ build the model from a .ocda.db. + . + The point of this module is that it returns *the same value* + ocda.load_ssp returns from the same document's .ssp. A writer is then + a function from the model to bytes, and cannot tell which + serialisation it was given, exactly as spine's own writers cannot. + . + Written from the sisudoc-abstraction-format specification, + ocda-db-format-specification.md, whose section 4 gives the column by + column correspondence this module implements. ++/ + +import std.array : join, split; +import std.json : JSONType, parseJSON; +import std.conv : to; +import std.algorithm : startsWith; +import std.string : strip; +import ocda.model; +import ocda.sqlite; + +/+ ↓ the schema major this loader knows (specification section 2) +/ +enum schema_major_read = "2"; + +class OcdaDbError : Exception { + this(string msg) { super(msg); } +} + +/+ ↓ every language the database holds, in manifest order +/ +struct DbLanguage { + int doc_id; + string lang; + string source_filename; + string ssp_digest; +} + +DbLanguage[] dbLanguages(ref Db db) { + DbLanguage[] out_; + foreach (r; db.rows( + "SELECT id, lang, coalesce(source_filename,''), coalesce(ssp_digest,'') + FROM documents ORDER BY id")) { + out_ ~= DbLanguage(r[0].to!int, r[1], r[2], r[3]); + } + return out_; +} + +/+ ↓ load one language of a database as the model. + lang == "" takes the first language the manifest gave. ++/ +OcdaDocument loadDbFile(string path, string lang = "", bool any_version = false) { + auto db = Db.open(path); + scope(exit) { db.close(); } + + if (!db.has("documents") || !db.has("objects") || !db.has("metadata")) { + throw new OcdaDbError(path ~ ": no documents/objects/metadata tables;" + ~ " it does not look like a .ocda.db"); + } + + /+ ↓ specification sections 2 and 3.2: the rows describing the file rather + than a document in it are the only ones with doc_id null +/ + string schema_version = db.one( + "SELECT value FROM metadata WHERE doc_id IS NULL AND key = 'schema.version'"); + if (schema_version.length > 0 && !any_version) { + auto major = schema_version.split(".")[0]; + if (major != schema_major_read) { + throw new OcdaDbError(path ~ " is schema v" ~ schema_version + ~ ", this loader knows v" ~ schema_major_read ~ ".x"); + } + } + + auto languages = dbLanguages(db); + if (languages.length == 0) { + throw new OcdaDbError(path ~ ": no documents rows"); + } + DbLanguage chosen = languages[0]; + if (lang.length > 0) { + bool found = false; + foreach (l; languages) { if (l.lang == lang) { chosen = l; found = true; break; } } + if (!found) { + string have; + foreach (i, l; languages) { have ~= (i > 0 ? " " : "") ~ l.lang; } + throw new OcdaDbError(path ~ ": no language '" ~ lang ~ "'; it holds: " ~ have); + } + } + + OcdaDocument doc; + doc.format_version = schema_version; + doc.source_filename = chosen.source_filename; + + /+ ↓ @source of the .ssp corresponds to the documents row and the + source.digest metadata (specification section 5.1 of the .ssp, and + section 7 here) +/ + doc.source["language"] = chosen.lang; + string languages_joined; + foreach (i, l; languages) { languages_joined ~= (i > 0 ? " " : "") ~ l.lang; } + doc.source["languages"] = languages_joined; + auto digest = db.one("SELECT value FROM metadata WHERE doc_id = " + ~ chosen.doc_id.to!string ~ " AND key = 'source.digest'"); + if (digest.length > 0) { doc.source["digest"] = digest; } + + /+ ↓ @meta: every metadata row of this language that is not a source.* + property. source.* describes the artefact rather than the document + and lives in @source in the text form. + +/ + foreach (r; db.rows("SELECT key, value FROM metadata WHERE doc_id = " + ~ chosen.doc_id.to!string ~ " ORDER BY key")) { + string key = r[0]; + if (key.startsWith("source.")) { continue; } + /+ ↓ translation.ocn_aligned says whether the languages of this + document share one numbering. A .ocda.db can know it because it + holds every language; a .ssp is one language and cannot. So it is + a fact about the *file* rather than about the document, and putting + it in the document's metadata would let a writer tell which + serialisation it was handed, which is the one thing this library + exists to prevent. Left out until the model has a place for + collection-wide facts. + +/ + if (key.startsWith("translation.")) { continue; } + /+ ↓ the database has one metadata table where the text form has three + blocks, and tells them apart by a key prefix. Unprefixing them + here is what makes the two loaders give one model: a writer asks + doc.make["footer"] whichever serialisation it was handed, and + never doc.metadata["make.footer"]. + +/ + if (key.startsWith("make.")) { + doc.make[key["make.".length .. $]] = r[1].strip; + } else if (key.startsWith("doc_has.")) { + doc.doc_has[key["doc_has.".length .. $]] = r[1].strip; + } else { + doc.metadata[key] = r[1].strip; + } + } + + /+ ↓ the objects, in document order within each section +/ + foreach (r; db.rows( + "SELECT section, seq, ocn, is_a, coalesce(is_of_part,''), + coalesce(is_of_section,''), coalesce(is_of_type,''), + coalesce(heading_level,-1), coalesce(identifier,''), + coalesce(parent_ocn,0), coalesce(last_descendant_ocn,0), + coalesce(children,''), coalesce(ancestors,''), + coalesce(dummy_heading,0), coalesce(object_number_off,0), + coalesce(indent_base,0), coalesce(indent_hang,0), + coalesce(bullet,0), coalesce(lang,''), + coalesce(has_links,0), coalesce(has_notes_reg,0), + coalesce(has_notes_star,0), coalesce(has_images,0), + coalesce(segment,''), coalesce(segment_prev,''), + coalesce(segment_next,''), coalesce(anchor,''), + coalesce(segment_html_is,''), + coalesce(table_cols,0), coalesce(table_widths,''), + coalesce(table_aligns,''), coalesce(table_header,0), + coalesce(code_linenumbers,0), + coalesce(heading_ancestors_text,''), coalesce(sha256,''), + coalesce(text,''), id, coalesce(heading_lev_anchor,''), + coalesce(segment_epub,''), coalesce(segment_epub_is,'') + FROM objects WHERE doc_id = " ~ chosen.doc_id.to!string + ~ " ORDER BY CASE section + WHEN 'head' THEN 1 WHEN 'toc' THEN 2 WHEN 'body' THEN 3 + WHEN 'endnotes' THEN 4 WHEN 'glossary' THEN 5 + WHEN 'bibliography' THEN 6 WHEN 'bookindex' THEN 7 + WHEN 'blurb' THEN 8 WHEN 'tail' THEN 9 ELSE 10 END, seq")) { + OcdaObject obj; + obj.section = r[0]; + obj.ocn = r[2].to!int; + obj.is_a = r[3]; + obj.part = r[4]; + obj.own_section = (r[5] == r[0]) ? "" : r[5]; + obj.is_of_type = r[6]; + /+ ↓ the database holds the level as a number, the text form as the + letter or digit the markup wrote. Model section 4 is the mapping. +/ + obj.heading_level = levelName(r[7].to!int); + obj.identifier = r[8]; + obj.parent_ocn = r[9].to!int; + obj.last_descendant_ocn = r[10].to!int; + obj.children = jsonInts(r[11]); + obj.ancestors = jsonInts(r[12]); + obj.dummy_heading = (r[13] != "0"); + obj.ocn_off = (r[14] != "0"); + obj.indent_base = r[15].to!int; + obj.indent_hang = r[16].to!int; + obj.bullet = (r[17] != "0"); + obj.lang = r[18]; + obj.has_links = (r[19] != "0"); + obj.has_notes_reg = (r[20] != "0"); + obj.has_notes_star = (r[21] != "0"); + obj.has_images = (r[22] != "0"); + obj.segment = r[23]; + obj.segment_prev = r[24]; + obj.segment_next = r[25]; + obj.anchor = r[26]; + obj.segment_html_is = r[27]; + obj.heading_lev_anchor = r[37]; + obj.segment_epub = r[38]; + obj.segment_epub_is = r[39]; + obj.table_cols = r[28].to!int; + obj.table_widths = jsonWords(r[29]); + obj.table_aligns = jsonWords(r[30]); + obj.table_header = (r[31] != "0"); + obj.code_linenumbers = (r[32] != "0"); + obj.heading_ancestors_text = jsonJoined(r[33]); + obj.sha256 = r[34]; + obj.text = r[35]; + + /+ ↓ the four repeatable properties of the text form are child tables + here (specification section 4) +/ + string object_id = r[36]; + foreach (i; db.rows("SELECT name, coalesce(sha256,''), coalesce(bytes,0), + coalesce(width,0), coalesce(height,0), + coalesce(missing,0) + FROM object_images WHERE object_id = " ~ object_id + ~ " ORDER BY seq")) { + OcdaImage img; + img.name = i[0]; + img.sha256 = i[1]; + img.bytes = i[2].to!ulong; + img.width = i[3].to!int; + img.height = i[4].to!int; + img.missing = (i[5] != "0"); + obj.images ~= img; + } + foreach (l; db.rows("SELECT url FROM object_links WHERE object_id = " + ~ object_id ~ " ORDER BY seq")) { + obj.stow_link ~= l[0]; + } + foreach (a; db.rows("SELECT anchor FROM object_anchors WHERE object_id = " + ~ object_id ~ " ORDER BY seq")) { + obj.anchor_tags ~= a[0]; + } + + auto section = doc.sectionByName(r[0]); + if (section !is null) { *section ~= obj; } + } + return doc; +} + +/+ ↓ model section 4: the eight levels and their indices +/ +private string levelName(int index) { + switch (index) { + case 0: return "A"; + case 1: return "B"; + case 2: return "C"; + case 3: return "D"; + case 4: return "1"; + case 5: return "2"; + case 6: return "3"; + case 7: return "4"; + default: return ""; + } +} + +/+ ↓ the JSON columns, read as JSON. + . + The fixed width arrays are JSON in the database and space separated + (or "|" separated) in the text form. This was first written as a + character scan that threw away brackets and quotes, which is wrong the + moment a value holds an escape: Phobos' JSONValue.toString escapes + "/" as "\/", so every heading containing a slash came back with a + backslash in it. 36 of 36 documents differed on + heading_ancestors_text and nothing else had shown it. + . + So: parse. The values are small, there are few of them, and a real + parser is shorter than the scan it replaces. ++/ +private string[] jsonStrings(string v) { + if (v.length == 0) { return []; } + string[] out_; + try { + auto j = parseJSON(v); + if (j.type != JSONType.array) { return []; } + foreach (e; j.array) { + switch (e.type) { + case JSONType.string: out_ ~= e.str; break; + case JSONType.integer: out_ ~= e.integer.to!string; break; + case JSONType.uinteger: out_ ~= e.uinteger.to!string; break; + case JSONType.float_: out_ ~= e.floating.to!string; break; + case JSONType.true_: out_ ~= "true"; break; + case JSONType.false_: out_ ~= "false"; break; + default: out_ ~= ""; break; + } + } + } catch (Exception) { + return []; + } + return out_; +} + +private int[] jsonInts(string v) { + int[] out_; + foreach (s; jsonStrings(v)) { + if (s.length == 0) { continue; } + try { out_ ~= s.to!int; } catch (Exception) {} + } + return out_; +} + +/+ ↓ space separated in the text form: table_widths, table_aligns +/ +private string jsonWords(string v) { + return jsonStrings(v).join(" "); +} + +/+ ↓ heading_ancestors_text is eight slots, joined with "|" in the text + form and a JSON array here. The empty slots are significant: they say + which levels the heading sits under, so the separators carry as much + as the text does. ++/ +private string jsonJoined(string v) { + auto parts = jsonStrings(v); + if (parts.length == 0) { return ""; } + return parts.join("|"); +} diff --git a/dlang/src/ocda/load_pod.d b/dlang/src/ocda/load_pod.d new file mode 100644 index 0000000..75993be --- /dev/null +++ b/dlang/src/ocda/load_pod.d @@ -0,0 +1,115 @@ +module ocda.load_pod; + +/+ ↓ the abstraction out of a pod archive. + . + A .sisupod is one zstd frame wrapping a stored zip; a .zip is the same + archive unwrapped, which is what spine published before it wrapped them. + Either way, a pod built with --pod2 carries the abstraction inside it, + at media/abstraction/<uid>.<lang>.ssp, and that is what is read. + . + So this is a third input form and *not* a third model: the .ssp inside + is parsed by ocda.load_ssp and comes back as the same value. A writer + over the library gains .sisupod without being touched, which is the + whole reason the dispatch is in one place. + . + A pod built without --pod2 carries markup and no abstraction. That is + refused, and the refusal says which flag makes a pod this reader can + read, because "no abstraction in it" is a fact about the pod rather + than about the reader. ++/ + +import std.algorithm : endsWith, sort; +import std.array : array; +import std.file : read; +import std.string : indexOf; +import ocda.model; +import ocda.load_ssp; +import ocda.zip; +import ocda.zstd; + +/+ ↓ a limit on what a frame may expand to. + A reader of an artefact it did not make declines to be told how much + memory to allocate. 512 MiB is far above the largest pod in the + reference collection (4 MiB) and far below anything that matters. ++/ +enum size_t max_pod_size = 512 * 1024 * 1024; + +class OcdaPodError : Exception { + this(string msg) { super(msg); } +} + +/+ ↓ which languages of the abstraction the pod carries, in sorted order +/ +string[] podLanguages(string path) { + auto bytes = archiveBytes(path); + string[] out_; + foreach (e; zipFindSuffix(zipEntries(bytes), ".ssp")) { + string lang = langOfSspName(e.name); + if (lang.length > 0) { out_ ~= lang; } + } + return out_.sort.array; +} + +/+ ↓ load one language of the abstraction a pod carries. + lang == "" takes the only one there is, and where there is a choice it + says so rather than choosing, as the .ocda.db loader does. ++/ +OcdaDocument loadPodFile(string path, string lang = "") { + auto bytes = archiveBytes(path); + auto entries = zipEntries(bytes); + auto ssps = zipFindSuffix(entries, ".ssp"); + if (ssps.length == 0) { + throw new OcdaPodError(path ~ ": carries no .ssp." + ~ " A pod carries the abstraction only when spine was asked for it:" + ~ " spine --pod2 puts it in, spine --source does not."); + } + ZipEntry chosen = ssps[0]; + if (lang.length > 0) { + bool found = false; + foreach (e; ssps) { + if (langOfSspName(e.name) == lang) { chosen = e; found = true; break; } + } + if (!found) { + string have; + foreach (i, e; ssps) { + have ~= (i > 0 ? " " : "") ~ langOfSspName(e.name); + } + throw new OcdaPodError(path ~ ": no language '" ~ lang + ~ "'; it holds: " ~ have); + } + } else if (ssps.length > 1) { + string have; + foreach (i, e; ssps) { + have ~= (i > 0 ? " " : "") ~ langOfSspName(e.name); + } + throw new OcdaPodError(path ~ ": holds more than one language (" + ~ have ~ "); name the one wanted"); + } + auto ssp = zipRead(bytes, chosen); + return parseSsp(cast(string) ssp.idup); +} + +/+ ↓ the archive, unwrapped if it was wrapped. + The bytes decide, not the name: a .sisupod is a zstd frame and a .zip + is not, and a file misnamed either way still reads. ++/ +private ubyte[] archiveBytes(string path) { + auto bytes = cast(ubyte[]) read(path); + return zstdIsFrame(bytes) + ? zstdDecompress(bytes, max_pod_size) + : bytes; +} + +/+ ↓ <uid>.<lang>.ssp -> <lang> +/ +private string langOfSspName(string name) { + if (!name.endsWith(".ssp")) { return ""; } + string stem = name[0 .. $ - 4]; + auto dot = stem.lastIndexOf('.'); + return (dot > 0) ? stem[dot + 1 .. $] : ""; +} + +private ptrdiff_t lastIndexOf(string s, char c) { + for (ptrdiff_t i = s.length - 1; i >= 0; i--) { + if (s[i] == c) { return i; } + } + return -1; +} diff --git a/dlang/src/ocda/load_ssp.d b/dlang/src/ocda/load_ssp.d new file mode 100644 index 0000000..34d156c --- /dev/null +++ b/dlang/src/ocda/load_ssp.d @@ -0,0 +1,337 @@ +module ocda.load_ssp; + +import std.algorithm : splitter, startsWith; +import std.array : array, split; +import std.conv : to, ConvException; +import std.file : readText; +import std.stdio : writeln; +import std.string : strip, stripLeft, indexOf; +import ocda.model; + +OcdaDocument parseSspFile(string filename) { + string content = readText(filename); + return parseSsp(content); +} + +OcdaDocument parseSsp(string content) { + OcdaDocument doc; + OcdaObject current_obj; + bool in_object = false; + /+ ↓ whether a text line has been seen for the object being read. + Not "is the text non-empty": an object whose first text line is blank + (a verse opening on an empty line, which the sample set has seven of) + would then lose it, the next line overwriting rather than appending. + +/ + bool seen_text = false; + string current_section = ""; + string current_block = ""; // meta, make, doc_has, or section name + + foreach (line; content.splitter("\n")) { + /+ ↓ comment. The first line of a .ssp declares the format version, and + it is the one comment that carries meaning: a reader checks the + major part and refuses what it does not know rather than loading + an unfamiliar artefact half populated in silence. + +/ + if (line.length > 0 && line[0] == '%') { + enum _banner = "% SiSU Document Abstraction v"; + enum _source = "% Source: "; + if (doc.format_version.length == 0 && line.startsWith(_banner)) { + doc.format_version = line[_banner.length .. $].strip; + } else if (doc.source_filename.length == 0 && line.startsWith(_source)) { + doc.source_filename = line[_source.length .. $].strip; + } + continue; + } + + /+ ↓ blank line - finalize current object +/ + if (line.length == 0 || line.strip.length == 0) { + if (in_object) { + commitObject(doc, current_section, current_obj); + current_obj = OcdaObject.init; + in_object = false; + seen_text = false; + } + continue; + } + + /+ ↓ section/block open +/ + if (line[0] == '@') { + auto brace_pos = line.indexOf('{'); + if (brace_pos > 0) { + string name = line[1 .. brace_pos].strip; + current_block = name; + if (!isPreambleBlock(name)) { + current_section = name; + /+ ↓ a block that is neither preamble nor a section this reader + knows: counted, so that --report-unknown can say so +/ + if (doc.sectionByName(name) is null) { + doc.unknown.block[name]++; + } + } + } + continue; + } + + /+ ↓ section/block close +/ + if (line[0] == '}') { + if (in_object) { + commitObject(doc, current_section, current_obj); + current_obj = OcdaObject.init; + in_object = false; + seen_text = false; + } + current_block = ""; + continue; + } + + /+ ↓ inside meta/make/doc_has blocks: key-value pairs +/ + if (isPreambleBlock(current_block)) { + string stripped = line.stripLeft; + auto colon_pos = stripped.indexOf(':'); + if (colon_pos > 0) { + string key = stripped[0 .. colon_pos].strip; + string value = (colon_pos + 1 < stripped.length) + ? stripped[colon_pos + 1 .. $].strip : ""; + if (current_block == "meta") { doc.metadata[key] = value; } + else if (current_block == "make") { doc.make[key] = value; } + else if (current_block == "source") { doc.source[key] = value; } + else { doc.doc_has[key] = value; } + } + continue; + } + + /+ ↓ object declaration: [N] type ... +/ + if (line[0] == '[') { + if (in_object) { + commitObject(doc, current_section, current_obj); + } + current_obj = OcdaObject.init; + current_obj.section = current_section; + in_object = true; + seen_text = false; + parseObjectDeclaration(line, current_obj); + continue; + } + + /+ ↓ property: .name: value +/ + if (line[0] == '.') { + if (in_object) { + parseProperty(line, current_obj, doc.unknown); + } + continue; + } + + /+ ↓ text content: | content +/ + if (line.length >= 1 && line[0] == '|') { + if (in_object) { + string text_content = (line.length > 2) ? line[2 .. $] : ""; + if (seen_text) { + current_obj.text ~= "\n" ~ text_content; + } else { + current_obj.text = text_content; + seen_text = true; + } + } + continue; + } + + /+ ↓ no known shape. Counted by its first character so that a reader + can be told the file held something this one did not expect + +/ + doc.unknown.line[line[0 .. 1].idup]++; + } + + /+ ↓ finalize last object +/ + if (in_object) { + commitObject(doc, current_section, current_obj); + } + + return doc; +} + +private void parseObjectDeclaration(string line, ref OcdaObject obj) { + /+ parse: [N] type :level identifier +/ + auto bracket_close = line.indexOf(']'); + if (bracket_close < 0) return; + try { + obj.ocn = line[1 .. bracket_close].strip.to!int; + } catch (ConvException) { + obj.ocn = 0; + } + string rest = (bracket_close + 1 < line.length) + ? line[bracket_close + 1 .. $].strip : ""; + if (rest.length == 0) return; + + auto parts = rest.split(" "); + if (parts.length == 0) return; + obj.is_a = parts[0]; + + if (obj.is_a == "heading" && parts.length >= 2) { + /+ parse :A, :B, :1, etc. +/ + string lev = parts[1]; + if (lev.length >= 2 && lev[0] == ':') { + obj.heading_level = lev[1 .. $]; + } + /+ optional identifier after level +/ + if (parts.length >= 3) { + obj.identifier = parts[2]; + } + } +} + +/+ ↓ the blocks that are document preamble rather than sections of objects +/ +private bool isPreambleBlock(string name) { + return name == "source" || name == "meta" || name == "make" || name == "doc_has"; +} + +private void parseProperty(string line, ref OcdaObject obj, ref OcdaUnknown unknown) { + /+ parse: .name: value +/ + auto colon_pos = line.indexOf(':'); + if (colon_pos < 1) return; + string name = line[1 .. colon_pos].strip; + string value = (colon_pos + 1 < line.length) + ? line[colon_pos + 1 .. $].strip : ""; + + switch (name) { + case "part": obj.part = value; break; + case "is_of_type": obj.is_of_type = value; break; + case "section": obj.own_section = value; break; + case "identifier": obj.identifier = value; break; + case "segment_html_is": obj.segment_html_is = value; break; + /+ ↓ the epub segments at heading levels 1 to 4 where the html + segments at level 4 alone, so the two are different files and + the abstraction records both +/ + case "segment_epub": obj.segment_epub = value; break; + case "segment_epub_is": obj.segment_epub_is = value; break; + case "table_aligns": obj.table_aligns = value; break; + case "stow_link": obj.stow_link ~= value; break; + case "anchor_tag": obj.anchor_tags ~= value; break; + case "parent": obj.parent_ocn = toIntSafe(value); break; + case "last_descendant": obj.last_descendant_ocn = toIntSafe(value); break; + case "children": obj.children = parseIntArray(value); break; + case "ancestors": obj.ancestors = parseIntArray(value); break; + case "dummy": obj.dummy_heading = (value == "true"); break; + case "ocn_off": obj.ocn_off = (value == "true"); break; + case "indent": + auto parts = value.split(" "); + if (parts.length >= 1) obj.indent_base = toIntSafe(parts[0]); + if (parts.length >= 2) obj.indent_hang = toIntSafe(parts[1]); + break; + case "bullet": obj.bullet = (value == "true"); break; + case "lang": obj.lang = value; break; + case "has": + foreach (flag; value.split(" ")) { + switch (flag) { + case "links": obj.has_links = true; break; + case "notes_reg": obj.has_notes_reg = true; break; + case "notes_star": obj.has_notes_star = true; break; + case "images": obj.has_images = true; break; + default: break; + } + } + break; + case "segment": obj.segment = value; break; + case "segment_prev": obj.segment_prev = value; break; + case "segment_next": obj.segment_next = value; break; + case "anchor": obj.anchor = value; break; + case "heading_lev_anchor": obj.heading_lev_anchor = value; break; + case "table_cols": obj.table_cols = toIntSafe(value); break; + case "table_widths": obj.table_widths = value; break; + case "table_header": obj.table_header = (value == "true"); break; + case "code_syntax": obj.code_syntax = value; break; + case "code_linenumbers": obj.code_linenumbers = (value == "true"); break; + case "heading_ancestors_text": obj.heading_ancestors_text = value; break; + case "sha256": obj.sha256 = value; break; + case "image": obj.images ~= parseImageRecord(value); break; + /+ ↓ a property this reader has no use for, or has never heard of. + Ignored, which is the conforming behaviour within a major + version, but counted rather than passed over in silence + +/ + default: unknown.property[name]++; break; + } +} + +private OcdaImage parseImageRecord(string value) { + /+ parse: <filename> sha256:<HEX> bytes:<n> [key:value ...] + filename first, remaining fields order-independent; unrecognised + fields are kept rather than discarded +/ + OcdaImage img; + bool first = true; + foreach (field; value.split(" ")) { + if (field.length == 0) continue; + if (first) { + img.name = field; + first = false; + continue; + } + auto colon_pos = field.indexOf(':'); + if (colon_pos < 1) continue; + string key = field[0 .. colon_pos]; + string val = (colon_pos + 1 < field.length) ? field[colon_pos + 1 .. $] : ""; + switch (key) { + case "sha256": img.sha256 = val; break; + case "bytes": img.bytes = toUlongSafe(val); break; + case "px": + auto dims = val.split("x"); + if (dims.length == 2) { + img.width = toIntSafe(dims[0]); + img.height = toIntSafe(dims[1]); + } + break; + case "missing": img.missing = (val == "true"); break; + default: img.fields[key] = val; break; + } + } + return img; +} + +/+ ↓ two values the text form leaves out because they can be worked out. + . + The .ssp emits a property only when it carries information, so + .identifier is absent when it is simply the ocn, and .anchor is absent + when it is the segment name. The .ocda.db, being columns rather than + lines, stores both unconditionally. So a loader that takes the text + form literally gives a *different model* from the one the database + gives, and the difference is not small: in the reference collection + the identifier is omitted on all but a handful of objects. + . + The model is the thing both serialisations describe, so the loader + fills them in. Found by dumping both models and diffing them, which no + amount of comparing section counts would have shown. ++/ +private void deriveOmitted(ref OcdaObject obj) { + import std.conv : to; + if (obj.identifier.length == 0 && obj.ocn != 0) { + obj.identifier = obj.ocn.to!string; + } + if (obj.anchor.length == 0 && obj.segment.length > 0) { + obj.anchor = obj.segment; + } +} + +private void commitObject(ref OcdaDocument doc, string section, ref OcdaObject obj) { + deriveOmitted(obj); + auto sec = doc.sectionByName(section); + if (sec !is null) { + *sec ~= obj; + } +} + +private int toIntSafe(string s) { + try { return s.strip.to!int; } + catch (ConvException) { return 0; } +} + +private ulong toUlongSafe(string s) { + try { return s.strip.to!ulong; } + catch (ConvException) { return 0; } +} + +private int[] parseIntArray(string s) { + int[] result; + foreach (part; s.split(" ")) { + auto v = toIntSafe(part); + result ~= v; + } + return result; +} diff --git a/dlang/src/ocda/model.d b/dlang/src/ocda/model.d new file mode 100644 index 0000000..403502a --- /dev/null +++ b/dlang/src/ocda/model.d @@ -0,0 +1,147 @@ +module ocda.model; + +struct OcdaImage { + string name; // image filename as referenced by the object + string sha256; // hex-encoded (uppercase) digest of the file + ulong bytes; // file size + int width; // file's own pixel width (not display width) + int height; // file's own pixel height + bool missing; // file absent or unreadable, no sha256/bytes + string[string] fields; // any further/unrecognised key:value fields +} + +struct OcdaObject { + int ocn; + string is_a; // heading, para, toc, table, code, group, block, verse, quote, endnote, glossary, bibliography, bookindex, blurb + string section; // head, toc, body, endnotes, glossary, bibliography, bookindex, blurb + string heading_level; // A, B, C, D, 1, 2, 3, 4 (for headings) + string identifier; // heading: from the declaration line; otherwise .identifier + string part; // frontmatter, body, backmatter + string is_of_type; // para, block, comment: how the text behaves + string own_section; // .section, when it differs from the block it sits in + string segment_html_is;// the html segment this object sits in + string segment_epub; // the epub segment this heading opens + string segment_epub_is;// the epub segment this object sits in + string table_aligns; // l c r per column + string[] stow_link; // links extracted from the object + string[] anchor_tags; // named targets the markup placed on this object + int parent_ocn; + int last_descendant_ocn; + int[] children; // direct child heading OCNs + int[] ancestors; // markedup_ancestors[8] + bool dummy_heading; + bool ocn_off; + int indent_base; + int indent_hang; + bool bullet; + string lang; + bool has_links; + bool has_notes_reg; + bool has_notes_star; + bool has_images; + string segment; + string segment_prev; + string segment_next; + string anchor; + string heading_lev_anchor; // a heading's own named target + int table_cols; + string table_widths; + bool table_header; + string code_syntax; + bool code_linenumbers; + string heading_ancestors_text; + string sha256; + OcdaImage[] images; // repeatable .image records, in order + string text; // content (lines joined with newline) + + /+ ↓ whether this object is rendered at all. + is_of_type "comment" is markup commentary: carried by the + abstraction so that nothing is lost, and not put into output. An + output that renders it leaks the author notes into the document, + which is what spine itself does not do. + +/ + bool renders() const { + return is_of_type != "comment"; + } + + /+ ↓ the section this object belongs to, which is the block it sits in + unless it says otherwise +/ + string sectionOf() const { + return (own_section.length > 0) ? own_section : section; + } +} + +/+ ↓ what the reader met and did not understand. + A reader that silently drops what it does not know cannot say whether + it understood the document, and cannot be used to check a + specification against an implementation: silence and understanding + look alike. So the parser counts what it ignored, and + --report-unknown prints it. ++/ +struct OcdaUnknown { + size_t[string] property; // .name -> times seen + size_t[string] block; // @name -> times seen + size_t[string] line; // first char -> times seen, lines of no known shape + + bool any() const { + return property.length > 0 || block.length > 0 || line.length > 0; + } +} + +struct OcdaDocument { + string format_version; // as declared on line 1, "" when absent + string source_filename; // from the "% Source:" line + string[string] source; // @source: language, languages, digest + OcdaUnknown unknown; // what was met and not understood + string[string] metadata; // key -> value + string[string] make; // key -> value + string[string] doc_has; // key -> value + OcdaObject[] head; + OcdaObject[] toc; + OcdaObject[] body; + OcdaObject[] endnotes; + OcdaObject[] glossary; + OcdaObject[] bibliography; + OcdaObject[] bookindex; + OcdaObject[] blurb; + OcdaObject[] tail; + + /+ ↓ the major part of the declared version, "" when none was declared +/ + string formatVersionMajor() const { + import std.string : indexOf; + if (format_version.length == 0) { return ""; } + auto dot = format_version.indexOf('.'); + return (dot > 0) ? format_version[0 .. dot] : format_version; + } + + string meta(string key) { + if (auto p = key in metadata) return *p; + return ""; + } + + OcdaObject[]* sectionByName(string name) { + switch (name) { + case "head": return &head; + case "toc": return &toc; + case "body": return &body; + case "endnotes": return &endnotes; + case "glossary": return &glossary; + case "bibliography": return &bibliography; + case "bookindex": return &bookindex; + case "blurb": return &blurb; + case "tail": return &tail; + default: return null; + } + } +} + +/+ ↓ the format major version this reader knows. + The .ssp and the .ocda.db carry one version between them, declared in + band: the .ssp on its first line, the database in its schema.version + row. The minor part is raised when a property is added, and a reader of + the newer format still reads the older one, so only the major part is + checked here. Raise this when the reader has been brought to a new + major, not before: the point of the gate is to refuse what has not been + looked at. ++/ +enum ssp_format_major_read = "2"; diff --git a/dlang/src/ocda/sqlite.d b/dlang/src/ocda/sqlite.d new file mode 100644 index 0000000..6f3753c --- /dev/null +++ b/dlang/src/ocda/sqlite.d @@ -0,0 +1,122 @@ +module ocda.sqlite; + +/+ ↓ the sqlite3 C API, declared rather than depended upon. + . + A binding from the dub registry would be a dependency to fetch, and + the point of this reader is that it builds in a sandbox with no + network. The dozen functions a reader needs are declared here and the + library is linked directly, which costs a page of extern(C) and saves + a dependency. ++/ + +import std.conv : to; +import std.string : fromStringz, toStringz; + +extern (C) { + struct sqlite3; + struct sqlite3_stmt; + + int sqlite3_open_v2(const(char)* filename, sqlite3** db, int flags, const(char)* vfs); + int sqlite3_close(sqlite3* db); + int sqlite3_prepare_v2(sqlite3* db, const(char)* sql, int nbyte, + sqlite3_stmt** stmt, const(char)** tail); + int sqlite3_step(sqlite3_stmt* stmt); + int sqlite3_finalize(sqlite3_stmt* stmt); + int sqlite3_column_count(sqlite3_stmt* stmt); + const(char)* sqlite3_column_name(sqlite3_stmt* stmt, int col); + const(ubyte)* sqlite3_column_text(sqlite3_stmt* stmt, int col); + int sqlite3_column_int(sqlite3_stmt* stmt, int col); + int sqlite3_column_type(sqlite3_stmt* stmt, int col); + const(char)* sqlite3_errmsg(sqlite3* db); + const(void)* sqlite3_column_blob(sqlite3_stmt* stmt, int col); + int sqlite3_column_bytes(sqlite3_stmt* stmt, int col); +} + +enum SQLITE_OK = 0; +enum SQLITE_ROW = 100; +enum SQLITE_DONE = 101; +enum SQLITE_NULL = 5; +enum SQLITE_OPEN_READONLY = 0x00000001; + +class SqliteError : Exception { + this(string msg) { super(msg); } +} + +/+ ↓ a database, opened read only. + Read only is deliberate: a reader has no business writing, and a + request that opens a database read-write can create an empty one where + a path is mistyped. ++/ +struct Db { + private sqlite3* _handle; + + static Db open(string path) { + sqlite3* handle; + auto rc = sqlite3_open_v2(path.toStringz, &handle, SQLITE_OPEN_READONLY, null); + if (rc != SQLITE_OK) { + string msg = (handle is null) + ? "cannot open " ~ path + : "cannot open " ~ path ~ ": " ~ sqlite3_errmsg(handle).fromStringz.idup; + if (handle !is null) { sqlite3_close(handle); } + throw new SqliteError(msg); + } + Db db; + db._handle = handle; + return db; + } + + void close() { + if (_handle !is null) { sqlite3_close(_handle); _handle = null; } + } + + /+ ↓ every row of a query, each row as its columns in order +/ + string[][] rows(string sql) { + sqlite3_stmt* stmt; + auto rc = sqlite3_prepare_v2(_handle, sql.toStringz, -1, &stmt, null); + if (rc != SQLITE_OK) { + throw new SqliteError(sqlite3_errmsg(_handle).fromStringz.idup ~ " in: " ~ sql); + } + scope(exit) { sqlite3_finalize(stmt); } + string[][] out_rows; + int cols = sqlite3_column_count(stmt); + while (sqlite3_step(stmt) == SQLITE_ROW) { + string[] row; + foreach (c; 0 .. cols) { + row ~= (sqlite3_column_type(stmt, c) == SQLITE_NULL) + ? "" : (cast(const(char)*) sqlite3_column_text(stmt, c)).fromStringz.idup; + } + out_rows ~= row; + } + return out_rows; + } + + /+ ↓ the first column of the first row, "" when there is none +/ + string one(string sql) { + auto r = rows(sql); + return (r.length > 0 && r[0].length > 0) ? r[0][0] : ""; + } + + /+ ↓ the first column of the first row as bytes, for a blob. + Separate from one(), because a blob is not text: an image read as text + stops at the first zero byte, and a png's ninth byte is zero. + +/ + ubyte[] blob(string sql) { + sqlite3_stmt* stmt; + if (sqlite3_prepare_v2(_handle, sql.toStringz, -1, &stmt, null) != SQLITE_OK) { + throw new SqliteError("prepare: " ~ sqlite3_errmsg(_handle).fromStringz.idup + ~ " [" ~ sql ~ "]"); + } + scope(exit) { sqlite3_finalize(stmt); } + if (sqlite3_step(stmt) != SQLITE_ROW) { return []; } + int n = sqlite3_column_bytes(stmt, 0); + if (n <= 0) { return []; } + auto p = cast(const(ubyte)*) sqlite3_column_blob(stmt, 0); + if (p is null) { return []; } + return p[0 .. n].dup; + } + + /+ ↓ does a table or view of this name exist +/ + bool has(string name) { + return one("SELECT count(*) FROM sqlite_master WHERE name = '" ~ name ~ "'") != "0"; + } +} diff --git a/dlang/src/ocda/zip.d b/dlang/src/ocda/zip.d new file mode 100644 index 0000000..9dff6f4 --- /dev/null +++ b/dlang/src/ocda/zip.d @@ -0,0 +1,115 @@ +module ocda.zip; + +/+ ↓ enough of the zip format to find one file in a pod and read it. + . + Not a zip library. A pod archive holds a handful of entries, spine + writes them **stored** (uncompressed) so that the outer zstd frame does + the compressing once over the whole thing, and what is wanted from it + here is one named entry. So: read the central directory, find the + entry, return its bytes. + . + Deflate is *refused rather than mishandled*. Phobos can inflate, but a + library that says it read an entry it in fact skipped is worse than one + that says it cannot, and spine's own archives do not need it. ++/ + +import std.algorithm : endsWith; +import std.conv : to; + +class OcdaZipError : Exception { + this(string msg) { super(msg); } +} + +struct ZipEntry { + string name; + uint method; // 0 stored, 8 deflate + uint compressed; + uint uncompressed; + uint local_offset; +} + +/+ ↓ the entries, from the central directory at the end of the file +/ +ZipEntry[] zipEntries(const(ubyte)[] bytes) { + enum uint eocd_sig = 0x06054b50; + enum uint cd_sig = 0x02014b50; + if (bytes.length < 22) { + throw new OcdaZipError("too short to be a zip archive"); + } + /+ ↓ the end of central directory record is last, and may be followed by a + comment of up to 65535 bytes, so it is searched for backwards +/ + ptrdiff_t eocd = -1; + ptrdiff_t lowest = bytes.length > (22 + 65535) ? bytes.length - (22 + 65535) : 0; + for (ptrdiff_t i = bytes.length - 22; i >= lowest; i--) { + if (u32(bytes, i) == eocd_sig) { eocd = i; break; } + } + if (eocd < 0) { + throw new OcdaZipError("no end of central directory record;" + ~ " it does not look like a zip archive"); + } + uint count = u16(bytes, eocd + 10); + uint offset = u32(bytes, eocd + 16); + ZipEntry[] out_; + size_t p = offset; + foreach (_; 0 .. count) { + if (p + 46 > bytes.length || u32(bytes, p) != cd_sig) { + throw new OcdaZipError("central directory entry " + ~ out_.length.to!string ~ " is malformed"); + } + ZipEntry e; + e.method = u16(bytes, p + 10); + e.compressed = u32(bytes, p + 20); + e.uncompressed = u32(bytes, p + 24); + uint name_len = u16(bytes, p + 28); + uint extra_len = u16(bytes, p + 30); + uint cmnt_len = u16(bytes, p + 32); + e.local_offset = u32(bytes, p + 42); + e.name = cast(string) bytes[p + 46 .. p + 46 + name_len].idup; + out_ ~= e; + p += 46 + name_len + extra_len + cmnt_len; + } + return out_; +} + +/+ ↓ one entry's bytes, from its local header +/ +ubyte[] zipRead(const(ubyte)[] bytes, ZipEntry e) { + enum uint local_sig = 0x04034b50; + size_t p = e.local_offset; + if (p + 30 > bytes.length || u32(bytes, p) != local_sig) { + throw new OcdaZipError(e.name ~ ": local header is malformed"); + } + uint name_len = u16(bytes, p + 26); + uint extra_len = u16(bytes, p + 28); + size_t start = p + 30 + name_len + extra_len; + size_t end = start + e.compressed; + if (end > bytes.length) { + throw new OcdaZipError(e.name ~ ": runs past the end of the archive"); + } + if (e.method != 0) { + throw new OcdaZipError(e.name ~ ": compression method " + ~ e.method.to!string ~ "; this reader takes only stored entries," + ~ " which is what spine writes"); + } + return bytes[start .. end].dup; +} + +/+ ↓ the first entry whose name ends this way, or none. + Used to find media/abstraction/<uid>.<lang>.ssp without knowing the uid. ++/ +ZipEntry[] zipFindSuffix(ZipEntry[] entries, string suffix) { + ZipEntry[] out_; + foreach (e; entries) { + if (e.name.endsWith(suffix)) { out_ ~= e; } + } + return out_; +} + +private uint u16(const(ubyte)[] b, size_t i) { + return cast(uint) b[i] | (cast(uint) b[i + 1] << 8); +} + +private uint u32(const(ubyte)[] b, size_t i) { + return cast(uint) b[i] + | (cast(uint) b[i + 1] << 8) + | (cast(uint) b[i + 2] << 16) + | (cast(uint) b[i + 3] << 24); +} diff --git a/dlang/src/ocda/zstd.d b/dlang/src/ocda/zstd.d new file mode 100644 index 0000000..401f1a3 --- /dev/null +++ b/dlang/src/ocda/zstd.d @@ -0,0 +1,79 @@ +module ocda.zstd; + +/+ ↓ libzstd, declared rather than bound. + . + A .sisupod is one zstd frame wrapping a zip archive, so reading one + needs exactly two things from the library: is this a frame, and what is + in it. Both are declared here and the C library is linked, which is the + same arrangement ocda.sqlite uses: no vendored tree, no generated + header, and the prototypes say what is being called. ++/ + +extern (C) @nogc nothrow private { + size_t ZSTD_decompress(void* dst, size_t dstCapacity, + const(void)* src, size_t compressedSize); + ulong ZSTD_getFrameContentSize(const(void)* src, size_t srcSize); + uint ZSTD_isError(size_t code); + const(char)* ZSTD_getErrorName(size_t code); +} + +/+ ↓ the two values ZSTD_getFrameContentSize returns in place of a size +/ +private enum ulong contentSizeUnknown = ulong.max; +private enum ulong contentSizeError = ulong.max - 1; + +/+ ↓ the magic number of a zstd frame, little endian 0xFD2FB528 +/ +private static immutable ubyte[4] zstd_magic = [0x28, 0xB5, 0x2F, 0xFD]; + +class OcdaZstdError : Exception { + this(string msg) { super(msg); } +} + +/+ ↓ does this begin with a zstd frame? + Asked of the bytes rather than of the filename, because what is inside + decides how a thing is opened. A .sisupod is a frame; a .zip published + before spine wrapped them is not, and both are read. ++/ +bool zstdIsFrame(const(ubyte)[] bytes) { + if (bytes.length < zstd_magic.length) { return false; } + foreach (i, b; zstd_magic) { + if (bytes[i] != b) { return false; } + } + return true; +} + +/+ ↓ one frame in, its contents out. + max_size is a refusal rather than a buffer size: a frame declares how + much it expands to, and a frame that claims more than the caller is + willing to hold is not decompressed at all. A reader of an artefact it + did not make wants that limit. ++/ +ubyte[] zstdDecompress(const(ubyte)[] bytes, size_t max_size) { + if (!zstdIsFrame(bytes)) { + throw new OcdaZstdError("not a zstd frame"); + } + ulong declared = ZSTD_getFrameContentSize(bytes.ptr, bytes.length); + if (declared == contentSizeError) { + throw new OcdaZstdError("zstd frame: could not read its content size"); + } + if (declared == contentSizeUnknown) { + throw new OcdaZstdError("zstd frame declares no content size;" + ~ " this reader takes only frames that do"); + } + if (declared > max_size) { + import std.conv : to; + throw new OcdaZstdError("zstd frame expands to " ~ declared.to!string + ~ " bytes, over the " ~ max_size.to!string ~ " byte limit"); + } + auto out_ = new ubyte[](cast(size_t) declared); + size_t got = ZSTD_decompress(out_.ptr, out_.length, bytes.ptr, bytes.length); + if (ZSTD_isError(got)) { + import std.string : fromStringz; + throw new OcdaZstdError("zstd: " + ~ ZSTD_getErrorName(got).fromStringz.idup); + } + if (got != declared) { + throw new OcdaZstdError("zstd frame expanded to a different size" + ~ " than it declared"); + } + return out_; +} diff --git a/dlang/test/agree.d b/dlang/test/agree.d new file mode 100644 index 0000000..bdf1b2e --- /dev/null +++ b/dlang/test/agree.d @@ -0,0 +1,84 @@ +/++ + Do the two loaders agree? + + The library's whole purpose is that a writer cannot tell which + serialisation it was handed. That is only true if loading a document's + .ssp and loading its .ocda.db give the same model, so this asks. + + Usage: agree <doc>.ssp <doc>.ocda.db [lang] ++/ +import std.conv : to; +import std.stdio; +import ocda.model; +import ocda.load; + +int main(string[] args) { + if (args.length < 3) { + stderr.writeln("usage: agree <doc>.ssp <doc>.ocda.db [lang]"); + return 2; + } + string lang = (args.length > 3) ? args[3] : ""; + OcdaDocument a, b; + try { + a = load(args[1]); + b = load(args[2], lang); + } catch (Exception e) { + stderr.writeln("error: ", e.msg); + return 2; + } + + int differences = 0; + void differ(string what, string from_ssp, string from_db) { + writeln(" differ: ", what); + writeln(" .ssp: ", from_ssp); + writeln(" .ocda.db: ", from_db); + differences++; + } + + if (a.format_version != b.format_version) { + differ("format_version", a.format_version, b.format_version); + } + if (a.source_filename != b.source_filename) { + differ("source_filename", a.source_filename, b.source_filename); + } + + /+ ↓ the sections, object by object. Position within a section is the + identity to compare on: an ocn is not unique and 0 is shared +/ + static immutable sections = ["head", "toc", "body", "endnotes", "glossary", + "bibliography", "bookindex", "blurb", "tail"]; + foreach (name; sections) { + auto sa = a.sectionByName(name); + auto sb = b.sectionByName(name); + if (sa is null || sb is null) { continue; } + if ((*sa).length != (*sb).length) { + differ(name ~ " object count", (*sa).length.to!string, (*sb).length.to!string); + continue; + } + foreach (i, oa; *sa) { + auto ob = (*sb)[i]; + string at = name ~ "[" ~ i.to!string ~ "]"; + if (oa.ocn != ob.ocn) { differ(at ~ " ocn", oa.ocn.to!string, ob.ocn.to!string); } + if (oa.is_a != ob.is_a) { differ(at ~ " is_a", oa.is_a, ob.is_a); } + if (oa.is_of_type != ob.is_of_type) { differ(at ~ " is_of_type", oa.is_of_type, ob.is_of_type); } + if (oa.part != ob.part) { differ(at ~ " part", oa.part, ob.part); } + if (oa.parent_ocn != ob.parent_ocn) { differ(at ~ " parent", oa.parent_ocn.to!string, ob.parent_ocn.to!string); } + if (oa.heading_level != ob.heading_level) { differ(at ~ " level", oa.heading_level, ob.heading_level); } + if (oa.sha256 != ob.sha256) { differ(at ~ " sha256", oa.sha256, ob.sha256); } + if (oa.text != ob.text) { differ(at ~ " text", oa.text, ob.text); } + if (oa.ancestors != ob.ancestors) { + differ(at ~ " ancestors", oa.ancestors.to!string, ob.ancestors.to!string); + } + if (oa.images.length != ob.images.length) { + differ(at ~ " image count", oa.images.length.to!string, ob.images.length.to!string); + } + if (differences > 12) { writeln(" ... stopping after 12"); return 1; } + } + } + + if (differences == 0) { + writeln("agree: the two loaders give the same model"); + return 0; + } + writeln(differences, " difference(s)"); + return 1; +} diff --git a/dlang/test/dump.d b/dlang/test/dump.d new file mode 100644 index 0000000..17aa9bb --- /dev/null +++ b/dlang/test/dump.d @@ -0,0 +1,35 @@ +/++ + Write a loaded abstraction out canonically. + + The oracle for everything that claims to implement this model: load a + .ssp or a .ocda.db and print the model, deterministically, so that two + implementations can be compared with diff. + + Usage: ocda-dump [--unknown] <file>.ssp|<file>.ocda.db [lang] ++/ +import std.stdio; +import ocda.model; +import ocda.load; +import ocda.dump; + +int main(string[] args) { + bool unknown = false; + string[] rest; + foreach (a; args[1 .. $]) { + if (a == "--unknown") { unknown = true; } else { rest ~= a; } + } + if (rest.length < 1) { + stderr.writeln("usage: ocda-dump [--unknown] <file>.ssp|<file>.ocda.db [lang]"); + return 2; + } + string lang = (rest.length > 1) ? rest[1] : ""; + OcdaDocument doc; + try { + doc = load(rest[0], lang); + } catch (Exception e) { + stderr.writeln("error: ", e.msg); + return 2; + } + write(unknown ? dumpUnknown(doc) : dumpDocument(doc)); + return 0; +} diff --git a/gleam/COPYRIGHT b/gleam/COPYRIGHT new file mode 100644 index 0000000..adc5446 --- /dev/null +++ b/gleam/COPYRIGHT @@ -0,0 +1,48 @@ +- Name: sisudoc-ocda-lib + - Description: part of SiSU spine (Doc Reform): documents, structuring, + processing, publishing and search + - [https://www.sisudoc.org] + + - Author: Ralph Amissah + [ralph.amissah@gmail.com] + + - Copyright: (C) 2026 Ralph Amissah, All Rights Reserved. + + - License: AGPL 3 or later, for everything in this repository unless a file or + a directory specifies otherwise.: + + sisudoc-ocda-lib + + part of SiSU spine (Doc Reform): documents, structuring, + processing, publishing and search + + Copyright (C) Ralph Amissah + + This program is free software: you can redistribute it and/or modify it + under the terms of the GNU Affero General Public License (AGPLv3) as + published by the Free Software Foundation, either version 3 of the License, + or (at your option) any later version. + + This program is distributed in the hope that it will be useful, but + WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY + or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public + License for more details. + + You should have received a copy of the GNU Affero General Public License + along with this program. If not, see [https://www.gnu.org/licenses/]. + + If you have Internet connection, the latest version of the AGPL should be + available at: [https://www.gnu.org/licenses/agpl-3.0.html] + + - Spine, Doc Reform (related to SiSU) uses standard: + - docReform markup syntax (based on SiSU markup) + - standard SiSU markup syntax with modified headers and minor + modifications + - docReform object numbering (based on SiSU object citation numbering) + - standard SiSU document object numbering + + - Homepages: + [https://www.sisudoc.org] + + - Copyright for ./nix-gleam/ 2023 Arnar Gauti Ingason + - License for ./nix-gleam/ MIT License diff --git a/gleam/README.md b/gleam/README.md new file mode 100644 index 0000000..d586bd3 --- /dev/null +++ b/gleam/README.md @@ -0,0 +1,102 @@ +# ocda-lib, in Gleam + +The abstraction's model and both its loaders, as a Gleam library, and the +two programs that put it to the question. + +| | | +|-----------------|----------------------------------------------------------------------------------| +| `ocda/model` | the document, the object, the image. Depends on nothing but the standard library | +| `ocda/load_ssp` | text -> model | +| `ocda/load_db` | sqlite -> model | +| `ocda/load` | what a path is, and loading it | +| `ocda/dump` | the model written out canonically, as an oracle | + +## Use + +```gleam +import ocda/load +import ocda/model + +case load.load(path, "en", False) { + Error(why) -> io.println_error(why) + Ok(doc) -> { + // and from here a writer neither knows nor cares which of the two + // files it was handed + list.each(model.section(doc, "body"), fn(obj) { ... }) + } +} +``` + +## The two programs + +```sh +nix build +./result/bin/ocda-lib agree <doc>.ssp <doc>.ocda.db [lang] +./result/bin/ocda-lib dump [--unknown] <file>.ssp|<file>.ocda.db [lang] +``` + +`agree` asks whether the two loaders give the same model. `dump` writes +the whole model out canonically, and is the stronger check of the two, +because it is **byte for byte what the D library's `ocda-dump` writes**. +So the four questions can all be asked with `diff`: + +| | | +|-----------------------------|-----------------------------| +| D: `.ssp` == `.ocda.db` | 36 of 36 document-languages | +| Gleam: `.ssp` == `.ocda.db` | 36 of 36 | +| `.ssp`: D == Gleam | 36 of 36 | +| `.ocda.db`: D == Gleam | 36 of 36 | + +## Deliberately a mirror of the D library + +Field for field, name for name, including where that is not the most +Gleamish choice: + +- `text` is one `String` with the lines joined by newline, not a + `List(String)`. So an object whose only text line is blank is + indistinguishable from an object with no text, *in both languages, in the + same place*. A difference that existed in one and not the other would be + worse than a loss both share, because it would show up in every dump and + mean nothing. +- the properties are typed fields rather than a list of pairs. A writer + wants `obj.parent_ocn`, not a lookup that might miss. + +The point is that the two agree. Making them agree is much cheaper when +they are shaped alike, and the shape is the specification's, not either +language's. + +## What the agreement found + +Four disagreements, none of which a comparison of section counts or of a +handful of fields would have shown. All four are recorded in the D +library's history; the short version: + +1. `.identifier` and `.anchor` are omitted by the text form when they can + be worked out (the identifier is the ocn, the anchor is the segment). + Both loaders now derive them. Over one document that is 1,314 + identifiers. +2. `make` and `doc_has` are one metadata table in the database, + distinguished by a key prefix. Both loaders unprefix them. +3. The JSON columns must be *parsed*, not scanned: the encoder writes `/` + as `\/` and every heading with a slash in it came back wrong. +4. `translation.ocn_aligned` is in the database and cannot be in a `.ssp`, + which is one language. It is a fact about the file, so neither loader + puts it in the document. + +## Build + +```sh +nix build # sandboxed, no network +nix develop -c gleam test +``` + +Shipped as an Erlang shipment rather than an escript, because sqlite on the +BEAM is a NIF and a NIF cannot be loaded from inside an escript archive. +The dependency hash in `flake.nix` pins what `manifest.toml` pins; change +one and the other must change with it. + +## Format + +Specified in **sisudoc-abstraction-format**: `ocda-model.md` for the model, +`ssp-format-specification.md` and `ocda-db-format-specification.md` for the +two serialisations. At format **2.x**. diff --git a/gleam/flake.lock b/gleam/flake.lock new file mode 100644 index 0000000..5f236ab --- /dev/null +++ b/gleam/flake.lock @@ -0,0 +1,27 @@ +{ + "nodes": { + "nixpkgs": { + "locked": { + "lastModified": 1790185690, + "narHash": "sha256-xJ+X4hBtOcAFGBOe5nAMyMUeF9foJBmIOu3NjBqBycU=", + "owner": "NixOS", + "repo": "nixpkgs", + "rev": "4975466d324710c576dc11ad614684e6bd8cad8e", + "type": "github" + }, + "original": { + "owner": "NixOS", + "ref": "nixos-unstable", + "repo": "nixpkgs", + "type": "github" + } + }, + "root": { + "inputs": { + "nixpkgs": "nixpkgs" + } + } + }, + "root": "root", + "version": 7 +} diff --git a/gleam/flake.nix b/gleam/flake.nix new file mode 100644 index 0000000..a5a3407 --- /dev/null +++ b/gleam/flake.nix @@ -0,0 +1,66 @@ +{ + description = "ocda-lib: the object-centric document abstraction as a Gleam library"; + inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable"; + outputs = { self, nixpkgs }: + let + system = "x86_64-linux"; # adjust for aarch64-linux / darwin + pkgs = import nixpkgs { inherit system; }; + + name = "ocda_lib"; + + /* The builder, nix-gleam's, kept here in nix-gleam/ (see the README + there). It fetches each Hex dependency on its own, with the + checksum gleam has already written into manifest.toml, so there is + no hash in this file to keep up to date, and nothing a run of gleam + could make come out differently. + */ + inherit (pkgs.callPackage ./nix-gleam { }) buildGleamApplication; + + /* rebar3 with the port compiler already in it. + esqlite builds a NIF, and its rebar3 configuration asks for the + `pc` plugin, which rebar3 would otherwise fetch from Hex at build + time: impossible in a sandbox, and the build fails with + "missing_package pc". nixpkgs carries the plugin, so the answer is + a rebar3 that has it before it starts. + */ + rebar3-pc = pkgs.beamPackages.rebar3WithPlugins { + plugins = [ pkgs.beamPackages.pc ]; + }; + in { + packages.${system} = { + default = self.packages.${system}.${name}; + /* Shipped as an Erlang shipment rather than as an escript, because + sqlite on the BEAM is a NIF: esqlite compiles its own sqlite into + a shared object, and a NIF cannot be loaded from inside an + escript archive, the runtime needing a real file on disk. The + shipment keeps priv/ on disk, so it loads. + */ + ${name} = buildGleamApplication { + pname = "ocda-lib"; + version = "2.0.0"; + src = ./.; + rebar3Package = rebar3-pc; + nativeBuildInputs = [ pkgs.makeWrapper ]; + preBuild = '' + export HOME=$TMPDIR + ''; + installPhase = '' + runHook preInstall + mkdir -p $out/lib $out/bin + cp -r build/erlang-shipment $out/lib/${name} + makeWrapper $out/lib/${name}/entrypoint.sh $out/bin/ocda-lib \ + --add-flags run \ + --prefix PATH : ${pkgs.lib.makeBinPath [ pkgs.beamPackages.erlang pkgs.coreutils ]} + runHook postInstall + ''; + meta = { + description = "the abstraction as a Gleam library"; + mainProgram = "ocda-lib"; + }; + }; + }; + devShells.${system}.default = pkgs.mkShell { + packages = [ pkgs.gleam pkgs.beamPackages.erlang pkgs.rebar3 pkgs.sqlite ]; + }; + }; +} diff --git a/gleam/gleam.toml b/gleam/gleam.toml new file mode 100644 index 0000000..7b474eb --- /dev/null +++ b/gleam/gleam.toml @@ -0,0 +1,14 @@ +name = "ocda_lib" +version = "2.0.0" +description = "the object-centric document abstraction as a library: the model, and loaders for its two serialisations" +licences = ["AGPL-3.0-or-later"] + +[dependencies] +gleam_stdlib = ">= 1.0.0 and < 2.0.0" +argv = ">= 1.1.0 and < 2.0.0" +simplifile = ">= 2.7.0 and < 3.0.0" +sqlight = ">= 1.0.0 and < 2.0.0" +gleam_json = ">= 3.0.0 and < 4.0.0" + +[dev-dependencies] +gleeunit = ">= 1.0.0 and < 2.0.0" diff --git a/gleam/manifest.toml b/gleam/manifest.toml new file mode 100644 index 0000000..2b0cac1 --- /dev/null +++ b/gleam/manifest.toml @@ -0,0 +1,26 @@ +# Do not manually edit this file, it is managed by Gleam. +# +# This file locks the dependency versions used, to make your build +# deterministic and to prevent unexpected versions from being included +# in your application. +# +# You should check this file into your source control repository. + +packages = [ + { name = "argv", version = "1.1.0", build_tools = ["gleam"], requirements = [], otp_app = "argv", source = "hex", outer_checksum = "3277D100448BDB4A29B6D58C0F36F631CBC349E8BDD09766C6309DF202831140" }, + { name = "esqlite", version = "0.9.0", build_tools = ["rebar3"], requirements = [], otp_app = "esqlite", source = "hex", outer_checksum = "CCF72258A4EE152EC7AD92AA9A03552EB6CA1B06B65C93AD5B6E55C302E05855" }, + { name = "filepath", version = "1.1.2", build_tools = ["gleam"], requirements = ["gleam_stdlib"], otp_app = "filepath", source = "hex", outer_checksum = "B06A9AF0BF10E51401D64B98E4B627F1D2E48C154967DA7AF4D0914780A6D40A" }, + { name = "gleam_json", version = "3.1.0", build_tools = ["gleam"], requirements = ["gleam_stdlib"], otp_app = "gleam_json", source = "hex", outer_checksum = "44FDAA8847BE8FC48CA7A1C089706BD54BADCC4C45B237A992EDDF9F2CDB2836" }, + { name = "gleam_stdlib", version = "1.0.5", build_tools = ["gleam"], requirements = [], otp_app = "gleam_stdlib", source = "hex", outer_checksum = "CEE5B6C076A85B45F60C585F4316C63EC8B7127C119D5738C3958A9C4D50404E" }, + { name = "gleeunit", version = "1.11.0", build_tools = ["gleam"], requirements = ["gleam_stdlib"], otp_app = "gleeunit", source = "hex", outer_checksum = "EC31ABA74256AEA531EDF8169931D775BBB384FED0A8A1BDC4DD9354E3E21826" }, + { name = "simplifile", version = "2.7.0", build_tools = ["gleam"], requirements = ["filepath", "gleam_stdlib"], otp_app = "simplifile", source = "hex", outer_checksum = "A2727627B063E87351934C7F7F008F2D1FDB16F6DE0B8C79F9E46459CFC9C164" }, + { name = "sqlight", version = "1.2.0", build_tools = ["gleam"], requirements = ["esqlite", "gleam_stdlib"], otp_app = "sqlight", source = "hex", outer_checksum = "841768D0A81107EE2DB46B949354F284A68B2E8098315C28A1FE3C07D206C17A" }, +] + +[requirements] +argv = { version = ">= 1.1.0 and < 2.0.0" } +gleam_json = { version = ">= 3.0.0 and < 4.0.0" } +gleam_stdlib = { version = ">= 1.0.0 and < 2.0.0" } +gleeunit = { version = ">= 1.0.0 and < 2.0.0" } +simplifile = { version = ">= 2.7.0 and < 3.0.0" } +sqlight = { version = ">= 1.0.0 and < 2.0.0" } diff --git a/gleam/nix-gleam/LICENSE b/gleam/nix-gleam/LICENSE new file mode 100644 index 0000000..c35ad09 --- /dev/null +++ b/gleam/nix-gleam/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2023 Arnar Gauti Ingason + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/gleam/nix-gleam/README.md b/gleam/nix-gleam/README.md new file mode 100644 index 0000000..39bd1b6 --- /dev/null +++ b/gleam/nix-gleam/README.md @@ -0,0 +1,32 @@ +# nix-gleam, kept here + +`default.nix` is the builder of **nix-gleam**, by Arnar Gauti Ingason, +copied unchanged from + +- https://github.com/arnarg/nix-gleam +- `builder/default.nix` at commit `1f38154` (2026-10-06) + +under the MIT licence, which is in `LICENSE` beside it. + +It is kept here rather than taken as a flake input so that the build does +not depend on a third repository being reachable, and so that what builds +this package is in the package. + +## Why it is used + +There is no Gleam builder in nixpkgs, and a sandboxed build cannot fetch +from Hex. The usual answer, one fixed output derivation that runs +`gleam deps download` and is pinned by a hash of everything it fetched, has +two faults: the hash has to be updated by hand whenever a dependency +changes, and gleam writes `packages.toml` in a different order on each run, +so the same dependencies do not always give the same hash. + +nix-gleam fetches each Hex package on its own with `fetchHex`, using the +checksum gleam has already written for it into `manifest.toml`, and writes +`packages.toml` itself. So there is no hash to maintain, and nothing a run +of gleam could make come out differently. + +## Updating it + +Copy `builder/default.nix` from a newer commit over `default.nix`, update +the commit above, and build. diff --git a/gleam/nix-gleam/default.nix b/gleam/nix-gleam/default.nix new file mode 100644 index 0000000..d074263 --- /dev/null +++ b/gleam/nix-gleam/default.nix @@ -0,0 +1,280 @@ +{ + stdenv, + lib, + git, + xxhash, + fetchHex, + gleam, + beamPackages, + rsync, + nodejs, +}: let + inherit (builtins) fromTOML readFile; +in { + buildGleamApplication = { + src, + nativeBuildInputs ? [], + localPackages ? [], + erlangPackage ? beamPackages.erlang, + rebar3Package ? beamPackages.rebar3, + ... + } @ attrs: let + # gleam.toml contains an application name and version. + gleamToml = lib.importTOML (src + "/gleam.toml"); + + # manifest.toml contains a list of required packages including a sha256 checksum + # that can be used by nix fetchHex fetcher. + manifestToml = lib.importTOML (src + "/manifest.toml"); + + # Specify which target to build for. + buildTarget = attrs.target or gleamToml.target or "erlang"; + + # Generates a packages.toml expected by gleam compiler. + packagesTOML = with lib; + concatStringsSep "\n" ( + ["[packages]"] + ++ (map + (p: "${p.name} = \"${p.version}\"") + manifestToml.packages) + ++ (lib.optional (gitRecordsTOML != "") "") + ++ (lib.optional (gitRecordsTOML != "") gitRecordsTOML) + ); + + # Gleam records a git dependency twice: once in `[packages]`, + # and once in a `[git.<name>]` table with the commit. + # Without the second, Gleam will try to download it. + gitRecordsTOML = with lib; + concatMapStringsSep "\n" ( + p: "[git.${p.name}]\ncommit = \"${p.commit}\"" + ) (filterPackagesBySource "git" manifestToml.packages); + + # Helper function to filter manifest.toml packages + filterPackagesBySource = type: packages: lib.lists.filter (p: p.source == type) packages; + + gitDerivs = + map + (p: { + name = p.name; + derivation = fetchGit { + url = p.repo; + rev = p.commit; + }; + }) + (filterPackagesBySource "git" manifestToml.packages); + + # Fetch all dependencies + depsDerivs = + map + (p: { + name = p.name; + derivation = fetchHex { + inherit (p) version; + pkg = p.name; + sha256 = p.outer_checksum; + }; + }) + (filterPackagesBySource "hex" manifestToml.packages); + + # Find replacement paths for `local` package dependencies + # from `localPackages` list. + localDeps = let + # Build a lookup attrset for local packages. + localDerivs = lib.mergeAttrsList (map ( + p: let + name = (fromTOML (readFile (p + "/gleam.toml"))).name; + in { + "${name}" = p; + } + ) + localPackages); + in + map ( + p: { + inherit (p) name path; + localSrc = + if localDerivs ? "${p.name}" + then localDerivs.${p.name} + else throw "Local dependency \"${p.name}\" not found in `localPackages`."; + # Keep local packages in a writable location during build. + newPath = p.path; + } + ) (filterPackagesBySource "local" manifestToml.packages); + + # Check if elixir is needed in nativeBuildInputs by checking if "mix" is in + # required build_tools. + isElixirProject = with lib; p: any (t: t == "mix") p.build_tools; + needsElixir = with lib; any isElixirProject manifestToml.packages; + + # nativeBuildInputs needed for both targets. + defaultNativeBuildInputs = [gleam beamPackages.hex rsync git xxhash]; + in + # Base common mkDerivation attributes + stdenv.mkDerivation (attrs + // { + pname = attrs.pname or gleamToml.name; + version = attrs.version or gleamToml.version; + + src = lib.cleanSource attrs.src; + + # Here we must copy the dependencies into the right spot and + # create a packages.toml file so the gleam compiler does not + # attempt to pull the dependencies from the internet. + configurePhase = + attrs.configurePhase + or '' + runHook preConfigure + + mkdir -p build/packages + + # Write the packages.toml file + cat <<EOF > build/packages/packages.toml + ${packagesTOML} + EOF + + ${ + # Gleam writes build output into the local dependency's own source + # directory, so it cannot be the read-only store path. + lib.concatStringsSep "\n" ( + lib.forEach localDeps ( + d: '' + mkdir -p "${d.newPath}" + rsync --chmod=Du=rwx,Dg=rx,Do=rx,Fu=rw,Fg=r,Fo=r -r ${d.localSrc}/* "${d.newPath}/" + '' + ) + ) + } + + ${ + lib.concatStringsSep "\n" ( + lib.forEach gitDerivs ( + d: '' + rsync --chmod=Du=rwx,Dg=rx,Do=rx,Fu=rw,Fg=r,Fo=r -r ${d.derivation}/* build/packages/${d.name}/ + '' + ) + ) + } + + # Copy all the dependencies into place + ${lib.concatStringsSep "\n" ( + lib.forEach depsDerivs ( + # gleam outputs files inside the dependency's source directory + # and therefor it needs to have permissive permissions. + d: '' + rsync --chmod=Du=rwx,Dg=rx,Do=rx,Fu=rw,Fg=r,Fo=r -r ${d.derivation}/* build/packages/${d.name}/ + '' + ) + )} + + # To prevent dependency resolution in Gleam 1.15+, local packages + # need to have their fingerprint up-to-date. + ${lib.concatStringsSep "\n" ( + lib.forEach localDeps (d: '' + printf "%u" 0x$(xxhsum -H3 ${d.newPath}/gleam.toml | cut -d' ' -f1 | cut -d '_' -f2) > build/packages/${d.name}.config_fingerprint + '') + )} + + ${ + # Prime local package dependency caches so they do not try to fetch. + # + # Gleam compiles a local dependency inside that dependency's own + # directory, using the package cache found there. + lib.concatStringsSep "\n" ( + lib.forEach localDeps ( + d: '' + mkdir -p "${d.newPath}/build/packages" + cp build/packages/packages.toml "${d.newPath}/build/packages/packages.toml" + rsync --chmod=Du=rwx,Dg=rx,Do=rx,Fu=rw,Fg=r,Fo=r -r build/packages/* "${d.newPath}/build/packages/" + '' + ) + ) + } + + runHook postConfigure + ''; + } + # When the build target is erlang + // lib.optionalAttrs (buildTarget == "erlang") { + nativeBuildInputs = + defaultNativeBuildInputs + ++ [erlangPackage rebar3Package] + ++ (lib.optional needsElixir beamPackages.elixir) + ++ nativeBuildInputs; + + # The gleam compiler has a nice export function for erlang shipment. + buildPhase = + attrs.buildPhase + or '' + runHook preBuild + + export REBAR_CACHE_DIR="$TMP/.rebar-cache" + + gleam export erlang-shipment + + runHook postBuild + ''; + + # Install all built packages into lib and create an entrypoint script + # that starts the application. + installPhase = + attrs.installPhase + or '' + runHook preInstall + + mkdir -p $out/{bin,lib} + + rsync --exclude=entrypoint.sh -r build/erlang-shipment/* $out/lib/ + + cat <<EOF > $out/bin/${gleamToml.name} + #!/usr/bin/env sh + ${erlangPackage}/bin/erl \ + -pa $out/lib/*/ebin \ + -eval "${gleamToml.name}@@main:run(${gleamToml.name})" \ + -noshell \ + -extra "\$@" + EOF + chmod +x $out/bin/${gleamToml.name} + + runHook postInstall + ''; + } + # When the build target is javascript + // lib.optionalAttrs (buildTarget == "javascript") { + nativeBuildInputs = defaultNativeBuildInputs ++ nativeBuildInputs; + + # The gleam compiler doesn't provide an export mechanism for javascript target. + buildPhase = + attrs.buildPhase + or '' + runHook prebuild + + gleam build --target javascript + + runHook postBuild + ''; + + # Install all built packages into lib and create an entrypoint script + # that starts the application. + installPhase = + attrs.installPhase + or '' + runHook preInstall + + mkdir -p $out/{bin,lib} + + rsync --exclude=gleam.lock --exclude=gleam_version -r build/dev/javascript/* $out/lib/ + + cat <<EOF > $out/lib/${gleamToml.name}/main.mjs + import { main } from "./${gleamToml.name}.mjs"; + main(); + EOF + + cat <<EOF > $out/bin/${gleamToml.name} + #!/usr/bin/env sh + ${nodejs}/bin/node $out/lib/${gleamToml.name}/main.mjs "\$@" + EOF + chmod +x $out/bin/${gleamToml.name} + + runHook postInstall + ''; + }); +} diff --git a/gleam/src/COPYRIGHT b/gleam/src/COPYRIGHT new file mode 100644 index 0000000..7e7dd56 --- /dev/null +++ b/gleam/src/COPYRIGHT @@ -0,0 +1,45 @@ +- Name: sisudoc-ocda-lib + - Description: part of SiSU spine (Doc Reform): documents, structuring, + processing, publishing and search + - [https://www.sisudoc.org] + + - Author: Ralph Amissah + [ralph.amissah@gmail.com] + + - Copyright: (C) 2026 Ralph Amissah, All Rights Reserved. + + - License: AGPL 3 or later, for everything in this repository unless a file or + a directory specifies otherwise.: + + sisudoc-ocda-lib + + part of SiSU spine (Doc Reform): documents, structuring, + processing, publishing and search + + Copyright (C) Ralph Amissah + + This program is free software: you can redistribute it and/or modify it + under the terms of the GNU Affero General Public License (AGPLv3) as + published by the Free Software Foundation, either version 3 of the License, + or (at your option) any later version. + + This program is distributed in the hope that it will be useful, but + WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY + or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public + License for more details. + + You should have received a copy of the GNU Affero General Public License + along with this program. If not, see [https://www.gnu.org/licenses/]. + + If you have Internet connection, the latest version of the AGPL should be + available at: [https://www.gnu.org/licenses/agpl-3.0.html] + + - Spine, Doc Reform (related to SiSU) uses standard: + - docReform markup syntax (based on SiSU markup) + - standard SiSU markup syntax with modified headers and minor + modifications + - docReform object numbering (based on SiSU object citation numbering) + - standard SiSU document object numbering + + - Homepages: + [https://www.sisudoc.org] diff --git a/gleam/src/ocda/dump.gleam b/gleam/src/ocda/dump.gleam new file mode 100644 index 0000000..a75da67 --- /dev/null +++ b/gleam/src/ocda/dump.gleam @@ -0,0 +1,247 @@ +//// The model, written out canonically. +//// +//// Not an output format: an *oracle*. Two libraries in two languages +//// implement the same model from the same specification, and the cheap way +//// to ask whether they agree is to have each write the whole loaded +//// document as text and diff the two. Byte for byte identical to the D +//// library's `ocda/dump.d`, which is the whole point of it. +//// +//// The rules, which both implementations follow: +//// +//// - every line is `<key> <value>`, the value running to end of line, +//// and only one free-form value per line; +//// - a field at its default is *omitted*, so the output says what the +//// document has rather than what the type declares; +//// - dictionaries come in sorted key order, because neither language's +//// map has an order worth relying on; +//// - sections come in the fixed order of the format. +//// +//// Anything that reads this to reconstruct a document has misunderstood +//// it. Load the `.ssp` or the `.ocda.db`. + +import gleam/dict.{type Dict} +import gleam/int +import gleam/list +import gleam/string +import ocda/model.{type Document, type Object} + +pub fn document(doc: Document) -> String { + let lines = + [] + |> kv("version", doc.format_version) + |> kv("source_filename", doc.source_filename) + |> entries("source", doc.source) + |> entries("meta", doc.metadata) + |> entries("make", doc.make) + |> entries("doc_has", doc.doc_has) + |> counts(doc) + |> objects(doc) + lines |> list.reverse |> join +} + +/// What the loader met and did not understand, sorted. +/// +/// Separate from the document dump: it describes the loader, not the +/// document, and the `.ocda.db` loader has nothing to report where the +/// `.ssp` loader does. +pub fn unknown(doc: Document) -> String { + [] + |> numbered("property", doc.unknown.property) + |> numbered("block", doc.unknown.block) + |> numbered("line", doc.unknown.line) + |> list.reverse + |> join +} + +fn join(lines: List(String)) -> String { + case lines { + [] -> "" + _ -> string.join(lines, "\n") <> "\n" + } +} + +/// Accumulated newest-first and reversed once, so that nothing appends to +/// the tail of a list that grows to a few hundred thousand lines. +fn kv(acc: List(String), key: String, value: String) -> List(String) { + case value { + "" -> acc + v -> [key <> " " <> v, ..acc] + } +} + +fn kn(acc: List(String), key: String, value: Int) -> List(String) { + case value { + 0 -> acc + n -> [key <> " " <> int.to_string(n), ..acc] + } +} + +fn kb(acc: List(String), key: String, value: Bool) -> List(String) { + case value { + False -> acc + True -> [key <> " 1", ..acc] + } +} + +fn entries( + acc: List(String), + prefix: String, + d: Dict(String, String), +) -> List(String) { + d + |> dict.to_list + |> list.sort(fn(a, b) { string.compare(a.0, b.0) }) + |> list.fold(acc, fn(lines, pair) { + [prefix <> " " <> pair.0 <> " " <> pair.1, ..lines] + }) +} + +fn numbered( + acc: List(String), + prefix: String, + d: Dict(String, Int), +) -> List(String) { + d + |> dict.to_list + |> list.sort(fn(a, b) { string.compare(a.0, b.0) }) + |> list.fold(acc, fn(lines, pair) { + [prefix <> " " <> pair.0 <> " " <> int.to_string(pair.1), ..lines] + }) +} + +fn counts(acc: List(String), doc: Document) -> List(String) { + list.fold(model.section_names, acc, fn(lines, name) { + case model.section(doc, name) { + [] -> lines + objects -> + [ + "section " <> name <> " " <> int.to_string(list.length(objects)), + ..lines + ] + } + }) +} + +fn objects(acc: List(String), doc: Document) -> List(String) { + list.fold(model.section_names, acc, fn(lines, name) { + model.section(doc, name) + |> list.index_fold(lines, fn(l, obj, i) { + object(l, name, i, obj) + }) + }) +} + +fn object( + acc: List(String), + section: String, + index: Int, + obj: Object, +) -> List(String) { + let acc = ["object " <> section <> " " <> int.to_string(index), ..acc] + acc + |> kn("ocn", obj.ocn) + |> kv("is_a", obj.is_a) + |> kv("own_section", obj.own_section) + |> kv("level", obj.heading_level) + |> kv("identifier", obj.identifier) + |> kv("part", obj.part) + |> kv("is_of_type", obj.is_of_type) + |> kv("segment_html_is", obj.segment_html_is) + |> kv("segment_epub", obj.segment_epub) + |> kv("segment_epub_is", obj.segment_epub_is) + |> kn("parent", obj.parent_ocn) + |> kn("last_descendant", obj.last_descendant_ocn) + |> kv("children", ints(obj.children)) + |> kv("ancestors", ints(obj.ancestors)) + |> kb("dummy", obj.dummy_heading) + |> kb("ocn_off", obj.ocn_off) + |> kn("indent_base", obj.indent_base) + |> kn("indent_hang", obj.indent_hang) + |> kb("bullet", obj.bullet) + |> kv("lang", obj.lang) + |> kb("has_links", obj.has_links) + |> kb("has_notes_reg", obj.has_notes_reg) + |> kb("has_notes_star", obj.has_notes_star) + |> kb("has_images", obj.has_images) + |> kv("segment", obj.segment) + |> kv("segment_prev", obj.segment_prev) + |> kv("segment_next", obj.segment_next) + |> kv("anchor", obj.anchor) + |> kv("heading_lev_anchor", obj.heading_lev_anchor) + |> kn("table_cols", obj.table_cols) + |> kv("table_widths", obj.table_widths) + |> kv("table_aligns", obj.table_aligns) + |> kb("table_header", obj.table_header) + |> kv("code_syntax", obj.code_syntax) + |> kb("code_linenumbers", obj.code_linenumbers) + |> kv("heading_ancestors_text", obj.heading_ancestors_text) + |> kv("sha256", obj.sha256) + |> links(obj) + |> anchor_tags(obj) + |> images(obj) + |> text(obj) +} + +fn links(acc: List(String), obj: Object) -> List(String) { + list.fold(obj.stow_link, acc, fn(lines, url) { kv(lines, "stow_link", url) }) +} + +fn anchor_tags(acc: List(String), obj: Object) -> List(String) { + list.fold(obj.anchor_tags, acc, fn(lines, a) { kv(lines, "anchor_tag", a) }) +} + +fn images(acc: List(String), obj: Object) -> List(String) { + list.fold(obj.images, acc, fn(lines, img) { + // the name last, because a filename may hold a space where none of the + // other fields can + let lines = + kv( + lines, + "image", + img.sha256 + <> " " + <> int.to_string(img.bytes) + <> " " + <> int.to_string(img.width) + <> "x" + <> int.to_string(img.height) + <> " " + <> case img.missing { + True -> "1" + False -> "0" + } + <> " " + <> img.name, + ) + img.fields + |> dict.to_list + |> list.sort(fn(a, b) { string.compare(a.0, b.0) }) + |> list.fold(lines, fn(l, pair) { + kv(l, "image_field", pair.0 <> " " <> pair.1) + }) + }) +} + +/// Text, when there is any. +/// +/// An object whose only text line is blank holds "" here, indistinguishable +/// from no text at all, because the model joins the lines into one string. +/// Both libraries lose the same distinction in the same place, which is +/// what makes them comparable; it is recorded rather than hidden. +fn text(acc: List(String), obj: Object) -> List(String) { + case obj.text { + "" -> acc + t -> { + let lines = string.split(t, on: "\n") + let acc = kv(acc, "text", int.to_string(list.length(lines))) + list.fold(lines, acc, fn(l, line) { ["| " <> line, ..l] }) + } + } +} + +fn ints(xs: List(Int)) -> String { + case xs { + [] -> "" + _ -> xs |> list.map(int.to_string) |> string.join(",") + } +} diff --git a/gleam/src/ocda/images.gleam b/gleam/src/ocda/images.gleam new file mode 100644 index 0000000..2f3a56e --- /dev/null +++ b/gleam/src/ocda/images.gleam @@ -0,0 +1,135 @@ +//// An image's bytes, from whichever source the document came from. +//// +//// The model *names* images, with digest, byte size and pixel size; it does +//// not hold them. That is right - an abstraction of a document is not a +//// bundle of its assets - but a writer of any format that shows images needs +//// the bytes, so something has to know where they are. This is that +//// something, and it is in the library rather than in a writer because the +//// answer depends on the *input form* and not on the output. +//// +//// | `.ocda.db` | the `files` table, role 'image', by name | +//// | `.zip` | the archive entry whose path ends in `/image/<name>` | +//// | `.sisupod` | the same, if the frame can be unwrapped, which on the BEAM it cannot | +//// | `.ssp` | beside it: a pod puts the abstraction in `media/abstraction/` and its images in `media/image/` | + +import gleam/bit_array +import gleam/dynamic/decode +import gleam/list +import gleam/result +import gleam/string +import ocda/load +import ocda/zip +import simplifile +import sqlight + +/// The bytes, or an error saying why not. +pub fn image_bytes(source: String, name: String) -> Result(BitArray, String) { + case load.source_of(source) { + load.OcdaDb -> from_db(source, name) + load.Pod -> from_archive(source, name) + load.Ssp -> from_beside(source, name) + load.Unknown -> Error(source <> ": not a source this library reads") + } +} + +/// Can this source give images at all? +/// +/// A writer asks before it decides whether to emit them, because an `<img>` +/// pointing at a file it did not include is worse than no image. +pub fn can_give_images(source: String) -> Bool { + case load.source_of(source) { + load.OcdaDb -> True + load.Pod -> zip_readable(source) + load.Ssp -> image_dir_beside(source) != "" + load.Unknown -> False + } +} + +fn zip_readable(path: String) -> Bool { + case simplifile.read_bits(path) { + Error(_) -> False + Ok(bytes) -> + case zip.entries(bytes) { + Ok(_) -> True + Error(_) -> False + } + } +} + +fn from_db(path: String, name: String) -> Result(BitArray, String) { + case sqlight.open("file:" <> path <> "?mode=ro") { + Error(e) -> Error(path <> ": " <> e.message) + Ok(conn) -> { + let out = + sqlight.query( + "select data from files where role = 'image' and name = ?1", + on: conn, + with: [sqlight.text(name)], + expecting: blob(), + ) + |> result.replace_error(path <> ": could not read " <> name) + let _ = sqlight.close(conn) + case out { + Ok([data, ..]) -> Ok(data) + Ok([]) -> Error(name <> ": not in this database") + Error(e) -> Error(e) + } + } + } +} + +/// A blob column, read as bytes rather than as text: an image read as text +/// stops at the first zero byte, and a png's ninth byte is zero. +fn blob() -> decode.Decoder(BitArray) { + use v <- decode.field(0, decode.bit_array) + decode.success(v) +} + +fn from_archive(path: String, name: String) -> Result(BitArray, String) { + use bytes <- result.try( + simplifile.read_bits(path) + |> result.replace_error(path <> ": cannot read"), + ) + use entries <- result.try(zip.entries(bytes)) + case list.find(entries, fn(e) { string.ends_with(e.name, "/image/" <> name) }) { + Error(_) -> Error(name <> ": not in this archive") + Ok(entry) -> zip.read(bytes, entry) + } +} + +fn from_beside(path: String, name: String) -> Result(BitArray, String) { + case image_dir_beside(path) { + "" -> Error(name <> ": a .ssp outside a pod has no image directory beside it") + dir -> + simplifile.read_bits(dir <> "/" <> name) + |> result.replace_error(dir <> "/" <> name <> ": cannot read") + } +} + +/// `media/abstraction/<x>.ssp` -> `media/image/` +fn image_dir_beside(path: String) -> String { + let parts = string.split(path, on: "/") + let dir = case list.reverse(parts) { + [_file, ..rest] -> string.join(list.reverse(rest), "/") + [] -> "" + } + case string.ends_with(dir, "abstraction") { + False -> "" + True -> { + let parent = case list.reverse(string.split(dir, on: "/")) { + [_, ..rest] -> string.join(list.reverse(rest), "/") + [] -> "" + } + let image_dir = parent <> "/image" + case simplifile.is_directory(image_dir) { + Ok(True) -> image_dir + _ -> "" + } + } + } +} + +/// Unused here, kept so that a caller can size a read before making it. +pub fn byte_size(b: BitArray) -> Int { + bit_array.byte_size(b) +} diff --git a/gleam/src/ocda/load.gleam b/gleam/src/ocda/load.gleam new file mode 100644 index 0000000..2a82ac9 --- /dev/null +++ b/gleam/src/ocda/load.gleam @@ -0,0 +1,79 @@ +//// What a path is, and loading it. +//// +//// One place that decides, so that every reader and every writer accepts +//// the same inputs and a new input form is added once. This mirrors +//// spine's own `ocda/abstraction/load.d`, which names the sources it can +//// enter the pipeline from and tells them apart by suffix. +//// +//// Deliberately *not* here: `.sst` and `.ssm` markup. Reading markup means +//// implementing the parser, which is spine's work and a far larger thing +//// than a reader or a sample writer needs. This library reads +//// abstractions. + +import gleam/list +import gleam/string +import ocda/load_db +import ocda/load_pod +import ocda/load_ssp +import ocda/model.{type Document} + +pub type Source { + /// `.ssp`, the abstraction as text + Ssp + /// `.ocda.db`, the abstraction as sqlite + OcdaDb + /// `.sisupod` or `.zip`, an archive carrying a `.ssp` + Pod + Unknown +} + +/// `.sisupod` is the archive spine writes now; `.zip` is what it wrote +/// before, and every pod already published carries that name, so both are +/// recognised. What is inside decides how it is opened, not the name. +const suffixes = [ + #(".ocda.db", OcdaDb), + #(".ssp", Ssp), + #(".sisupod", Pod), + #(".zip", Pod), + #(".db", OcdaDb), +] + +pub fn source_of(path: String) -> Source { + case list.find(suffixes, fn(s) { string.ends_with(path, s.0) }) { + Ok(#(_, source)) -> source + Error(_) -> Unknown + } +} + +pub fn source_name(source: Source) -> String { + case source { + Ssp -> ".ssp (the abstraction as text)" + OcdaDb -> ".ocda.db (the abstraction as sqlite)" + Pod -> "a pod archive carrying a .ssp" + Unknown -> "not an abstraction this library reads" + } +} + +/// Load whatever the path is into the model. +/// +/// `lang` names which language to take from a `.ocda.db`, which holds +/// every language of its document, and from a pod, which may carry more than +/// one `.ssp`. A `.ssp` is one language by construction and ignores it. +pub fn load( + path: String, + lang: String, + any_version: Bool, +) -> Result(Document, String) { + case source_of(path) { + Ssp -> load_ssp.load_file(path) + OcdaDb -> load_db.load_file(path, lang, any_version) + Pod -> load_pod.load_file(path, lang) + Unknown -> + Error( + path + <> ": " + <> source_name(Unknown) + <> " (expected .ssp, .ocda.db, .sisupod or .zip)", + ) + } +} diff --git a/gleam/src/ocda/load_db.gleam b/gleam/src/ocda/load_db.gleam new file mode 100644 index 0000000..a5f044b --- /dev/null +++ b/gleam/src/ocda/load_db.gleam @@ -0,0 +1,532 @@ +//// Loading a `.ocda.db` into the model. +//// +//// The point of this module is that it returns *the same value* +//// `ocda/load_ssp` returns from the same document's `.ssp`. A writer is +//// then a function from the model to bytes and cannot tell which +//// serialisation it was given, exactly as spine's own writers cannot. +//// +//// Written against `ocda-db-format-specification.md`, whose section 4 +//// gives the column by column correspondence this module implements. + +import gleam/dict +import gleam/dynamic/decode +import gleam/float +import gleam/int +import gleam/json +import gleam/list +import gleam/option +import gleam/result +import gleam/string +import ocda/model.{type Document, Image, Object} +import sqlight + +/// The schema major version this loader knows. +pub const schema_major_read = "2" + +/// One language of the database, as the manifest gives it. +pub type Language { + Language(doc_id: Int, lang: String, source_filename: String, ssp_digest: String) +} + +/// Load one language of a database as the model. `lang` == "" takes the +/// first language the manifest gives. +pub fn load_file( + path: String, + lang: String, + any_version: Bool, +) -> Result(Document, String) { + // read only: a loader has no business writing, and a read-write open + // *creates* an empty database where a path is mistyped, which then reads + // as a document with nothing in it, a failure that looks like an answer + use conn <- with_connection("file:" <> path <> "?mode=ro", path) + + use _ <- result.try(case + has(conn, "documents") && has(conn, "objects") && has(conn, "metadata") + { + True -> Ok(Nil) + False -> + Error( + path + <> ": no documents/objects/metadata tables; it does not look like a .ocda.db", + ) + }) + + let schema_version = + one(conn, "select value from metadata where doc_id is null + and key = 'schema.version'") + use _ <- result.try(case schema_version, any_version { + "", _ -> Ok(Nil) + _, True -> Ok(Nil) + v, False -> + case major_of(v) == schema_major_read { + True -> Ok(Nil) + False -> + Error( + path + <> " is schema v" + <> v + <> ", this loader knows v" + <> schema_major_read + <> ".x", + ) + } + }) + + let languages = languages(conn) + use chosen <- result.try(case languages, lang { + [], _ -> Error(path <> ": no documents rows") + [first, ..], "" -> Ok(first) + _, wanted -> + case list.find(languages, fn(l) { l.lang == wanted }) { + Ok(found) -> Ok(found) + Error(_) -> + Error( + path + <> ": no language '" + <> wanted + <> "'; it holds: " + <> string.join(list.map(languages, fn(l) { l.lang }), " "), + ) + } + }) + + let source = + dict.new() + |> dict.insert("language", chosen.lang) + |> dict.insert( + "languages", + string.join(list.map(languages, fn(l) { l.lang }), " "), + ) + let digest = meta_value(conn, chosen.doc_id, "source.digest") + let source = case digest { + "" -> source + d -> dict.insert(source, "digest", d) + } + + let document = + model.Document( + ..model.new_document(), + format_version: schema_version, + source_filename: chosen.source_filename, + source: source, + ) + let document = metadata(conn, chosen.doc_id, document) + let document = objects(conn, chosen.doc_id, document) + Ok(document) +} + +fn with_connection( + uri: String, + path: String, + then: fn(sqlight.Connection) -> Result(a, String), +) -> Result(a, String) { + case sqlight.open(uri) { + Error(e) -> Error(path <> ": " <> e.message) + Ok(conn) -> { + let outcome = then(conn) + let _ = sqlight.close(conn) + outcome + } + } +} + +/// Every language the database holds, in manifest order. +pub fn languages(conn: sqlight.Connection) -> List(Language) { + let d = { + use id <- decode.field(0, decode.int) + use lang <- decode.field(1, decode.string) + use source <- decode.field(2, decode.string) + use digest <- decode.field(3, decode.string) + decode.success(Language(id, lang, source, digest)) + } + sqlight.query( + "select id, lang, coalesce(source_filename,''), coalesce(ssp_digest,'') + from documents order by id", + on: conn, + with: [], + expecting: d, + ) + |> result.unwrap([]) +} + +/// The three preamble blocks of the text form, out of the one metadata +/// table the database keeps them in. +fn metadata( + conn: sqlight.Connection, + doc_id: Int, + document: Document, +) -> Document { + let d = { + use key <- decode.field(0, decode.string) + use value <- decode.field(1, decode.string) + decode.success(#(key, value)) + } + let rows = + sqlight.query( + "select key, value from metadata where doc_id = " + <> int.to_string(doc_id) + <> " order by key", + on: conn, + with: [], + expecting: d, + ) + |> result.unwrap([]) + + list.fold(rows, document, fn(doc, row) { + let #(key, raw) = row + let value = string.trim(raw) + case key { + // source.* describes the artefact rather than the document, and is + // @source in the text form + "source." <> _ -> doc + // translation.ocn_aligned says whether the languages of this document + // share one numbering. A .ocda.db can know it because it holds every + // language; a .ssp is one language and cannot. It is a fact about the + // file, not the document, and carrying it here would let a writer + // tell which serialisation it was handed. + "translation." <> _ -> doc + // the database tells the three blocks apart by a key prefix. + // Unprefixing them is what makes the two loaders give one model: a + // writer asks make["footer"], never metadata["make.footer"]. + "make." <> rest -> + model.Document(..doc, make: dict.insert(doc.make, rest, value)) + "doc_has." <> rest -> + model.Document(..doc, doc_has: dict.insert(doc.doc_has, rest, value)) + _ -> + model.Document(..doc, metadata: dict.insert(doc.metadata, key, value)) + } + }) +} + +/// The objects, in document order within each section, the sections in the +/// order the format gives rather than the alphabetical one sqlite would. +fn objects( + conn: sqlight.Connection, + doc_id: Int, + document: Document, +) -> Document { + let d = { + use section <- decode.field(0, decode.string) + use ocn <- decode.field(2, decode.int) + use is_a <- decode.field(3, decode.string) + use part <- decode.field(4, decode.string) + use is_of_section <- decode.field(5, decode.string) + use is_of_type <- decode.field(6, decode.string) + use heading_level <- decode.field(7, decode.int) + use identifier <- decode.field(8, decode.string) + use parent <- decode.field(9, decode.int) + use last_descendant <- decode.field(10, decode.int) + use children <- decode.field(11, decode.string) + use ancestors <- decode.field(12, decode.string) + use dummy <- decode.field(13, decode.int) + use ocn_off <- decode.field(14, decode.int) + use indent_base <- decode.field(15, decode.int) + use indent_hang <- decode.field(16, decode.int) + use bullet <- decode.field(17, decode.int) + use lang <- decode.field(18, decode.string) + use has_links <- decode.field(19, decode.int) + use has_notes_reg <- decode.field(20, decode.int) + use has_notes_star <- decode.field(21, decode.int) + use has_images <- decode.field(22, decode.int) + use segment <- decode.field(23, decode.string) + use segment_prev <- decode.field(24, decode.string) + use segment_next <- decode.field(25, decode.string) + use anchor <- decode.field(26, decode.string) + use segment_html_is <- decode.field(27, decode.string) + use table_cols <- decode.field(28, decode.int) + use table_widths <- decode.field(29, decode.string) + use table_aligns <- decode.field(30, decode.string) + use table_header <- decode.field(31, decode.int) + use code_linenumbers <- decode.field(32, decode.int) + use hat <- decode.field(33, decode.string) + use sha256 <- decode.field(34, decode.string) + use text <- decode.field(35, decode.string) + use id <- decode.field(36, decode.int) + use hla <- decode.field(37, decode.string) + use seg_epub <- decode.field(38, decode.string) + use seg_epub_is <- decode.field(39, decode.string) + decode.success(#( + id, + Object( + ocn: ocn, + is_a: is_a, + section: section, + heading_level: level_name(heading_level), + identifier: identifier, + part: part, + is_of_type: is_of_type, + own_section: case is_of_section == section { + True -> "" + False -> is_of_section + }, + segment_html_is: segment_html_is, + table_aligns: json_words(table_aligns), + stow_link: [], + anchor_tags: [], + parent_ocn: parent, + last_descendant_ocn: last_descendant, + children: json_ints(children), + ancestors: json_ints(ancestors), + dummy_heading: dummy != 0, + ocn_off: ocn_off != 0, + indent_base: indent_base, + indent_hang: indent_hang, + bullet: bullet != 0, + lang: lang, + has_links: has_links != 0, + has_notes_reg: has_notes_reg != 0, + has_notes_star: has_notes_star != 0, + has_images: has_images != 0, + segment: segment, + segment_prev: segment_prev, + segment_next: segment_next, + anchor: anchor, + heading_lev_anchor: hla, + segment_epub: seg_epub, + segment_epub_is: seg_epub_is, + table_cols: table_cols, + table_widths: json_words(table_widths), + table_header: table_header != 0, + code_syntax: "", + code_linenumbers: code_linenumbers != 0, + heading_ancestors_text: json_joined(hat), + sha256: sha256, + images: [], + text: text, + ), + )) + } + let rows = + sqlight.query(object_sql(doc_id), on: conn, with: [], expecting: d) + |> result.unwrap([]) + + // Built newest-first per section and reversed once at the end. Appending + // to the tail of each section instead would be quadratic, and the largest + // document in the reference collection has some thousands of objects in + // one section. + let sections = + list.fold(rows, dict.new(), fn(acc, row) { + let #(id, object) = row + // the four repeatable properties of the text form are child tables + // here, one query each + let object = + Object( + ..object, + images: images(conn, id), + stow_link: links(conn, id), + anchor_tags: anchors(conn, id), + ) + case model.is_section(object.section) { + False -> acc + True -> + dict.upsert(acc, object.section, fn(existing) { + case existing { + option.Some(previous) -> [object, ..previous] + option.None -> [object] + } + }) + } + }) + |> dict.map_values(fn(_, objects) { list.reverse(objects) }) + model.Document(..document, sections: sections) +} + +fn object_sql(doc_id: Int) -> String { + "select section, seq, ocn, is_a, coalesce(is_of_part,''), + coalesce(is_of_section,''), coalesce(is_of_type,''), + coalesce(heading_level,-1), coalesce(identifier,''), + coalesce(parent_ocn,0), coalesce(last_descendant_ocn,0), + coalesce(children,''), coalesce(ancestors,''), + coalesce(dummy_heading,0), coalesce(object_number_off,0), + coalesce(indent_base,0), coalesce(indent_hang,0), + coalesce(bullet,0), coalesce(lang,''), + coalesce(has_links,0), coalesce(has_notes_reg,0), + coalesce(has_notes_star,0), coalesce(has_images,0), + coalesce(segment,''), coalesce(segment_prev,''), + coalesce(segment_next,''), coalesce(anchor,''), + coalesce(segment_html_is,''), + coalesce(table_cols,0), coalesce(table_widths,''), + coalesce(table_aligns,''), coalesce(table_header,0), + coalesce(code_linenumbers,0), + coalesce(heading_ancestors_text,''), coalesce(sha256,''), + coalesce(text,''), id, coalesce(heading_lev_anchor,''), + coalesce(segment_epub,''), coalesce(segment_epub_is,'') + from objects where doc_id = " <> int.to_string(doc_id) <> " + order by case section + when 'head' then 1 when 'toc' then 2 when 'body' then 3 + when 'endnotes' then 4 when 'glossary' then 5 + when 'bibliography' then 6 when 'bookindex' then 7 + when 'blurb' then 8 when 'tail' then 9 else 10 end, seq" +} + +fn images(conn: sqlight.Connection, object_id: Int) -> List(model.Image) { + let d = { + use name <- decode.field(0, decode.string) + use sha256 <- decode.field(1, decode.string) + use bytes <- decode.field(2, decode.int) + use width <- decode.field(3, decode.int) + use height <- decode.field(4, decode.int) + use missing <- decode.field(5, decode.int) + decode.success(Image( + name: name, + sha256: sha256, + bytes: bytes, + width: width, + height: height, + missing: missing != 0, + fields: dict.new(), + )) + } + sqlight.query( + "select name, coalesce(sha256,''), coalesce(bytes,0), + coalesce(width,0), coalesce(height,0), coalesce(missing,0) + from object_images where object_id = " + <> int.to_string(object_id) + <> " order by seq", + on: conn, + with: [], + expecting: d, + ) + |> result.unwrap([]) +} + +fn links(conn: sqlight.Connection, object_id: Int) -> List(String) { + let d = { + use url <- decode.field(0, decode.string) + decode.success(url) + } + sqlight.query( + "select url from object_links where object_id = " + <> int.to_string(object_id) + <> " order by seq", + on: conn, + with: [], + expecting: d, + ) + |> result.unwrap([]) +} + +fn anchors(conn: sqlight.Connection, object_id: Int) -> List(String) { + let d = { + use anchor <- decode.field(0, decode.string) + decode.success(anchor) + } + sqlight.query( + "select anchor from object_anchors where object_id = " + <> int.to_string(object_id) + <> " order by seq", + on: conn, + with: [], + expecting: d, + ) + |> result.unwrap([]) +} + +fn one(conn: sqlight.Connection, sql: String) -> String { + let d = { + use v <- decode.field(0, decode.string) + decode.success(v) + } + case sqlight.query(sql, on: conn, with: [], expecting: d) { + Ok([v, ..]) -> v + _ -> "" + } +} + +fn has(conn: sqlight.Connection, name: String) -> Bool { + one( + conn, + "select cast(count(*) as text) from sqlite_master where name = '" + <> name + <> "'", + ) + != "0" +} + +fn meta_value(conn: sqlight.Connection, doc_id: Int, key: String) -> String { + one( + conn, + "select value from metadata where doc_id = " + <> int.to_string(doc_id) + <> " and key = '" + <> key + <> "'", + ) +} + +fn major_of(version: String) -> String { + case string.split_once(version, ".") { + Ok(#(major, _)) -> major + Error(_) -> version + } +} + +/// The eight heading levels and their indices. +fn level_name(index: Int) -> String { + case index { + 0 -> "A" + 1 -> "B" + 2 -> "C" + 3 -> "D" + 4 -> "1" + 5 -> "2" + 6 -> "3" + 7 -> "4" + _ -> "" + } +} + +/// The JSON columns, read as JSON. +/// +/// Not as a character scan that strips brackets and quotes: that is wrong +/// the moment a value holds an escape, and spine's encoder writes "/" as +/// "\/", so every heading containing a slash comes back with a backslash in +/// it. The D library had exactly that bug and all 36 document-languages +/// differed on heading_ancestors_text because of it. +fn json_strings(value: String) -> List(String) { + case value { + "" -> [] + v -> + case json.parse(v, decode.list(of: loose_string())) { + Ok(items) -> items + Error(_) -> [] + } + } +} + +/// An element that may be a string or a number, which is what these columns +/// hold: children and ancestors are int arrays, table_aligns and +/// heading_ancestors_text string arrays, and table_widths an int array +/// except where the columns divide evenly, when it is [33.3333,...]. One +/// table in the reference collection does that, and without the float +/// branch the widths came back as three empty strings: the fallback turned +/// a value it could not read into a value that looked read. +fn loose_string() -> decode.Decoder(String) { + decode.one_of(decode.string, [ + decode.int |> decode.map(int.to_string), + decode.float |> decode.map(float.to_string), + decode.success(""), + ]) +} + +fn json_ints(value: String) -> List(Int) { + value + |> json_strings + |> list.filter(fn(s) { s != "" }) + |> list.map(fn(s) { s |> int.parse |> result.unwrap(0) }) +} + +/// Space separated in the text form: table_widths, table_aligns. +fn json_words(value: String) -> String { + value |> json_strings |> string.join(" ") +} + +/// heading_ancestors_text is eight slots, joined with "|" in the text form +/// and a JSON array here. The empty slots are significant: they say which +/// levels the heading sits under. +fn json_joined(value: String) -> String { + case json_strings(value) { + [] -> "" + parts -> string.join(parts, "|") + } +} diff --git a/gleam/src/ocda/load_pod.gleam b/gleam/src/ocda/load_pod.gleam new file mode 100644 index 0000000..35483c6 --- /dev/null +++ b/gleam/src/ocda/load_pod.gleam @@ -0,0 +1,138 @@ +//// The abstraction out of a pod archive. +//// +//// A pod built with `--pod2` carries the abstraction inside it, at +//// `media/abstraction/<uid>.<lang>.ssp`, and that is what is read. So this is +//// a third input form and *not* a third model: the `.ssp` inside is parsed by +//// `ocda/load_ssp` and comes back as the same value. A writer over the +//// library gains pods without being touched, which is the whole reason the +//// dispatch is in one place. +//// +//// ## The one thing this cannot do +//// +//// A `.sisupod` is **one zstd frame wrapping** the zip. This reads the zip +//// and **not the frame**, because there is no zstd on the BEAM without a NIF, +//// and the one that exists, `ezstd`, clones zstd from github in its build, so +//// it cannot be built in a sandbox without patching it. The D library has +//// both, through a declared `libzstd`. +//// +//// So: a `.zip` pod is read, a `.sisupod` is refused, and the refusal says +//// why rather than pretending the file is malformed. It is the only place the +//// two libraries differ in what they accept, and it is a packaging limit +//// rather than anything about the format. + +import gleam/list +import gleam/result +import gleam/string +import ocda/load_ssp +import ocda/model.{type Document} +import ocda/zip +import simplifile + +/// The magic number of a zstd frame, little endian 0xFD2FB528. +const zstd_magic = <<0x28, 0xB5, 0x2F, 0xFD>> + +/// Which languages of the abstraction the pod carries, sorted. +pub fn languages(path: String) -> Result(List(String), String) { + use bytes <- result.try(archive_bytes(path)) + use entries <- result.try(zip.entries(bytes)) + zip.find_suffix(entries, ".ssp") + |> list.filter_map(fn(e) { lang_of(e.name) }) + |> list.sort(string.compare) + |> Ok +} + +/// Load one language of the abstraction a pod carries. +/// +/// `lang` == "" takes the only one there is, and where there is a choice it +/// says so rather than choosing, as the `.ocda.db` loader does. +pub fn load_file(path: String, lang: String) -> Result(Document, String) { + use bytes <- result.try(archive_bytes(path)) + use entries <- result.try(zip.entries(bytes)) + let ssps = zip.find_suffix(entries, ".ssp") + use chosen <- result.try(case ssps, lang { + [], _ -> + Error( + path + <> ": carries no .ssp. A pod carries the abstraction only when spine" + <> " was asked for it: spine --pod2 puts it in, spine --source does not.", + ) + [only], "" -> Ok(only) + _, "" -> + Error( + path + <> ": holds more than one language (" + <> string.join(langs(ssps), " ") + <> "); name the one wanted", + ) + _, wanted -> + case list.find(ssps, fn(e) { lang_of(e.name) == Ok(wanted) }) { + Ok(found) -> Ok(found) + Error(_) -> + Error( + path + <> ": no language '" + <> wanted + <> "'; it holds: " + <> string.join(langs(ssps), " "), + ) + } + }) + use content <- result.try(zip.read(bytes, chosen)) + case bit_array_to_string(content) { + Ok(text) -> Ok(load_ssp.parse(text)) + Error(_) -> Error(chosen.name <> ": not valid UTF-8") + } +} + +fn langs(entries: List(zip.Entry)) -> List(String) { + list.filter_map(entries, fn(e) { lang_of(e.name) }) +} + +/// `<uid>.<lang>.ssp` -> `<lang>` +fn lang_of(name: String) -> Result(String, Nil) { + case string.ends_with(name, ".ssp") { + False -> Error(Nil) + True -> { + let stem = string.drop_end(name, 4) + case string.split(stem, on: ".") |> list.last { + Ok(lang) if lang != stem -> Ok(lang) + _ -> Error(Nil) + } + } + } +} + +/// The bytes decide, not the name: a file misnamed either way is treated by +/// what it holds. +fn archive_bytes(path: String) -> Result(BitArray, String) { + case simplifile.read_bits(path) { + Error(e) -> Error(path <> ": " <> simplifile.describe_error(e)) + Ok(bytes) -> + case is_zstd_frame(bytes) { + False -> Ok(bytes) + True -> + Error( + path + <> " is a zstd frame wrapping the archive, and this library has no" + <> " zstd: the only one on the BEAM is a NIF that fetches zstd in" + <> " its own build, so it cannot be built sandboxed. Unwrap it" + <> " first (zstd -d) and read the archive, or use the D library," + <> " which reads both.", + ) + } + } +} + +fn is_zstd_frame(bytes: BitArray) -> Bool { + case bytes { + <<head:bytes-size(4), _:bits>> -> head == zstd_magic + _ -> False + } +} + +@external(erlang, "unicode", "characters_to_binary") +fn bit_array_to_string_ffi(b: BitArray) -> String + +fn bit_array_to_string(b: BitArray) -> Result(String, Nil) { + Ok(bit_array_to_string_ffi(b)) +} diff --git a/gleam/src/ocda/load_ssp.gleam b/gleam/src/ocda/load_ssp.gleam new file mode 100644 index 0000000..b48e1c5 --- /dev/null +++ b/gleam/src/ocda/load_ssp.gleam @@ -0,0 +1,478 @@ +//// Loading a `.ssp` into the model. +//// +//// The format is line oriented and every line is one of eight shapes told +//// apart by its first character, so the loader is a fold over the lines +//// carrying the state the shapes need. +//// +//// Written against `ssp-format-specification.md`. Where it and the D +//// library differ the D library is the bug, or this is: they are meant to +//// give the same model, and `ocda-dump` says whether they do. + +import gleam/dict.{type Dict} +import gleam/int +import gleam/list +import gleam/option +import gleam/result +import gleam/string +import ocda/model.{type Document, type Image, type Object, Image, Object} +import simplifile + +pub fn load_file(path: String) -> Result(Document, String) { + case simplifile.read(path) { + Ok(content) -> Ok(parse(content)) + Error(e) -> Error(path <> ": " <> simplifile.describe_error(e)) + } +} + +type State { + State( + document: Document, + /// the block currently open, "" when none + block: String, + /// the section currently open, "" when in the preamble + section: String, + /// objects of the current section, newest first + objects: List(Object), + /// the object being read, if any + current: Result(Object, Nil), + /// text lines of the object being read, newest first + text: List(String), + /// whether a text line has been seen for the object being read. + /// Not "is the text non-empty": an object whose first text line is + /// blank, which the reference collection has seven of, would then lose + /// it. + seen_text: Bool, + ) +} + +pub fn parse(content: String) -> Document { + content + |> string.split(on: "\n") + |> list.fold(from: initial(), with: line) + |> finish +} + +fn initial() -> State { + State( + document: model.new_document(), + block: "", + section: "", + objects: [], + current: Error(Nil), + text: [], + seen_text: False, + ) +} + +fn line(state: State, raw: String) -> State { + case raw { + "% SiSU Document Abstraction v" <> version -> + case state.document.format_version { + "" -> + State( + ..state, + document: model.Document( + ..state.document, + format_version: string.trim(version), + ), + ) + _ -> state + } + "% Source: " <> filename -> + case state.document.source_filename { + "" -> + State( + ..state, + document: model.Document( + ..state.document, + source_filename: string.trim(filename), + ), + ) + _ -> state + } + "%" <> _ -> state + "@" <> rest -> open_block(state, rest) + "}" <> _ -> close_block(state) + "[" <> rest -> declaration(state, rest) + "." <> rest -> property(state, rest) + "|" <> rest -> text_line(state, rest) + _ -> + case string.trim(raw) { + // a blank line ends the object being read + "" -> commit(state) + _ -> + case model.is_preamble_block(state.block) { + True -> preamble_pair(state, raw) + False -> unknown_line(state, raw) + } + } + } +} + +fn open_block(state: State, rest: String) -> State { + let name = case string.split_once(rest, "{") { + Ok(#(name, _)) -> string.trim(name) + Error(_) -> string.trim(rest) + } + case model.is_preamble_block(name) { + True -> State(..flush(state), block: name) + False -> { + let state = flush(state) + let document = case model.is_section(name) { + True -> state.document + // a block that is neither preamble nor one of the nine sections: + // counted, so that a reader can say the file held something it did + // not expect + False -> + model.Document( + ..state.document, + unknown: model.Unknown( + ..state.document.unknown, + block: bump(state.document.unknown.block, name), + ), + ) + } + State(..state, block: name, section: name, document: document) + } + } +} + +fn close_block(state: State) -> State { + State(..flush(state), block: "") +} + +/// Commit the object being read, then put the section's objects into the +/// document and close the section. +fn flush(state: State) -> State { + let state = commit(state) + case state.section, state.objects { + "", _ -> state + _, [] -> State(..state, section: "", objects: []) + name, objects -> + State( + ..state, + section: "", + objects: [], + document: model.Document( + ..state.document, + sections: add_section( + state.document.sections, + name, + list.reverse(objects), + ), + ), + ) + } +} + +fn preamble_pair(state: State, raw: String) -> State { + case string.split_once(string.trim_start(raw), ":") { + Error(_) -> state + Ok(#(key, value)) -> { + let key = string.trim(key) + let value = string.trim(value) + let document = state.document + let document = case state.block { + "meta" -> + model.Document( + ..document, + metadata: dict.insert(document.metadata, key, value), + ) + "make" -> + model.Document(..document, make: dict.insert(document.make, key, value)) + "source" -> + model.Document( + ..document, + source: dict.insert(document.source, key, value), + ) + _ -> + model.Document( + ..document, + doc_has: dict.insert(document.doc_has, key, value), + ) + } + State(..state, document: document) + } + } +} + +/// `[<ocn>] <kind>`, and for a heading `[<ocn>] heading :<level> +/// [<identifier>]`. +fn declaration(state: State, rest: String) -> State { + let state = commit(state) + case string.split_once(rest, "]") { + Error(_) -> state + Ok(#(ocn_text, tail)) -> { + let ocn = parse_int(string.trim(ocn_text)) + let object = + Object(..model.new_object(), ocn: ocn, section: state.section) + let object = case string.split(string.trim(tail), on: " ") { + ["heading", level, ..identifier] -> + Object( + ..object, + is_a: "heading", + heading_level: string.drop_start(level, 1), + identifier: string.join(identifier, " "), + ) + [""] -> object + [kind, ..] -> Object(..object, is_a: kind) + [] -> object + } + State(..state, current: Ok(object), text: [], seen_text: False) + } + } +} + +fn property(state: State, rest: String) -> State { + case state.current, string.split_once(rest, ":") { + Ok(object), Ok(#(name, raw_value)) -> { + let name = string.trim(name) + let value = string.trim(raw_value) + let #(object, known) = apply_property(object, name, value) + let document = case known { + True -> state.document + False -> + model.Document( + ..state.document, + unknown: model.Unknown( + ..state.document.unknown, + property: bump(state.document.unknown.property, name), + ), + ) + } + State(..state, current: Ok(object), document: document) + } + _, _ -> state + } +} + +/// The properties this library reads, and what each sets. +/// +/// A property it has no use for is counted rather than passed over: +/// ignoring one is conforming within a major version, doing so in silence +/// is what stops a reader being a check on the specification. +fn apply_property( + object: Object, + name: String, + value: String, +) -> #(Object, Bool) { + case name { + "part" -> #(Object(..object, part: value), True) + "is_of_type" -> #(Object(..object, is_of_type: value), True) + "section" -> #(Object(..object, own_section: value), True) + "identifier" -> #(Object(..object, identifier: value), True) + "segment_html_is" -> #(Object(..object, segment_html_is: value), True) + "segment_epub" -> #(Object(..object, segment_epub: value), True) + "segment_epub_is" -> #(Object(..object, segment_epub_is: value), True) + "table_aligns" -> #(Object(..object, table_aligns: value), True) + "stow_link" -> #( + Object(..object, stow_link: list.append(object.stow_link, [value])), + True, + ) + "anchor_tag" -> #( + Object(..object, anchor_tags: list.append(object.anchor_tags, [value])), + True, + ) + "parent" -> #(Object(..object, parent_ocn: parse_int(value)), True) + "last_descendant" -> #( + Object(..object, last_descendant_ocn: parse_int(value)), + True, + ) + "children" -> #(Object(..object, children: numbers(value)), True) + "ancestors" -> #(Object(..object, ancestors: numbers(value)), True) + "dummy" -> #(Object(..object, dummy_heading: value == "true"), True) + "ocn_off" -> #(Object(..object, ocn_off: value == "true"), True) + "indent" -> { + let parts = string.split(value, on: " ") + let base = case parts { + [first, ..] -> parse_int(first) + [] -> 0 + } + let hang = case parts { + [_, second, ..] -> parse_int(second) + _ -> 0 + } + #(Object(..object, indent_base: base, indent_hang: hang), True) + } + "bullet" -> #(Object(..object, bullet: value == "true"), True) + "lang" -> #(Object(..object, lang: value), True) + "has" -> #(has_flags(object, value), True) + "segment" -> #(Object(..object, segment: value), True) + "segment_prev" -> #(Object(..object, segment_prev: value), True) + "segment_next" -> #(Object(..object, segment_next: value), True) + "anchor" -> #(Object(..object, anchor: value), True) + "heading_lev_anchor" -> #( + Object(..object, heading_lev_anchor: value), + True, + ) + "table_cols" -> #(Object(..object, table_cols: parse_int(value)), True) + "table_widths" -> #(Object(..object, table_widths: value), True) + "table_header" -> #(Object(..object, table_header: value == "true"), True) + "code_syntax" -> #(Object(..object, code_syntax: value), True) + "code_linenumbers" -> #( + Object(..object, code_linenumbers: value == "true"), + True, + ) + "heading_ancestors_text" -> #( + Object(..object, heading_ancestors_text: value), + True, + ) + "sha256" -> #(Object(..object, sha256: value), True) + "image" -> #( + Object(..object, images: list.append(object.images, [image(value)])), + True, + ) + _ -> #(object, False) + } +} + +fn has_flags(object: Object, value: String) -> Object { + value + |> string.split(on: " ") + |> list.fold(from: object, with: fn(obj, flag) { + case flag { + "links" -> Object(..obj, has_links: True) + "notes_reg" -> Object(..obj, has_notes_reg: True) + "notes_star" -> Object(..obj, has_notes_star: True) + "images" -> Object(..obj, has_images: True) + _ -> obj + } + }) +} + +/// `<filename> sha256:<HEX> bytes:<n> [px:<w>x<h>] [missing:true] [...]` +/// +/// The filename comes first, the rest are order-independent, and a field +/// this library does not recognise is kept rather than discarded. +fn image(value: String) -> Image { + case string.split(value, on: " ") { + [] -> model.new_image() + [name, ..fields] -> + list.fold(fields, model.new_image(), fn(img, field) { + case string.split_once(field, ":") { + Error(_) -> img + Ok(#(key, val)) -> + case key { + "sha256" -> Image(..img, sha256: val) + "bytes" -> Image(..img, bytes: parse_int(val)) + "px" -> + case string.split_once(val, "x") { + Ok(#(w, h)) -> + Image(..img, width: parse_int(w), height: parse_int(h)) + Error(_) -> img + } + "missing" -> Image(..img, missing: val == "true") + _ -> Image(..img, fields: dict.insert(img.fields, key, val)) + } + } + }) + |> fn(img) { Image(..img, name: name) } + } +} + +/// `| <text>`, and `| ` alone for a blank line. Never a bare `|`: the +/// emitter writes the space. +fn text_line(state: State, rest: String) -> State { + case state.current { + Error(_) -> state + Ok(_) -> { + let content = string.drop_start(rest, 1) + State(..state, text: [content, ..state.text], seen_text: True) + } + } +} + +fn unknown_line(state: State, raw: String) -> State { + let first = string.slice(raw, 0, 1) + State( + ..state, + document: model.Document( + ..state.document, + unknown: model.Unknown( + ..state.document.unknown, + line: bump(state.document.unknown.line, first), + ), + ), + ) +} + +fn commit(state: State) -> State { + case state.current { + Error(_) -> state + Ok(object) -> { + let text = case state.seen_text { + True -> string.join(list.reverse(state.text), "\n") + False -> "" + } + let object = derive_omitted(Object(..object, text: text)) + let objects = case model.is_section(state.section) { + True -> [object, ..state.objects] + False -> state.objects + } + State( + ..state, + current: Error(Nil), + text: [], + seen_text: False, + objects: objects, + ) + } + } +} + +/// Two values the text form leaves out because they can be worked out. +/// +/// `.identifier` is absent when it is simply the ocn, `.anchor` when it is +/// the segment name. The `.ocda.db`, being columns rather than lines, +/// stores both unconditionally. So a loader that takes the text form +/// literally gives a different model from the one the database gives, and +/// over one document that is 1,314 identifiers and 69 anchors. +fn derive_omitted(object: Object) -> Object { + let object = case object.identifier, object.ocn { + "", 0 -> object + "", ocn -> Object(..object, identifier: int.to_string(ocn)) + _, _ -> object + } + case object.anchor, object.segment { + "", "" -> object + "", segment -> Object(..object, anchor: segment) + _, _ -> object + } +} + +fn finish(state: State) -> Document { + flush(state).document +} + +fn add_section( + sections: Dict(String, List(Object)), + name: String, + objects: List(Object), +) -> Dict(String, List(Object)) { + dict.upsert(sections, name, fn(existing) { + case existing { + option.Some(previous) -> list.append(previous, objects) + option.None -> objects + } + }) +} + +fn bump(counts: Dict(String, Int), key: String) -> Dict(String, Int) { + dict.upsert(counts, key, fn(existing) { + case existing { + option.Some(n) -> n + 1 + option.None -> 1 + } + }) +} + +fn parse_int(value: String) -> Int { + value |> string.trim |> int.parse |> result.unwrap(0) +} + +fn numbers(value: String) -> List(Int) { + value + |> string.split(on: " ") + |> list.filter(fn(s) { s != "" }) + |> list.map(parse_int) +} diff --git a/gleam/src/ocda/model.gleam b/gleam/src/ocda/model.gleam new file mode 100644 index 0000000..050a97b --- /dev/null +++ b/gleam/src/ocda/model.gleam @@ -0,0 +1,278 @@ +//// The object-centric document abstraction, as Gleam types. +//// +//// A deliberate mirror of the D library's `ocda.model`, field for field. +//// The two are meant to agree, and the cheapest way to make that testable +//// is for them to hold the same things under the same names: the dump of a +//// loaded document is then identical by construction rather than by +//// negotiation. +//// +//// Written against specs/sisudoc-abstraction-format: `ocda-model.md` for +//// the model, the two format specifications for what each serialisation +//// says about it. + +import gleam/dict.{type Dict} +import gleam/list +import gleam/string + +/// The format major version this library knows. +/// +/// The `.ssp` and the `.ocda.db` carry one version between them, declared +/// in band. The minor part rises when a property is added and a reader of +/// the newer format still reads the older, so only the major is checked. +pub const format_major_read = "2" + +/// The nine sections, in the order the format gives them, which is not the +/// order a sorted key list or a `group by` would give. +pub const section_names = [ + "head", "toc", "body", "endnotes", "glossary", "bibliography", "bookindex", + "blurb", "tail", +] + +/// The four blocks that are document preamble rather than sections of +/// objects. +pub const preamble_blocks = ["source", "meta", "make", "doc_has"] + +/// A repeatable `.image` record: the file as the abstraction found it. +pub type Image { + Image( + /// filename as the object refers to it + name: String, + /// hex, uppercase + sha256: String, + bytes: Int, + /// the file's own pixel dimensions, not any display width + width: Int, + height: Int, + /// absent or unreadable, so no digest and no size + missing: Bool, + /// any further key:value fields, kept rather than dropped + fields: Dict(String, String), + ) +} + +pub fn new_image() -> Image { + Image( + name: "", + sha256: "", + bytes: 0, + width: 0, + height: 0, + missing: False, + fields: dict.new(), + ) +} + +/// One object of the abstraction. +/// +/// `text` is one string with the lines joined by newline, as in the D +/// library and for the same reason: an object whose only text line is +/// blank is then indistinguishable from an object with no text, in both +/// libraries, in the same place. A difference that exists in one language +/// and not the other would be worse than a loss both share. +pub type Object { + Object( + ocn: Int, + /// one of the sixteen kinds + is_a: String, + /// the section this object was found in + section: String, + /// headings: A B C D 1 2 3 4 + heading_level: String, + identifier: String, + part: String, + /// how the text behaves: para, block, comment + is_of_type: String, + /// the object's own section, when it differs from the block it sits in + own_section: String, + segment_html_is: String, + segment_epub: String, + segment_epub_is: String, + table_aligns: String, + stow_link: List(String), + anchor_tags: List(String), + parent_ocn: Int, + last_descendant_ocn: Int, + children: List(Int), + ancestors: List(Int), + dummy_heading: Bool, + ocn_off: Bool, + indent_base: Int, + indent_hang: Int, + bullet: Bool, + lang: String, + has_links: Bool, + has_notes_reg: Bool, + has_notes_star: Bool, + has_images: Bool, + segment: String, + segment_prev: String, + segment_next: String, + anchor: String, + heading_lev_anchor: String, + table_cols: Int, + table_widths: String, + table_header: Bool, + code_syntax: String, + code_linenumbers: Bool, + heading_ancestors_text: String, + sha256: String, + images: List(Image), + text: String, + ) +} + +pub fn new_object() -> Object { + Object( + ocn: 0, + is_a: "", + section: "", + heading_level: "", + identifier: "", + part: "", + is_of_type: "", + own_section: "", + segment_html_is: "", + segment_epub: "", + segment_epub_is: "", + table_aligns: "", + stow_link: [], + anchor_tags: [], + parent_ocn: 0, + last_descendant_ocn: 0, + children: [], + ancestors: [], + dummy_heading: False, + ocn_off: False, + indent_base: 0, + indent_hang: 0, + bullet: False, + lang: "", + has_links: False, + has_notes_reg: False, + has_notes_star: False, + has_images: False, + segment: "", + segment_prev: "", + segment_next: "", + anchor: "", + heading_lev_anchor: "", + table_cols: 0, + table_widths: "", + table_header: False, + code_syntax: "", + code_linenumbers: False, + heading_ancestors_text: "", + sha256: "", + images: [], + text: "", + ) +} + +/// Whether this object is rendered at all. +/// +/// `is_of_type` "comment" is markup commentary: carried by the abstraction +/// so that nothing is lost, and not put into output. An output that +/// renders it leaks the author's notes into the document, which is what +/// spine itself does not do. +pub fn renders(object: Object) -> Bool { + object.is_of_type != "comment" +} + +/// The section this object belongs to: the block it sits in, unless it +/// says otherwise. +pub fn section_of(object: Object) -> String { + case object.own_section { + "" -> object.section + own -> own + } +} + +/// What the loader met and did not understand. +/// +/// A loader that silently drops what it does not know cannot say whether +/// it understood the document, and so cannot be used to check a +/// specification against an implementation: silence and understanding look +/// alike. +pub type Unknown { + Unknown( + /// `.name` -> times seen + property: Dict(String, Int), + /// `@name` -> times seen + block: Dict(String, Int), + /// first character -> times seen, for lines of no known shape + line: Dict(String, Int), + ) +} + +pub fn new_unknown() -> Unknown { + Unknown(property: dict.new(), block: dict.new(), line: dict.new()) +} + +pub fn any_unknown(unknown: Unknown) -> Bool { + dict.size(unknown.property) > 0 + || dict.size(unknown.block) > 0 + || dict.size(unknown.line) > 0 +} + +pub type Document { + Document( + /// as declared on line 1 of a `.ssp` or in `schema.version`, "" when + /// absent + format_version: String, + source_filename: String, + /// language, languages, digest + source: Dict(String, String), + unknown: Unknown, + metadata: Dict(String, String), + make: Dict(String, String), + doc_has: Dict(String, String), + /// section name -> objects, in document order + sections: Dict(String, List(Object)), + ) +} + +pub fn new_document() -> Document { + Document( + format_version: "", + source_filename: "", + source: dict.new(), + unknown: new_unknown(), + metadata: dict.new(), + make: dict.new(), + doc_has: dict.new(), + sections: dict.new(), + ) +} + +/// The major part of the declared version, "" when none was declared. +pub fn version_major(document: Document) -> String { + case string.split_once(document.format_version, ".") { + Ok(#(major, _)) -> major + Error(_) -> document.format_version + } +} + +pub fn meta(document: Document, key: String) -> String { + case dict.get(document.metadata, key) { + Ok(value) -> value + Error(_) -> "" + } +} + +/// The objects of a named section, empty when the document has none. +pub fn section(document: Document, name: String) -> List(Object) { + case dict.get(document.sections, name) { + Ok(objects) -> objects + Error(_) -> [] + } +} + +/// Is this one of the nine sections of the format? +pub fn is_section(name: String) -> Bool { + list.contains(section_names, name) +} + +/// Is this one of the four preamble blocks? +pub fn is_preamble_block(name: String) -> Bool { + list.contains(preamble_blocks, name) +} diff --git a/gleam/src/ocda/zip.gleam b/gleam/src/ocda/zip.gleam new file mode 100644 index 0000000..5ce6e09 --- /dev/null +++ b/gleam/src/ocda/zip.gleam @@ -0,0 +1,170 @@ +//// Enough of the zip format to find one file in a pod and read it. +//// +//// Not a zip library. A pod archive holds a handful of entries, spine writes +//// them **stored** (uncompressed) so that the outer zstd frame does the +//// compressing once over the whole thing, and what is wanted from it here is +//// one named entry. +//// +//// Deflate is *refused rather than mishandled*. Erlang's `zlib` could +//// inflate it, but a library that says it read an entry it in fact skipped is +//// worse than one that says it cannot, and spine's own archives do not need +//// it. + +import gleam/bit_array +import gleam/int +import gleam/list +import gleam/result +import gleam/string + +pub type Entry { + Entry( + name: String, + /// 0 stored, 8 deflate + method: Int, + compressed: Int, + uncompressed: Int, + local_offset: Int, + ) +} + +const eocd_sig = 0x06054b50 + +const cd_sig = 0x02014b50 + +const local_sig = 0x04034b50 + +/// The entries, from the central directory at the end of the file. +pub fn entries(bytes: BitArray) -> Result(List(Entry), String) { + let size = bit_array.byte_size(bytes) + use _ <- result.try(case size >= 22 { + True -> Ok(Nil) + False -> Error("too short to be a zip archive") + }) + use eocd <- result.try(find_eocd(bytes, size - 22, lowest(size))) + use count <- result.try(u16(bytes, eocd + 10)) + use offset <- result.try(u32(bytes, eocd + 16)) + read_central(bytes, offset, count, []) +} + +/// The end of central directory record is last, and may be followed by a +/// comment of up to 65535 bytes, so it is searched for backwards. +fn find_eocd(bytes: BitArray, at: Int, lowest: Int) -> Result(Int, String) { + case at < lowest { + True -> + Error( + "no end of central directory record; it does not look like a zip archive", + ) + False -> + case u32(bytes, at) { + Ok(sig) if sig == eocd_sig -> Ok(at) + _ -> find_eocd(bytes, at - 1, lowest) + } + } +} + +fn lowest(size: Int) -> Int { + case size > 22 + 65_535 { + True -> size - 22 - 65_535 + False -> 0 + } +} + +fn read_central( + bytes: BitArray, + at: Int, + remaining: Int, + acc: List(Entry), +) -> Result(List(Entry), String) { + case remaining { + 0 -> Ok(list.reverse(acc)) + _ -> { + use sig <- result.try(u32(bytes, at)) + use _ <- result.try(case sig == cd_sig { + True -> Ok(Nil) + False -> + Error( + "central directory entry " + <> int.to_string(list.length(acc)) + <> " is malformed", + ) + }) + use method <- result.try(u16(bytes, at + 10)) + use compressed <- result.try(u32(bytes, at + 20)) + use uncompressed <- result.try(u32(bytes, at + 24)) + use name_len <- result.try(u16(bytes, at + 28)) + use extra_len <- result.try(u16(bytes, at + 30)) + use cmnt_len <- result.try(u16(bytes, at + 32)) + use local_offset <- result.try(u32(bytes, at + 42)) + use name <- result.try(text(bytes, at + 46, name_len)) + let entry = + Entry( + name: name, + method: method, + compressed: compressed, + uncompressed: uncompressed, + local_offset: local_offset, + ) + read_central( + bytes, + at + 46 + name_len + extra_len + cmnt_len, + remaining - 1, + [entry, ..acc], + ) + } + } +} + +/// One entry's bytes, from its local header. +pub fn read(bytes: BitArray, entry: Entry) -> Result(BitArray, String) { + use sig <- result.try(u32(bytes, entry.local_offset)) + use _ <- result.try(case sig == local_sig { + True -> Ok(Nil) + False -> Error(entry.name <> ": local header is malformed") + }) + use name_len <- result.try(u16(bytes, entry.local_offset + 26)) + use extra_len <- result.try(u16(bytes, entry.local_offset + 28)) + use _ <- result.try(case entry.method { + 0 -> Ok(Nil) + m -> + Error( + entry.name + <> ": compression method " + <> int.to_string(m) + <> "; this reader takes only stored entries, which is what spine writes", + ) + }) + let start = entry.local_offset + 30 + name_len + extra_len + case bit_array.slice(bytes, start, entry.compressed) { + Ok(slice) -> Ok(slice) + Error(_) -> Error(entry.name <> ": runs past the end of the archive") + } +} + +/// The entries whose names end this way, in archive order. Used to find +/// media/abstraction/<uid>.<lang>.ssp without knowing the uid. +pub fn find_suffix(entries: List(Entry), suffix: String) -> List(Entry) { + list.filter(entries, fn(e) { string.ends_with(e.name, suffix) }) +} + +fn u16(bytes: BitArray, at: Int) -> Result(Int, String) { + case bit_array.slice(bytes, at, 2) { + Ok(<<v:little-size(16)>>) -> Ok(v) + _ -> Error("read past the end of the archive at " <> int.to_string(at)) + } +} + +fn u32(bytes: BitArray, at: Int) -> Result(Int, String) { + case bit_array.slice(bytes, at, 4) { + Ok(<<v:little-size(32)>>) -> Ok(v) + _ -> Error("read past the end of the archive at " <> int.to_string(at)) + } +} + +fn text(bytes: BitArray, at: Int, len: Int) -> Result(String, String) { + case bit_array.slice(bytes, at, len) { + Ok(slice) -> + bit_array.to_string(slice) + |> result.replace_error("an entry name is not valid UTF-8") + Error(_) -> Error("an entry name runs past the end of the archive") + } +} diff --git a/gleam/src/ocda_lib.gleam b/gleam/src/ocda_lib.gleam new file mode 100644 index 0000000..1e8a5ae --- /dev/null +++ b/gleam/src/ocda_lib.gleam @@ -0,0 +1,136 @@ +//// ocda-lib, in Gleam: the abstraction as a library, put to the question. +//// +//// A library has no binary, so this is the two programs that test its +//// claims. +//// +//// ocda-lib dump [--unknown] <file>.ssp|<file>.ocda.db [lang] +//// ocda-lib agree <doc>.ssp <doc>.ocda.db [lang] +//// +//// `dump` writes the whole loaded model out canonically, byte for byte as +//// the D library's `ocda-dump` writes it, so that the two implementations +//// can be diffed. `agree` asks the same question the D library's +//// `ocda-agree` asks, within this language: does loading a document's +//// `.ssp` give what loading its `.ocda.db` gives? + +import argv +import gleam/int +import gleam/io +import gleam/list +import gleam/string +import ocda/dump +import ocda/load +import ocda/model.{type Document} + +pub fn main() { + case argv.load().arguments { + ["dump", "--unknown", path] -> dump_file(path, "", True) + ["dump", "--unknown", path, lang] -> dump_file(path, lang, True) + ["dump", path] -> dump_file(path, "", False) + ["dump", path, lang] -> dump_file(path, lang, False) + ["agree", ssp, db] -> agree(ssp, db, "") + ["agree", ssp, db, lang] -> agree(ssp, db, lang) + _ -> usage() + } +} + +fn usage() -> Nil { + io.println_error( + "usage: ocda-lib dump [--unknown] <file>.ssp|<file>.ocda.db [lang] + ocda-lib agree <doc>.ssp <doc>.ocda.db [lang]", + ) + halt(2) +} + +fn dump_file(path: String, lang: String, unknown: Bool) -> Nil { + case load.load(path, lang, False) { + Error(message) -> fail(message) + Ok(doc) -> + case unknown { + True -> io.print(dump.unknown(doc)) + False -> io.print(dump.document(doc)) + } + } +} + +/// The library's whole purpose is that a writer cannot tell which +/// serialisation it was handed. That is only true if the two loaders give +/// the same model, so this asks, and says where they part company rather +/// than only that they did. +fn agree(ssp_path: String, db_path: String, lang: String) -> Nil { + case load.load(ssp_path, "", False), load.load(db_path, lang, False) { + Error(message), _ -> fail(message) + _, Error(message) -> fail(message) + Ok(from_ssp), Ok(from_db) -> { + let differences = compare(from_ssp, from_db) + case differences { + [] -> io.println("agree: the two loaders give the same model") + _ -> { + list.each(list.take(differences, 12), io.println) + case list.length(differences) > 12 { + True -> io.println(" ... stopping after 12") + False -> Nil + } + io.println(int.to_string(list.length(differences)) <> " difference(s)") + halt(1) + } + } + } + } +} + +/// Compared through the dump rather than field by field, which is both +/// shorter and stricter: the dump carries every field, so nothing can +/// differ without showing here. +fn compare(from_ssp: Document, from_db: Document) -> List(String) { + let a = dump.document(from_ssp) + let b = dump.document(from_db) + case a == b { + True -> [] + False -> diff_lines(a, b) + } +} + +fn diff_lines(a: String, b: String) -> List(String) { + let la = split_lines(a) + let lb = split_lines(b) + case list.length(la) == list.length(lb) { + False -> [ + " differ: line count", + " .ssp: " <> int.to_string(list.length(la)), + " .ocda.db: " <> int.to_string(list.length(lb)), + ..pairwise(la, lb) + ] + True -> pairwise(la, lb) + } +} + +fn pairwise(la: List(String), lb: List(String)) -> List(String) { + list.zip(la, lb) + |> list.index_map(fn(pair, i) { #(i, pair.0, pair.1) }) + |> list.filter(fn(t) { t.1 != t.2 }) + |> list.map(fn(t) { + " differ: line " + <> int.to_string(t.0 + 1) + <> "\n .ssp: " + <> t.1 + <> "\n .ocda.db: " + <> t.2 + }) +} + +fn split_lines(text: String) -> List(String) { + case text { + "" -> [] + _ -> string.split(text, on: "\n") + } +} + +fn fail(message: String) -> Nil { + io.println_error("error: " <> message) + halt(2) +} + +/// Gleam has no standard exit, and the exit status is what a sweep script +/// reads. +@external(erlang, "erlang", "halt") +fn halt(code: Int) -> Nil diff --git a/gleam/test/ocda_lib_test.gleam b/gleam/test/ocda_lib_test.gleam new file mode 100644 index 0000000..02fe463 --- /dev/null +++ b/gleam/test/ocda_lib_test.gleam @@ -0,0 +1,148 @@ +import gleam/dict +import gleam/list +import simplifile +import gleam/string +import gleeunit +import gleeunit/should +import ocda/dump +import ocda/load +import ocda/load_ssp +import ocda/model + +pub fn main() { + gleeunit.main() +} + +const minimal = "% SiSU Document Abstraction v2.0 +% Source: example.sst + +@source { + language: en + languages: en + digest: ABC +} + +@meta { + title.main: An Example +} + +@body { +[1] heading :A the_title +.part: frontmatter +.sha256: DEAD +| An Example + +[2] para +.part: body +.parent: 1 +| First line +| +| Third line, after a blank one + +} +" + +pub fn version_test() { + let doc = load_ssp.parse(minimal) + doc.format_version |> should.equal("2.0") + model.version_major(doc) |> should.equal("2") + doc.source_filename |> should.equal("example.sst") +} + +pub fn preamble_test() { + let doc = load_ssp.parse(minimal) + model.meta(doc, "title.main") |> should.equal("An Example") +} + +pub fn objects_test() { + let doc = load_ssp.parse(minimal) + let body = model.section(doc, "body") + body |> list.length |> should.equal(2) +} + +/// The blank line in the middle of an object's text is kept, and so is the +/// count: three lines, not two. +pub fn blank_text_line_test() { + let doc = load_ssp.parse(minimal) + case model.section(doc, "body") { + [_, second] -> + second.text + |> should.equal("First line\n\nThird line, after a blank one") + _ -> should.fail() + } +} + +/// A heading's identifier comes from its declaration line; an object whose +/// identifier is simply its ocn has it derived, because the text form does +/// not write what it can work out. +pub fn derived_identifier_test() { + let doc = load_ssp.parse(minimal) + case model.section(doc, "body") { + [first, second] -> { + first.identifier |> should.equal("the_title") + second.identifier |> should.equal("2") + } + _ -> should.fail() + } +} + +pub fn source_of_test() { + load.source_of("a/b.ssp") |> should.equal(load.Ssp) + load.source_of("a/b.ocda.db") |> should.equal(load.OcdaDb) + load.source_of("a/b.sst") |> should.equal(load.Unknown) +} + +/// The dump omits a field at its default, so a minimal document gives a +/// short dump and every line in it is a fact. +pub fn dump_test() { + let text = dump.document(load_ssp.parse(minimal)) + text |> string.contains(contain: "version 2.0") |> should.be_true + text |> string.contains(contain: "meta title.main An Example") |> should.be_true + text |> string.contains(contain: "object body 0") |> should.be_true + text |> string.contains(contain: "level A") |> should.be_true + // a false boolean is not written at all + text |> string.contains(contain: "bullet") |> should.be_false +} + + +/// A property is read, which the fixture above had to be corrected to show: +/// a property line is `.name: value` at the start of the line, and an +/// indented one is a line of no known shape. The fixture indented them, so +/// every property in it was being counted as malformed and the tests around +/// it passed for the wrong reason. +pub fn property_test() { + let doc = load_ssp.parse(minimal) + case model.section(doc, "body") { + [first, second] -> { + first.part |> should.equal("frontmatter") + first.sha256 |> should.equal("DEAD") + second.parent_ocn |> should.equal(1) + } + _ -> should.fail() + } + // and nothing in the fixture is malformed any more + dict_size(doc.unknown.line) |> should.equal(0) +} + +@external(erlang, "maps", "size") +fn dict_size(d: dict.Dict(a, b)) -> Int + +/// What a path is, including the two archive forms. +pub fn pod_source_test() { + load.source_of("a/b.sisupod") |> should.equal(load.Pod) + load.source_of("a/b.zip") |> should.equal(load.Pod) +} + +/// A zstd-wrapped archive is refused, and the refusal says why rather than +/// claiming the file is malformed. This is the one place the D and Gleam +/// libraries differ in what they accept, and it is a packaging limit: the +/// only zstd on the BEAM is a NIF that fetches zstd in its own build. +pub fn zstd_refusal_test() { + let path = "test/fixture-zstd-frame.sisupod" + let assert Ok(_) = simplifile.write_bits(path, <<0x28, 0xB5, 0x2F, 0xFD, 0>>) + let assert Error(message) = load.load(path, "", False) + message |> string.contains(contain: "zstd") |> should.be_true + message |> string.contains(contain: "zstd -d") |> should.be_true + let _ = simplifile.delete(path) + Nil +} diff --git a/org/COPYRIGHT b/org/COPYRIGHT new file mode 100644 index 0000000..23b22f5 --- /dev/null +++ b/org/COPYRIGHT @@ -0,0 +1,19 @@ +- Name: sisudoc-ocda-lib + - Description: part of SiSU spine (Doc Reform): documents, structuring, + processing, publishing and search + - [https://www.sisudoc.org] + + - Author: Ralph Amissah + [ralph.amissah@gmail.com] + + - Copyright: (C) 2026 Ralph Amissah, All Rights Reserved. + + - Spine, Doc Reform (related to SiSU) uses standard: + - docReform markup syntax (based on SiSU markup) + - standard SiSU markup syntax with modified headers and minor + modifications + - docReform object numbering (based on SiSU object citation numbering) + - standard SiSU document object numbering + + - Homepages: + [https://www.sisudoc.org] diff --git a/org/config-lib.org b/org/config-lib.org new file mode 100644 index 0000000..7b7ba18 --- /dev/null +++ b/org/config-lib.org @@ -0,0 +1,279 @@ +-*- mode: org -*- +#+TITLE: configuration: .gitignore and copyright +#+DESCRIPTION: setup only; the code, READMEs and build files are kept as files +#+FILETAGS: :spine:build:tools: +#+AUTHOR: Ralph Amissah +#+EMAIL: [[mailto:ralph.amissah@gmail.com][ralph.amissah@gmail.com]] +#+COPYRIGHT: Copyright (C) 2026 Ralph Amissah +#+LANGUAGE: en +#+STARTUP: show3levels hideblocks hidestars noindent entitiespretty +#+PROPERTY: header-args+ :eval never-export :exports code +#+PROPERTY: header-args+ :noweb yes :padline no +#+PROPERTY: header-args+ :results silent :cache no +#+PROPERTY: header-args+ :mkdirp yes +#+OPTIONS: H:3 num:nil toc:t \n:t ::t |:t ^:nil -:t f:t *:t + +* ./ +** git .gitignore :gitignore: + +- git ls-files --others --exclude-from=.git/info/exclude +- git ls-files --others --ignored --exclude-standard --directory +- git check-ignore -v flake.lock +- git clean -ix +- find . -name "*_" | xargs rm -rf +- find . -name "*.o" | xargs rm -rf + +#+HEADER: :tangle "../.gitignore" +#+BEGIN_SRC gitignore +# Deny everything, then whitelist what belongs in the repository. +# +# The practice is deliberate: a repository should hold what was decided to +# put in it, not whatever happened to be in the directory when someone ran +# "git add .". Build output, editor droppings, downloaded dependencies and +# exported binaries are numerous, are regenerated, and differ between +# machines; source is few and is chosen. So the default is no, and every +# yes below is a decision that can be read. +# +# To see what is being excluded, and why: +# git status --ignored +# git check-ignore -v <path> +# git ls-files --others --ignored --exclude-standard --directory + +* + +# ── the repository itself ──────────────────────────────────────────────── +!.gitignore +!README.md +!LICENSE +!**/LICENSE + +# ── directories have to be walked into before a file inside can match ──── +!gleam/ +!dlang/ +!org/ +!gleam/**/ +!dlang/**/ +!org/**/ +# but not these, whatever they contain +build/ +.build/ +.direnv/ +.gleam/ +bin/ +result +result-* + +# ── documentation and notes, in any language directory ─────────────────── +!**/COPYRIGHT +!**/*.md +!**/*.org + +# ── nix ────────────────────────────────────────────────────────────────── +!**/*.nix +!**/flake.lock +!**/.envrc + +# ── dlang ──────────────────────────────────────────────────────────────── +!**/*.d +!**/dub.json +!**/dub.sdl +!**/dub.selections.json +*.o +*.a +*.lst + +# ── gleam ──────────────────────────────────────────────────────────────── +!**/*.gleam +# Erlang, for what Gleam reaches through FFI (a sha256, from OTP crypto) +!**/src/*.erl +# gleam.toml is the manifest, manifest.toml is the lockfile: both belong +!**/*.toml +*.beam +*.ez +erl_crash.dump + +# ── ruby, when it arrives ──────────────────────────────────────────────── +!**/*.rb +!**/Gemfile +!**/Gemfile.lock +!**/*.gemspec + +# ── continuous integration ─────────────────────────────────────────────── +!**/*.yml +!**/*.yaml +#+END_SRC + +* COPYRIGHT & LICENSE +** notices +*** project (project root) ./ + +#+HEADER: :tangle "../COPYRIGHT" +#+HEADER: :noweb yes +#+BEGIN_SRC text +- Name: <<project_name>> + <<sisudoc_spine_copyright>> + + <<sisudoc_spine_license_agpl3>> + + <<sisudoc_spine_summary>> +#+END_SRC + +#+HEADER: :tangle "../LICENSE" +#+HEADER: :noweb yes +#+BEGIN_SRC text +<<sisudoc_spine_license_agpl3_the_license>> +#+END_SRC + +*** code org ./org + +- .org literate programming: + - copyright asserted + - omitted + - license + - dependency information + +#+HEADER: :tangle "../org/COPYRIGHT" +#+HEADER: :noweb yes +#+BEGIN_SRC text +- Name: <<project_name>> + <<sisudoc_spine_copyright>> + + <<sisudoc_spine_summary>> +#+END_SRC + + +#+HEADER: :noweb yes +#+BEGIN_SRC text +- Name: <<project_name>> + <<sisudoc_spine_copyright>> + + <<sisudoc_spine_license_agpl3>> + + <<sisudoc_spine_summary>> +#+END_SRC + +*** code source +**** gleam + +#+HEADER: :tangle "../gleam/COPYRIGHT" +#+HEADER: :noweb yes +#+BEGIN_SRC org +- Name: <<project_name>> + <<sisudoc_spine_copyright>> + + <<sisudoc_spine_license_agpl3>> + + <<sisudoc_spine_summary>> + + - Copyright for ./nix-gleam/ 2023 Arnar Gauti Ingason + - License for ./nix-gleam/ MIT License +#+END_SRC + +***** gleam src + +#+HEADER: :tangle "../gleam/src/COPYRIGHT" +#+HEADER: :noweb yes +#+BEGIN_SRC org +- Name: <<project_name>> + <<sisudoc_spine_copyright>> + + <<sisudoc_spine_license_agpl3>> + + <<sisudoc_spine_summary>> +#+END_SRC + +**** dlang +***** dlang + +#+HEADER: :tangle "../dlang/COPYRIGHT" +#+HEADER: :noweb yes +#+BEGIN_SRC org +- Name: <<project_name>> + <<sisudoc_spine_copyright>> + + <<sisudoc_spine_license_agpl3>> + + <<sisudoc_spine_summary>> +#+END_SRC + +***** dlang src + +#+HEADER: :tangle "../dlang/src/COPYRIGHT" +#+HEADER: :noweb yes +#+BEGIN_SRC org +- Name: <<project_name>> + <<sisudoc_spine_copyright>> + + <<sisudoc_spine_license_agpl3>> + + <<sisudoc_spine_summary>> +#+END_SRC + +* org includes +** project name + +#+NAME: project_name +#+HEADER: :noweb yes +#+BEGIN_SRC emacs-lisp +<<./sisudoc_spine_version_info_and_doc_header_including_copyright_and_license.org:project_name()>> +#+END_SRC + +** project version + +#+NAME: spine_version +#+HEADER: :noweb yes +#+BEGIN_SRC emacs-lisp +<<./sisudoc_spine_version_info_and_doc_header_including_copyright_and_license.org:spine_project_version()>> +#+END_SRC + +** ocda ssp format version + +#+NAME: ocda_ssp_format_version +#+HEADER: :noweb yes +#+BEGIN_SRC emacs-lisp +<<./sisudoc_spine_version_info_and_doc_header_including_copyright_and_license.org:ocda_ssp_format_version()>> +#+END_SRC + +** copyright + +#+NAME: sisudoc_spine_copyright +#+BEGIN_SRC emacs-lisp +<<./sisudoc_spine_version_info_and_doc_header_including_copyright_and_license.org:sisudoc_spine_copyright()>> +#+END_SRC + +** license agpl3 + +#+NAME: sisudoc_spine_license_agpl3 +#+BEGIN_SRC emacs-lisp +<<./sisudoc_spine_version_info_and_doc_header_including_copyright_and_license.org:sisudoc_spine_license_agpl3()>> +#+END_SRC + +#+NAME: sisudoc_spine_license_agpl3_the_license +#+BEGIN_SRC emacs-lisp +<<./sisudoc_spine_version_info_and_doc_header_including_copyright_and_license.org:sisudoc_spine_license_agpl3_the_license()>> +#+END_SRC + +** spine_summary + +#+NAME: sisudoc_spine_summary +#+BEGIN_SRC emacs-lisp +<<./sisudoc_spine_version_info_and_doc_header_including_copyright_and_license.org:sisudoc_spine_summary()>> +#+END_SRC + +** year + +#+NAME: year +#+HEADER: :noweb yes +#+BEGIN_SRC emacs-lisp +<<./sisudoc_spine_version_info_and_doc_header_including_copyright_and_license.org:year()>> +#+END_SRC + +* nixpkgs source + +#+NAME: nixpkgs_unstable +#+HEADER: :noweb yes +#+BEGIN_SRC emacs-lisp +<<./sisudoc_spine_version_info_and_doc_header_including_copyright_and_license.org:nixpkgs_unstable()>> +#+END_SRC + +* __END__ diff --git a/org/sisudoc_spine_version_info_and_doc_header_including_copyright_and_license.org b/org/sisudoc_spine_version_info_and_doc_header_including_copyright_and_license.org new file mode 100644 index 0000000..7b6fd66 --- /dev/null +++ b/org/sisudoc_spine_version_info_and_doc_header_including_copyright_and_license.org @@ -0,0 +1,972 @@ +-*- mode: org -*- +#+TITLE: sisudoc spine (doc_reform) object-centric document abstraction +#+DESCRIPTION: documents - structuring, publishing in multiple formats & search +#+FILETAGS: :spine:abstraction: +#+AUTHOR: Ralph Amissah +#+EMAIL: [[mailto:ralph.amissah@gmail.com][ralph.amissah@gmail.com]] +#+COPYRIGHT: Copyright (C) 2026 Ralph Amissah +#+LANGUAGE: en +#+STARTUP: content hideblocks hidestars noindent entitiespretty +#+PROPERTY: header-args+ :eval never-export :exports code +#+PROPERTY: header-args+ :noweb yes :padline no +#+PROPERTY: header-args+ :results silent :cache no +#+PROPERTY: header-args+ :mkdirp yes +#+OPTIONS: H:3 num:nil toc:t \n:t ::t |:t ^:nil -:t f:t *:t + +- [[./doc-reform.org][doc-reform.org]] [[./][org/]] + +* spine doc header including copyright & license + +#+NAME: spine_doc_header_including_copyright_and_license +#+HEADER: :noweb yes +#+BEGIN_SRC org +/+ +- Name: <<project_name>> + - Description: <<project_description>> + - static content generator + + - Author: Ralph Amissah + [ralph.amissah@gmail.com] + + - Copyright: (C) <<project_year_started>> Ralph Amissah, All Rights Reserved. + + - License: AGPL 3 or later, for everything in this repository unless a file or + a directory specifies otherwise.: + + Spine (SiSU), a framework for document structuring, processing, publishing + and search + + Copyright (C) Ralph Amissah + + This program is free software: you can redistribute it and/or modify it + under the terms of the GNU Affero General Public License (AGPLv3) as + published by the Free Software Foundation (FSF), either version 3 of the + License, or (at your option) any later version. + + This program is distributed in the hope that it will be useful, but WITHOUT + ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License + for more details. + + You should have received a copy of the GNU Affero General Public License + along with this program. If not, see [https://www.gnu.org/licenses/]. + + If you have Internet connection, the latest version of the AGPL should be + available at: [https://www.gnu.org/licenses/agpl-3.0.html] + + - Spine (by Doc Reform, related to SiSU) uses standard: + - docReform markup syntax + - standard SiSU markup syntax with modified headers and minor modifications + - docReform object numbering + - standard SiSU object citation numbering & system + + - Homepages: + [https://www.sisudoc.org] + [https://www.doc-reform.org] + + - Git + [https://git.sisudoc.org/] + ++/ +#+END_SRC + +** incorporate +*** copyright + +#+NAME: sisudoc_spine_copyright +#+BEGIN_SRC org +- Description: part of SiSU spine (Doc Reform): documents, structuring, + processing, publishing and search + - [https://www.sisudoc.org] + +- Author: Ralph Amissah + [ralph.amissah@gmail.com] + +- Copyright: (C) <<project_year_started>> Ralph Amissah, All Rights Reserved. +#+END_SRC + +*** license +**** AGPLv3 notice + +#+NAME: sisudoc_spine_license_agpl3 +#+HEADER: :noweb yes +#+BEGIN_SRC org +- License: AGPL 3 or later, for everything in this repository unless a file or + a directory specifies otherwise.: + + <<project_name>> + + <<project_description>> + + Copyright (C) Ralph Amissah + + This program is free software: you can redistribute it and/or modify it + under the terms of the GNU Affero General Public License (AGPLv3) as + published by the Free Software Foundation, either version 3 of the License, + or (at your option) any later version. + + This program is distributed in the hope that it will be useful, but + WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY + or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public + License for more details. + + You should have received a copy of the GNU Affero General Public License + along with this program. If not, see [https://www.gnu.org/licenses/]. + + If you have Internet connection, the latest version of the AGPL should be + available at: [https://www.gnu.org/licenses/agpl-3.0.html] +#+END_SRC + +**** the AGPLv3 License + +#+NAME: sisudoc_spine_license_agpl3_the_license +#+BEGIN_SRC org +# GNU AFFERO GENERAL PUBLIC LICENSE + +Version 3, 19 November 2007 + +Copyright (C) 2007 Free Software Foundation, Inc. +<https://fsf.org/> + +Everyone is permitted to copy and distribute verbatim copies of this +license document, but changing it is not allowed. + +## Preamble + +The GNU Affero General Public License is a free, copyleft license for +software and other kinds of works, specifically designed to ensure +cooperation with the community in the case of network server software. + +The licenses for most software and other practical works are designed +to take away your freedom to share and change the works. By contrast, +our General Public Licenses are intended to guarantee your freedom to +share and change all versions of a program--to make sure it remains +free software for all its users. + +When we speak of free software, we are referring to freedom, not +price. Our General Public Licenses are designed to make sure that you +have the freedom to distribute copies of free software (and charge for +them if you wish), that you receive source code or can get it if you +want it, that you can change the software or use pieces of it in new +free programs, and that you know you can do these things. + +Developers that use our General Public Licenses protect your rights +with two steps: (1) assert copyright on the software, and (2) offer +you this License which gives you legal permission to copy, distribute +and/or modify the software. + +A secondary benefit of defending all users' freedom is that +improvements made in alternate versions of the program, if they +receive widespread use, become available for other developers to +incorporate. Many developers of free software are heartened and +encouraged by the resulting cooperation. However, in the case of +software used on network servers, this result may fail to come about. +The GNU General Public License permits making a modified version and +letting the public access it on a server without ever releasing its +source code to the public. + +The GNU Affero General Public License is designed specifically to +ensure that, in such cases, the modified source code becomes available +to the community. It requires the operator of a network server to +provide the source code of the modified version running there to the +users of that server. Therefore, public use of a modified version, on +a publicly accessible server, gives the public access to the source +code of the modified version. + +An older license, called the Affero General Public License and +published by Affero, was designed to accomplish similar goals. This is +a different license, not a version of the Affero GPL, but Affero has +released a new version of the Affero GPL which permits relicensing +under this license. + +The precise terms and conditions for copying, distribution and +modification follow. + +## TERMS AND CONDITIONS + +### 0. Definitions. + +"This License" refers to version 3 of the GNU Affero General Public +License. + +"Copyright" also means copyright-like laws that apply to other kinds +of works, such as semiconductor masks. + +"The Program" refers to any copyrightable work licensed under this +License. Each licensee is addressed as "you". "Licensees" and +"recipients" may be individuals or organizations. + +To "modify" a work means to copy from or adapt all or part of the work +in a fashion requiring copyright permission, other than the making of +an exact copy. The resulting work is called a "modified version" of +the earlier work or a work "based on" the earlier work. + +A "covered work" means either the unmodified Program or a work based +on the Program. + +To "propagate" a work means to do anything with it that, without +permission, would make you directly or secondarily liable for +infringement under applicable copyright law, except executing it on a +computer or modifying a private copy. Propagation includes copying, +distribution (with or without modification), making available to the +public, and in some countries other activities as well. + +To "convey" a work means any kind of propagation that enables other +parties to make or receive copies. Mere interaction with a user +through a computer network, with no transfer of a copy, is not +conveying. + +An interactive user interface displays "Appropriate Legal Notices" to +the extent that it includes a convenient and prominently visible +feature that (1) displays an appropriate copyright notice, and (2) +tells the user that there is no warranty for the work (except to the +extent that warranties are provided), that licensees may convey the +work under this License, and how to view a copy of this License. If +the interface presents a list of user commands or options, such as a +menu, a prominent item in the list meets this criterion. + +### 1. Source Code. + +The "source code" for a work means the preferred form of the work for +making modifications to it. "Object code" means any non-source form of +a work. + +A "Standard Interface" means an interface that either is an official +standard defined by a recognized standards body, or, in the case of +interfaces specified for a particular programming language, one that +is widely used among developers working in that language. + +The "System Libraries" of an executable work include anything, other +than the work as a whole, that (a) is included in the normal form of +packaging a Major Component, but which is not part of that Major +Component, and (b) serves only to enable use of the work with that +Major Component, or to implement a Standard Interface for which an +implementation is available to the public in source code form. A +"Major Component", in this context, means a major essential component +(kernel, window system, and so on) of the specific operating system +(if any) on which the executable work runs, or a compiler used to +produce the work, or an object code interpreter used to run it. + +The "Corresponding Source" for a work in object code form means all +the source code needed to generate, install, and (for an executable +work) run the object code and to modify the work, including scripts to +control those activities. However, it does not include the work's +System Libraries, or general-purpose tools or generally available free +programs which are used unmodified in performing those activities but +which are not part of the work. For example, Corresponding Source +includes interface definition files associated with source files for +the work, and the source code for shared libraries and dynamically +linked subprograms that the work is specifically designed to require, +such as by intimate data communication or control flow between those +subprograms and other parts of the work. + +The Corresponding Source need not include anything that users can +regenerate automatically from other parts of the Corresponding Source. + +The Corresponding Source for a work in source code form is that same +work. + +### 2. Basic Permissions. + +All rights granted under this License are granted for the term of +copyright on the Program, and are irrevocable provided the stated +conditions are met. This License explicitly affirms your unlimited +permission to run the unmodified Program. The output from running a +covered work is covered by this License only if the output, given its +content, constitutes a covered work. This License acknowledges your +rights of fair use or other equivalent, as provided by copyright law. + +You may make, run and propagate covered works that you do not convey, +without conditions so long as your license otherwise remains in force. +You may convey covered works to others for the sole purpose of having +them make modifications exclusively for you, or provide you with +facilities for running those works, provided that you comply with the +terms of this License in conveying all material for which you do not +control copyright. Those thus making or running the covered works for +you must do so exclusively on your behalf, under your direction and +control, on terms that prohibit them from making any copies of your +copyrighted material outside their relationship with you. + +Conveying under any other circumstances is permitted solely under the +conditions stated below. Sublicensing is not allowed; section 10 makes +it unnecessary. + +### 3. Protecting Users' Legal Rights From Anti-Circumvention Law. + +No covered work shall be deemed part of an effective technological +measure under any applicable law fulfilling obligations under article +11 of the WIPO copyright treaty adopted on 20 December 1996, or +similar laws prohibiting or restricting circumvention of such +measures. + +When you convey a covered work, you waive any legal power to forbid +circumvention of technological measures to the extent such +circumvention is effected by exercising rights under this License with +respect to the covered work, and you disclaim any intention to limit +operation or modification of the work as a means of enforcing, against +the work's users, your or third parties' legal rights to forbid +circumvention of technological measures. + +### 4. Conveying Verbatim Copies. + +You may convey verbatim copies of the Program's source code as you +receive it, in any medium, provided that you conspicuously and +appropriately publish on each copy an appropriate copyright notice; +keep intact all notices stating that this License and any +non-permissive terms added in accord with section 7 apply to the code; +keep intact all notices of the absence of any warranty; and give all +recipients a copy of this License along with the Program. + +You may charge any price or no price for each copy that you convey, +and you may offer support or warranty protection for a fee. + +### 5. Conveying Modified Source Versions. + +You may convey a work based on the Program, or the modifications to +produce it from the Program, in the form of source code under the +terms of section 4, provided that you also meet all of these +conditions: + +- a) The work must carry prominent notices stating that you modified + it, and giving a relevant date. +- b) The work must carry prominent notices stating that it is + released under this License and any conditions added under + section 7. This requirement modifies the requirement in section 4 + to "keep intact all notices". +- c) You must license the entire work, as a whole, under this + License to anyone who comes into possession of a copy. This + License will therefore apply, along with any applicable section 7 + additional terms, to the whole of the work, and all its parts, + regardless of how they are packaged. This License gives no + permission to license the work in any other way, but it does not + invalidate such permission if you have separately received it. +- d) If the work has interactive user interfaces, each must display + Appropriate Legal Notices; however, if the Program has interactive + interfaces that do not display Appropriate Legal Notices, your + work need not make them do so. + +A compilation of a covered work with other separate and independent +works, which are not by their nature extensions of the covered work, +and which are not combined with it such as to form a larger program, +in or on a volume of a storage or distribution medium, is called an +"aggregate" if the compilation and its resulting copyright are not +used to limit the access or legal rights of the compilation's users +beyond what the individual works permit. Inclusion of a covered work +in an aggregate does not cause this License to apply to the other +parts of the aggregate. + +### 6. Conveying Non-Source Forms. + +You may convey a covered work in object code form under the terms of +sections 4 and 5, provided that you also convey the machine-readable +Corresponding Source under the terms of this License, in one of these +ways: + +- a) Convey the object code in, or embodied in, a physical product + (including a physical distribution medium), accompanied by the + Corresponding Source fixed on a durable physical medium + customarily used for software interchange. +- b) Convey the object code in, or embodied in, a physical product + (including a physical distribution medium), accompanied by a + written offer, valid for at least three years and valid for as + long as you offer spare parts or customer support for that product + model, to give anyone who possesses the object code either (1) a + copy of the Corresponding Source for all the software in the + product that is covered by this License, on a durable physical + medium customarily used for software interchange, for a price no + more than your reasonable cost of physically performing this + conveying of source, or (2) access to copy the Corresponding + Source from a network server at no charge. +- c) Convey individual copies of the object code with a copy of the + written offer to provide the Corresponding Source. This + alternative is allowed only occasionally and noncommercially, and + only if you received the object code with such an offer, in accord + with subsection 6b. +- d) Convey the object code by offering access from a designated + place (gratis or for a charge), and offer equivalent access to the + Corresponding Source in the same way through the same place at no + further charge. You need not require recipients to copy the + Corresponding Source along with the object code. If the place to + copy the object code is a network server, the Corresponding Source + may be on a different server (operated by you or a third party) + that supports equivalent copying facilities, provided you maintain + clear directions next to the object code saying where to find the + Corresponding Source. Regardless of what server hosts the + Corresponding Source, you remain obligated to ensure that it is + available for as long as needed to satisfy these requirements. +- e) Convey the object code using peer-to-peer transmission, + provided you inform other peers where the object code and + Corresponding Source of the work are being offered to the general + public at no charge under subsection 6d. + +A separable portion of the object code, whose source code is excluded +from the Corresponding Source as a System Library, need not be +included in conveying the object code work. + +A "User Product" is either (1) a "consumer product", which means any +tangible personal property which is normally used for personal, +family, or household purposes, or (2) anything designed or sold for +incorporation into a dwelling. In determining whether a product is a +consumer product, doubtful cases shall be resolved in favor of +coverage. For a particular product received by a particular user, +"normally used" refers to a typical or common use of that class of +product, regardless of the status of the particular user or of the way +in which the particular user actually uses, or expects or is expected +to use, the product. A product is a consumer product regardless of +whether the product has substantial commercial, industrial or +non-consumer uses, unless such uses represent the only significant +mode of use of the product. + +"Installation Information" for a User Product means any methods, +procedures, authorization keys, or other information required to +install and execute modified versions of a covered work in that User +Product from a modified version of its Corresponding Source. The +information must suffice to ensure that the continued functioning of +the modified object code is in no case prevented or interfered with +solely because modification has been made. + +If you convey an object code work under this section in, or with, or +specifically for use in, a User Product, and the conveying occurs as +part of a transaction in which the right of possession and use of the +User Product is transferred to the recipient in perpetuity or for a +fixed term (regardless of how the transaction is characterized), the +Corresponding Source conveyed under this section must be accompanied +by the Installation Information. But this requirement does not apply +if neither you nor any third party retains the ability to install +modified object code on the User Product (for example, the work has +been installed in ROM). + +The requirement to provide Installation Information does not include a +requirement to continue to provide support service, warranty, or +updates for a work that has been modified or installed by the +recipient, or for the User Product in which it has been modified or +installed. Access to a network may be denied when the modification +itself materially and adversely affects the operation of the network +or violates the rules and protocols for communication across the +network. + +Corresponding Source conveyed, and Installation Information provided, +in accord with this section must be in a format that is publicly +documented (and with an implementation available to the public in +source code form), and must require no special password or key for +unpacking, reading or copying. + +### 7. Additional Terms. + +"Additional permissions" are terms that supplement the terms of this +License by making exceptions from one or more of its conditions. +Additional permissions that are applicable to the entire Program shall +be treated as though they were included in this License, to the extent +that they are valid under applicable law. If additional permissions +apply only to part of the Program, that part may be used separately +under those permissions, but the entire Program remains governed by +this License without regard to the additional permissions. + +When you convey a copy of a covered work, you may at your option +remove any additional permissions from that copy, or from any part of +it. (Additional permissions may be written to require their own +removal in certain cases when you modify the work.) You may place +additional permissions on material, added by you to a covered work, +for which you have or can give appropriate copyright permission. + +Notwithstanding any other provision of this License, for material you +add to a covered work, you may (if authorized by the copyright holders +of that material) supplement the terms of this License with terms: + +- a) Disclaiming warranty or limiting liability differently from the + terms of sections 15 and 16 of this License; or +- b) Requiring preservation of specified reasonable legal notices or + author attributions in that material or in the Appropriate Legal + Notices displayed by works containing it; or +- c) Prohibiting misrepresentation of the origin of that material, + or requiring that modified versions of such material be marked in + reasonable ways as different from the original version; or +- d) Limiting the use for publicity purposes of names of licensors + or authors of the material; or +- e) Declining to grant rights under trademark law for use of some + trade names, trademarks, or service marks; or +- f) Requiring indemnification of licensors and authors of that + material by anyone who conveys the material (or modified versions + of it) with contractual assumptions of liability to the recipient, + for any liability that these contractual assumptions directly + impose on those licensors and authors. + +All other non-permissive additional terms are considered "further +restrictions" within the meaning of section 10. If the Program as you +received it, or any part of it, contains a notice stating that it is +governed by this License along with a term that is a further +restriction, you may remove that term. If a license document contains +a further restriction but permits relicensing or conveying under this +License, you may add to a covered work material governed by the terms +of that license document, provided that the further restriction does +not survive such relicensing or conveying. + +If you add terms to a covered work in accord with this section, you +must place, in the relevant source files, a statement of the +additional terms that apply to those files, or a notice indicating +where to find the applicable terms. + +Additional terms, permissive or non-permissive, may be stated in the +form of a separately written license, or stated as exceptions; the +above requirements apply either way. + +### 8. Termination. + +You may not propagate or modify a covered work except as expressly +provided under this License. Any attempt otherwise to propagate or +modify it is void, and will automatically terminate your rights under +this License (including any patent licenses granted under the third +paragraph of section 11). + +However, if you cease all violation of this License, then your license +from a particular copyright holder is reinstated (a) provisionally, +unless and until the copyright holder explicitly and finally +terminates your license, and (b) permanently, if the copyright holder +fails to notify you of the violation by some reasonable means prior to +60 days after the cessation. + +Moreover, your license from a particular copyright holder is +reinstated permanently if the copyright holder notifies you of the +violation by some reasonable means, this is the first time you have +received notice of violation of this License (for any work) from that +copyright holder, and you cure the violation prior to 30 days after +your receipt of the notice. + +Termination of your rights under this section does not terminate the +licenses of parties who have received copies or rights from you under +this License. If your rights have been terminated and not permanently +reinstated, you do not qualify to receive new licenses for the same +material under section 10. + +### 9. Acceptance Not Required for Having Copies. + +You are not required to accept this License in order to receive or run +a copy of the Program. Ancillary propagation of a covered work +occurring solely as a consequence of using peer-to-peer transmission +to receive a copy likewise does not require acceptance. However, +nothing other than this License grants you permission to propagate or +modify any covered work. These actions infringe copyright if you do +not accept this License. Therefore, by modifying or propagating a +covered work, you indicate your acceptance of this License to do so. + +### 10. Automatic Licensing of Downstream Recipients. + +Each time you convey a covered work, the recipient automatically +receives a license from the original licensors, to run, modify and +propagate that work, subject to this License. You are not responsible +for enforcing compliance by third parties with this License. + +An "entity transaction" is a transaction transferring control of an +organization, or substantially all assets of one, or subdividing an +organization, or merging organizations. If propagation of a covered +work results from an entity transaction, each party to that +transaction who receives a copy of the work also receives whatever +licenses to the work the party's predecessor in interest had or could +give under the previous paragraph, plus a right to possession of the +Corresponding Source of the work from the predecessor in interest, if +the predecessor has it or can get it with reasonable efforts. + +You may not impose any further restrictions on the exercise of the +rights granted or affirmed under this License. For example, you may +not impose a license fee, royalty, or other charge for exercise of +rights granted under this License, and you may not initiate litigation +(including a cross-claim or counterclaim in a lawsuit) alleging that +any patent claim is infringed by making, using, selling, offering for +sale, or importing the Program or any portion of it. + +### 11. Patents. + +A "contributor" is a copyright holder who authorizes use under this +License of the Program or a work on which the Program is based. The +work thus licensed is called the contributor's "contributor version". + +A contributor's "essential patent claims" are all patent claims owned +or controlled by the contributor, whether already acquired or +hereafter acquired, that would be infringed by some manner, permitted +by this License, of making, using, or selling its contributor version, +but do not include claims that would be infringed only as a +consequence of further modification of the contributor version. For +purposes of this definition, "control" includes the right to grant +patent sublicenses in a manner consistent with the requirements of +this License. + +Each contributor grants you a non-exclusive, worldwide, royalty-free +patent license under the contributor's essential patent claims, to +make, use, sell, offer for sale, import and otherwise run, modify and +propagate the contents of its contributor version. + +In the following three paragraphs, a "patent license" is any express +agreement or commitment, however denominated, not to enforce a patent +(such as an express permission to practice a patent or covenant not to +sue for patent infringement). To "grant" such a patent license to a +party means to make such an agreement or commitment not to enforce a +patent against the party. + +If you convey a covered work, knowingly relying on a patent license, +and the Corresponding Source of the work is not available for anyone +to copy, free of charge and under the terms of this License, through a +publicly available network server or other readily accessible means, +then you must either (1) cause the Corresponding Source to be so +available, or (2) arrange to deprive yourself of the benefit of the +patent license for this particular work, or (3) arrange, in a manner +consistent with the requirements of this License, to extend the patent +license to downstream recipients. "Knowingly relying" means you have +actual knowledge that, but for the patent license, your conveying the +covered work in a country, or your recipient's use of the covered work +in a country, would infringe one or more identifiable patents in that +country that you have reason to believe are valid. + +If, pursuant to or in connection with a single transaction or +arrangement, you convey, or propagate by procuring conveyance of, a +covered work, and grant a patent license to some of the parties +receiving the covered work authorizing them to use, propagate, modify +or convey a specific copy of the covered work, then the patent license +you grant is automatically extended to all recipients of the covered +work and works based on it. + +A patent license is "discriminatory" if it does not include within the +scope of its coverage, prohibits the exercise of, or is conditioned on +the non-exercise of one or more of the rights that are specifically +granted under this License. You may not convey a covered work if you +are a party to an arrangement with a third party that is in the +business of distributing software, under which you make payment to the +third party based on the extent of your activity of conveying the +work, and under which the third party grants, to any of the parties +who would receive the covered work from you, a discriminatory patent +license (a) in connection with copies of the covered work conveyed by +you (or copies made from those copies), or (b) primarily for and in +connection with specific products or compilations that contain the +covered work, unless you entered into that arrangement, or that patent +license was granted, prior to 28 March 2007. + +Nothing in this License shall be construed as excluding or limiting +any implied license or other defenses to infringement that may +otherwise be available to you under applicable patent law. + +### 12. No Surrender of Others' Freedom. + +If conditions are imposed on you (whether by court order, agreement or +otherwise) that contradict the conditions of this License, they do not +excuse you from the conditions of this License. If you cannot convey a +covered work so as to satisfy simultaneously your obligations under +this License and any other pertinent obligations, then as a +consequence you may not convey it at all. For example, if you agree to +terms that obligate you to collect a royalty for further conveying +from those to whom you convey the Program, the only way you could +satisfy both those terms and this License would be to refrain entirely +from conveying the Program. + +### 13. Remote Network Interaction; Use with the GNU General Public License. + +Notwithstanding any other provision of this License, if you modify the +Program, your modified version must prominently offer all users +interacting with it remotely through a computer network (if your +version supports such interaction) an opportunity to receive the +Corresponding Source of your version by providing access to the +Corresponding Source from a network server at no charge, through some +standard or customary means of facilitating copying of software. This +Corresponding Source shall include the Corresponding Source for any +work covered by version 3 of the GNU General Public License that is +incorporated pursuant to the following paragraph. + +Notwithstanding any other provision of this License, you have +permission to link or combine any covered work with a work licensed +under version 3 of the GNU General Public License into a single +combined work, and to convey the resulting work. The terms of this +License will continue to apply to the part which is the covered work, +but the work with which it is combined will remain governed by version +3 of the GNU General Public License. + +### 14. Revised Versions of this License. + +The Free Software Foundation may publish revised and/or new versions +of the GNU Affero General Public License from time to time. Such new +versions will be similar in spirit to the present version, but may +differ in detail to address new problems or concerns. + +Each version is given a distinguishing version number. If the Program +specifies that a certain numbered version of the GNU Affero General +Public License "or any later version" applies to it, you have the +option of following the terms and conditions either of that numbered +version or of any later version published by the Free Software +Foundation. If the Program does not specify a version number of the +GNU Affero General Public License, you may choose any version ever +published by the Free Software Foundation. + +If the Program specifies that a proxy can decide which future versions +of the GNU Affero General Public License can be used, that proxy's +public statement of acceptance of a version permanently authorizes you +to choose that version for the Program. + +Later license versions may give you additional or different +permissions. However, no additional obligations are imposed on any +author or copyright holder as a result of your choosing to follow a +later version. + +### 15. Disclaimer of Warranty. + +THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY +APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT +HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT +WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND +PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE +DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR +CORRECTION. + +### 16. Limitation of Liability. + +IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING +WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR +CONVEYS THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, +INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES +ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT +NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR +LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM +TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER +PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. + +### 17. Interpretation of Sections 15 and 16. + +If the disclaimer of warranty and limitation of liability provided +above cannot be given local legal effect according to their terms, +reviewing courts shall apply local law that most closely approximates +an absolute waiver of all civil liability in connection with the +Program, unless a warranty or assumption of liability accompanies a +copy of the Program in return for a fee. + +END OF TERMS AND CONDITIONS + +## How to Apply These Terms to Your New Programs + +If you develop a new program, and you want it to be of the greatest +possible use to the public, the best way to achieve this is to make it +free software which everyone can redistribute and change under these +terms. + +To do so, attach the following notices to the program. It is safest to +attach them to the start of each source file to most effectively state +the exclusion of warranty; and each file should have at least the +"copyright" line and a pointer to where the full notice is found. + + <one line to give the program's name and a brief idea of what it does.> + Copyright (C) <year> <name of author> + + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU Affero General Public License as + published by the Free Software Foundation, either version 3 of the + License, or (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU Affero General Public License for more details. + + You should have received a copy of the GNU Affero General Public License + along with this program. If not, see <https://www.gnu.org/licenses/>. + +Also add information on how to contact you by electronic and paper +mail. + +If your software can interact with users remotely through a computer +network, you should also make sure that it provides a way for users to +get its source. For example, if your program is a web application, its +interface could display a "Source" link that leads users to an archive +of the code. There are many ways you could offer source, and different +solutions will be better for different programs; see section 13 for +the specific requirements. + +You should also get your employer (if you work as a programmer) or +school, if any, to sign a "copyright disclaimer" for the program, if +necessary. For more information on this, and how to apply and follow +the GNU AGPL, see <https://www.gnu.org/licenses/>. +#+END_SRC + +*** spine_summary + +#+NAME: sisudoc_spine_summary +#+BEGIN_SRC org +- Spine, Doc Reform (related to SiSU) uses standard: + - docReform markup syntax (based on SiSU markup) + - standard SiSU markup syntax with modified headers and minor + modifications + - docReform object numbering (based on SiSU object citation numbering) + - standard SiSU document object numbering + +- Homepages: + [https://www.sisudoc.org] +#+END_SRC + +#+BEGIN_SRC org + - Homepages: + [https://www.sisudoc.org] + + - Git + [] +#+END_SRC + +*** markup_samples + +#+NAME: sisudoc_spine_markup_samples +#+BEGIN_SRC org +- Spine, Doc Reform (SiSU) markup samples + Individual document content Copyright (Author) [as stated in document header] + Individual document content License (Author) [as stated in document header] +#+END_SRC + +* project variables +** project name + +#+NAME: project_name +#+BEGIN_SRC org +sisudoc-ocda-lib +#+END_SRC + +#+BEGIN_SRC org +SisuDoc Spine, Doc Reform [a part of] +#+END_SRC + +** project description + +#+NAME: project_description +#+BEGIN_SRC org +part of SiSU spine (Doc Reform): documents, structuring, +processing, publishing and search +#+END_SRC + +#+BEGIN_SRC org +Spine, Doc Reform (SiSU), a framework for document structuring, publishing and search +#+END_SRC + +#+BEGIN_SRC org +documents, structuring, processing, publishing, search +#+END_SRC + +** project start year + +#+NAME: project_year_started +#+BEGIN_SRC org +2026 +#+END_SRC + +** project paths + +#+NAME: project_code_paths +#+BEGIN_SRC org +dlang/src/* gleam/src/* +#+END_SRC + +#+BEGIN_SRC org +src/* src/sisudoc/* +#+END_SRC + +* spine version info SET VERSION +** version in full + +#+NAME: spine_project_version +#+BEGIN_SRC org +<<spine_project_version_part_major>>.<<spine_project_version_part_minor>>.<<spine_project_version_part_patch>> +#+END_SRC + +** version parts (major, minor, patch) + +#+NAME: spine_project_version_major_minor_patch +#+BEGIN_SRC org +<<spine_project_version_part_major>>, <<spine_project_version_part_minor>>, <<spine_project_version_part_patch>> +#+END_SRC + +*** major + +#+NAME: spine_project_version_part_major +#+BEGIN_SRC org +0 +#+END_SRC + +*** minor + +#+NAME: spine_project_version_part_minor +#+BEGIN_SRC org +25 +#+END_SRC + +*** patch + +#+NAME: spine_project_version_part_patch +#+BEGIN_SRC org +2 +#+END_SRC + +* year + +#+NAME: year +#+BEGIN_SRC org +2026 +#+END_SRC + +* sisudoc spine git source SET +** sisudoc-spine src + +#+NAME: sisudoc_spine_nix_src +#+BEGIN_SRC org +git://git.sisudoc.org/software/sisudoc-spine +#+END_SRC + +#+BEGIN_SRC org +git+file:///home/ralph/grotto/repo/git.repo/projects/doc-reform/code/software/sisudoc-spine +#+END_SRC + +** dlang overlay src + +#+NAME: sisudoc_dlang_nix_overlay_src +#+BEGIN_SRC org +git://git.sisudoc.org/package/dlang-nix-overlays-and-flake +#+END_SRC + +#+BEGIN_SRC org +git+file:///home/ralph/grotto/repo/git.repo/projects/doc-reform/packages/nix-flakes/dlang/dlang-nix-flakes +#+END_SRC + +* ocda ssp version format SET +** ocda ssp version in full + +#+NAME: ocda_ssp_format_version +#+BEGIN_SRC org +<<ocda_ssp_format_version_part_major>>.<<ocda_ssp_format_version_part_minor>> +#+END_SRC + +** parts +*** major + +#+NAME: ocda_ssp_format_version_part_major +#+BEGIN_SRC org +2 +#+END_SRC + +*** minor + +#+NAME: ocda_ssp_format_version_part_minor +#+BEGIN_SRC org +1 +#+END_SRC + +* sqlite collection search db schema version + +#+NAME: sqlite_collection_search_db_schema_version +#+BEGIN_SRC org +1 +#+END_SRC + +* nipkgs source + +- github:NixOS/nixpkgs/nixpkgs-unstable +- https://channels.nixos.org/nixpkgs-unstable/nixexprs.tar.zst + +#+NAME: nixpkgs_unstable +#+BEGIN_SRC org +https://channels.nixos.org/nixpkgs-unstable/nixexprs.tar.zst +#+END_SRC + +#+BEGIN_SRC org +github:NixOS/nixpkgs/nixpkgs-unstable +#+END_SRC |
