diff options
| -rw-r--r-- | README.md | 86 | ||||
| -rw-r--r-- | claude-org/config-lib.org | 274 | ||||
| -rw-r--r-- | claude-org/sisudoc_spine_version_info_and_doc_header_including_copyright_and_license.org | 260 | ||||
| -rw-r--r-- | dlang/dub.json | 2 | ||||
| -rw-r--r-- | dlang/flake.nix | 2 | ||||
| -rw-r--r-- | gleam/README.md | 23 | ||||
| -rw-r--r-- | gleam/flake.lock | 17 | ||||
| -rw-r--r-- | gleam/gleam.toml | 2 | ||||
| -rw-r--r-- | org/d-lib.org | 3900 | ||||
| -rw-r--r-- | org/gleam-lib.org | 3662 |
10 files changed, 609 insertions, 7619 deletions
@@ -1,9 +1,9 @@ # sisudoc-ocda-lib The **object-centric document abstraction as a library**: the model, and -loaders for both of the forms it is written in. +loaders for each of the forms it is written in. -**One reader per language, in a directory named for the language.** +**One implementation per language, in a directory named for the language.** | | | |--------------------|---------| @@ -11,10 +11,13 @@ loaders for both of the forms it is written in. | [`gleam/`](gleam/) | Gleam | | `ruby/` | to come | -## What it is for (NO LONGER RELEASED, NOW USED DIRECTLY BY WRITER) FIX +## What it is for -The readers in `sisudoc-ssp-reader` (NO LONGER RELEASED, NOW USED DIRECTLY BY WRITER) FIX and `sisudoc-ocda-db-reader` (NO LONGER RELEASED, NOW USED DIRECTLY BY WRITER) FIX each read -one serialisation and print. A **writer** needs neither: it needs the document. +The first programs written from the specification were two readers, +`sisudoc-ssp-reader` and `sisudoc-ocda-db-reader`, each reading one +serialisation and printing. They are published frozen, as the record of +readers written from the specification alone, and their work is now done by +`ocda-read`, below. A **writer** needs neither: it needs the document. ``` .ssp ─┐ @@ -37,14 +40,13 @@ ocda-read [--tree|--html|--llm-text|--llm-jsonl|--images|--verify --image-dir=<d <file>.ssp | <file>.ocda.db | <file>.sisupod [lang] ``` -One reader, in each language, where `sisudoc-ssp-reader` (NO LONGER RELEASED, NOW USED DIRECTLY BY WRITER) FIX and -`sisudoc-ocda-db-reader` (NO LONGER RELEASED, NOW USED DIRECTLY BY WRITER) FIX were two, each with a parser of its own. Built on -this library, **every mode reads every form and says the same thing of a -document's `.ssp`, `.ocda.db` and `.sisupod`**, 36 of 36 document-languages. -The D reader has every mode of both old readers; the Gleam one the modes the -Gleam readers had (the summary, `--tree`, `--languages`, `--files`, -`--report-unknown`), and in those **its output is the D one's, byte for -byte**, 36 of 36. +One reader, in each language, where the two earlier readers had a parser +each. Built on this library, **every mode reads every form and says the same +thing of a document's `.ssp`, `.ocda.db` and `.sisupod`**, for every +document-language of the markup document samples. The D reader has every +mode of both earlier readers; the Gleam one the modes the Gleam readers had +(the summary, `--tree`, `--languages`, `--files`, `--report-unknown`), and in +those **its output is the D one's, byte for byte**. It is in each language's `src/read/`, and not part of the library: a program using the library does not build it (in D, by dub configuration; @@ -59,6 +61,11 @@ writes the same property, and the report is the same from either form. ## The checks that matter +The checks are run over the markup document samples +([sisudoc-spine-samples](https://git.sisudoc.org/sisudoc-spine-samples)), +built by spine with `--show-abstraction --ocda-db --pod2`, which writes each +document's `.ssp`, its `.ocda.db` and its pod. + The claim above is only true if the loaders agree, so the library asks. Two programs, in each language: @@ -71,14 +78,14 @@ ocda-dump <file>.ssp|<file>.ocda.db [lang] # the whole model, canonically of the two**, and the reason it exists: it writes the entire loaded model out in a fully determined form, and the D and the Gleam library write it *byte for byte alike*. So four questions can be asked, all of them with -`diff`: +`diff`, and each holds for every document-language: -| | | -|-----------------------------|---------------------------------| -| D: `.ssp` == `.ocda.db` | **36 of 36** document-languages | -| Gleam: `.ssp` == `.ocda.db` | **36 of 36** | -| `.ssp`: D == Gleam | **36 of 36** | -| `.ocda.db`: D == Gleam | **36 of 36** | +| | | +|-----------------------------|---------------| +| D: `.ssp` == `.ocda.db` | **every one** | +| Gleam: `.ssp` == `.ocda.db` | **every one** | +| `.ssp`: D == Gleam | **every one** | +| `.ocda.db`: D == Gleam | **every one** | None of those held at first. @@ -89,15 +96,15 @@ None of those held at first. An object whose **first text line is blank** lost it. The loader decided whether to start a new line by asking whether the accumulated text was empty, which is not the same question as whether a line has been seen. Of -the sample collection **seven documents have verse opening on a blank -line**, and in every one of them the blank was dropped. Eight of -thirty-six documents disagreed. +the markup document samples **seven documents have verse opening on a blank +line**, and in every one of them the blank was dropped. Nothing else had caught it. The two `.ssp` readers had been compared on *section counts*, which this does not change. It took loading the same document two ways and comparing the objects. -Fixed here and in `sisudoc-ssp-reader/dlang`, which carried the same code. (NO LONGER RELEASED, NOW USED DIRECTLY BY WRITER) FIX +Fixed here and in the `.ssp` reader, `sisudoc-ssp-reader`, which carried the +same code. ### the whole-model dump: four more @@ -117,8 +124,8 @@ the two loaders: serialisation it was handed. 3. **The JSON columns were scanned rather than parsed.** Phobos writes `/` as `\/`, so every heading containing a slash came back with a backslash - in it: **all 36 document-languages** differed on - `heading_ancestors_text` and nothing else had shown it. + in it: **every document-language** differed on `heading_ancestors_text` + and nothing else had shown it. 4. **`translation.ocn_aligned`** is in the database and cannot be in a `.ssp`, which is one language and cannot know whether the languages share a numbering. It is a fact about the *file*, so the loaders leave @@ -144,8 +151,8 @@ puts it in, `--source` does not. A pod without one is refused with a message saying which flag makes a pod this library can read, because that is a fact about the pod rather than about the library. -**Over the reference collection, a `.sisupod` gives the same model as the -`.ssp` beside it: 36 of 36 document-languages, byte identical dumps.** A +**Over the markup document samples, a `.sisupod` gives the same model as the +`.ssp` beside it, for every document-language: byte identical dumps.** A writer over the library therefore gained a third input form without being touched, which is the point of the dispatch being in one place. @@ -165,11 +172,24 @@ the only place the two libraries differ in what they accept. `ocda/load` is the one place in each language that decides what a path is, so a new input form is added once and every reader and writer gains it. +## Building and use + +`nix build` in `dlang/` or `gleam/`. See [`dlang/README.md`](dlang/README.md) +and [`gleam/README.md`](gleam/README.md) for the modules, the programs and +using the library from another program. + ## The format -Specified in **sisudoc-abstraction-format**: `ocda-model.md` for the model -this library implements, and the two format specifications for the loaders. -Section 4 of `ocda-db-format-specification.md` gives the column by column -correspondence the `load_db` of each language follows. +Specified in +[**sisudoc-abstraction-format**](https://git.sisudoc.org/sisudoc-abstraction-format): +`ocda-model.md` for the model this library implements, and the two format +specifications for the loaders. Section 4 of +`ocda-db-format-specification.md` gives the column by column correspondence +the `load_db` of each language follows. + +At format **2.1**; the library's version follows the format's major and +minor parts. + +## Licence -At format **2.x**. +AGPL-3.0-or-later. See [`COPYRIGHT`](COPYRIGHT). diff --git a/claude-org/config-lib.org b/claude-org/config-lib.org new file mode 100644 index 0000000..3e9a31e --- /dev/null +++ b/claude-org/config-lib.org @@ -0,0 +1,274 @@ +-*- mode: org -*- +#+TITLE: configuration: .gitignore and copyright +#+DESCRIPTION: setup only; the code, READMEs and build files are kept as files +#+FILETAGS: :spine:build:tools: +#+AUTHOR: Ralph Amissah +#+EMAIL: [[mailto:ralph.amissah@gmail.com][ralph.amissah@gmail.com]] +#+COPYRIGHT: Copyright (C) 2015 (continuously updated, current 2026) Ralph Amissah +#+LANGUAGE: en +#+STARTUP: show3levels hideblocks hidestars noindent entitiespretty +#+PROPERTY: header-args+ :eval never-export :exports code +#+PROPERTY: header-args+ :noweb yes :padline no +#+PROPERTY: header-args+ :results silent :cache no +#+PROPERTY: header-args+ :mkdirp yes +#+OPTIONS: H:3 num:nil toc:t \n:t ::t |:t ^:nil -:t f:t *:t + +* ./ +** git .gitignore :gitignore: + +#+HEADER: :tangle "../.gitignore" +#+BEGIN_SRC gitignore +# git ls-files --others --exclude-from=.git/info/exclude +# git ls-files --others --ignored --exclude-standard --directory +# git check-ignore -v flake.lock +# git clean -ix +# find . -name "*_" | xargs rm -rf +# find . -name "*.o" | xargs rm -rf + +# Deny everything, then whitelist what belongs in the repository. +# +# The practice is deliberate: a repository should hold what was decided to +# put in it, not whatever happened to be in the directory when someone ran +# "git add .". Build output, editor droppings, downloaded dependencies and +# exported binaries are numerous, are regenerated, and differ between +# machines; source is few and is chosen. So the default is no, and every +# yes below is a decision that can be read. +# +# To see what is being excluded, and why: +# git status --ignored +# git check-ignore -v <path> +# git ls-files --others --ignored --exclude-standard --directory + +* + +# ── the repository itself ──────────────────────────────────────────────── +!.gitignore +!README.md +!LICENSE +!**/LICENSE + +# ── directories have to be walked into before a file inside can match ──── +!*/ +# but not these, whatever they contain +build/ +.build/ +.direnv/ +.gleam/ +bin/ +result +result-* + +# ── documentation and notes, in any language directory ─────────────────── +!**/COPYRIGHT +!**/*.md +!**/*.org + +# ── nix ────────────────────────────────────────────────────────────────── +!**/*.nix +!**/flake.lock +!**/.envrc + +# ── dlang ──────────────────────────────────────────────────────────────── +!**/*.d +!**/dub.json +!**/dub.sdl +!**/dub.selections.json +*.o +*.a +*.lst + +# ── gleam ──────────────────────────────────────────────────────────────── +!**/*.gleam +# Erlang, for what Gleam reaches through FFI (a sha256, from OTP crypto) +!**/src/*.erl +# gleam.toml is the manifest, manifest.toml is the lockfile: both belong +!**/*.toml +*.beam +*.ez +erl_crash.dump + +# ── ruby, when it arrives ──────────────────────────────────────────────── +!**/*.rb +!**/Gemfile +!**/Gemfile.lock +!**/*.gemspec + +# ── continuous integration ─────────────────────────────────────────────── +!**/*.yml +!**/*.yaml +#+END_SRC + +* COPYRIGHT & LICENSE +** notices +*** project (project root) ./ + +#+HEADER: :tangle "../COPYRIGHT" +#+HEADER: :noweb yes +#+BEGIN_SRC text +- Name: spine - SiSU Spine, Doc Reform + <<sisudoc_spine_copyright>> + + <<sisudoc_spine_license_agpl3>> + + <<sisudoc_spine_summary>> + +<<sisudoc_spine_markup_samples>> +#+END_SRC + +*** code org ./org + +- .org literate programming: + - copyright asserted + - omitted + - license + - dependency information + +#+HEADER: :tangle "../org/COPYRIGHT" +#+HEADER: :noweb yes +#+BEGIN_SRC text +- Name: spine - SiSU Spine, Doc Reform + <<sisudoc_spine_copyright>> + + <<sisudoc_spine_summary>> + +<<sisudoc_spine_markup_samples>> +#+END_SRC + + +#+HEADER: :noweb yes +- Name: spine - SiSU Spine, Doc Reform + <<sisudoc_spine_copyright>> + + <<sisudoc_spine_license_agpl3>> + + <<sisudoc_spine_summary>> + +<<sisudoc_spine_markup_samples>> +#+END_SRC + +*** code source +**** gleam +**** gleam + +#+HEADER: :tangle "../gleam/COPYRIGHT" +#+HEADER: :noweb yes +#+BEGIN_SRC org +- Name: spine - SiSU Spine, Doc Reform + <<sisudoc_spine_copyright>> + + <<sisudoc_spine_license_agpl3>> + + <<sisudoc_spine_summary>> + +<<sisudoc_spine_markup_samples>> +#+END_SRC + +***** gleam src + +#+HEADER: :tangle "../gleam/src/COPYRIGHT" +#+HEADER: :noweb yes +#+BEGIN_SRC org +- Name: spine - SiSU Spine, Doc Reform + <<sisudoc_spine_copyright>> + + <<sisudoc_spine_license_agpl3>> + + <<sisudoc_spine_summary>> + +<<sisudoc_spine_markup_samples>> +#+END_SRC + +**** dlang +***** dlang + +#+HEADER: :tangle "../dlang/COPYRIGHT" +#+HEADER: :noweb yes +#+BEGIN_SRC org +- Name: spine - SiSU Spine, Doc Reform + <<sisudoc_spine_copyright>> + + <<sisudoc_spine_license_agpl3>> + + <<sisudoc_spine_summary>> + +<<sisudoc_spine_markup_samples>> +#+END_SRC + +***** dlang src + +#+HEADER: :tangle "../dlang/src/COPYRIGHT" +#+HEADER: :noweb yes +#+BEGIN_SRC org +- Name: spine - SiSU Spine, Doc Reform + <<sisudoc_spine_copyright>> + + <<sisudoc_spine_license_agpl3>> + + <<sisudoc_spine_summary>> + +<<sisudoc_spine_markup_samples>> +#+END_SRC + +** incorporate +*** copyright + +#+NAME: sisudoc_spine_copyright +#+BEGIN_SRC emacs-lisp +<<./sisudoc_spine_version_info_and_doc_header_including_copyright_and_license.org:sisudoc_spine_copyright()>> +#+END_SRC + +*** agpl3 + +#+NAME: sisudoc_spine_license_agpl3 +#+BEGIN_SRC emacs-lisp +<<./sisudoc_spine_version_info_and_doc_header_including_copyright_and_license.org:sisudoc_spine_license_agpl3()>> +#+END_SRC + +*** spine_summary + +#+NAME: sisudoc_spine_summary +#+BEGIN_SRC emacs-lisp +<<./sisudoc_spine_version_info_and_doc_header_including_copyright_and_license.org:sisudoc_spine_summary()>> +#+END_SRC + +*** markup_samples + +#+NAME: sisudoc_spine_markup_samples +#+BEGIN_SRC emacs-lisp +<<./sisudoc_spine_version_info_and_doc_header_including_copyright_and_license.org:sisudoc_spine_markup_samples()>> +#+END_SRC + +* org includes +** project version + +#+NAME: spine_version +#+HEADER: :noweb yes +#+BEGIN_SRC emacs-lisp +<<./sisudoc_spine_version_info_and_doc_header_including_copyright_and_license.org:spine_project_version()>> +#+END_SRC + +** year + +#+NAME: year +#+HEADER: :noweb yes +#+BEGIN_SRC emacs-lisp +<<./sisudoc_spine_version_info_and_doc_header_including_copyright_and_license.org:year()>> +#+END_SRC + +** document header including copyright & license + +#+NAME: doc_header_including_copyright_and_license +#+HEADER: :noweb yes +#+BEGIN_SRC emacs-lisp +<<./sisudoc_spine_version_info_and_doc_header_including_copyright_and_license.org:spine_doc_header_including_copyright_and_license()>> +#+END_SRC + +* nixpkgs source + +#+NAME: nixpkgs_unstable +#+HEADER: :noweb yes +#+BEGIN_SRC emacs-lisp +<<./sisudoc_spine_version_info_and_doc_header_including_copyright_and_license.org:nixpkgs_unstable()>> +#+END_SRC + +* __END__ diff --git a/claude-org/sisudoc_spine_version_info_and_doc_header_including_copyright_and_license.org b/claude-org/sisudoc_spine_version_info_and_doc_header_including_copyright_and_license.org new file mode 100644 index 0000000..a474009 --- /dev/null +++ b/claude-org/sisudoc_spine_version_info_and_doc_header_including_copyright_and_license.org @@ -0,0 +1,260 @@ +-*- mode: org -*- +#+TITLE: sisudoc spine (doc_reform) object-centric document abstraction +#+DESCRIPTION: documents - structuring, publishing in multiple formats & search +#+FILETAGS: :spine:abstraction: +#+AUTHOR: Ralph Amissah +#+EMAIL: [[mailto:ralph.amissah@gmail.com][ralph.amissah@gmail.com]] +#+COPYRIGHT: Copyright (C) 2015 (continuously updated, current 2026) Ralph Amissah +#+LANGUAGE: en +#+STARTUP: content hideblocks hidestars noindent entitiespretty +#+PROPERTY: header-args+ :eval never-export :exports code +#+PROPERTY: header-args+ :noweb yes :padline no +#+PROPERTY: header-args+ :results silent :cache no +#+PROPERTY: header-args+ :mkdirp yes +#+OPTIONS: H:3 num:nil toc:t \n:t ::t |:t ^:nil -:t f:t *:t + +- [[./doc-reform.org][doc-reform.org]] [[./][org/]] + +* spine doc header including copyright & license + +#+NAME: spine_doc_header_including_copyright_and_license +#+BEGIN_SRC org +/+ +- Name: SisuDoc Spine, Doc Reform [a part of] + - Description: documents, structuring, processing, publishing, search + - static content generator + + - Author: Ralph Amissah + [ralph.amissah@gmail.com] + + - Copyright: (C) 2015 (continuously updated, current 2026) Ralph Amissah, All Rights Reserved. + + - License: AGPL 3 or later: + + Spine (SiSU), a framework for document structuring, publishing and + search + + Copyright (C) Ralph Amissah + + This program is free software: you can redistribute it and/or modify it + under the terms of the GNU AFERO General Public License as published by the + Free Software Foundation, either version 3 of the License, or (at your + option) any later version. + + This program is distributed in the hope that it will be useful, but WITHOUT + ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + more details. + + You should have received a copy of the GNU General Public License along with + this program. If not, see [https://www.gnu.org/licenses/]. + + If you have Internet connection, the latest version of the AGPL should be + available at these locations: + [https://www.fsf.org/licensing/licenses/agpl.html] + [https://www.gnu.org/licenses/agpl.html] + + - Spine (by Doc Reform, related to SiSU) uses standard: + - docReform markup syntax + - standard SiSU markup syntax with modified headers and minor modifications + - docReform object numbering + - standard SiSU object citation numbering & system + + - Homepages: + [https://www.sisudoc.org] + [https://www.doc-reform.org] + + - Git + [https://git.sisudoc.org/] + ++/ +#+END_SRC + +** incorporate +*** copyright + +#+NAME: sisudoc_spine_copyright +#+BEGIN_SRC org +- Description: documents, structuring, processing, publishing, search + - static content generator + +- Author: Ralph Amissah + [ralph.amissah@gmail.com] + +- Copyright: (C) 2015 - 2026 Ralph Amissah +#+END_SRC + +*** license +**** AGPLv3 + +#+NAME: sisudoc_spine_license_agpl3 +#+BEGIN_SRC org +- code under src/* src/sisudoc/* + - License: AGPL 3 or later: + + Spine, Doc Reform (SiSU), a framework for document structuring, publishing and + search + + Copyright (C) Ralph Amissah + + This program is free software: you can redistribute it and/or modify it + under the terms of the GNU AFERO General Public License as published by + the Free Software Foundation, either version 3 of the License, or (at your + option) any later version. + + This program is distributed in the hope that it will be useful, but + WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY + or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License + for more details. + + You should have received a copy of the GNU General Public License along + with this program. If not, see [https://www.gnu.org/licenses/]. + + If you have Internet connection, the latest version of the AGPL should be + available at these locations: + [https://www.fsf.org/licensing/licenses/agpl.html] + [https://www.gnu.org/licenses/agpl.html] +#+END_SRC + +*** spine_summary + +#+NAME: sisudoc_spine_summary +#+BEGIN_SRC org +- Spine, Doc Reform (related to SiSU) uses standard: + - docReform markup syntax (based on SiSU markup) + - standard SiSU markup syntax with modified headers and minor + modifications + - docReform object numbering (based on SiSU object citation numbering) + - standard SiSU document object numbering + +- Homepages: + [https://www.sisudoc.org] +#+END_SRC + +#+BEGIN_SRC org + - Homepages: + [https://www.sisudoc.org] + [https://www.sisudoc.org] + + - Git + [] +#+END_SRC + +*** markup_samples + +#+NAME: sisudoc_spine_markup_samples +#+BEGIN_SRC org +- Spine, Doc Reform (SiSU) markup samples + Individual document content Copyright (Author) [as stated in document header] + Individual document content License (Author) [as stated in document header] +#+END_SRC + +* spine version info SET VERSION +** version in full + +#+NAME: spine_project_version +#+BEGIN_SRC org +<<spine_project_version_part_major>>.<<spine_project_version_part_minor>>.<<spine_project_version_part_patch>> +#+END_SRC + +** version parts (major, minor, patch) + +#+NAME: spine_project_version_major_minor_patch +#+BEGIN_SRC org +<<spine_project_version_part_major>>, <<spine_project_version_part_minor>>, <<spine_project_version_part_patch>> +#+END_SRC + +*** major + +#+NAME: spine_project_version_part_major +#+BEGIN_SRC org +0 +#+END_SRC + +*** minor + +#+NAME: spine_project_version_part_minor +#+BEGIN_SRC org +25 +#+END_SRC + +*** patch + +#+NAME: spine_project_version_part_patch +#+BEGIN_SRC org +2 +#+END_SRC + +* year + +#+NAME: year +#+BEGIN_SRC org +2026 +#+END_SRC + +* sisudoc spine git source SET +** sisudoc-spine src + +#+NAME: sisudoc_spine_nix_src +#+BEGIN_SRC org +git://git.sisudoc.org/software/sisudoc-spine +#+END_SRC + +#+BEGIN_SRC org +git+file:///home/ralph/grotto/repo/git.repo/projects/doc-reform/code/software/sisudoc-spine +#+END_SRC + +** dlang overlay src + +#+NAME: sisudoc_dlang_nix_overlay_src +#+BEGIN_SRC org +git://git.sisudoc.org/package/dlang-nix-overlays-and-flake +#+END_SRC + +#+BEGIN_SRC org +git+file:///home/ralph/grotto/repo/git.repo/projects/doc-reform/packages/nix-flakes/dlang/dlang-nix-flakes" +#+END_SRC + +* ocda ssp version format SET +** ocda ssp version in full + +#+NAME: ocda_ssp_format_version +#+BEGIN_SRC org +<<ocda_ssp_format_version_part_major>>.<<ocda_ssp_format_version_part_minor>> +#+END_SRC + +** parts +*** major + +#+NAME: ocda_ssp_format_version_part_major +#+BEGIN_SRC org +2 +#+END_SRC + +*** minor + +#+NAME: ocda_ssp_format_version_part_minor +#+BEGIN_SRC org +1 +#+END_SRC + +* sqlite collection search db schema version + +#+NAME: sqlite_collection_search_db_schema_version +#+BEGIN_SRC org +1 +#+END_SRC + +* nipkgs source + +- github:NixOS/nixpkgs/nixpkgs-unstable +- https://channels.nixos.org/nixpkgs-unstable/nixexprs.tar.zst + +#+NAME: nixpkgs_unstable +#+BEGIN_SRC org +https://channels.nixos.org/nixpkgs-unstable/nixexprs.tar.zst +#+END_SRC + +#+BEGIN_SRC org +github:NixOS/nixpkgs/nixpkgs-unstable +#+END_SRC diff --git a/dlang/dub.json b/dlang/dub.json index 80be675..6b77a17 100644 --- a/dlang/dub.json +++ b/dlang/dub.json @@ -2,7 +2,7 @@ "authors": [ "Ralph Amissah" ], "copyright": "Copyright (C) 2026 Ralph Amissah", "name": "ocda-lib", - "version": "0.1.0", + "version": "2.1.0", "description": "the object-centric document abstraction as a library: the model, and loaders for its two serialisations", "homepage": "https://sisudoc.org", "license": "AGPL-3.0+", diff --git a/dlang/flake.nix b/dlang/flake.nix index 94b3e37..232d738 100644 --- a/dlang/flake.nix +++ b/dlang/flake.nix @@ -20,7 +20,7 @@ mkTool = { tool, binary, what, mains, compiler, compilerBin }: pkgs.stdenv.mkDerivation { pname = "ocda-lib-${tool}"; - version = "0.1.0"; + version = "2.1.0"; src = self; nativeBuildInputs = [ compiler ]; buildInputs = [ pkgs.sqlite pkgs.zstd ]; diff --git a/gleam/README.md b/gleam/README.md index 482db38..8135005 100644 --- a/gleam/README.md +++ b/gleam/README.md @@ -60,12 +60,12 @@ the whole model out canonically, and is the stronger check of the two, because it is **byte for byte what the D library's `ocda-dump` writes**. So the four questions can all be asked with `diff`: -| | | -|-----------------------------|-----------------------------| -| D: `.ssp` == `.ocda.db` | 36 of 36 document-languages | -| Gleam: `.ssp` == `.ocda.db` | 36 of 36 | -| `.ssp`: D == Gleam | 36 of 36 | -| `.ocda.db`: D == Gleam | 36 of 36 | +| | | +|-----------------------------|-------------------------| +| D: `.ssp` == `.ocda.db` | every document-language | +| Gleam: `.ssp` == `.ocda.db` | every document-language | +| `.ssp`: D == Gleam | every document-language | +| `.ocda.db`: D == Gleam | every document-language | ## Deliberately a mirror of the D library @@ -112,11 +112,12 @@ nix develop -c gleam test Shipped as an Erlang shipment rather than an escript, because sqlite on the BEAM is a NIF and a NIF cannot be loaded from inside an escript archive. -The dependency hash in `flake.nix` pins what `manifest.toml` pins; change -one and the other must change with it. +The build is nix-gleam's, kept in `nix-gleam/` (see its README): each Hex +dependency is fetched on its own, with the checksum gleam wrote into +`manifest.toml`, so there is no hash in `flake.nix` to keep up to date. ## Format -Specified in **sisudoc-abstraction-format**: `ocda-model.org` for the model, -`ssp-format-specification.org` and `ocda-db-format-specification.org` for the -two serialisations. At format **2.x**. +Specified in **sisudoc-abstraction-format**: `ocda-model.md` for the model, +`ssp-format-specification.md` and `ocda-db-format-specification.md` for the +two serialisations. At format **2.1**. diff --git a/gleam/flake.lock b/gleam/flake.lock index 5f236ab..0ccb16f 100644 --- a/gleam/flake.lock +++ b/gleam/flake.lock @@ -2,18 +2,15 @@ "nodes": { "nixpkgs": { "locked": { - "lastModified": 1790185690, - "narHash": "sha256-xJ+X4hBtOcAFGBOe5nAMyMUeF9foJBmIOu3NjBqBycU=", - "owner": "NixOS", - "repo": "nixpkgs", - "rev": "4975466d324710c576dc11ad614684e6bd8cad8e", - "type": "github" + "lastModified": 1791487292, + "narHash": "sha256-6BIvCXtdLW4HQwsOGkgH444aB3vCbHypE9TA78ireh8=", + "rev": "8edc0c72e3a38faf5434e40d1f19431125fe4b30", + "type": "tarball", + "url": "https://releases.nixos.org/nixpkgs/nixpkgs-26.11pre1088030.8edc0c72e3a3/nixexprs.tar.zst" }, "original": { - "owner": "NixOS", - "ref": "nixos-unstable", - "repo": "nixpkgs", - "type": "github" + "type": "tarball", + "url": "https://channels.nixos.org/nixpkgs-unstable/nixexprs.tar.zst" } }, "root": { diff --git a/gleam/gleam.toml b/gleam/gleam.toml index 8e18279..66d3280 100644 --- a/gleam/gleam.toml +++ b/gleam/gleam.toml @@ -1,5 +1,5 @@ name = "ocda_lib" -version = "0.1.0" +version = "2.1.0" description = "the object-centric document abstraction as a library: the model, and loaders for its two serialisations" licences = ["AGPL-3.0-or-later"] diff --git a/org/d-lib.org b/org/d-lib.org deleted file mode 100644 index 1a56d68..0000000 --- a/org/d-lib.org +++ /dev/null @@ -1,3900 +0,0 @@ --*- mode: org -*- -#+TITLE: dlang load -#+DESCRIPTION: dlang lib -#+FILETAGS: :spine:build:tools: -#+AUTHOR: Ralph Amissah -#+EMAIL: [[mailto:ralph.amissah@gmail.com][ralph.amissah@gmail.com]] -#+COPYRIGHT: Copyright (C) 2015 (continuously updated, current 2026) Ralph Amissah -#+LANGUAGE: en -#+STARTUP: content hideblocks hidestars noindent entitiespretty -#+PROPERTY: header-args+ :eval never-export :exports code -#+PROPERTY: header-args+ :noweb yes :padline no -#+PROPERTY: header-args+ :results silent :cache no -#+PROPERTY: header-args+ :mkdirp yes -#+OPTIONS: H:3 num:nil toc:t \n:t ::t |:t ^:nil -:t f:t *:t - -* ocda -** model - -#+HEADER: :tangle "../dlang/src/ocda/model.d" -#+BEGIN_SRC d -<<doc_header_including_copyright_and_license>> -module ocda.model; - -struct OcdaImage { - string name; // image filename as referenced by the object - string sha256; // hex-encoded (uppercase) digest of the file - ulong bytes; // file size - int width; // file's own pixel width (not display width) - int height; // file's own pixel height - bool missing; // file absent or unreadable, no sha256/bytes - string[string] fields; // any further/unrecognised key:value fields -} - -struct OcdaObject { - int ocn; - string is_a; // heading, para, toc, table, code, group, block, verse, quote, endnote, glossary, bibliography, bookindex, blurb - string section; // head, toc, body, endnotes, glossary, bibliography, bookindex, blurb - string heading_level; // A, B, C, D, 1, 2, 3, 4 (for headings) - string identifier; // heading: from the declaration line; otherwise .identifier - string part; // frontmatter, body, backmatter - string is_of_type; // para, block, comment: how the text behaves - string own_section; // .section, when it differs from the block it sits in - string segment_html_is;// the html segment this object sits in - string segment_epub; // the epub segment this heading opens - string segment_epub_is;// the epub segment this object sits in - string table_aligns; // l c r per column - string[] stow_link; // links extracted from the object - string[] anchor_tags; // named targets the markup placed on this object - int parent_ocn; - int last_descendant_ocn; - int ocn_range_first;// a container (a poem): the first and last - int ocn_range_last; // ocn of what it holds; 0 otherwise (2.1) - int[] children; // direct child heading OCNs - int[] ancestors; // markedup_ancestors[8] - bool dummy_heading; - bool ocn_off; - int indent_base; - int indent_hang; - bool bullet; - string lang; - bool has_links; - bool has_notes_reg; - bool has_notes_star; - bool has_images; - string segment; - string segment_prev; - string segment_next; - string anchor; - string heading_lev_anchor; // a heading's own named target - int table_cols; - string table_widths; - bool table_header; - string code_syntax; - bool code_linenumbers; - string heading_ancestors_text; - string sha256; - OcdaImage[] images; // repeatable .image records, in order - string text; // content (lines joined with newline) - - /+ ↓ whether this object is rendered at all. - is_of_type "comment" is markup commentary: carried by the - abstraction so that nothing is lost, and not put into output. An - output that renders it leaks the author notes into the document, - which is what spine itself does not do. - +/ - bool renders() const { - return is_of_type != "comment"; - } - - /+ ↓ the section this object belongs to, which is the block it sits in - unless it says otherwise +/ - string sectionOf() const { - return (own_section.length > 0) ? own_section : section; - } -} - -/+ ↓ what the reader met and did not understand. - A reader that silently drops what it does not know cannot say whether - it understood the document, and cannot be used to check a - specification against an implementation: silence and understanding - look alike. So the parser counts what it ignored, and - --report-unknown prints it. -+/ -struct OcdaUnknown { - size_t[string] property; // .name -> times seen - size_t[string] block; // @name -> times seen - size_t[string] line; // first char -> times seen, lines of no known shape - - bool any() const { - return property.length > 0 || block.length > 0 || line.length > 0; - } -} - -struct OcdaDocument { - string format_version; // as declared on line 1, "" when absent - string source_filename; // from the "% Source:" line - string[string] source; // @source: language, languages, digest - OcdaUnknown unknown; // what was met and not understood - string[string] metadata; // key -> value - string[string] make; // key -> value - string[string] doc_has; // key -> value - OcdaObject[] head; - OcdaObject[] toc; - OcdaObject[] body; - OcdaObject[] endnotes; - OcdaObject[] glossary; - OcdaObject[] bibliography; - OcdaObject[] bookindex; - OcdaObject[] blurb; - OcdaObject[] tail; - - /+ ↓ the major part of the declared version, "" when none was declared +/ - string formatVersionMajor() const { - import std.string : indexOf; - if (format_version.length == 0) { return ""; } - auto dot = format_version.indexOf('.'); - return (dot > 0) ? format_version[0 .. dot] : format_version; - } - - string meta(string key) { - if (auto p = key in metadata) return *p; - return ""; - } - - OcdaObject[]* sectionByName(string name) { - switch (name) { - case "head": return &head; - case "toc": return &toc; - case "body": return &body; - case "endnotes": return &endnotes; - case "glossary": return &glossary; - case "bibliography": return &bibliography; - case "bookindex": return &bookindex; - case "blurb": return &blurb; - case "tail": return &tail; - default: return null; - } - } -} - -/+ ↓ the format major version this reader knows. - The .ssp and the .ocda.db carry one version between them, declared in - band: the .ssp on its first line, the database in its schema.version - row. The minor part is raised when a property is added, and a reader of - the newer format still reads the older one, so only the major part is - checked here. Raise this when the reader has been brought to a new - major, not before: the point of the gate is to refuse what has not been - looked at. -+/ -enum ssp_format_major_read = "2"; -#+END_SRC - -** load -** what to load -*** inventory - -#+HEADER: :tangle "../dlang/src/ocda/inventory.d" -#+BEGIN_SRC d -<<doc_header_including_copyright_and_license>> -module ocda.inventory; - -/+ ↓ what a source holds, beyond the one document the model is loaded from. - . - The model is one language of one document. Two questions a reader asks - are about the file instead: which languages it holds, and, for a - database, what else it carries. They are here rather than in a reader - because the answer depends on the input form, which is the library's - to know, and so that a reader asking them does not learn which form it - was handed. - . - languages() answers for every form, and answers the same: the - languages in the order the manifest gave them (which every .ssp - records, in @source languages), each with the markup file it was made - from and the sha256 of its .ssp. The database stores that digest; for a - .ssp and a pod it is taken from the bytes, which are the same bytes. - . - carried() is the database's: a .ocda.db holds the markup, images, - configuration and catalogues it was built from, a .ssp holds none of - them, and a pod's are its archive entries rather than anything the - abstraction says. So it answers for a database and says so otherwise. -+/ - -import std.algorithm : endsWith, countUntil; -import std.array : array, split; -import std.conv : to; -import std.digest : toHexString, LetterCase; -import std.digest.sha : sha256Of; -import std.file : read; -import std.string : strip; -import ocda.model; -import ocda.load; -import ocda.load_ssp; -import ocda.load_db; -import ocda.load_pod; -import ocda.sqlite; -import ocda.zip; - -struct OcdaLanguage { - string lang; - string source_filename; // the markup file the abstraction was made from - string ssp_digest; // sha256 of that language's .ssp, upper case hex -} - -struct OcdaCarried { - string role; // source, image, conf, manifest, tools - string compression; // how it is stored: none, zstd - string count; - string bytes; // the files' own size, whatever the storage - string stored; // what they take in the database -} - -OcdaLanguage[] languages(string path) { - final switch (sourceOf(path)) { - case Source.ocda_db: - auto db = Db.open(path); - scope(exit) { db.close(); } - OcdaLanguage[] out_; - foreach (l; dbLanguages(db)) { - out_ ~= OcdaLanguage(l.lang, l.source_filename, l.ssp_digest); - } - return out_; - case Source.ssp: - auto bytes = cast(ubyte[]) read(path); - auto doc = parseSsp(cast(string) bytes.idup); - return [ OcdaLanguage(sourceValue(doc, "language"), doc.source_filename, hexOf(bytes)) ]; - case Source.pod: - auto archive = archiveBytes(path); - OcdaLanguage[string] found; - string[] order; - foreach (e; zipFindSuffix(zipEntries(archive), ".ssp")) { - auto bytes = zipRead(archive, e); - auto doc = parseSsp(cast(string) bytes.idup); - string lang = langOfSspName(e.name); - found[lang] = OcdaLanguage(lang, doc.source_filename, hexOf(bytes)); - if (order.length == 0) { - order = sourceValue(doc, "languages").split; - } - } - /+ ↓ the manifest's order, and anything it did not name after it +/ - OcdaLanguage[] out_; - foreach (l; order) { if (auto p = l in found) { out_ ~= *p; } } - foreach (l, v; found) { if (order.countUntil(l) < 0) { out_ ~= v; } } - return out_; - case Source.unknown: - throw new OcdaLoadError(path ~ ": " ~ sourceName(Source.unknown)); - } -} - -/+ ↓ what a database carries, by role and storage; empty for any other form, - which isCarrier() says in advance +/ -bool isCarrier(string path) { - return sourceOf(path) == Source.ocda_db; -} - -OcdaCarried[] carried(string path) { - if (!isCarrier(path)) { return []; } - auto db = Db.open(path); - scope(exit) { db.close(); } - if (!db.has("files")) { return []; } - OcdaCarried[] out_; - foreach (r; db.rows( - "SELECT role, compression, count(*), sum(bytes), sum(length(data)) - FROM files GROUP BY role, compression ORDER BY role")) { - out_ ~= OcdaCarried(r[0], r[1], r[2], r[3], r[4]); - } - return out_; -} - -private string sourceValue(ref OcdaDocument doc, string key) { - if (auto p = key in doc.source) { return *p; } - return ""; -} - -private string hexOf(const(ubyte)[] bytes) { - return toHexString!(LetterCase.upper)(sha256Of(bytes)).idup; -} -#+END_SRC - -*** load -**** load - -#+HEADER: :tangle "../dlang/src/ocda/load.d" -#+BEGIN_SRC d -<<doc_header_including_copyright_and_license>> -module ocda.load; - -/+ ↓ what a path is, and loading it. - . - One place that decides, so that every reader and every writer accepts - the same inputs and a new input type is added once. This mirrors - spine's own ocda/abstraction/load.d, which names the sources it can - enter the pipeline from and tells them apart by suffix. - . - Deliberately *not* here: .sst and .ssm markup. Reading markup means - implementing the parser, which is spine's work and a far larger thing - than a reader or a sample writer needs. This library reads - abstractions. -+/ - -import std.algorithm : endsWith; -import ocda.model; -import ocda.load_ssp; -import ocda.load_db; -import ocda.load_pod; - -enum Source { - ssp, // .ssp, the abstraction as text - ocda_db, // .ocda.db, the abstraction as sqlite - pod, // .sisupod or .zip, an archive carrying a .ssp - unknown, -} - -Source sourceOf(string path) { - if (path.endsWith(".ocda.db")) { return Source.ocda_db; } - if (path.endsWith(".ssp")) { return Source.ssp; } - /+ ↓ .sisupod is the archive spine writes now; .zip is what it wrote - before, and every pod already published carries that name, so both - are read. What is inside decides how it is opened, not the name. - +/ - if (path.endsWith(".sisupod")) { return Source.pod; } - if (path.endsWith(".zip")) { return Source.pod; } - if (path.endsWith(".db")) { return Source.ocda_db; } - return Source.unknown; -} - -string sourceName(Source s) { - final switch (s) { - case Source.ssp: return ".ssp (the abstraction as text)"; - case Source.ocda_db: return ".ocda.db (the abstraction as sqlite)"; - case Source.pod: return "a pod archive carrying a .ssp"; - case Source.unknown: return "not an abstraction this library reads"; - } -} - -class OcdaLoadError : Exception { - this(string msg) { super(msg); } -} - -/+ ↓ load whatever the path is into the model. - lang names which language to take from a .ocda.db, which holds - every language of its document, and from a pod, which may carry more - than one .ssp. A .ssp is one language by construction and ignores it. -+/ -OcdaDocument load(string path, string lang = "", bool any_version = false) { - final switch (sourceOf(path)) { - case Source.ssp: return parseSspFile(path); - case Source.ocda_db: return loadDbFile(path, lang, any_version); - case Source.pod: return loadPodFile(path, lang); - case Source.unknown: - throw new OcdaLoadError(path ~ ": " ~ sourceName(Source.unknown) - ~ " (expected .ssp, .ocda.db, .sisupod or .zip)"); - } -} -#+END_SRC - -**** load db - -#+HEADER: :tangle "../dlang/src/ocda/load_db.d" -#+BEGIN_SRC d -<<doc_header_including_copyright_and_license>> -module ocda.load_db; - -/+ ↓ build the model from a .ocda.db. - . - The point of this module is that it returns *the same value* - ocda.load_ssp returns from the same document's .ssp. A writer is then - a function from the model to bytes, and cannot tell which - serialisation it was given, exactly as spine's own writers cannot. - . - Written from the sisudoc-abstraction-format specification, - ocda-db-format-specification.md, whose section 4 gives the column by - column correspondence this module implements. -+/ - -import std.array : join, split; -import std.json : JSONType, parseJSON; -import std.conv : to; -import std.algorithm : startsWith; -import std.string : strip; -import ocda.model; -import ocda.sqlite; - -/+ ↓ the schema major this loader knows (specification section 2) +/ -enum schema_major_read = "2"; - -class OcdaDbError : Exception { - this(string msg) { super(msg); } -} - -/+ ↓ every language the database holds, in manifest order +/ -struct DbLanguage { - int doc_id; - string lang; - string source_filename; - string ssp_digest; -} - -DbLanguage[] dbLanguages(ref Db db) { - DbLanguage[] out_; - foreach (r; db.rows( - "SELECT id, lang, coalesce(source_filename,''), coalesce(ssp_digest,'') - FROM documents ORDER BY id")) { - out_ ~= DbLanguage(r[0].to!int, r[1], r[2], r[3]); - } - return out_; -} - -/+ ↓ load one language of a database as the model. - lang == "" takes the first language the manifest gave. -+/ -OcdaDocument loadDbFile(string path, string lang = "", bool any_version = false) { - auto db = Db.open(path); - scope(exit) { db.close(); } - - if (!db.has("documents") || !db.has("objects") || !db.has("metadata")) { - throw new OcdaDbError(path ~ ": no documents/objects/metadata tables;" - ~ " it does not look like a .ocda.db"); - } - - /+ ↓ specification sections 2 and 3.2: the rows describing the file rather - than a document in it are the only ones with doc_id null +/ - string schema_version = db.one( - "SELECT value FROM metadata WHERE doc_id IS NULL AND key = 'schema.version'"); - if (schema_version.length > 0 && !any_version) { - auto major = schema_version.split(".")[0]; - if (major != schema_major_read) { - throw new OcdaDbError(path ~ " is schema v" ~ schema_version - ~ ", this loader knows v" ~ schema_major_read ~ ".x"); - } - } - - auto languages = dbLanguages(db); - if (languages.length == 0) { - throw new OcdaDbError(path ~ ": no documents rows"); - } - DbLanguage chosen = languages[0]; - if (lang.length > 0) { - bool found = false; - foreach (l; languages) { if (l.lang == lang) { chosen = l; found = true; break; } } - if (!found) { - string have; - foreach (i, l; languages) { have ~= (i > 0 ? " " : "") ~ l.lang; } - throw new OcdaDbError(path ~ ": no language '" ~ lang ~ "'; it holds: " ~ have); - } - } - - OcdaDocument doc; - doc.format_version = schema_version; - doc.source_filename = chosen.source_filename; - - /+ ↓ @source of the .ssp corresponds to the documents row and the - source.digest metadata (specification section 5.1 of the .ssp, and - section 7 here) +/ - doc.source["language"] = chosen.lang; - string languages_joined; - foreach (i, l; languages) { languages_joined ~= (i > 0 ? " " : "") ~ l.lang; } - doc.source["languages"] = languages_joined; - auto digest = db.one("SELECT value FROM metadata WHERE doc_id = " - ~ chosen.doc_id.to!string ~ " AND key = 'source.digest'"); - if (digest.length > 0) { doc.source["digest"] = digest; } - - /+ ↓ @meta: every metadata row of this language that is not a source.* - property. source.* describes the artefact rather than the document - and lives in @source in the text form. - +/ - foreach (r; db.rows("SELECT key, value FROM metadata WHERE doc_id = " - ~ chosen.doc_id.to!string ~ " ORDER BY key")) { - string key = r[0]; - if (key.startsWith("source.")) { continue; } - /+ ↓ translation.ocn_aligned says whether the languages of this - document share one numbering. A .ocda.db can know it because it - holds every language; a .ssp is one language and cannot. So it is - a fact about the *file* rather than about the document, and putting - it in the document's metadata would let a writer tell which - serialisation it was handed, which is the one thing this library - exists to prevent. Left out until the model has a place for - collection-wide facts. - +/ - if (key.startsWith("translation.")) { continue; } - /+ ↓ the database has one metadata table where the text form has three - blocks, and tells them apart by a key prefix. Unprefixing them - here is what makes the two loaders give one model: a writer asks - doc.make["footer"] whichever serialisation it was handed, and - never doc.metadata["make.footer"]. - +/ - if (key.startsWith("make.")) { - doc.make[key["make.".length .. $]] = r[1].strip; - } else if (key.startsWith("doc_has.")) { - doc.doc_has[key["doc_has.".length .. $]] = r[1].strip; - } else { - doc.metadata[key] = r[1].strip; - } - } - - /+ ↓ 2.1 added the container range as two columns. A 2.0 database has - neither, and reads as having no range, which is what 2.0 meant. +/ - bool has_range = false; - foreach (c; db.rows("PRAGMA table_info(objects)")) { - if (c[1] == "ocn_range_first") { has_range = true; } - } - string range_cols = has_range - ? "coalesce(ocn_range_first,0), coalesce(ocn_range_last,0)" - : "0, 0"; - - /+ ↓ the objects, in document order within each section +/ - foreach (r; db.rows( - "SELECT section, seq, ocn, is_a, coalesce(is_of_part,''), - coalesce(is_of_section,''), coalesce(is_of_type,''), - coalesce(heading_level,-1), coalesce(identifier,''), - coalesce(parent_ocn,0), coalesce(last_descendant_ocn,0), - coalesce(children,''), coalesce(ancestors,''), - coalesce(dummy_heading,0), coalesce(object_number_off,0), - coalesce(indent_base,0), coalesce(indent_hang,0), - coalesce(bullet,0), coalesce(lang,''), - coalesce(has_links,0), coalesce(has_notes_reg,0), - coalesce(has_notes_star,0), coalesce(has_images,0), - coalesce(segment,''), coalesce(segment_prev,''), - coalesce(segment_next,''), coalesce(anchor,''), - coalesce(segment_html_is,''), - coalesce(table_cols,0), coalesce(table_widths,''), - coalesce(table_aligns,''), coalesce(table_header,0), - coalesce(code_linenumbers,0), - coalesce(heading_ancestors_text,''), coalesce(sha256,''), - coalesce(text,''), id, coalesce(heading_lev_anchor,''), - coalesce(segment_epub,''), coalesce(segment_epub_is,''), " ~ range_cols ~ " - FROM objects WHERE doc_id = " ~ chosen.doc_id.to!string - ~ " ORDER BY CASE section - WHEN 'head' THEN 1 WHEN 'toc' THEN 2 WHEN 'body' THEN 3 - WHEN 'endnotes' THEN 4 WHEN 'glossary' THEN 5 - WHEN 'bibliography' THEN 6 WHEN 'bookindex' THEN 7 - WHEN 'blurb' THEN 8 WHEN 'tail' THEN 9 ELSE 10 END, seq")) { - OcdaObject obj; - obj.section = r[0]; - obj.ocn = r[2].to!int; - obj.is_a = r[3]; - obj.part = r[4]; - obj.own_section = (r[5] == r[0]) ? "" : r[5]; - obj.is_of_type = r[6]; - /+ ↓ the database holds the level as a number, the text form as the - letter or digit the markup wrote. Model section 4 is the mapping. +/ - obj.heading_level = levelName(r[7].to!int); - obj.identifier = r[8]; - obj.parent_ocn = r[9].to!int; - obj.last_descendant_ocn = r[10].to!int; - obj.children = jsonInts(r[11]); - obj.ancestors = jsonInts(r[12]); - obj.dummy_heading = (r[13] != "0"); - obj.ocn_off = (r[14] != "0"); - obj.indent_base = r[15].to!int; - obj.indent_hang = r[16].to!int; - obj.bullet = (r[17] != "0"); - obj.lang = r[18]; - obj.has_links = (r[19] != "0"); - obj.has_notes_reg = (r[20] != "0"); - obj.has_notes_star = (r[21] != "0"); - obj.has_images = (r[22] != "0"); - obj.segment = r[23]; - obj.segment_prev = r[24]; - obj.segment_next = r[25]; - obj.anchor = r[26]; - obj.segment_html_is = r[27]; - obj.heading_lev_anchor = r[37]; - obj.segment_epub = r[38]; - obj.segment_epub_is = r[39]; - obj.ocn_range_first = r[40].to!int; - obj.ocn_range_last = r[41].to!int; - obj.table_cols = r[28].to!int; - obj.table_widths = jsonWords(r[29]); - obj.table_aligns = jsonWords(r[30]); - obj.table_header = (r[31] != "0"); - obj.code_linenumbers = (r[32] != "0"); - obj.heading_ancestors_text = jsonJoined(r[33]); - obj.sha256 = r[34]; - obj.text = r[35]; - - /+ ↓ the four repeatable properties of the text form are child tables - here (specification section 4) +/ - string object_id = r[36]; - foreach (i; db.rows("SELECT name, coalesce(sha256,''), coalesce(bytes,0), - coalesce(width,0), coalesce(height,0), - coalesce(missing,0) - FROM object_images WHERE object_id = " ~ object_id - ~ " ORDER BY seq")) { - OcdaImage img; - img.name = i[0]; - img.sha256 = i[1]; - img.bytes = i[2].to!ulong; - img.width = i[3].to!int; - img.height = i[4].to!int; - img.missing = (i[5] != "0"); - obj.images ~= img; - } - foreach (l; db.rows("SELECT url FROM object_links WHERE object_id = " - ~ object_id ~ " ORDER BY seq")) { - obj.stow_link ~= l[0]; - } - foreach (a; db.rows("SELECT anchor FROM object_anchors WHERE object_id = " - ~ object_id ~ " ORDER BY seq")) { - obj.anchor_tags ~= a[0]; - } - - auto section = doc.sectionByName(r[0]); - if (section !is null) { *section ~= obj; } - } - countUnread(db, chosen.doc_id, doc); - return doc; -} - -/+ ↓ what the database holds and the model does not. - . - The text loader meets every property line and counts those it does not - keep, so a reader can say what was read and not understood. This - loader asks for columns by name, and would never see the ones it did - not ask for: it would report that it had understood everything, which - is the silence the count exists to prevent. So it counts them, on the - conditions under which the .ssp writes the same property (specification - section 6.3), and the report is the same from either form: checked, - 36 of 36. -+/ -private void countUnread(ref Db db, int doc_id, ref OcdaDocument doc) { - string id = doc_id.to!string; - void count(string property, string where) { - auto n = db.one("SELECT count(*) FROM objects WHERE doc_id = " ~ id - ~ " AND " ~ where).to!size_t; - if (n > 0) { doc.unknown.property[property] = n; } - } - count("ancestors_collapsed", "coalesce(ancestors_collapsed,'') != ''"); - count("dom_status", "coalesce(dom_status,'') != ''"); - count("dom_status_collapsed", "coalesce(dom_status_collapsed,'') != ''"); - count("heading_lev_collapsed", "heading_lev_collapsed IS NOT NULL"); - count("parent_lev", "coalesce(parent_lev,0) != 0"); - count("segment_lv4_is", "coalesce(segment_lv4_is,'') != ''"); - count("attrib", "coalesce(attrib,'') != ''"); - if (db.has("object_subtoc")) { - auto n = db.one("SELECT count(*) FROM object_subtoc s JOIN objects o" - ~ " ON o.id = s.object_id WHERE o.doc_id = " ~ id).to!size_t; - if (n > 0) { doc.unknown.property["lev4_subtoc"] = n; } - } -} - -/+ ↓ model section 4: the eight levels and their indices +/ -private string levelName(int index) { - switch (index) { - case 0: return "A"; - case 1: return "B"; - case 2: return "C"; - case 3: return "D"; - case 4: return "1"; - case 5: return "2"; - case 6: return "3"; - case 7: return "4"; - default: return ""; - } -} - -/+ ↓ the JSON columns, read as JSON. - . - The fixed width arrays are JSON in the database and space separated - (or "|" separated) in the text form. This was first written as a - character scan that threw away brackets and quotes, which is wrong the - moment a value holds an escape: Phobos' JSONValue.toString escapes - "/" as "\/", so every heading containing a slash came back with a - backslash in it. 36 of 36 documents differed on - heading_ancestors_text and nothing else had shown it. - . - So: parse. The values are small, there are few of them, and a real - parser is shorter than the scan it replaces. -+/ -private string[] jsonStrings(string v) { - if (v.length == 0) { return []; } - string[] out_; - try { - auto j = parseJSON(v); - if (j.type != JSONType.array) { return []; } - foreach (e; j.array) { - switch (e.type) { - case JSONType.string: out_ ~= e.str; break; - case JSONType.integer: out_ ~= e.integer.to!string; break; - case JSONType.uinteger: out_ ~= e.uinteger.to!string; break; - case JSONType.float_: out_ ~= e.floating.to!string; break; - case JSONType.true_: out_ ~= "true"; break; - case JSONType.false_: out_ ~= "false"; break; - default: out_ ~= ""; break; - } - } - } catch (Exception) { - return []; - } - return out_; -} - -private int[] jsonInts(string v) { - int[] out_; - foreach (s; jsonStrings(v)) { - if (s.length == 0) { continue; } - try { out_ ~= s.to!int; } catch (Exception) {} - } - return out_; -} - -/+ ↓ space separated in the text form: table_widths, table_aligns +/ -private string jsonWords(string v) { - return jsonStrings(v).join(" "); -} - -/+ ↓ heading_ancestors_text is eight slots, joined with "|" in the text - form and a JSON array here. The empty slots are significant: they say - which levels the heading sits under, so the separators carry as much - as the text does. -+/ -private string jsonJoined(string v) { - auto parts = jsonStrings(v); - if (parts.length == 0) { return ""; } - return parts.join("|"); -} -#+END_SRC - -**** load pod - -#+HEADER: :tangle "../dlang/src/ocda/load_pod.d" -#+BEGIN_SRC d -<<doc_header_including_copyright_and_license>> -module ocda.load_pod; - -/+ ↓ the abstraction out of a pod archive. - . - A .sisupod is one zstd frame wrapping a stored zip; a .zip is the same - archive unwrapped, which is what spine published before it wrapped them. - Either way, a pod built with --pod2 carries the abstraction inside it, - at media/abstraction/<uid>.<lang>.ssp, and that is what is read. - . - So this is a third input form and *not* a third model: the .ssp inside - is parsed by ocda.load_ssp and comes back as the same value. A writer - over the library gains .sisupod without being touched, which is the - whole reason the dispatch is in one place. - . - A pod built without --pod2 carries markup and no abstraction. That is - refused, and the refusal says which flag makes a pod this reader can - read, because "no abstraction in it" is a fact about the pod rather - than about the reader. -+/ - -import std.algorithm : endsWith, sort; -import std.array : array; -import std.file : read; -import std.string : indexOf; -import ocda.model; -import ocda.load_ssp; -import ocda.zip; -import ocda.zstd; - -/+ ↓ a limit on what a frame may expand to. - A reader of an artefact it did not make declines to be told how much - memory to allocate. 512 MiB is far above the largest pod in the - reference collection (4 MiB) and far below anything that matters. -+/ -enum size_t max_pod_size = 512 * 1024 * 1024; - -class OcdaPodError : Exception { - this(string msg) { super(msg); } -} - -/+ ↓ which languages of the abstraction the pod carries, in sorted order +/ -string[] podLanguages(string path) { - auto bytes = archiveBytes(path); - string[] out_; - foreach (e; zipFindSuffix(zipEntries(bytes), ".ssp")) { - string lang = langOfSspName(e.name); - if (lang.length > 0) { out_ ~= lang; } - } - return out_.sort.array; -} - -/+ ↓ load one language of the abstraction a pod carries. - lang == "" takes the only one there is, and where there is a choice it - says so rather than choosing, as the .ocda.db loader does. -+/ -OcdaDocument loadPodFile(string path, string lang = "") { - auto bytes = archiveBytes(path); - auto entries = zipEntries(bytes); - auto ssps = zipFindSuffix(entries, ".ssp"); - if (ssps.length == 0) { - throw new OcdaPodError(path ~ ": carries no .ssp." - ~ " A pod carries the abstraction only when spine was asked for it:" - ~ " spine --pod2 puts it in, spine --source does not."); - } - ZipEntry chosen = ssps[0]; - if (lang.length > 0) { - bool found = false; - foreach (e; ssps) { - if (langOfSspName(e.name) == lang) { chosen = e; found = true; break; } - } - if (!found) { - string have; - foreach (i, e; ssps) { - have ~= (i > 0 ? " " : "") ~ langOfSspName(e.name); - } - throw new OcdaPodError(path ~ ": no language '" ~ lang - ~ "'; it holds: " ~ have); - } - } else if (ssps.length > 1) { - string have; - foreach (i, e; ssps) { - have ~= (i > 0 ? " " : "") ~ langOfSspName(e.name); - } - throw new OcdaPodError(path ~ ": holds more than one language (" - ~ have ~ "); name the one wanted"); - } - auto ssp = zipRead(bytes, chosen); - return parseSsp(cast(string) ssp.idup); -} - -/+ ↓ the archive, unwrapped if it was wrapped. - The bytes decide, not the name: a .sisupod is a zstd frame and a .zip - is not, and a file misnamed either way still reads. -+/ -package ubyte[] archiveBytes(string path) { - auto bytes = cast(ubyte[]) read(path); - return zstdIsFrame(bytes) - ? zstdDecompress(bytes, max_pod_size) - : bytes; -} - -/+ ↓ <uid>.<lang>.ssp -> <lang> +/ -package string langOfSspName(string name) { - if (!name.endsWith(".ssp")) { return ""; } - string stem = name[0 .. $ - 4]; - auto dot = stem.lastIndexOf('.'); - return (dot > 0) ? stem[dot + 1 .. $] : ""; -} - -private ptrdiff_t lastIndexOf(string s, char c) { - for (ptrdiff_t i = s.length - 1; i >= 0; i--) { - if (s[i] == c) { return i; } - } - return -1; -} -#+END_SRC - -**** load ssp - -#+HEADER: :tangle "../dlang/src/ocda/load_ssp.d" -#+BEGIN_SRC d -<<doc_header_including_copyright_and_license>> -module ocda.load_ssp; - -import std.algorithm : splitter, startsWith; -import std.array : array, split; -import std.conv : to, ConvException; -import std.file : readText; -import std.stdio : writeln; -import std.string : strip, stripLeft, indexOf; -import ocda.model; - -OcdaDocument parseSspFile(string filename) { - string content = readText(filename); - return parseSsp(content); -} - -OcdaDocument parseSsp(string content) { - OcdaDocument doc; - OcdaObject current_obj; - bool in_object = false; - /+ ↓ whether a text line has been seen for the object being read. - Not "is the text non-empty": an object whose first text line is blank - (a verse opening on an empty line, which the sample set has seven of) - would then lose it, the next line overwriting rather than appending. - +/ - bool seen_text = false; - string current_section = ""; - string current_block = ""; // meta, make, doc_has, or section name - - foreach (line; content.splitter("\n")) { - /+ ↓ comment. The first line of a .ssp declares the format version, and - it is the one comment that carries meaning: a reader checks the - major part and refuses what it does not know rather than loading - an unfamiliar artefact half populated in silence. - +/ - if (line.length > 0 && line[0] == '%') { - enum _banner = "% SiSU Document Abstraction v"; - enum _source = "% Source: "; - if (doc.format_version.length == 0 && line.startsWith(_banner)) { - doc.format_version = line[_banner.length .. $].strip; - } else if (doc.source_filename.length == 0 && line.startsWith(_source)) { - doc.source_filename = line[_source.length .. $].strip; - } - continue; - } - - /+ ↓ blank line - finalize current object +/ - if (line.length == 0 || line.strip.length == 0) { - if (in_object) { - commitObject(doc, current_section, current_obj); - current_obj = OcdaObject.init; - in_object = false; - seen_text = false; - } - continue; - } - - /+ ↓ section/block open +/ - if (line[0] == '@') { - auto brace_pos = line.indexOf('{'); - if (brace_pos > 0) { - string name = line[1 .. brace_pos].strip; - current_block = name; - if (!isPreambleBlock(name)) { - current_section = name; - /+ ↓ a block that is neither preamble nor a section this reader - knows: counted, so that --report-unknown can say so +/ - if (doc.sectionByName(name) is null) { - doc.unknown.block[name]++; - } - } - } - continue; - } - - /+ ↓ section/block close +/ - if (line[0] == '}') { - if (in_object) { - commitObject(doc, current_section, current_obj); - current_obj = OcdaObject.init; - in_object = false; - seen_text = false; - } - current_block = ""; - continue; - } - - /+ ↓ inside meta/make/doc_has blocks: key-value pairs +/ - if (isPreambleBlock(current_block)) { - string stripped = line.stripLeft; - auto colon_pos = stripped.indexOf(':'); - if (colon_pos > 0) { - string key = stripped[0 .. colon_pos].strip; - string value = (colon_pos + 1 < stripped.length) - ? stripped[colon_pos + 1 .. $].strip : ""; - if (current_block == "meta") { doc.metadata[key] = value; } - else if (current_block == "make") { doc.make[key] = value; } - else if (current_block == "source") { doc.source[key] = value; } - else { doc.doc_has[key] = value; } - } - continue; - } - - /+ ↓ object declaration: [N] type ... +/ - if (line[0] == '[') { - if (in_object) { - commitObject(doc, current_section, current_obj); - } - current_obj = OcdaObject.init; - current_obj.section = current_section; - in_object = true; - seen_text = false; - parseObjectDeclaration(line, current_obj); - continue; - } - - /+ ↓ property: .name: value +/ - if (line[0] == '.') { - if (in_object) { - parseProperty(line, current_obj, doc.unknown); - } - continue; - } - - /+ ↓ text content: | content +/ - if (line.length >= 1 && line[0] == '|') { - if (in_object) { - string text_content = (line.length > 2) ? line[2 .. $] : ""; - if (seen_text) { - current_obj.text ~= "\n" ~ text_content; - } else { - current_obj.text = text_content; - seen_text = true; - } - } - continue; - } - - /+ ↓ no known shape. Counted by its first character so that a reader - can be told the file held something this one did not expect - +/ - doc.unknown.line[line[0 .. 1].idup]++; - } - - /+ ↓ finalize last object +/ - if (in_object) { - commitObject(doc, current_section, current_obj); - } - - return doc; -} - -private void parseObjectDeclaration(string line, ref OcdaObject obj) { - /+ parse: [N] type :level identifier +/ - auto bracket_close = line.indexOf(']'); - if (bracket_close < 0) return; - try { - obj.ocn = line[1 .. bracket_close].strip.to!int; - } catch (ConvException) { - obj.ocn = 0; - } - string rest = (bracket_close + 1 < line.length) - ? line[bracket_close + 1 .. $].strip : ""; - if (rest.length == 0) return; - - auto parts = rest.split(" "); - if (parts.length == 0) return; - obj.is_a = parts[0]; - - if (obj.is_a == "heading" && parts.length >= 2) { - /+ parse :A, :B, :1, etc. +/ - string lev = parts[1]; - if (lev.length >= 2 && lev[0] == ':') { - obj.heading_level = lev[1 .. $]; - } - /+ optional identifier after level +/ - if (parts.length >= 3) { - obj.identifier = parts[2]; - } - } -} - -/+ ↓ the blocks that are document preamble rather than sections of objects +/ -private bool isPreambleBlock(string name) { - return name == "source" || name == "meta" || name == "make" || name == "doc_has"; -} - -private void parseProperty(string line, ref OcdaObject obj, ref OcdaUnknown unknown) { - /+ parse: .name: value +/ - auto colon_pos = line.indexOf(':'); - if (colon_pos < 1) return; - string name = line[1 .. colon_pos].strip; - string value = (colon_pos + 1 < line.length) - ? line[colon_pos + 1 .. $].strip : ""; - - switch (name) { - case "part": obj.part = value; break; - case "is_of_type": obj.is_of_type = value; break; - case "section": obj.own_section = value; break; - case "identifier": obj.identifier = value; break; - case "segment_html_is": obj.segment_html_is = value; break; - /+ ↓ the epub segments at heading levels 1 to 4 where the html - segments at level 4 alone, so the two are different files and - the abstraction records both +/ - case "segment_epub": obj.segment_epub = value; break; - case "segment_epub_is": obj.segment_epub_is = value; break; - case "table_aligns": obj.table_aligns = value; break; - case "stow_link": obj.stow_link ~= value; break; - case "anchor_tag": obj.anchor_tags ~= value; break; - case "parent": obj.parent_ocn = toIntSafe(value); break; - case "last_descendant": obj.last_descendant_ocn = toIntSafe(value); break; - case "ocn_range": { - auto r = parseIntArray(value); - if (r.length == 2) { obj.ocn_range_first = r[0]; obj.ocn_range_last = r[1]; } - break; - } - case "children": obj.children = parseIntArray(value); break; - case "ancestors": obj.ancestors = parseIntArray(value); break; - case "dummy": obj.dummy_heading = (value == "true"); break; - case "ocn_off": obj.ocn_off = (value == "true"); break; - case "indent": - auto parts = value.split(" "); - if (parts.length >= 1) obj.indent_base = toIntSafe(parts[0]); - if (parts.length >= 2) obj.indent_hang = toIntSafe(parts[1]); - break; - case "bullet": obj.bullet = (value == "true"); break; - case "lang": obj.lang = value; break; - case "has": - foreach (flag; value.split(" ")) { - switch (flag) { - case "links": obj.has_links = true; break; - case "notes_reg": obj.has_notes_reg = true; break; - case "notes_star": obj.has_notes_star = true; break; - case "images": obj.has_images = true; break; - default: break; - } - } - break; - case "segment": obj.segment = value; break; - case "segment_prev": obj.segment_prev = value; break; - case "segment_next": obj.segment_next = value; break; - case "anchor": obj.anchor = value; break; - case "heading_lev_anchor": obj.heading_lev_anchor = value; break; - case "table_cols": obj.table_cols = toIntSafe(value); break; - case "table_widths": obj.table_widths = value; break; - case "table_header": obj.table_header = (value == "true"); break; - case "code_syntax": obj.code_syntax = value; break; - case "code_linenumbers": obj.code_linenumbers = (value == "true"); break; - case "heading_ancestors_text": obj.heading_ancestors_text = value; break; - case "sha256": obj.sha256 = value; break; - case "image": obj.images ~= parseImageRecord(value); break; - /+ ↓ a property this reader has no use for, or has never heard of. - Ignored, which is the conforming behaviour within a major - version, but counted rather than passed over in silence - +/ - default: unknown.property[name]++; break; - } -} - -private OcdaImage parseImageRecord(string value) { - /+ parse: <filename> sha256:<HEX> bytes:<n> [key:value ...] - filename first, remaining fields order-independent; unrecognised - fields are kept rather than discarded +/ - OcdaImage img; - bool first = true; - foreach (field; value.split(" ")) { - if (field.length == 0) continue; - if (first) { - img.name = field; - first = false; - continue; - } - auto colon_pos = field.indexOf(':'); - if (colon_pos < 1) continue; - string key = field[0 .. colon_pos]; - string val = (colon_pos + 1 < field.length) ? field[colon_pos + 1 .. $] : ""; - switch (key) { - case "sha256": img.sha256 = val; break; - case "bytes": img.bytes = toUlongSafe(val); break; - case "px": - auto dims = val.split("x"); - if (dims.length == 2) { - img.width = toIntSafe(dims[0]); - img.height = toIntSafe(dims[1]); - } - break; - case "missing": img.missing = (val == "true"); break; - default: img.fields[key] = val; break; - } - } - return img; -} - -/+ ↓ two values the text form leaves out because they can be worked out. - . - The .ssp emits a property only when it carries information, so - .identifier is absent when it is simply the ocn, and .anchor is absent - when it is the segment name. The .ocda.db, being columns rather than - lines, stores both unconditionally. So a loader that takes the text - form literally gives a *different model* from the one the database - gives, and the difference is not small: in the reference collection - the identifier is omitted on all but a handful of objects. - . - The model is the thing both serialisations describe, so the loader - fills them in. Found by dumping both models and diffing them, which no - amount of comparing section counts would have shown. -+/ -private void deriveOmitted(ref OcdaObject obj) { - import std.conv : to; - if (obj.identifier.length == 0 && obj.ocn != 0) { - obj.identifier = obj.ocn.to!string; - } - if (obj.anchor.length == 0 && obj.segment.length > 0) { - obj.anchor = obj.segment; - } -} - -private void commitObject(ref OcdaDocument doc, string section, ref OcdaObject obj) { - deriveOmitted(obj); - auto sec = doc.sectionByName(section); - if (sec !is null) { - ,*sec ~= obj; - } -} - -private int toIntSafe(string s) { - try { return s.strip.to!int; } - catch (ConvException) { return 0; } -} - -private ulong toUlongSafe(string s) { - try { return s.strip.to!ulong; } - catch (ConvException) { return 0; } -} - -private int[] parseIntArray(string s) { - int[] result; - foreach (part; s.split(" ")) { - auto v = toIntSafe(part); - result ~= v; - } - return result; -} -#+END_SRC - -*** images - -#+HEADER: :tangle "../dlang/src/ocda/images.d" -#+BEGIN_SRC d -<<doc_header_including_copyright_and_license>> -module ocda.images; - -/+ ↓ an image's bytes, from whichever source the document came from. - . - The model *names* images, with digest, byte size and pixel size; it does - not hold them. That is right - an abstraction of a document is not a - bundle of its assets - but a writer of any format that shows images - needs the bytes, so something has to know where they are. This is that - something, and it is in the library rather than in a writer because the - answer depends on the *input form* and not on the output. - . - Three answers, and one refusal: - . - .ocda.db the files table, role 'image', by name - .sisupod the archive entry whose path ends in /image/<name> - .zip the same - .ssp beside it: a pod puts the abstraction in - media/abstraction/ and its images in media/image/, so the - path is documented rather than guessed - . - A .ssp that is not in a pod has nowhere to look, and that is reported as - absent rather than as an error: a reader that cannot find an image - should say so and carry on, because the document is still readable. -+/ - -import std.algorithm : endsWith; -import std.array : array; -import std.file : exists, isFile, read; -import std.path : buildPath, dirName; -import ocda.load; -import ocda.sqlite; -import ocda.zip; -import ocda.zstd; - -/+ ↓ the bytes, or an empty slice when the source has no such image +/ -ubyte[] imageBytes(string source, string name) { - final switch (sourceOf(source)) { - case Source.ocda_db: return fromDb(source, name); - case Source.pod: return fromArchive(source, name); - case Source.ssp: return fromBeside(source, name); - case Source.unknown: return []; - } -} - -/+ ↓ can this source give images at all? - A writer asks before it decides whether to emit them, because an <img> - pointing at a file it did not include is worse than no image. -+/ -bool canGiveImages(string source) { - final switch (sourceOf(source)) { - case Source.ocda_db: return true; - case Source.pod: return true; - case Source.ssp: return imageDirBeside(source).length > 0; - case Source.unknown: return false; - } -} - -private ubyte[] fromDb(string path, string name) { - auto db = Db.open(path); - scope(exit) { db.close(); } - if (!db.has("files")) { return []; } - /+ ↓ the name is from the abstraction, not from a caller, and the quote - doubling is what keeps it a value rather than syntax +/ - return db.blob("SELECT data FROM files WHERE role = 'image' AND name = '" - ~ sqlQuote(name) ~ "'"); -} - -private ubyte[] fromArchive(string path, string name) { - auto bytes = cast(ubyte[]) read(path); - if (zstdIsFrame(bytes)) { bytes = zstdDecompress(bytes, 512 * 1024 * 1024); } - auto entries = zipEntries(bytes); - foreach (e; entries) { - if (e.name.endsWith("/image/" ~ name)) { return zipRead(bytes, e); } - } - return []; -} - -private ubyte[] fromBeside(string path, string name) { - string dir = imageDirBeside(path); - if (dir.length == 0) { return []; } - string file = buildPath(dir, name); - return file.exists && file.isFile ? cast(ubyte[]) read(file) : []; -} - -/+ ↓ media/abstraction/<x>.ssp -> media/image/ +/ -private string imageDirBeside(string path) { - string dir = path.dirName; - if (!dir.endsWith("abstraction")) { return ""; } - string image_dir = buildPath(dir.dirName, "image"); - return image_dir.exists ? image_dir : ""; -} - -/+ ↓ a single quote doubled is how sql says a single quote +/ -private string sqlQuote(string s) { - string out_; - foreach (c; s) { - out_ ~= (c == '\'') ? "''" : [c]; - } - return out_; -} -#+END_SRC - -** sqlite - -#+HEADER: :tangle "../dlang/src/ocda/sqlite.d" -#+BEGIN_SRC d -<<doc_header_including_copyright_and_license>> -module ocda.sqlite; - -/+ ↓ the sqlite3 C API, declared rather than depended upon. - . - A binding from the dub registry would be a dependency to fetch, and - the point of this reader is that it builds in a sandbox with no - network. The dozen functions a reader needs are declared here and the - library is linked directly, which costs a page of extern(C) and saves - a dependency. -+/ - -import std.conv : to; -import std.string : fromStringz, toStringz; - -extern (C) { - struct sqlite3; - struct sqlite3_stmt; - - int sqlite3_open_v2(const(char)* filename, sqlite3** db, int flags, const(char)* vfs); - int sqlite3_close(sqlite3* db); - int sqlite3_prepare_v2(sqlite3* db, const(char)* sql, int nbyte, - sqlite3_stmt** stmt, const(char)** tail); - int sqlite3_step(sqlite3_stmt* stmt); - int sqlite3_finalize(sqlite3_stmt* stmt); - int sqlite3_column_count(sqlite3_stmt* stmt); - const(char)* sqlite3_column_name(sqlite3_stmt* stmt, int col); - const(ubyte)* sqlite3_column_text(sqlite3_stmt* stmt, int col); - int sqlite3_column_int(sqlite3_stmt* stmt, int col); - int sqlite3_column_type(sqlite3_stmt* stmt, int col); - const(char)* sqlite3_errmsg(sqlite3* db); - const(void)* sqlite3_column_blob(sqlite3_stmt* stmt, int col); - int sqlite3_column_bytes(sqlite3_stmt* stmt, int col); -} - -enum SQLITE_OK = 0; -enum SQLITE_ROW = 100; -enum SQLITE_DONE = 101; -enum SQLITE_NULL = 5; -enum SQLITE_OPEN_READONLY = 0x00000001; - -class SqliteError : Exception { - this(string msg) { super(msg); } -} - -/+ ↓ a database, opened read only. - Read only is deliberate: a reader has no business writing, and a - request that opens a database read-write can create an empty one where - a path is mistyped. -+/ -struct Db { - private sqlite3* _handle; - - static Db open(string path) { - sqlite3* handle; - auto rc = sqlite3_open_v2(path.toStringz, &handle, SQLITE_OPEN_READONLY, null); - if (rc != SQLITE_OK) { - string msg = (handle is null) - ? "cannot open " ~ path - : "cannot open " ~ path ~ ": " ~ sqlite3_errmsg(handle).fromStringz.idup; - if (handle !is null) { sqlite3_close(handle); } - throw new SqliteError(msg); - } - Db db; - db._handle = handle; - return db; - } - - void close() { - if (_handle !is null) { sqlite3_close(_handle); _handle = null; } - } - - /+ ↓ every row of a query, each row as its columns in order +/ - string[][] rows(string sql) { - sqlite3_stmt* stmt; - auto rc = sqlite3_prepare_v2(_handle, sql.toStringz, -1, &stmt, null); - if (rc != SQLITE_OK) { - throw new SqliteError(sqlite3_errmsg(_handle).fromStringz.idup ~ " in: " ~ sql); - } - scope(exit) { sqlite3_finalize(stmt); } - string[][] out_rows; - int cols = sqlite3_column_count(stmt); - while (sqlite3_step(stmt) == SQLITE_ROW) { - string[] row; - foreach (c; 0 .. cols) { - row ~= (sqlite3_column_type(stmt, c) == SQLITE_NULL) - ? "" : (cast(const(char)*) sqlite3_column_text(stmt, c)).fromStringz.idup; - } - out_rows ~= row; - } - return out_rows; - } - - /+ ↓ the first column of the first row, "" when there is none +/ - string one(string sql) { - auto r = rows(sql); - return (r.length > 0 && r[0].length > 0) ? r[0][0] : ""; - } - - /+ ↓ the first column of the first row as bytes, for a blob. - Separate from one(), because a blob is not text: an image read as text - stops at the first zero byte, and a png's ninth byte is zero. - +/ - ubyte[] blob(string sql) { - sqlite3_stmt* stmt; - if (sqlite3_prepare_v2(_handle, sql.toStringz, -1, &stmt, null) != SQLITE_OK) { - throw new SqliteError("prepare: " ~ sqlite3_errmsg(_handle).fromStringz.idup - ~ " [" ~ sql ~ "]"); - } - scope(exit) { sqlite3_finalize(stmt); } - if (sqlite3_step(stmt) != SQLITE_ROW) { return []; } - int n = sqlite3_column_bytes(stmt, 0); - if (n <= 0) { return []; } - auto p = cast(const(ubyte)*) sqlite3_column_blob(stmt, 0); - if (p is null) { return []; } - return p[0 .. n].dup; - } - - /+ ↓ does a table or view of this name exist +/ - bool has(string name) { - return one("SELECT count(*) FROM sqlite_master WHERE name = '" ~ name ~ "'") != "0"; - } -} -#+END_SRC - -** compression -*** zip - -#+HEADER: :tangle "../dlang/src/ocda/zip.d" -#+BEGIN_SRC d -<<doc_header_including_copyright_and_license>> -module ocda.zip; - -/+ ↓ enough of the zip format to find one file in a pod and read it. - . - Not a zip library. A pod archive holds a handful of entries, spine - writes them **stored** (uncompressed) so that the outer zstd frame does - the compressing once over the whole thing, and what is wanted from it - here is one named entry. So: read the central directory, find the - entry, return its bytes. - . - Deflate is *refused rather than mishandled*. Phobos can inflate, but a - library that says it read an entry it in fact skipped is worse than one - that says it cannot, and spine's own archives do not need it. -+/ - -import std.algorithm : endsWith; -import std.conv : to; - -class OcdaZipError : Exception { - this(string msg) { super(msg); } -} - -struct ZipEntry { - string name; - uint method; // 0 stored, 8 deflate - uint compressed; - uint uncompressed; - uint local_offset; -} - -/+ ↓ the entries, from the central directory at the end of the file +/ -ZipEntry[] zipEntries(const(ubyte)[] bytes) { - enum uint eocd_sig = 0x06054b50; - enum uint cd_sig = 0x02014b50; - if (bytes.length < 22) { - throw new OcdaZipError("too short to be a zip archive"); - } - /+ ↓ the end of central directory record is last, and may be followed by a - comment of up to 65535 bytes, so it is searched for backwards +/ - ptrdiff_t eocd = -1; - ptrdiff_t lowest = bytes.length > (22 + 65535) ? bytes.length - (22 + 65535) : 0; - for (ptrdiff_t i = bytes.length - 22; i >= lowest; i--) { - if (u32(bytes, i) == eocd_sig) { eocd = i; break; } - } - if (eocd < 0) { - throw new OcdaZipError("no end of central directory record;" - ~ " it does not look like a zip archive"); - } - uint count = u16(bytes, eocd + 10); - uint offset = u32(bytes, eocd + 16); - ZipEntry[] out_; - size_t p = offset; - foreach (_; 0 .. count) { - if (p + 46 > bytes.length || u32(bytes, p) != cd_sig) { - throw new OcdaZipError("central directory entry " - ~ out_.length.to!string ~ " is malformed"); - } - ZipEntry e; - e.method = u16(bytes, p + 10); - e.compressed = u32(bytes, p + 20); - e.uncompressed = u32(bytes, p + 24); - uint name_len = u16(bytes, p + 28); - uint extra_len = u16(bytes, p + 30); - uint cmnt_len = u16(bytes, p + 32); - e.local_offset = u32(bytes, p + 42); - e.name = cast(string) bytes[p + 46 .. p + 46 + name_len].idup; - out_ ~= e; - p += 46 + name_len + extra_len + cmnt_len; - } - return out_; -} - -/+ ↓ one entry's bytes, from its local header +/ -ubyte[] zipRead(const(ubyte)[] bytes, ZipEntry e) { - enum uint local_sig = 0x04034b50; - size_t p = e.local_offset; - if (p + 30 > bytes.length || u32(bytes, p) != local_sig) { - throw new OcdaZipError(e.name ~ ": local header is malformed"); - } - uint name_len = u16(bytes, p + 26); - uint extra_len = u16(bytes, p + 28); - size_t start = p + 30 + name_len + extra_len; - size_t end = start + e.compressed; - if (end > bytes.length) { - throw new OcdaZipError(e.name ~ ": runs past the end of the archive"); - } - if (e.method != 0) { - throw new OcdaZipError(e.name ~ ": compression method " - ~ e.method.to!string ~ "; this reader takes only stored entries," - ~ " which is what spine writes"); - } - return bytes[start .. end].dup; -} - -/+ ↓ the first entry whose name ends this way, or none. - Used to find media/abstraction/<uid>.<lang>.ssp without knowing the uid. -+/ -ZipEntry[] zipFindSuffix(ZipEntry[] entries, string suffix) { - ZipEntry[] out_; - foreach (e; entries) { - if (e.name.endsWith(suffix)) { out_ ~= e; } - } - return out_; -} - -private uint u16(const(ubyte)[] b, size_t i) { - return cast(uint) b[i] | (cast(uint) b[i + 1] << 8); -} - -private uint u32(const(ubyte)[] b, size_t i) { - return cast(uint) b[i] - | (cast(uint) b[i + 1] << 8) - | (cast(uint) b[i + 2] << 16) - | (cast(uint) b[i + 3] << 24); -} -#+END_SRC - -*** zstd - -#+HEADER: :tangle "../dlang/src/ocda/zstd.d" -#+BEGIN_SRC d -<<doc_header_including_copyright_and_license>> -module ocda.zstd; - -/+ ↓ libzstd, declared rather than bound. - . - A .sisupod is one zstd frame wrapping a zip archive, so reading one - needs exactly two things from the library: is this a frame, and what is - in it. Both are declared here and the C library is linked, which is the - same arrangement ocda.sqlite uses: no vendored tree, no generated - header, and the prototypes say what is being called. -+/ - -extern (C) @nogc nothrow private { - size_t ZSTD_decompress(void* dst, size_t dstCapacity, - const(void)* src, size_t compressedSize); - ulong ZSTD_getFrameContentSize(const(void)* src, size_t srcSize); - uint ZSTD_isError(size_t code); - const(char)* ZSTD_getErrorName(size_t code); -} - -/+ ↓ the two values ZSTD_getFrameContentSize returns in place of a size +/ -private enum ulong contentSizeUnknown = ulong.max; -private enum ulong contentSizeError = ulong.max - 1; - -/+ ↓ the magic number of a zstd frame, little endian 0xFD2FB528 +/ -private static immutable ubyte[4] zstd_magic = [0x28, 0xB5, 0x2F, 0xFD]; - -class OcdaZstdError : Exception { - this(string msg) { super(msg); } -} - -/+ ↓ does this begin with a zstd frame? - Asked of the bytes rather than of the filename, because what is inside - decides how a thing is opened. A .sisupod is a frame; a .zip published - before spine wrapped them is not, and both are read. -+/ -bool zstdIsFrame(const(ubyte)[] bytes) { - if (bytes.length < zstd_magic.length) { return false; } - foreach (i, b; zstd_magic) { - if (bytes[i] != b) { return false; } - } - return true; -} - -/+ ↓ one frame in, its contents out. - max_size is a refusal rather than a buffer size: a frame declares how - much it expands to, and a frame that claims more than the caller is - willing to hold is not decompressed at all. A reader of an artefact it - did not make wants that limit. -+/ -ubyte[] zstdDecompress(const(ubyte)[] bytes, size_t max_size) { - if (!zstdIsFrame(bytes)) { - throw new OcdaZstdError("not a zstd frame"); - } - ulong declared = ZSTD_getFrameContentSize(bytes.ptr, bytes.length); - if (declared == contentSizeError) { - throw new OcdaZstdError("zstd frame: could not read its content size"); - } - if (declared == contentSizeUnknown) { - throw new OcdaZstdError("zstd frame declares no content size;" - ~ " this reader takes only frames that do"); - } - if (declared > max_size) { - import std.conv : to; - throw new OcdaZstdError("zstd frame expands to " ~ declared.to!string - ~ " bytes, over the " ~ max_size.to!string ~ " byte limit"); - } - auto out_ = new ubyte[](cast(size_t) declared); - size_t got = ZSTD_decompress(out_.ptr, out_.length, bytes.ptr, bytes.length); - if (ZSTD_isError(got)) { - import std.string : fromStringz; - throw new OcdaZstdError("zstd: " - ~ ZSTD_getErrorName(got).fromStringz.idup); - } - if (got != declared) { - throw new OcdaZstdError("zstd frame expanded to a different size" - ~ " than it declared"); - } - return out_; -} -#+END_SRC - -** dump - -#+HEADER: :tangle "../dlang/src/ocda/dump.d" -#+BEGIN_SRC d -<<doc_header_including_copyright_and_license>> -module ocda.dump; - -/+ ↓ the model, written out canonically. - . - Not an output format: an *oracle*. Two libraries in two languages - implement the same model from the same specification, and the cheap - way to ask whether they agree is to have each write the whole loaded - document as text and diff the two. That is only useful if the text is - fully determined by the model, so: - . - - every line is "<key> <value>", the value running to end of line, - and only one free-form value per line; - - a field at its default is *omitted*, so the output says what the - document has rather than what the struct declares; - - dictionaries are emitted in sorted key order, because neither - language's hash map has an order worth relying on; - - sections come in the fixed order of the format, not - alphabetically. - . - Anything that reads this to reconstruct a document has misunderstood - it. Load the .ssp or the .ocda.db. -+/ - -import std.algorithm : sort, splitter; -import std.array : appender, array, join; -import std.conv : to; -import ocda.model; - -/+ ↓ the nine sections, in format order +/ -static immutable string[] dump_sections = ["head", "toc", "body", "endnotes", - "glossary", "bibliography", "bookindex", "blurb", "tail"]; - -string dumpDocument(ref OcdaDocument doc) { - auto out_ = appender!string; - - void kv(string key, string value) { - if (value.length == 0) { return; } - out_ ~= key; out_ ~= " "; out_ ~= value; out_ ~= "\n"; - } - void kn(string key, long value) { - if (value == 0) { return; } - kv(key, value.to!string); - } - void kb(string key, bool value) { - if (!value) { return; } - kv(key, "1"); - } - void dict(string prefix, string[string] d) { - foreach (key; d.keys.dup.sort) { - out_ ~= prefix; out_ ~= " "; out_ ~= key; out_ ~= " "; - out_ ~= d[key]; out_ ~= "\n"; - } - } - - kv("version", doc.format_version); - kv("source_filename", doc.source_filename); - dict("source", doc.source); - dict("meta", doc.metadata); - dict("make", doc.make); - dict("doc_has", doc.doc_has); - - foreach (name; dump_sections) { - auto sec = doc.sectionByName(name); - if (sec is null || (*sec).length == 0) { continue; } - kv("section", name ~ " " ~ (*sec).length.to!string); - } - - foreach (name; dump_sections) { - auto sec = doc.sectionByName(name); - if (sec is null) { continue; } - foreach (i, obj; *sec) { - out_ ~= "object "; out_ ~= name; out_ ~= " "; - out_ ~= i.to!string; out_ ~= "\n"; - kn("ocn", obj.ocn); - kv("is_a", obj.is_a); - kv("own_section", obj.own_section); - kv("level", obj.heading_level); - kv("identifier", obj.identifier); - kv("part", obj.part); - kv("is_of_type", obj.is_of_type); - kv("segment_html_is", obj.segment_html_is); - kv("segment_epub", obj.segment_epub); - kv("segment_epub_is", obj.segment_epub_is); - kn("parent", obj.parent_ocn); - kn("last_descendant", obj.last_descendant_ocn); - kn("ocn_range_first", obj.ocn_range_first); - kn("ocn_range_last", obj.ocn_range_last); - kv("children", ints(obj.children)); - kv("ancestors", ints(obj.ancestors)); - kb("dummy", obj.dummy_heading); - kb("ocn_off", obj.ocn_off); - kn("indent_base", obj.indent_base); - kn("indent_hang", obj.indent_hang); - kb("bullet", obj.bullet); - kv("lang", obj.lang); - kb("has_links", obj.has_links); - kb("has_notes_reg", obj.has_notes_reg); - kb("has_notes_star", obj.has_notes_star); - kb("has_images", obj.has_images); - kv("segment", obj.segment); - kv("segment_prev", obj.segment_prev); - kv("segment_next", obj.segment_next); - kv("anchor", obj.anchor); - kv("heading_lev_anchor", obj.heading_lev_anchor); - kn("table_cols", obj.table_cols); - kv("table_widths", obj.table_widths); - kv("table_aligns", obj.table_aligns); - kb("table_header", obj.table_header); - kv("code_syntax", obj.code_syntax); - kb("code_linenumbers", obj.code_linenumbers); - kv("heading_ancestors_text", obj.heading_ancestors_text); - kv("sha256", obj.sha256); - foreach (url; obj.stow_link) { kv("stow_link", url); } - foreach (a; obj.anchor_tags) { kv("anchor_tag", a); } - foreach (img; obj.images) { - /+ ↓ the name last, because a filename may hold a space where - none of the other fields can +/ - kv("image", img.sha256 ~ " " ~ img.bytes.to!string ~ " " - ~ img.width.to!string ~ "x" ~ img.height.to!string ~ " " - ~ (img.missing ? "1" : "0") ~ " " ~ img.name); - foreach (key; img.fields.keys.dup.sort) { - kv("image_field", key ~ " " ~ img.fields[key]); - } - } - /+ ↓ text, when there is any. An object whose only text line is - blank holds "" here, indistinguishable from no text at all, - because the model joins the lines into one string. Both - libraries lose the same distinction in the same place, which is - what makes them comparable; it is recorded rather than hidden. - +/ - if (obj.text.length > 0) { - auto lines = obj.text.splitter("\n").array; - kv("text", lines.length.to!string); - foreach (line; lines) { - out_ ~= "| "; out_ ~= line; out_ ~= "\n"; - } - } - } - } - return out_.data; -} - -/+ ↓ what the reader met and did not understand, sorted. Separate from the - document dump: it describes the reader, not the document, and the - .ocda.db loader has nothing to report where the .ssp loader does. -+/ -string dumpUnknown(ref OcdaDocument doc) { - auto out_ = appender!string; - void counts(string prefix, size_t[string] d) { - foreach (key; d.keys.dup.sort) { - out_ ~= prefix; out_ ~= " "; out_ ~= key; out_ ~= " "; - out_ ~= d[key].to!string; out_ ~= "\n"; - } - } - counts("property", doc.unknown.property); - counts("block", doc.unknown.block); - counts("line", doc.unknown.line); - return out_.data; -} - -private string ints(int[] xs) { - if (xs.length == 0) { return ""; } - string[] parts; - foreach (x; xs) { parts ~= x.to!string; } - return parts.join(","); -} -#+END_SRC - -* read -** main - -#+HEADER: :tangle "../dlang/src/read/main.d" -#+BEGIN_SRC d -<<doc_header_including_copyright_and_license>> -/++ - ocda-read - reads the object-centric document abstraction, in any of the - forms it is written in, and says what is in it. - - ocda-read [options] <file>.ssp | <file>.ocda.db | <file>.sisupod [lang] - - One reader where there were two: sisudoc-ssp-reader (NO LONGER RELEASED, NOW USED DIRECTLY BY WRITER) FIX read a .ssp and - sisudoc-ocda-db-reader (NO LONGER RELEASED, NOW USED DIRECTLY BY WRITER) FIX a .ocda.db, each with its own parser. This one is - built on the library, so every mode reads every form, and a mode given the - .ssp, the .ocda.db and the .sisupod of a document says the same thing of - each: it is told what the document is, not which file it came from. - - Two things are about the file rather than the document, and say so: - --files, what a .ocda.db carries besides the abstraction, and --validate, - which checks a .ssp as text. Everything else is the model's. - - The output modules are the .ssp reader's, moved onto the library's model. -+/ -module read.main; - -import std.algorithm : canFind, map; -import std.array : array, join; -import std.getopt; -import std.stdio : writeln, writefln, stderr; -import ocda.model; -import ocda.load; -import ocda.inventory; -import read.tree; -import read.html; -import read.llm_text; -import read.llm_jsonl; -import read.images; -import read.verify; -import read.report_unknown; -import read.validate; - -int main(string[] args) { - bool do_tree = false; - bool do_html = false; - bool do_llm_text = false; - bool do_llm_jsonl = false; - bool do_images = false; - bool do_verify = false; - bool do_report_unknown = false; - bool do_validate = false; - bool do_languages = false; - bool do_files = false; - bool any_version = false; - string image_dir = ""; - string output_file = ""; - string lang = ""; - - GetoptResult helpInfo; - try { - helpInfo = getopt(args, - "tree", "the headings, as a tree", &do_tree, - "outline", "the same as --tree", &do_tree, - "html", "the document as one html page", &do_html, - "llm-text", "plain text with heading breadcrumbs", &do_llm_text, - "llm-jsonl", "JSONL, chunks ready for embedding", &do_llm_jsonl, - "images", "the image records: digest, size, pixels", &do_images, - "verify", "check the image records against --image-dir", &do_verify, - "image-dir", "the images to verify against", &image_dir, - "languages", "the languages the file holds", &do_languages, - "files", "what a .ocda.db carries besides the abstraction", &do_files, - "report-unknown", "what was read and not understood", &do_report_unknown, - "validate", "check a .ssp against its specification, as text", &do_validate, - "lang", "=<code> the language to read (default: the first)", &lang, - "output", "write the mode's output to a file", &output_file, - "any-version", "read a format major this reader does not know", &any_version, - ); - } catch (GetOptException e) { - stderr.writeln("ocda-read: ", e.msg, "; --help lists the options"); - return 2; - } - - if (helpInfo.helpWanted || args.length < 2) { - defaultGetoptPrinter( - "ocda-read - reads the document abstraction, in any of its forms\n\n" - ~ "Usage: ocda-read [options] <file>.ssp|<file>.ocda.db|<file>.sisupod [lang]\n", - helpInfo.options, - ); - return 0; - } - - string input_file = args[1]; - if (lang.length == 0 && args.length > 2) { lang = args[2]; } - bool lang_named = (lang.length > 0); - - /+ ↓ --validate is about the .ssp as text, so it is asked of a .ssp or not - at all: there is no text form of a database to check, and a pod's - .ssp is checked by taking it out +/ - if (do_validate && sourceOf(input_file) != Source.ssp) { - stderr.writeln("--validate checks a .ssp as text, against the .ssp", - " specification; ", input_file, " is ", sourceName(sourceOf(input_file))); - return 2; - } - - /+ ↓ which language. A .ocda.db and a pod may hold several; where none is - named, every form reads the first in the manifest's order and says - so, rather than the database taking one and the pod refusing +/ - OcdaLanguage[] held; - OcdaDocument doc; - try { - held = languages(input_file); - if (lang.length == 0 && held.length > 0) { lang = held[0].lang; } - if (held.length > 0 && !held.map!(l => l.lang).canFind(lang)) { - stderr.writeln("error: no language '", lang, "' in ", input_file, - "; it holds: ", held.map!(l => l.lang).array.join(" ")); - return 2; - } - doc = load(input_file, lang, any_version); - } catch (Exception e) { - stderr.writeln("Error reading ", input_file, ": ", e.msg); - return 2; - } - - /+ ↓ the version gate: the major part, and refuse what is not known +/ - if (doc.format_version.length == 0) { - stderr.writeln("warning: ", input_file, " declares no format version"); - } else if (doc.formatVersionMajor != ssp_format_major_read) { - if (any_version) { - stderr.writeln("warning: ", input_file, " is format v", doc.format_version, - ", this reader knows v", ssp_format_major_read, ".x; reading anyway"); - } else { - stderr.writeln("error: ", input_file, " is format v", doc.format_version, - ", this reader knows v", ssp_format_major_read, ".x"); - stderr.writeln(" a major version differs: a property may have", - " changed meaning or gone away."); - stderr.writeln(" --any-version reads it regardless."); - return 3; - } - } - - writeln("Read: ", input_file); - if (doc.format_version.length > 0) { - writeln(" Format: v", doc.format_version); - } - if (auto _lang = "language" in doc.source) { - writeln(" Source: ", *_lang, - ("languages" in doc.source) ? " of " ~ doc.source["languages"] : "", - ("digest" in doc.source) ? ", markup digest " ~ doc.source["digest"][0 .. 16] ~ "..." : ""); - } - if (held.length > 1 && !lang_named) { - writeln(" (reading ", lang, "; --lang=<code> for another)"); - } - writeln(" Title: ", doc.meta("title.main")); - writeln(" Author: ", doc.meta("creator.author")); - writeln(" Sections: head=", doc.head.length, - " toc=", doc.toc.length, - " body=", doc.body.length, - " endnotes=", doc.endnotes.length, - " glossary=", doc.glossary.length, - " bibliography=", doc.bibliography.length, - " bookindex=", doc.bookindex.length, - " blurb=", doc.blurb.length, - " tail=", doc.tail.length); - auto image_records = imageRecordCount(doc); - if (image_records > 0) { - writeln(" Images: ", image_records, " record", - (image_records == 1) ? "" : "s"); - } - - if (do_languages) { printLanguages(held); } - if (do_files) { printFiles(input_file); } - if (do_tree) { outputTree(doc, output_file); } - if (do_html) { outputHtmlScroll(doc, output_file); } - if (do_llm_text) { outputLlmBreadcrumbs(doc, output_file); } - if (do_llm_jsonl) { outputLlmJsonl(doc, output_file); } - if (do_images) { outputImages(doc, output_file); } - - int _verify_failures = 0; - if (do_verify) { - if (image_dir.length == 0) { - stderr.writeln("--verify needs --image-dir=<path to the pod's media/image>"); - return 2; - } - _verify_failures = verifyImages(doc, image_dir, output_file); - } - - if (do_report_unknown) { reportUnknown(doc); } - - int _validation_findings = 0; - if (do_validate) { - auto _found = validateSspFile(input_file, ssp_format_major_read); - reportValidation(input_file, _found); - _validation_findings = cast(int) _found.length; - } - - if (!do_tree && !do_html && !do_llm_text && !do_llm_jsonl && !do_images - && !do_verify && !do_report_unknown && !do_validate - && !do_languages && !do_files) { - writeln("\n(this is the summary: --help lists the modes, each adding its output)"); - } - - return (_verify_failures > 0 || _validation_findings > 0) ? 1 : 0; -} - -/+ ↓ the languages, each with the markup it was made from and its .ssp's - digest: the same from every form, the database storing the digest and - the others taking it from the bytes +/ -void printLanguages(OcdaLanguage[] held) { - writeln(); - writeln(" lang source ssp digest"); - foreach (l; held) { - writefln(" %-7s %-31s %s", l.lang, l.source_filename, - (l.ssp_digest.length > 16) ? l.ssp_digest[0 .. 16] ~ "..." : l.ssp_digest); - } -} - -/+ ↓ what a .ocda.db carries besides the abstraction. A .ssp carries - nothing, and a pod's files are its archive entries rather than anything - the abstraction records, so for those it says so +/ -void printFiles(string path) { - writeln(); - if (!isCarrier(path)) { - writeln(" carried files: only a .ocda.db carries files; this is ", - sourceName(sourceOf(path))); - return; - } - auto rows = carried(path); - if (rows.length == 0) { writeln(" carries no files"); return; } - writeln(" carried files:"); - writeln(" role stored n original in the file"); - foreach (r; rows) { - writefln(" %-9s %-7s %-5s %-13s %s", r.role, r.compression, r.count, r.bytes, r.stored); - } -} -#+END_SRC - -** check -*** validate =.ssp= - -#+HEADER: :tangle "../dlang/src/read/validate.d" -#+BEGIN_SRC d -<<doc_header_including_copyright_and_license>> -module read.validate; - -/+ ↓ check a .ssp against the format specification. - . - Written from specs/sisudoc-abstraction-format/ssp-format-specification.md - and from nothing else: the point of it is to be what the specification - says, executed, so that a file can be held against the document rather - than against an implementation. Each check names the section it comes - from, so a disagreement can be taken back to the text. -+/ - -import std.algorithm : canFind, startsWith; -import std.array : array, split; -import std.conv : to; -import std.file : readText; -import std.stdio : writeln, writefln; -import std.string : indexOf, strip; - -struct SspFinding { - size_t line; // 1-based - string section; // the specification section the rule is from - string detail; -} - -/+ ↓ specification section 6.3: the properties, in the order they are - emitted. A reader may rely on the order, so a validator can check it +/ -private immutable string[] property_order = [ - "identifier", "part", "section", "parent", "last_descendant", "ocn_range", - "children", - "ancestors", "ancestors_collapsed", "dom_status", "dom_status_collapsed", - "heading_lev_collapsed", "parent_lev", "dummy", "ocn_off", "is_of_type", - "attrib", "meta_lang", "meta_syntax", "sha256", "image", "indent", - "bullet", "lang", "has", "table_cols", "table_widths", "table_aligns", - "table_header", "code_linenumbers", "stow_link", "segment", "anchor", - "segment_prev", "segment_next", "heading_lev_anchor", "segment_epub", - "segment_html_is", "segment_epub_is", "segment_lv4_is", - "heading_ancestors_text", "lev4_subtoc", "anchor_tag", -]; - -/+ ↓ section 6.1: the sixteen kinds +/ -private immutable string[] kinds = [ - "para", "heading", "toc", "bookindex", "endnote", "code", "bibliography", - "comment", "verse", "glossary", "quote", "poem", "block", "table", - "blurb", "group", -]; - -/+ ↓ section 4: the nine sections, in the order they appear +/ -private immutable string[] section_order = [ - "head", "toc", "body", "endnotes", "glossary", "bibliography", - "bookindex", "blurb", "tail", -]; - -/+ ↓ section 4: the four preamble blocks, in the order they appear +/ -private immutable string[] preamble_order = ["source", "meta", "make", "doc_has"]; - -/+ ↓ section 6.1: the heading levels +/ -private immutable string[] levels = ["A", "B", "C", "D", "1", "2", "3", "4"]; - -private int propertyRank(string name) { - foreach (i, p; property_order) { if (p == name) { return cast(int) i; } } - return -1; -} - -SspFinding[] validateSspFile(string filename, string major_known) { - SspFinding[] found; - auto lines = readText(filename).split("\n"); - - bool seen_banner = false; - bool seen_source_line = false; - string block = ""; // the block currently open, "" when none - int preamble_at = -1; // how far through the preamble order - int section_at = -1; // how far through the section order - bool in_object = false; - int prev_rank = -1; - size_t doc_has_keys = 0; - bool in_text = false; // text lines come after properties - - foreach (i, line; lines) { - size_t no = i + 1; - - /+ ↓ section 2: the version, on the first line +/ - if (no == 1) { - enum banner = "% SiSU Document Abstraction v"; - if (!line.startsWith(banner)) { - found ~= SspFinding(no, "2", "first line is not the version banner"); - } else { - seen_banner = true; - string v = line[banner.length .. $].strip; - auto dot = v.indexOf('.'); - string major = (dot > 0) ? v[0 .. dot] : v; - if (major != major_known) { - found ~= SspFinding(no, "2.1", - "format major v" ~ major ~ ", this validator knows v" ~ major_known); - } - } - continue; - } - if (no == 2) { - if (!line.startsWith("% Source: ")) { - found ~= SspFinding(no, "4", "second line is not \"% Source: \""); - } else { seen_source_line = true; } - continue; - } - - if (line.length == 0) { in_object = false; in_text = false; prev_rank = -1; continue; } - - /+ ↓ section 3: a comment is the file header only +/ - if (line[0] == '%') { - found ~= SspFinding(no, "3", "a comment outside the two header lines"); - continue; - } - - /+ ↓ block open +/ - if (line[0] == '@') { - auto brace = line.indexOf(" {"); - if (brace < 1 || line[$ - 1] != '{') { - found ~= SspFinding(no, "3", "block open is not \"@name {\""); - continue; - } - string name = line[1 .. brace]; - if (preamble_order.canFind(name)) { - int at = 0; - foreach (k, p; preamble_order) { if (p == name) { at = cast(int) k; } } - if (at <= preamble_at) { - found ~= SspFinding(no, "4", "preamble block @" ~ name ~ " out of order"); - } - preamble_at = at; - if (section_at >= 0) { - found ~= SspFinding(no, "4", "preamble block @" ~ name ~ " after a section"); - } - } else if (section_order.canFind(name)) { - int at = 0; - foreach (k, sname; section_order) { if (sname == name) { at = cast(int) k; } } - if (at <= section_at) { - found ~= SspFinding(no, "4", "section @" ~ name ~ " out of order"); - } - section_at = at; - } else { - found ~= SspFinding(no, "4", "@" ~ name ~ " is neither a preamble block nor a section"); - } - block = name; - doc_has_keys = 0; - continue; - } - - if (line[0] == '}') { - if (line.length != 1) { - found ~= SspFinding(no, "3", "block close carries more than \"}\""); - } - /+ ↓ section 5.4: ten keys, always +/ - if (block == "doc_has" && doc_has_keys != 10) { - found ~= SspFinding(no, "5.4", - "@doc_has has " ~ doc_has_keys.to!string ~ " keys, the specification says ten"); - } - block = ""; - in_object = false; - prev_rank = -1; - continue; - } - - /+ ↓ section 5: inside a preamble block, " key: value" +/ - if (preamble_order.canFind(block)) { - if (!line.startsWith(" ") || line.indexOf(": ") < 3) { - found ~= SspFinding(no, "5", "not \" key: value\" inside @" ~ block); - } else if (block == "doc_has") { - doc_has_keys++; - } - continue; - } - - /+ ↓ section 6.1: the object declaration line +/ - if (line[0] == '[') { - auto close = line.indexOf("] "); - if (close < 2) { - found ~= SspFinding(no, "6.1", "declaration is not \"[ocn] ...\""); - continue; - } - string ocn = line[1 .. close]; - foreach (c; ocn) { - if (c < '0' || c > '9') { - found ~= SspFinding(no, "6.1", "ocn \"" ~ ocn ~ "\" is not a number"); - break; - } - } - auto rest = line[close + 2 .. $].split(" "); - if (rest.length == 0 || !kinds.canFind(rest[0])) { - found ~= SspFinding(no, "6.1", - "\"" ~ ((rest.length > 0) ? rest[0] : "") ~ "\" is not one of the sixteen kinds"); - } else if (rest[0] == "heading") { - if (rest.length < 2 || rest[1].length < 2 || rest[1][0] != ':') { - found ~= SspFinding(no, "6.1", "a heading declaration carries no \":level\""); - } else if (!levels.canFind(rest[1][1 .. $])) { - found ~= SspFinding(no, "6.1", "level \"" ~ rest[1][1 .. $] ~ "\" is not one of the eight"); - } - } - in_object = true; - in_text = false; - prev_rank = -1; - continue; - } - - /+ ↓ section 6.3: properties, known and in order +/ - if (line[0] == '.') { - if (!in_object) { - found ~= SspFinding(no, "6", "a property outside an object record"); - continue; - } - if (in_text) { - found ~= SspFinding(no, "6", "a property after the object text began"); - } - auto colon = line.indexOf(": "); - if (colon < 2) { - found ~= SspFinding(no, "3", "property is not \".name: value\""); - continue; - } - string name = line[1 .. colon]; - int rank = propertyRank(name); - if (rank < 0) { - found ~= SspFinding(no, "6.3", "." ~ name ~ " is not one of the " ~ property_order.length.to!string ~ " properties"); - } else { - /+ ↓ the four repeatable properties may recur at the same rank +/ - if (rank < prev_rank) { - found ~= SspFinding(no, "6.3", "." ~ name ~ " is out of the emitted order"); - } - prev_rank = rank; - } - continue; - } - - /+ ↓ section 6.2: text lines +/ - if (line[0] == '|') { - if (!in_object) { - found ~= SspFinding(no, "6.2", "a text line outside an object record"); - } - if (line.length < 2 || line[1] != ' ') { - found ~= SspFinding(no, "6.2", "a text line is not \"| \" then the text"); - } - in_text = true; - continue; - } - - found ~= SspFinding(no, "3", "line of no shape the specification defines"); - } - - if (!seen_banner) { found ~= SspFinding(0, "2", "no version banner"); } - if (!seen_source_line) { found ~= SspFinding(0, "4", "no \"% Source:\" line"); } - return found; -} - -void reportValidation(string filename, SspFinding[] found) { - if (found.length == 0) { - writeln("\nvalid: ", filename, - " conforms to the .ssp specification as this validator reads it"); - return; - } - writefln("\n%d finding%s against the specification:", - found.length, (found.length == 1) ? "" : "s"); - foreach (f; found) { - if (f.line > 0) { - writefln(" line %-7d [%-4s] %s", f.line, f.section, f.detail); - } else { - writefln(" %-13s [%-4s] %s", "(file)", f.section, f.detail); - } - } -} -#+END_SRC - -*** verify images - -#+HEADER: :tangle "../dlang/src/read/verify.d" -#+BEGIN_SRC d -<<doc_header_including_copyright_and_license>> -module read.verify; - -import std.algorithm : sort; -import std.array : array, join; -import std.conv : to; -import std.digest : toHexString; -import std.digest.sha : SHA256, digest; -import std.file : dirEntries, exists, isDir, SpanMode, getSize; -import std.path : baseName, buildPath, extension; -import std.stdio : writeln, File; -import std.string : toLower; -import ocda.model; -import read.images : gatherImageRecords; - -/+ ↓ verification: recompute what the .image records claim, against the - images as they stand in a directory (a pod's media/image). - - ssp-reader --verify --image-dir=<pod>/media/image doc.ssp - - Reports one line per distinct file: - - OK won_benkler_2_1.png - MISMATCH won_benkler_2_1.png sha256 recorded 5BA4... file 1C3D... - ABSENT won_benkler_2_1.png (recorded sha256 5BA4..., not in directory) - PRESENT ffa.png (recorded missing, but the file is there) - EXTRA unused.png (in directory, no record in this .ssp) - - Returns the number of failures. EXTRA is reported but is not a failure: - a pod carries the images of every language version of a document, while - one .ssp covers one language. -+/ -int verifyImages(OcdaDocument doc, string image_dir, string output_file) { - string[] lines; - int failures = 0; - int ok = 0; - int mismatch = 0; - int absent = 0; - int extra = 0; - - if (!(image_dir.exists) || !(image_dir.isDir)) { - writeln("ERROR: image directory not found: ", image_dir); - return 1; - } - auto gathered = gatherImageRecords(doc); - bool[string] recorded; - - foreach (e; gathered.entries) { - recorded[e.name] = true; - string pth = buildPath(image_dir, e.name); - if (!(pth.exists)) { - if (e.missing) { - /+ recorded missing, and missing here too: consistent +/ - ok += 1; - lines ~= "OK " ~ e.name ~ " (recorded missing, absent here)"; - } else { - absent += 1; - failures += 1; - lines ~= "ABSENT " ~ e.name - ~ " (recorded sha256 " ~ e.sha256 ~ ", not in directory)"; - } - continue; - } - if (e.missing) { - mismatch += 1; - failures += 1; - lines ~= "PRESENT " ~ e.name ~ " (recorded missing, but the file is there)"; - continue; - } - auto file_ = readFileStat(pth); - string[] differs; - if (file_.sha256 != e.sha256) { - differs ~= "sha256 recorded " ~ e.sha256 ~ " file " ~ file_.sha256; - } - if (file_.bytes != e.bytes) { - differs ~= "bytes recorded " ~ e.bytes.to!string - ~ " file " ~ file_.bytes.to!string; - } - /+ px is written only where the dimensions could be read, and read here - only for the formats below, so compare only when both are known +/ - if (e.width > 0 && e.height > 0 && file_.width > 0 && file_.height > 0 - && (file_.width != e.width || file_.height != e.height)) { - differs ~= "px recorded " ~ e.width.to!string ~ "x" ~ e.height.to!string - ~ " file " ~ file_.width.to!string ~ "x" ~ file_.height.to!string; - } - if (differs.length > 0) { - mismatch += 1; - failures += 1; - lines ~= "MISMATCH " ~ e.name ~ " " ~ differs.join("; "); - } else { - ok += 1; - lines ~= "OK " ~ e.name; - } - } - - /+ ↓ files in the directory that no record names +/ - string[] in_dir; - foreach (entry; dirEntries(image_dir, SpanMode.shallow)) { - if (!(entry.isFile)) continue; - string nm = entry.name.baseName; - switch (nm.extension.toLower) { - case ".png": case ".jpg": case ".jpeg": case ".gif": - in_dir ~= nm; - break; - default: break; - } - } - in_dir.sort; - foreach (nm; in_dir) { - if (nm !in recorded) { - extra += 1; - lines ~= "EXTRA " ~ nm ~ " (in directory, no record in this .ssp)"; - } - } - - string head = "Verify: " ~ gathered.entries.length.to!string ~ " distinct file" - ~ ((gathered.entries.length == 1) ? "" : "s") - ~ " in " ~ gathered.records.to!string ~ " record" - ~ ((gathered.records == 1) ? "" : "s") - ~ ", against " ~ image_dir; - lines = [head, ""] ~ lines; - lines ~= ""; - lines ~= "verify: " ~ ok.to!string ~ " ok, " - ~ mismatch.to!string ~ " mismatch, " - ~ absent.to!string ~ " absent, " - ~ extra.to!string ~ " extra" - ~ ((failures == 0) ? " PASS" : " FAIL"); - - /+ ↓ write output +/ - if (output_file.length > 0) { - auto f = File(output_file, "w"); - foreach (line; lines) f.writeln(line); - } else { - foreach (line; lines) writeln(line); - } - return failures; -} - -private struct FileStat_ { - string sha256; - ulong bytes; - int width; - int height; -} - -private FileStat_ readFileStat(string pth) { - FileStat_ ret; - auto f = File(pth, "rb"); - ubyte[32] sha = digest!SHA256(f.byChunk(4096 * 1024)); - ret.sha256 = sha.toHexString.to!string; - ret.bytes = pth.getSize; - auto dim = readImageDimensions(pth); - ret.width = dim[0]; - ret.height = dim[1]; - return ret; -} - -/+ ↓ pixel dimensions from the file header, for the formats spine's markup - admits (png, gif, jpg). Header read only, no decoding, and no dependency - beyond phobos. Returns [0, 0] where the header cannot be read -+/ -private int[2] readImageDimensions(string pth) { - int[2] ret = [0, 0]; - ubyte[] head; - try { - auto f = File(pth, "rb"); - head = new ubyte[](64 * 1024); - head = f.rawRead(head); - } catch (Exception ex) { - return ret; - } - if (head.length >= 24 - && head[0 .. 8] == cast(ubyte[])[0x89, 'P', 'N', 'G', 0x0d, 0x0a, 0x1a, 0x0a]) { - /+ png: IHDR width & height, big endian, at 16 and 20 +/ - ret[0] = beInt(head[16 .. 20]); - ret[1] = beInt(head[20 .. 24]); - return ret; - } - if (head.length >= 10 && (head[0 .. 3] == cast(ubyte[])['G', 'I', 'F'])) { - /+ gif: logical screen width & height, little endian, at 6 and 8 +/ - ret[0] = head[6] | (head[7] << 8); - ret[1] = head[8] | (head[9] << 8); - return ret; - } - if (head.length >= 4 && head[0] == 0xff && head[1] == 0xd8) { - /+ jpeg: walk the segments to a start of frame, height then width, - big endian, at offsets 5 and 7 of the segment - +/ - size_t i = 2; - while (i + 9 < head.length) { - if (head[i] != 0xff) { i += 1; continue; } - ubyte marker = head[i + 1]; - if (marker == 0xd8 || marker == 0x01 - || (marker >= 0xd0 && marker <= 0xd7)) { i += 2; continue; } - if (marker == 0xff) { i += 1; continue; } - size_t seg_len = (head[i + 2] << 8) | head[i + 3]; - bool is_sof = ((marker >= 0xc0 && marker <= 0xc3) - || (marker >= 0xc5 && marker <= 0xc7) - || (marker >= 0xc9 && marker <= 0xcb) - || (marker >= 0xcd && marker <= 0xcf)); - if (is_sof) { - ret[1] = (head[i + 5] << 8) | head[i + 6]; - ret[0] = (head[i + 7] << 8) | head[i + 8]; - return ret; - } - if (seg_len < 2) break; - i += 2 + seg_len; - } - } - return ret; -} - -private int beInt(const(ubyte)[] b) { - return cast(int)((b[0] << 24) | (b[1] << 16) | (b[2] << 8) | b[3]); -} -#+END_SRC - -*** report unknown - -#+HEADER: :tangle "../dlang/src/read/report_unknown.d" -#+BEGIN_SRC d -<<doc_header_including_copyright_and_license>> -module read.report_unknown; - -import std.algorithm : sort; -import std.array : array; -import std.stdio : writeln, writefln; -import ocda.model; - -/+ ↓ say what was read and not understood. - This is what makes the reader usable as a check on the format - specification: a reader that ignores what it does not know is - conforming, but one that cannot say what it ignored cannot tell you - whether the specification it was written from was complete. Silence - and understanding look the same from outside. -+/ -void reportUnknown(ref OcdaDocument doc) { - if (!doc.unknown.any) { - writeln("\nread and understood: every property, block and line was known"); - return; - } - writeln("\nread and NOT understood:"); - if (doc.unknown.property.length > 0) { - writeln(" properties ignored (", doc.unknown.property.length, " distinct):"); - foreach (name; doc.unknown.property.keys.sort) { - writefln(" .%-26s %8d", name, doc.unknown.property[name]); - } - } - if (doc.unknown.block.length > 0) { - writeln(" blocks ignored (", doc.unknown.block.length, " distinct):"); - foreach (name; doc.unknown.block.keys.sort) { - writefln(" @%-26s %8d", name, doc.unknown.block[name]); - } - } - if (doc.unknown.line.length > 0) { - writeln(" lines of no known shape, by first character:"); - foreach (c; doc.unknown.line.keys.sort) { - writefln(" %-27s %8d", "\"" ~ c ~ "\"", doc.unknown.line[c]); - } - } - writeln(" (ignoring these is conforming within a major version;", - " the count is so that it is not silent)"); -} -#+END_SRC - -** misc -*** tree - -#+HEADER: :tangle "../dlang/src/read/tree.d" -#+BEGIN_SRC d -<<doc_header_including_copyright_and_license>> -module read.tree; - -import std.conv : to; -import std.stdio : writeln, writef, writefln, File; -import std.array : replicate; -import ocda.model; - -void outputTree(OcdaDocument doc, string output_file) { - string[] lines; - - lines ~= "Document: " ~ doc.meta("title.main"); - if (doc.meta("title.subtitle").length > 0) - lines ~= " " ~ doc.meta("title.subtitle"); - lines ~= "Author: " ~ doc.meta("creator.author"); - lines ~= ""; - - /+ ↓ build OCN -> object lookup for body headings +/ - OcdaObject[int] headings; - foreach (obj; doc.body) { - if (obj.is_a == "heading" && obj.ocn > 0) { - headings[obj.ocn] = obj; - } - } - - /+ ↓ find root heading (from head section, or first body heading) +/ - if (doc.head.length > 0 && doc.head[0].is_a == "heading") { - auto root = doc.head[0]; - lines ~= formatHeading(root, 0); - walkChildren(root, headings, lines, 1); - } - - /+ ↓ write output +/ - if (output_file.length > 0) { - auto f = File(output_file, "w"); - foreach (line; lines) f.writeln(line); - } else { - foreach (line; lines) writeln(line); - } -} - -private void walkChildren(OcdaObject parent, ref OcdaObject[int] headings, ref string[] lines, int depth) { - foreach (child_ocn; parent.children) { - if (auto p = child_ocn in headings) { - lines ~= formatHeading(*p, depth); - walkChildren(*p, headings, lines, depth + 1); - } - } -} - -private string formatHeading(OcdaObject obj, int depth) { - string indent = replicate(" ", depth); - string level_tag = (obj.heading_level.length > 0) ? obj.heading_level : "?"; - string ocn_str = (obj.ocn > 0) ? " [" ~ obj.ocn.to!string ~ "]" : ""; - /+ ↓ strip internal markup from heading text for clean display +/ - string text = stripInternalMarkup(obj.text); - return indent ~ level_tag ~ "~" ~ ocn_str ~ " " ~ text; -} - -private string stripInternalMarkup(string text) { - /+ ↓ simple removal of common internal markers for display +/ - import std.regex : replaceAll, regex; - import std.array : replace; - string result = text; - result = result.replace("⑆*", "").replace("⑆!", "").replace("⑆/", "") - .replace("⑆_", "").replace("⑆^", "").replace("⑆,", "") - .replace("┨", "").replace("┣", "") - .replace("┙", " ").replace("┘", "\n"); - return result; -} -#+END_SRC - -*** html - -#+HEADER: :tangle "../dlang/src/read/html.d" -#+BEGIN_SRC d -<<doc_header_including_copyright_and_license>> -module read.html; - -import std.array : replace, split, join, appender; -import std.conv : to; -import std.stdio : writeln, File; -import std.string : strip, indexOf, startsWith; -import ocda.model; - -void outputHtmlScroll(OcdaDocument doc, string output_file) { - auto buf = appender!(string[]); - - buf ~= htmlHead(doc); - - /+ ↓ title +/ - if (doc.head.length > 0) { - auto title_obj = doc.head[0]; - buf ~= `<div class="title">`; - buf ~= `<h1>` ~ htmlEscape(doc.meta("title.main")) ~ `</h1>`; - if (doc.meta("title.subtitle").length > 0) - buf ~= `<h2 class="subtitle">` ~ htmlEscape(doc.meta("title.subtitle")) ~ `</h2>`; - buf ~= `<p class="author">` ~ htmlEscape(doc.meta("creator.author")) ~ `</p>`; - buf ~= `</div>`; - } - - /+ ↓ table of contents +/ - if (doc.toc.length > 0) { - buf ~= `<div class="toc">`; - buf ~= `<h2>Table of Contents</h2>`; - foreach (obj; doc.toc) { - if (obj.is_a == "toc") { - string text = convertInlineMarkup(obj.text); - int indent = obj.indent_base; - string cls = (indent > 4) ? "toc-sub" : "toc-entry"; - buf ~= `<p class="` ~ cls ~ `">` ~ text ~ `</p>`; - } - } - buf ~= `</div>`; - buf ~= `<hr/>`; - } - - /+ ↓ body +/ - buf ~= `<div class="body">`; - foreach (obj; doc.body) { - buf ~= renderObject(obj); - } - buf ~= `</div>`; - - /+ ↓ endnotes +/ - bool has_endnotes = false; - foreach (obj; doc.endnotes) { - if (obj.is_a == "endnote") { has_endnotes = true; break; } - } - if (has_endnotes) { - buf ~= `<div class="endnotes">`; - buf ~= `<hr/>`; - buf ~= `<h2>Endnotes</h2>`; - foreach (obj; doc.endnotes) { - if (obj.is_a == "endnote") { - string text = convertInlineMarkup(obj.text); - string parent_ref = (obj.parent_ocn > 0) - ? ` <a href="#o` ~ obj.parent_ocn.to!string ~ `">[` ~ obj.parent_ocn.to!string ~ `]</a>` - : ""; - buf ~= `<p class="endnote">` ~ text ~ parent_ref ~ `</p>`; - } - } - buf ~= `</div>`; - } - - /+ ↓ glossary +/ - if (doc.glossary.length > 0) { - buf ~= `<div class="glossary">`; - buf ~= `<hr/>`; - buf ~= `<h2>Glossary</h2>`; - foreach (obj; doc.glossary) { - string text = convertInlineMarkup(obj.text); - buf ~= `<p class="glossary-entry">` ~ text ~ `</p>`; - } - buf ~= `</div>`; - } - - /+ ↓ bookindex +/ - bool has_bookindex = false; - foreach (obj; doc.bookindex) { - if (obj.is_a == "bookindex") { has_bookindex = true; break; } - } - if (has_bookindex) { - buf ~= `<div class="bookindex">`; - buf ~= `<hr/>`; - buf ~= `<h2>Book Index</h2>`; - foreach (obj; doc.bookindex) { - if (obj.is_a == "bookindex") { - string text = convertInlineMarkup(obj.text); - buf ~= `<p class="index-entry">` ~ text ~ `</p>`; - } - } - buf ~= `</div>`; - } - - buf ~= htmlTail(); - - /+ ↓ write output +/ - string[] output = buf.data; - if (output_file.length > 0) { - auto f = File(output_file, "w"); - foreach (line; output) f.writeln(line); - } else { - foreach (line; output) writeln(line); - } -} - -private string[] renderObject(OcdaObject obj) { - /+ ↓ markup commentary is carried by the abstraction and is not output. - Rendering it would put the author notes into the document, which - spine itself does not do - +/ - if (!obj.renders) { return []; } - string[] lines; - string text = convertInlineMarkup(obj.text); - string ocn_anchor = (obj.ocn > 0 && !obj.ocn_off) - ? ` id="o` ~ obj.ocn.to!string ~ `"` - : ""; - string ocn_display = (obj.ocn > 0 && !obj.ocn_off) - ? `<span class="ocn">` ~ obj.ocn.to!string ~ `</span>` - : ""; - - switch (obj.is_a) { - case "heading": - string tag = headingTag(obj.heading_level); - string seg_anchor = (obj.segment.length > 0) - ? ` id="` ~ htmlEscape(obj.segment) ~ `"` - : ocn_anchor; - lines ~= `<` ~ tag ~ seg_anchor ~ `>` ~ text ~ `</` ~ tag ~ `>`; - break; - case "para": - string indent_cls = ""; - if (obj.indent_base > 0) indent_cls = ` class="indent-` ~ obj.indent_base.to!string ~ `"`; - else if (obj.bullet) indent_cls = ` class="bullet"`; - lines ~= `<p` ~ ocn_anchor ~ indent_cls ~ `>` ~ text ~ ocn_display ~ `</p>`; - break; - case "code": - lines ~= `<pre` ~ ocn_anchor ~ `><code>` ~ htmlEscape(obj.text) ~ `</code></pre>`; - break; - case "table": - lines ~= renderTable(obj, ocn_anchor); - break; - case "group": - lines ~= `<div` ~ ocn_anchor ~ ` class="group">` ~ text ~ `</div>`; - break; - case "block": - lines ~= `<pre` ~ ocn_anchor ~ ` class="block">` ~ htmlEscape(obj.text) ~ `</pre>`; - break; - case "verse": - lines ~= `<p` ~ ocn_anchor ~ ` class="verse">` ~ text.replace("\n", "<br/>\n") ~ ocn_display ~ `</p>`; - break; - case "quote": - lines ~= `<blockquote` ~ ocn_anchor ~ `>` ~ text ~ ocn_display ~ `</blockquote>`; - break; - default: - if (text.length > 0) - lines ~= `<p` ~ ocn_anchor ~ `>` ~ text ~ ocn_display ~ `</p>`; - break; - } - return lines; -} - -private string headingTag(string level) { - switch (level) { - case "A": return "h1"; - case "B": return "h2"; - case "C": return "h3"; - case "D": return "h4"; - case "1": return "h3"; - case "2": return "h4"; - case "3": return "h5"; - case "4": return "h6"; - default: return "h4"; - } -} - -private string renderTable(OcdaObject obj, string ocn_anchor) { - string[] lines; - lines ~= `<table` ~ ocn_anchor ~ `>`; - /+ ↓ table content uses ┊ as cell separator, rows separated by newlines +/ - auto rows = obj.text.split("\n"); - bool first_row = obj.table_header; - foreach (row; rows) { - if (row.strip.length == 0) continue; - string tag = first_row ? "th" : "td"; - lines ~= `<tr>`; - foreach (cell; row.split("┊")) { - lines ~= `<` ~ tag ~ `>` ~ convertInlineMarkup(cell.strip) ~ `</` ~ tag ~ `>`; - } - lines ~= `</tr>`; - first_row = false; - } - lines ~= `</table>`; - return lines.join("\n"); -} - -/+ ↓ convert spine internal Unicode markup to HTML +/ -string convertInlineMarkup(string text) { - if (text.length == 0) return ""; - string result = htmlEscape(text); - - /+ ↓ font face markers: ⑆*bold⑆*, ⑆/italic⑆/, etc. - Pattern: ⑆X┨content┣X or ⑆Xcontent⑆X - The ┨┣ form is used within heading content +/ - - /+ ↓ bold +/ - result = replaceInlineMarker(result, "⑆!", "!", "<b>", "</b>"); - result = replaceInlineMarkerBracketed(result, "⑆!", "┨", "┣!", "<b>", "</b>"); - - /+ ↓ emphasis +/ - result = replaceInlineMarker(result, "⑆*", "*", "<em>", "</em>"); - result = replaceInlineMarkerBracketed(result, "⑆*", "┨", "┣*", "<em>", "</em>"); - - /+ ↓ italic +/ - result = replaceInlineMarker(result, "⑆/", "/", "<i>", "</i>"); - result = replaceInlineMarkerBracketed(result, "⑆/", "┨", "┣/", "<i>", "</i>"); - - /+ ↓ underscore +/ - result = replaceInlineMarker(result, "⑆_", "_", "<u>", "</u>"); - result = replaceInlineMarkerBracketed(result, "⑆_", "┨", "┣_", "<u>", "</u>"); - - /+ ↓ superscript +/ - result = replaceInlineMarker(result, "⑆^", "^", "<sup>", "</sup>"); - result = replaceInlineMarkerBracketed(result, "⑆^", "┨", "┣^", "<sup>", "</sup>"); - - /+ ↓ subscript +/ - result = replaceInlineMarker(result, "⑆,", ",", "<sub>", "</sub>"); - result = replaceInlineMarkerBracketed(result, "⑆,", "┨", "┣,", "<sub>", "</sub>"); - - /+ ↓ monospace +/ - result = replaceInlineMarker(result, "⑆■", "■", "<code>", "</code>"); - - /+ ↓ links: ┥link text┝┤url├ +/ - result = replaceLinks(result); - - /+ ↓ footnote markers: 【N text】 +/ - result = replaceFootnotes(result); - - /+ ↓ line breaks +/ - result = result.replace("┘", "<br/>"); - result = result.replace("┙", "<br/>"); - - /+ ↓ non-breaking spaces +/ - result = result.replace("░", " "); - - /+ ↓ bullet markers +/ - result = result.replace("● ", "• "); - - /+ ↓ page break markers +/ - result = result.replace("┼", "").replace("┿", "").replace("╂", ""); - - return result; -} - -private string replaceInlineMarker(string text, string open_mark, string close_suffix, string html_open, string html_close) { - /+ ↓ replace ⑆Xcontent⑆X pattern (no bracket form) +/ - string result = text; - while (true) { - auto open_pos = result.indexOf(open_mark); - if (open_pos < 0) break; - auto after_open = open_pos + open_mark.length; - /+ ↓ look for closing ⑆X +/ - string close_mark = "⑆" ~ close_suffix; - auto close_pos = indexOf(result[after_open .. $], close_mark); - if (close_pos < 0) break; - close_pos += after_open; - string content = result[after_open .. close_pos]; - result = result[0 .. open_pos] ~ html_open ~ content ~ html_close - ~ result[close_pos + close_mark.length .. $]; - } - return result; -} - -private string replaceInlineMarkerBracketed(string text, string open_mark, string bracket_open, string bracket_close, string html_open, string html_close) { - /+ ↓ replace ⑆X┨content┣X pattern (bracketed form) +/ - string result = text; - string full_open = open_mark ~ bracket_open; - while (true) { - auto open_pos = result.indexOf(full_open); - if (open_pos < 0) break; - auto after_open = open_pos + full_open.length; - auto close_pos = indexOf(result[after_open .. $], bracket_close); - if (close_pos < 0) break; - close_pos += after_open; - string content = result[after_open .. close_pos]; - result = result[0 .. open_pos] ~ html_open ~ content ~ html_close - ~ result[close_pos + bracket_close.length .. $]; - } - return result; -} - -private string replaceLinks(string text) { - /+ ↓ replace ┥link text┝┤url├ with <a href="url">link text</a> +/ - string result = text; - while (true) { - auto link_open = result.indexOf("┥"); - if (link_open < 0) break; - auto link_mid = indexOf(result[link_open .. $], "┝┤"); - if (link_mid < 0) break; - link_mid += link_open; - auto link_close = indexOf(result[link_mid .. $], "├"); - if (link_close < 0) break; - link_close += link_mid; - string link_text = result[link_open + "┥".length .. link_mid]; - string url = result[link_mid + "┝┤".length .. link_close]; - /+ ↓ handle internal links (¤ prefix) +/ - if (url.length > 0 && url[0] == '¤') { - url = url[("¤").length .. $]; - } - /+ ↓ handle OCN-only links (#N) +/ - if (url.length > 0 && url[0] == '#') { - url = "#o" ~ url[1 .. $]; - } - result = result[0 .. link_open] - ~ `<a href="` ~ url ~ `">` ~ link_text ~ `</a>` - ~ result[link_close + "├".length .. $]; - } - return result; -} - -private string replaceFootnotes(string text) { - /+ ↓ replace 【N content】 with superscript note number +/ - string result = text; - while (true) { - auto fn_open = result.indexOf("【"); - if (fn_open < 0) break; - auto fn_close = indexOf(result[fn_open .. $], "】"); - if (fn_close < 0) break; - fn_close += fn_open; - string fn_content = result[fn_open + "【".length .. fn_close]; - /+ ↓ extract note number (first token before space) +/ - auto space_pos = fn_content.indexOf(' '); - string note_num = (space_pos > 0) ? fn_content[0 .. space_pos] : fn_content; - result = result[0 .. fn_open] - ~ `<sup class="fn">` ~ note_num ~ `</sup>` - ~ result[fn_close + "】".length .. $]; - } - return result; -} - -string htmlEscape(string text) { - return text.replace("&", "&").replace("<", "<").replace(">", ">") - .replace("\"", """); -} - -private string[] htmlHead(OcdaDocument doc) { - string[] lines; - lines ~= `<!DOCTYPE html>`; - lines ~= `<html>`; - lines ~= `<head>`; - lines ~= `<meta charset="utf-8"/>`; - lines ~= `<title>` ~ htmlEscape(doc.meta("title.main")) ~ `</title>`; - lines ~= `<style>`; - lines ~= ` body { max-width: 48em; margin: 2em auto; padding: 0 1em; font-family: serif; line-height: 1.5; }`; - lines ~= ` .title { text-align: center; margin-bottom: 2em; }`; - lines ~= ` .subtitle { font-weight: normal; font-style: italic; }`; - lines ~= ` .ocn { float: right; font-size: 0.7em; color: #999; }`; - lines ~= ` .toc { margin-bottom: 2em; }`; - lines ~= ` .toc-entry { margin: 0.2em 0; }`; - lines ~= ` .toc-sub { margin: 0.2em 0 0.2em 2em; }`; - lines ~= ` .indent-1 { margin-left: 2em; }`; - lines ~= ` .indent-2 { margin-left: 4em; }`; - lines ~= ` .indent-3 { margin-left: 6em; }`; - lines ~= ` .bullet { margin-left: 2em; }`; - lines ~= ` .bullet:before { content: "\2022 "; }`; - lines ~= ` .verse { white-space: pre-wrap; font-style: italic; }`; - lines ~= ` .group { margin: 1em 2em; }`; - lines ~= ` .block { margin: 1em 2em; background: #f8f8f8; padding: 1em; }`; - lines ~= ` pre code { display: block; background: #f4f4f4; padding: 1em; overflow-x: auto; }`; - lines ~= ` table { border-collapse: collapse; margin: 1em 0; }`; - lines ~= ` th, td { border: 1px solid #ccc; padding: 0.4em 0.8em; }`; - lines ~= ` th { background: #f0f0f0; }`; - lines ~= ` .endnote { font-size: 0.9em; }`; - lines ~= ` .fn { color: #666; }`; - lines ~= ` .index-entry { font-size: 0.9em; }`; - lines ~= ` blockquote { margin: 1em 2em; padding-left: 1em; border-left: 3px solid #ccc; }`; - lines ~= `</style>`; - lines ~= `</head>`; - lines ~= `<body>`; - return lines; -} - -private string[] htmlTail() { - return [`</body>`, `</html>`]; -} -#+END_SRC - -*** images - -#+HEADER: :tangle "../dlang/src/read/images.d" -#+BEGIN_SRC d -<<doc_header_including_copyright_and_license>> -module read.images; - -import std.array : join; -import std.conv : to; -import std.stdio : writeln, File; -import ocda.model; - -/+ ↓ the .image records of a document, gathered per distinct file, with the - objects referencing each file, in order of first appearance +/ -struct ImageEntry { - string name; - string sha256; - ulong bytes; - int width; - int height; - bool missing; - string[] refs; - string[] other_sha256; // same filename, differing digest -} - -struct ImageRecords { - int records; // .image lines, an image referenced twice counts twice - ImageEntry[] entries; // distinct files -} - -ImageRecords gatherImageRecords(OcdaDocument doc) { - ImageRecords ret; - size_t[string] seen; - foreach (section; sectionNames) { - auto objs = doc.sectionByName(section); - if (objs is null) continue; - foreach (obj; *objs) { - foreach (img; obj.images) { - ret.records += 1; - string _ref = section ~ "[" ~ obj.ocn.to!string ~ "]"; - if (auto at = img.name in seen) { - if (ret.entries[*at].sha256 != img.sha256 - && img.sha256.length > 0 - && ret.entries[*at].sha256.length > 0) { - ret.entries[*at].other_sha256 ~= img.sha256; - } - ret.entries[*at].refs ~= _ref; - } else { - ImageEntry e_; - e_.name = img.name; - e_.sha256 = img.sha256; - e_.bytes = img.bytes; - e_.width = img.width; - e_.height = img.height; - e_.missing = img.missing; - e_.refs = [_ref]; - seen[img.name] = ret.entries.length; - ret.entries ~= e_; - } - } - } - } - return ret; -} - -/+ ↓ image manifest, line oriented and greppable: - - <sha256> <bytes> <px> <filename> refs: body[97] body[412] - MISSING <filename> refs: body[840] - - The point of the record in the .ssp is verification (is the image - shipped in this pod the image the abstraction was built from), so this - output is the shape a verifier wants: one line per distinct file. -+/ -void outputImages(OcdaDocument doc, string output_file) { - auto gathered = gatherImageRecords(doc); - string[] lines; - int missing = 0; - foreach (e; gathered.entries) { - if (e.missing) missing += 1; - } - lines ~= "Images: " ~ gathered.records.to!string ~ " record" - ~ ((gathered.records == 1) ? "" : "s") - ~ ", " ~ gathered.entries.length.to!string ~ " distinct file" - ~ ((gathered.entries.length == 1) ? "" : "s") - ~ ", " ~ missing.to!string ~ " missing"; - lines ~= ""; - foreach (e; gathered.entries) { - string px = (e.width > 0 && e.height > 0) - ? e.width.to!string ~ "x" ~ e.height.to!string : "-"; - lines ~= (e.missing) - ? "MISSING" ~ " " ~ e.name ~ " refs: " ~ e.refs.join(" ") - : e.sha256 ~ " " ~ e.bytes.to!string ~ " " ~ px - ~ " " ~ e.name ~ " refs: " ~ e.refs.join(" "); - /+ ↓ one filename, more than one digest: the .ssp is inconsistent, - or two distinct files share a name +/ - foreach (other; e.other_sha256) { - lines ~= " WARNING differing digest for same filename: " ~ other; - } - } - - /+ ↓ write output +/ - if (output_file.length > 0) { - auto f = File(output_file, "w"); - foreach (line; lines) f.writeln(line); - } else { - foreach (line; lines) writeln(line); - } -} - -/+ ↓ count of .image records in a document, for the parse summary +/ -int imageRecordCount(OcdaDocument doc) { - return gatherImageRecords(doc).records; -} - -package immutable string[] sectionNames = [ - "head", "toc", "body", "endnotes", - "glossary", "bibliography", "bookindex", "blurb", -]; -#+END_SRC - -** llm -*** llm text - -#+HEADER: :tangle "../dlang/src/read/llm_text.d" -#+BEGIN_SRC d -<<doc_header_including_copyright_and_license>> -module read.llm_text; - -import std.array : replace, split, join; -import std.conv : to; -import std.stdio : writeln, File; -import std.string : strip, indexOf, stripRight; -import ocda.model; - -/+ ↓ plain structured text with heading breadcrumbs for direct LLM consumption - format: - ## Document Title > Chapter > Section > [OCN] - - Clean paragraph text with inline markup stripped. -+/ -void outputLlmBreadcrumbs(OcdaDocument doc, string output_file) { - string[] lines; - string doc_title = doc.meta("title.main"); - - /+ ↓ document header +/ - lines ~= "# " ~ doc_title; - if (doc.meta("title.subtitle").length > 0) - lines ~= "# " ~ doc.meta("title.subtitle"); - lines ~= "# " ~ doc.meta("creator.author"); - lines ~= ""; - - /+ ↓ build heading text lookup from ancestors +/ - string[int] heading_text; - foreach (obj; doc.head) { - if (obj.is_a == "heading" && obj.ocn > 0) - heading_text[obj.ocn] = stripMarkup(obj.text); - } - foreach (obj; doc.body) { - if (obj.is_a == "heading") - heading_text[obj.ocn] = stripMarkup(obj.text); - } - - /+ ↓ body objects +/ - string current_breadcrumb = ""; - foreach (obj; doc.body) { - if (obj.is_a == "heading") { - current_breadcrumb = buildBreadcrumb(doc_title, obj.heading_ancestors_text, obj); - lines ~= ""; - lines ~= "## " ~ current_breadcrumb; - lines ~= ""; - continue; - } - - /+ ↓ markup commentary is not content +/ - if (!obj.renders) { continue; } - - string text = stripMarkup(obj.text); - if (text.strip.length == 0) continue; - - string ocn_ref = (obj.ocn > 0 && !obj.ocn_off) - ? " [" ~ obj.ocn.to!string ~ "]" - : ""; - - switch (obj.is_a) { - case "para": - lines ~= text ~ ocn_ref; - lines ~= ""; - break; - case "code": - lines ~= "```"; - lines ~= obj.text; /+ preserve raw text for code +/ - lines ~= "```"; - lines ~= ""; - break; - case "table": - lines ~= text ~ ocn_ref; - lines ~= ""; - break; - case "verse": - lines ~= text ~ ocn_ref; - lines ~= ""; - break; - case "quote": - lines ~= "> " ~ text ~ ocn_ref; - lines ~= ""; - break; - case "group": - case "block": - lines ~= text ~ ocn_ref; - lines ~= ""; - break; - default: - if (text.length > 0) { - lines ~= text ~ ocn_ref; - lines ~= ""; - } - break; - } - } - - /+ ↓ write output +/ - if (output_file.length > 0) { - auto f = File(output_file, "w"); - foreach (line; lines) f.writeln(line); - } else { - foreach (line; lines) writeln(line); - } -} - -private string buildBreadcrumb(string doc_title, string ancestors_text, OcdaObject obj) { - /+ ↓ ancestors_text is pipe-separated: "Title|Part||Chapter|||" +/ - string[] parts; - if (ancestors_text.length > 0) { - foreach (part; ancestors_text.split("|")) { - string clean = stripMarkup(part).strip; - if (clean.length > 0 && clean != doc_title) { - /+ ↓ truncate very long heading text for breadcrumb +/ - if (clean.length > 60) - clean = clean[0 .. 57] ~ "..."; - parts ~= clean; - } - } - } - /+ ↓ add the current heading itself +/ - string heading_text = stripMarkup(obj.text).strip; - if (heading_text.length > 60) - heading_text = heading_text[0 .. 57] ~ "..."; - - string ocn_ref = (obj.ocn > 0) ? " [" ~ obj.ocn.to!string ~ "]" : ""; - - if (parts.length > 0) - return doc_title ~ " > " ~ parts.join(" > ") ~ " > " ~ heading_text ~ ocn_ref; - else - return doc_title ~ " > " ~ heading_text ~ ocn_ref; -} - -/+ ↓ strip all internal Unicode markup to produce clean plain text +/ -string stripMarkup(string text) { - if (text.length == 0) return ""; - string r = text; - - /+ ↓ links: ┥link text┝┤url├ -> link text +/ - while (true) { - auto lo = r.indexOf("┥"); - if (lo < 0) break; - auto lm = indexOf(r[lo .. $], "┝┤"); - if (lm < 0) break; - lm += lo; - auto lc = indexOf(r[lm .. $], "├"); - if (lc < 0) break; - lc += lm; - string link_text = r[lo + "┥".length .. lm]; - r = r[0 .. lo] ~ link_text ~ r[lc + "├".length .. $]; - } - - /+ ↓ footnotes: 【N content】 -> (remove entirely for clean text) +/ - while (true) { - auto fo = r.indexOf("【"); - if (fo < 0) break; - auto fc = indexOf(r[fo .. $], "】"); - if (fc < 0) break; - fc += fo; - r = r[0 .. fo] ~ r[fc + "】".length .. $]; - } - - /+ ↓ font face markers +/ - r = r.replace("⑆*", "").replace("⑆!", "").replace("⑆/", "") - .replace("⑆_", "").replace("⑆^", "").replace("⑆,", "") - .replace("⑆■", "") - .replace("┨", "").replace("┣*", "").replace("┣!", "") - .replace("┣/", "").replace("┣_", "").replace("┣^", "") - .replace("┣,", "").replace("┣", ""); - - /+ ↓ line breaks and special chars +/ - r = r.replace("┘", "\n").replace("┙", " ").replace("┚", "\n"); - r = r.replace("░", " "); - r = r.replace("● ", "- "); - r = r.replace("┼", "").replace("┿", "").replace("╂", ""); - - /+ ↓ table cell separators +/ - r = r.replace("┊", " | "); - - /+ ↓ clean up multiple spaces +/ - while (r.indexOf(" ") >= 0) - r = r.replace(" ", " "); - - return r.strip; -} -#+END_SRC - -*** llm jsonl - -#+HEADER: :tangle "../dlang/src/read/llm_jsonl.d" -#+BEGIN_SRC d -<<doc_header_including_copyright_and_license>> -module read.llm_jsonl; - -import std.array : split, join, replace; -import std.conv : to; -import std.stdio : writeln, File; -import std.string : strip, indexOf, stripRight; -import ocda.model; -import read.llm_text : stripMarkup; - -/+ ↓ JSONL output with embeddings-ready chunks for RAG pipelines - one JSON object per line, each containing: - doc, ocn, path (breadcrumb), type, text (clean), - images (only where the object embeds one, so that a chunk whose - text reads around a figure carries the figure's identity), - index_terms (from bookindex entries associated with this object) -+/ -void outputLlmJsonl(OcdaDocument doc, string output_file) { - string[] lines; - string doc_id = doc.meta("title.main"); - string doc_author = doc.meta("creator.author"); - - /+ ↓ build bookindex -> OCN mapping for index_terms - bookindex objects in the .ssp have parent_ocn linking to body objects - but the index terms are in the text of bookindex objects, not - directly attached to body objects. We use a simpler approach: - headings carry heading_ancestors_text which gives context. +/ - - /+ ↓ track current heading path +/ - string current_path = ""; - - foreach (obj; doc.body) { - if (obj.is_a == "heading") { - current_path = buildPath(doc_id, obj.heading_ancestors_text, obj); - /+ ↓ emit heading as a chunk too +/ - string heading_text = stripMarkup(obj.text); - if (heading_text.strip.length > 0) { - lines ~= jsonLine(doc_id, doc_author, obj.ocn, current_path, "heading", heading_text, obj.images); - } - continue; - } - - /+ ↓ markup commentary is not content, and least of all is it a chunk - to embed: it would be indexed as though the author had written it - into the document - +/ - if (!obj.renders) { continue; } - - string text = stripMarkup(obj.text); - if (text.strip.length == 0) continue; - - lines ~= jsonLine(doc_id, doc_author, obj.ocn, current_path, obj.is_a, text, obj.images); - } - - /+ ↓ write output +/ - if (output_file.length > 0) { - auto f = File(output_file, "w"); - foreach (line; lines) f.writeln(line); - } else { - foreach (line; lines) writeln(line); - } -} - -private string jsonLine(string doc_id, string doc_author, int ocn, string path, string type, string text, OcdaImage[] images) { - return `{"doc":` ~ jsonStr(doc_id) - ~ `,"author":` ~ jsonStr(doc_author) - ~ `,"ocn":` ~ ocn.to!string - ~ `,"path":` ~ jsonStr(path) - ~ `,"type":` ~ jsonStr(type) - ~ `,"text":` ~ jsonStr(text) - ~ jsonImages(images) - ~ `}`; -} - -private string jsonImages(OcdaImage[] images) { - /+ ↓ omitted entirely where the object embeds no image +/ - if (images.length == 0) return ""; - string[] items; - foreach (img; images) { - string item = `{"name":` ~ jsonStr(img.name); - if (img.missing) { - item ~= `,"missing":true`; - } else { - item ~= `,"sha256":` ~ jsonStr(img.sha256) - ~ `,"bytes":` ~ img.bytes.to!string; - if (img.width > 0 && img.height > 0) { - item ~= `,"width":` ~ img.width.to!string - ~ `,"height":` ~ img.height.to!string; - } - } - items ~= item ~ `}`; - } - return `,"images":[` ~ items.join(",") ~ `]`; -} - -private string buildPath(string doc_title, string ancestors_text, OcdaObject obj) { - string[] parts; - if (ancestors_text.length > 0) { - foreach (part; ancestors_text.split("|")) { - string clean = stripMarkup(part).strip; - if (clean.length > 0 && clean != doc_title) { - parts ~= clean; - } - } - } - string heading_text = stripMarkup(obj.text).strip; - if (parts.length > 0) - return parts.join(" > ") ~ " > " ~ heading_text; - else - return heading_text; -} - -private string jsonStr(string s) { - /+ ↓ JSON string escaping +/ - string r = s; - r = r.replace(`\`, `\\`); - r = r.replace(`"`, `\"`); - r = r.replace("\n", `\n`); - r = r.replace("\r", `\r`); - r = r.replace("\t", `\t`); - return `"` ~ r ~ `"`; -} -#+END_SRC - -* test -** test agree - -#+HEADER: :tangle "../dlang/test/agree.d" -#+BEGIN_SRC d -<<doc_header_including_copyright_and_license>> -/++ - Do the two loaders agree? - - The library's whole purpose is that a writer cannot tell which - serialisation it was handed. That is only true if loading a document's - .ssp and loading its .ocda.db give the same model, so this asks. - - Usage: agree <doc>.ssp <doc>.ocda.db [lang] -+/ -import std.conv : to; -import std.stdio; -import ocda.model; -import ocda.load; - -int main(string[] args) { - if (args.length < 3) { - stderr.writeln("usage: agree <doc>.ssp <doc>.ocda.db [lang]"); - return 2; - } - string lang = (args.length > 3) ? args[3] : ""; - OcdaDocument a, b; - try { - a = load(args[1]); - b = load(args[2], lang); - } catch (Exception e) { - stderr.writeln("error: ", e.msg); - return 2; - } - - int differences = 0; - void differ(string what, string from_ssp, string from_db) { - writeln(" differ: ", what); - writeln(" .ssp: ", from_ssp); - writeln(" .ocda.db: ", from_db); - differences++; - } - - if (a.format_version != b.format_version) { - differ("format_version", a.format_version, b.format_version); - } - if (a.source_filename != b.source_filename) { - differ("source_filename", a.source_filename, b.source_filename); - } - - /+ ↓ the sections, object by object. Position within a section is the - identity to compare on: an ocn is not unique and 0 is shared +/ - static immutable sections = ["head", "toc", "body", "endnotes", "glossary", - "bibliography", "bookindex", "blurb", "tail"]; - foreach (name; sections) { - auto sa = a.sectionByName(name); - auto sb = b.sectionByName(name); - if (sa is null || sb is null) { continue; } - if ((*sa).length != (*sb).length) { - differ(name ~ " object count", (*sa).length.to!string, (*sb).length.to!string); - continue; - } - foreach (i, oa; *sa) { - auto ob = (*sb)[i]; - string at = name ~ "[" ~ i.to!string ~ "]"; - if (oa.ocn != ob.ocn) { differ(at ~ " ocn", oa.ocn.to!string, ob.ocn.to!string); } - if (oa.is_a != ob.is_a) { differ(at ~ " is_a", oa.is_a, ob.is_a); } - if (oa.is_of_type != ob.is_of_type) { differ(at ~ " is_of_type", oa.is_of_type, ob.is_of_type); } - if (oa.part != ob.part) { differ(at ~ " part", oa.part, ob.part); } - if (oa.parent_ocn != ob.parent_ocn) { differ(at ~ " parent", oa.parent_ocn.to!string, ob.parent_ocn.to!string); } - if (oa.heading_level != ob.heading_level) { differ(at ~ " level", oa.heading_level, ob.heading_level); } - if (oa.sha256 != ob.sha256) { differ(at ~ " sha256", oa.sha256, ob.sha256); } - if (oa.text != ob.text) { differ(at ~ " text", oa.text, ob.text); } - if (oa.ancestors != ob.ancestors) { - differ(at ~ " ancestors", oa.ancestors.to!string, ob.ancestors.to!string); - } - if (oa.images.length != ob.images.length) { - differ(at ~ " image count", oa.images.length.to!string, ob.images.length.to!string); - } - if (differences > 12) { writeln(" ... stopping after 12"); return 1; } - } - } - - if (differences == 0) { - writeln("agree: the two loaders give the same model"); - return 0; - } - writeln(differences, " difference(s)"); - return 1; -} -#+END_SRC - -** test dump - -#+HEADER: :tangle "../dlang/test/dump.d" -#+BEGIN_SRC d -<<doc_header_including_copyright_and_license>> -/++ - Write a loaded abstraction out canonically. - - The oracle for everything that claims to implement this model: load a - .ssp or a .ocda.db and print the model, deterministically, so that two - implementations can be compared with diff. - - Usage: ocda-dump [--unknown] <file>.ssp|<file>.ocda.db [lang] -+/ -import std.stdio; -import ocda.model; -import ocda.load; -import ocda.dump; - -int main(string[] args) { - bool unknown = false; - string[] rest; - foreach (a; args[1 .. $]) { - if (a == "--unknown") { unknown = true; } else { rest ~= a; } - } - if (rest.length < 1) { - stderr.writeln("usage: ocda-dump [--unknown] <file>.ssp|<file>.ocda.db [lang]"); - return 2; - } - string lang = (rest.length > 1) ? rest[1] : ""; - OcdaDocument doc; - try { - doc = load(rest[0], lang); - } catch (Exception e) { - stderr.writeln("error: ", e.msg); - return 2; - } - write(unknown ? dumpUnknown(doc) : dumpDocument(doc)); - return 0; -} -#+END_SRC - -* config -** nix flake.nix :flake: - -#+HEADER: :tangle ../dlang/flake.nix -#+HEADER: :noweb yes -#+BEGIN_SRC nix -{ - description = "ocda-lib: the object-centric document abstraction as a D library"; - inputs.nixpkgs.url = "<<nixpkgs_unstable>>"; - outputs = { self, nixpkgs }: - let - system = "x86_64-linux"; # adjust for aarch64-linux / darwin - pkgs = import nixpkgs { inherit system; }; - /* A library, so the library itself is not a build product: a program - that uses it compiles its modules alongside its own, as the writer's - flake does, or takes it as a dub dependency. What is built here are - programs over it: the reader, and the two that put the library to - the question - the agreement check, which loads a document's .ssp - and its .ocda.db and compares the two models, and the dump, which - writes a loaded model out canonically so that an implementation in - another language can be diffed against this one. - . - `mains` is what is compiled besides the library: the checks are one - file each in test/, the reader is src/read/. - */ - mkTool = { tool, binary, what, mains, compiler, compilerBin }: - pkgs.stdenv.mkDerivation { - pname = "ocda-lib-${tool}"; - version = "0.1.0"; - src = self; - nativeBuildInputs = [ compiler ]; - buildInputs = [ pkgs.sqlite pkgs.zstd ]; - buildPhase = '' - runHook preBuild - export HOME=$TMPDIR - ${compilerBin} -I=src -of=${tool} src/ocda/*.d ${mains} -L-lsqlite3 -L-lzstd - runHook postBuild - ''; - installPhase = '' - runHook preInstall - mkdir -p $out/bin - install -m755 ${tool} $out/bin/${binary} - runHook postInstall - ''; - meta = { - description = what; - mainProgram = binary; - }; - }; - ldc = { compiler = pkgs.ldc; compilerBin = "ldmd2"; }; - ocda-read = mkTool (ldc // { - tool = "read"; binary = "ocda-read"; mains = "src/read/*.d"; - what = "reads the abstraction in any of its forms and says what is in it"; - }); - ocda-agree = mkTool (ldc // { - tool = "agree"; binary = "ocda-agree"; mains = "test/agree.d"; - what = "checks that the .ssp and .ocda.db loaders give one model"; - }); - ocda-dump = mkTool (ldc // { - tool = "dump"; binary = "ocda-dump"; mains = "test/dump.d"; - what = "writes a loaded abstraction out canonically, as an oracle"; - }); - in { - packages.${system} = { - /* All three in one bin/, so that one build leaves every program in - one result link. Built separately into the same link, each build - replaced the one before, and a script looking for ocda-dump beside - ocda-agree found nothing. - */ - default = pkgs.symlinkJoin { - name = "ocda-lib-programs-0.1.0"; - paths = [ ocda-read ocda-agree ocda-dump ]; - }; - inherit ocda-read ocda-agree ocda-dump; - }; - - devShells.${system}.default = pkgs.mkShell { - packages = [ pkgs.ldc pkgs.dub pkgs.dtools pkgs.sqlite pkgs.zstd ]; - }; - }; -} -#+END_SRC - -** README.md - -#+HEADER: :tangle ../dlang/README.md -#+HEADER: :noweb yes -#+BEGIN_SRC md -# ocda-lib, in D - -The abstraction's model and both its loaders, as a dub library. - -## Modules - -| | | -|---|---| -| `ocda.model` | the document, the object, the eight sections. Imports nothing | -| `ocda.load_ssp` | text -> model | -| `ocda.load_db` | sqlite -> model | -| `ocda.load` | what a path is, and loading it | -| `ocda.inventory` | what a file holds beyond the one document: its languages (every form alike), and what a `.ocda.db` carries | -| `ocda.dump` | the model written out canonically, as an oracle | -| `ocda.sqlite` | the sqlite3 C API, declared rather than depended upon | -| `read.*` | `ocda-read`, the reader: in `src/read/`, and *not* part of the library (below) | - -`ocda.model` importing nothing is the property that matters: a writer -depends on the model and on neither loader. - -## Use - -```json -"dependencies": { "ocda-lib": { "path": "../../sisudoc-ocda-lib/dlang" } } -``` - -```d -import ocda.load; -auto doc = load("document.ocda.db", "de"); // or "document.ssp" -foreach (obj; doc.body) { ... } -``` - -Link `sqlite3`; the library declares the dozen functions it needs rather -than taking a binding from the registry, so there is nothing to fetch. - -## The reader - -```sh -nix build .#ocda-read && ./result/bin/ocda-read [--tree|--html|--llm-text|--llm-jsonl| - --images|--verify --image-dir=<d>|--languages|--files|--report-unknown|--validate] - <file>.ssp|<file>.ocda.db|<file>.sisupod [lang] -``` - -The library's reader, one program where `sisudoc-ssp-reader` (NO LONGER RELEASED, NOW USED DIRECTLY BY WRITER) FIX and -`sisudoc-ocda-db-reader` (NO LONGER RELEASED, NOW USED DIRECTLY BY WRITER) FIX were two. Its output modules are the `.ssp` -reader's, moved onto the library's model, so every mode reads every form -and **says the same thing of a document's `.ssp`, `.ocda.db` and -`.sisupod`**. `--files` answers for a `.ocda.db`, the one form that carries -files; `--validate` checks a `.ssp` as text and refuses anything else. - -It lives in `src/read/` beside the library, and dub keeps it out of the -library: the configuration `library` excludes `src/read/*`, the -configuration `ocda-read` builds it. A program using the library by dub, or -by naming `src/ocda/*.d` as the writer's flake does, never compiles it. - -## The two checks - -```sh -nix build .#ocda-agree && ./result/bin/ocda-agree <doc>.ssp <doc>.ocda.db [lang] -nix build .#ocda-dump && ./result/bin/ocda-dump [--unknown] <file> [lang] -``` - -`agree` loads both and compares ten fields of each object, naming the field -where they part company. `agree` found the blank-first-text-line defect and -was not enough for anything subtler. - -**`ocda-dump` is the stronger check**: the whole loaded model, every field, -written out in a form fully determined by the model, so that two -implementations can be compared with `diff`. It is an oracle, not an output -format; anything reading it to reconstruct a document has misunderstood it. -It agrees byte for byte with the Gleam library's `ocda-lib dump`, which is -the point of having it. - -It found four disagreements `agree` could not see. They are in the -repository README and in the history. - -## Build - -```sh -nix build # all three programs, in one result/bin -nix build .#ocda-read # the reader -nix build .#ocda-agree # the agreement check -nix build .#ocda-dump # the canonical dump -nix develop # ldc, dub, dtools, sqlite, zstd -dub build # the library -dub build --config=ocda-read # the reader -``` -#+END_SRC - -** dub.json - -#+HEADER: :tangle ../dlang/dub.json -#+HEADER: :noweb yes -#+BEGIN_SRC json -{ - "authors": [ "Ralph Amissah" ], - "copyright": "Copyright (C) 2026 Ralph Amissah", - "name": "ocda-lib", - "version": "0.1.0", - "description": "the object-centric document abstraction as a library: the model, and loaders for its two serialisations", - "homepage": "https://sisudoc.org", - "license": "AGPL-3.0+", - "sourcePaths": [ "./src" ], - "importPaths": [ "./src" ], - "libs": [ "sqlite3", "zstd" ], - "configurations": [ - { - "name": "library", - "targetType": "library", - "excludedSourceFiles": [ "src/read/*" ] - }, - { - "name": "ocda-read", - "targetType": "executable", - "targetName": "ocda-read", - "mainSourceFile": "src/read/main.d" - } - ] -} -#+END_SRC - -** dub generated dub.selections.json - -#+HEADER: :tangle-NO ../dlang/dub.selections.json_ -#+HEADER: :noweb yes -#+BEGIN_SRC json -{ "fileVersion": 1, "versions": {} } -#+END_SRC - -* org includes -** project version - -#+NAME: spine_version -#+HEADER: :noweb yes -#+BEGIN_SRC emacs-lisp -<<./sisudoc_spine_version_info_and_doc_header_including_copyright_and_license.org:spine_project_version()>> -#+END_SRC - -** year - -#+NAME: year -#+HEADER: :noweb yes -#+BEGIN_SRC emacs-lisp -<<./sisudoc_spine_version_info_and_doc_header_including_copyright_and_license.org:year()>> -#+END_SRC - -** document header including copyright & license - -#+NAME: doc_header_including_copyright_and_license -#+HEADER: :noweb yes -#+BEGIN_SRC emacs-lisp -<<./sisudoc_spine_version_info_and_doc_header_including_copyright_and_license.org:spine_doc_header_including_copyright_and_license()>> -#+END_SRC - -* nixpkgs source - -#+NAME: nixpkgs_unstable -#+HEADER: :noweb yes -#+BEGIN_SRC emacs-lisp -<<./sisudoc_spine_version_info_and_doc_header_including_copyright_and_license.org:nixpkgs_unstable()>> -#+END_SRC - -* __END__ diff --git a/org/gleam-lib.org b/org/gleam-lib.org deleted file mode 100644 index 708dce3..0000000 --- a/org/gleam-lib.org +++ /dev/null @@ -1,3662 +0,0 @@ --*- mode: org -*- -#+TITLE: gleam lib -#+DESCRIPTION: gleam lib -#+FILETAGS: :spine:build:tools: -#+AUTHOR: Ralph Amissah -#+EMAIL: [[mailto:ralph.amissah@gmail.com][ralph.amissah@gmail.com]] -#+COPYRIGHT: Copyright (C) 2015 (continuously updated, current 2026) Ralph Amissah -#+LANGUAGE: en -#+STARTUP: show2levels hideblocks hidestars noindent entitiespretty -#+PROPERTY: header-args+ :eval never-export :exports code -#+PROPERTY: header-args+ :tangle-mode :noweb no :padline no :tangle-mode :no-expand -#+PROPERTY: header-args+ :results silent :cache no -#+PROPERTY: header-args+ :mkdirp yes -#+OPTIONS: H:3 num:nil toc:t \n:t ::t |:t ^:nil -:t f:t *:t - -* ocda -** ocda lib - -#+NAME: ocda_lib -#+HEADER: :no-expand -#+BEGIN_SRC gleam -//// ocda-lib, in Gleam: the abstraction as a library, put to the question. -//// -//// A library has no binary, so this is the programs over it: the reader, -//// and the two that test its claims. -//// -//// ocda-lib dump [--unknown] <file>.ssp|<file>.ocda.db [lang] -//// ocda-lib agree <doc>.ssp <doc>.ocda.db [lang] -//// ocda-lib read [options] <file> [lang] (also installed as ocda-read) -//// -//// `dump` writes the whole loaded model out canonically, byte for byte as -//// the D library's `ocda-dump` writes it, so that the two implementations -//// can be diffed. `agree` asks the same question the D library's -//// `ocda-agree` asks, within this language: does loading a document's -//// `.ssp` give what loading its `.ocda.db` gives? - -import argv -import gleam/int -import gleam/io -import gleam/list -import gleam/string -import ocda/dump -import ocda/load -import ocda/model.{type Document} -import read/main as reader - -pub fn main() { - case argv.load().arguments { - ["dump", "--unknown", path] -> dump_file(path, "", True) - ["dump", "--unknown", path, lang] -> dump_file(path, lang, True) - ["dump", path] -> dump_file(path, "", False) - ["dump", path, lang] -> dump_file(path, lang, False) - ["agree", ssp, db] -> agree(ssp, db, "") - ["agree", ssp, db, lang] -> agree(ssp, db, lang) - ["read", ..rest] -> - case reader.run(rest) { - 0 -> Nil - code -> halt(code) - } - _ -> usage() - } -} - -fn usage() -> Nil { - io.println_error( - "usage: ocda-lib dump [--unknown] <file>.ssp|<file>.ocda.db [lang] - ocda-lib agree <doc>.ssp <doc>.ocda.db [lang] - ocda-lib read [--help] [options] <file> [lang]", - ) - halt(2) -} - -fn dump_file(path: String, lang: String, unknown: Bool) -> Nil { - case load.load(path, lang, False) { - Error(message) -> fail(message) - Ok(doc) -> - case unknown { - True -> io.print(dump.unknown(doc)) - False -> io.print(dump.document(doc)) - } - } -} - -/// The library's whole purpose is that a writer cannot tell which -/// serialisation it was handed. That is only true if the two loaders give -/// the same model, so this asks, and says where they part company rather -/// than only that they did. -fn agree(ssp_path: String, db_path: String, lang: String) -> Nil { - case load.load(ssp_path, "", False), load.load(db_path, lang, False) { - Error(message), _ -> fail(message) - _, Error(message) -> fail(message) - Ok(from_ssp), Ok(from_db) -> { - let differences = compare(from_ssp, from_db) - case differences { - [] -> io.println("agree: the two loaders give the same model") - _ -> { - list.each(list.take(differences, 12), io.println) - case list.length(differences) > 12 { - True -> io.println(" ... stopping after 12") - False -> Nil - } - io.println(int.to_string(list.length(differences)) <> " difference(s)") - halt(1) - } - } - } - } -} - -/// Compared through the dump rather than field by field, which is both -/// shorter and stricter: the dump carries every field, so nothing can -/// differ without showing here. -fn compare(from_ssp: Document, from_db: Document) -> List(String) { - let a = dump.document(from_ssp) - let b = dump.document(from_db) - case a == b { - True -> [] - False -> diff_lines(a, b) - } -} - -fn diff_lines(a: String, b: String) -> List(String) { - let la = split_lines(a) - let lb = split_lines(b) - case list.length(la) == list.length(lb) { - False -> [ - " differ: line count", - " .ssp: " <> int.to_string(list.length(la)), - " .ocda.db: " <> int.to_string(list.length(lb)), - ..pairwise(la, lb) - ] - True -> pairwise(la, lb) - } -} - -fn pairwise(la: List(String), lb: List(String)) -> List(String) { - list.zip(la, lb) - |> list.index_map(fn(pair, i) { #(i, pair.0, pair.1) }) - |> list.filter(fn(t) { t.1 != t.2 }) - |> list.map(fn(t) { - " differ: line " - <> int.to_string(t.0 + 1) - <> "\n .ssp: " - <> t.1 - <> "\n .ocda.db: " - <> t.2 - }) -} - -fn split_lines(text: String) -> List(String) { - case text { - "" -> [] - _ -> string.split(text, on: "\n") - } -} - -fn fail(message: String) -> Nil { - io.println_error("error: " <> message) - halt(2) -} - -/// Gleam has no standard exit, and the exit status is what a sweep script -/// reads. -@external(erlang, "erlang", "halt") -fn halt(code: Int) -> Nil -#+END_SRC - -** ocda -*** model - -#+NAME: model -#+HEADER: :no-expand -#+BEGIN_SRC gleam -//// The object-centric document abstraction, as Gleam types. -//// -//// A deliberate mirror of the D library's `ocda.model`, field for field. -//// The two are meant to agree, and the cheapest way to make that testable -//// is for them to hold the same things under the same names: the dump of a -//// loaded document is then identical by construction rather than by -//// negotiation. -//// -//// Written against specs/sisudoc-abstraction-format: `ocda-model.md` for -//// the model, the two format specifications for what each serialisation -//// says about it. - -import gleam/dict.{type Dict} -import gleam/list -import gleam/string - -/// The format major version this library knows. -/// -/// The `.ssp` and the `.ocda.db` carry one version between them, declared -/// in band. The minor part rises when a property is added and a reader of -/// the newer format still reads the older, so only the major is checked. -pub const format_major_read = "2" - -/// The nine sections, in the order the format gives them, which is not the -/// order a sorted key list or a `group by` would give. -pub const section_names = [ - "head", "toc", "body", "endnotes", "glossary", "bibliography", "bookindex", - "blurb", "tail", -] - -/// The four blocks that are document preamble rather than sections of -/// objects. -pub const preamble_blocks = ["source", "meta", "make", "doc_has"] - -/// A repeatable `.image` record: the file as the abstraction found it. -pub type Image { - Image( - /// filename as the object refers to it - name: String, - /// hex, uppercase - sha256: String, - bytes: Int, - /// the file's own pixel dimensions, not any display width - width: Int, - height: Int, - /// absent or unreadable, so no digest and no size - missing: Bool, - /// any further key:value fields, kept rather than dropped - fields: Dict(String, String), - ) -} - -pub fn new_image() -> Image { - Image( - name: "", - sha256: "", - bytes: 0, - width: 0, - height: 0, - missing: False, - fields: dict.new(), - ) -} - -/// One object of the abstraction. -/// -/// `text` is one string with the lines joined by newline, as in the D -/// library and for the same reason: an object whose only text line is -/// blank is then indistinguishable from an object with no text, in both -/// libraries, in the same place. A difference that exists in one language -/// and not the other would be worse than a loss both share. -pub type Object { - Object( - ocn: Int, - /// one of the sixteen kinds - is_a: String, - /// the section this object was found in - section: String, - /// headings: A B C D 1 2 3 4 - heading_level: String, - identifier: String, - part: String, - /// how the text behaves: para, block, comment - is_of_type: String, - /// the object's own section, when it differs from the block it sits in - own_section: String, - segment_html_is: String, - segment_epub: String, - segment_epub_is: String, - table_aligns: String, - stow_link: List(String), - anchor_tags: List(String), - parent_ocn: Int, - last_descendant_ocn: Int, - ocn_range_first: Int, - ocn_range_last: Int, - children: List(Int), - ancestors: List(Int), - dummy_heading: Bool, - ocn_off: Bool, - indent_base: Int, - indent_hang: Int, - bullet: Bool, - lang: String, - has_links: Bool, - has_notes_reg: Bool, - has_notes_star: Bool, - has_images: Bool, - segment: String, - segment_prev: String, - segment_next: String, - anchor: String, - heading_lev_anchor: String, - table_cols: Int, - table_widths: String, - table_header: Bool, - code_syntax: String, - code_linenumbers: Bool, - heading_ancestors_text: String, - sha256: String, - images: List(Image), - text: String, - ) -} - -pub fn new_object() -> Object { - Object( - ocn: 0, - is_a: "", - section: "", - heading_level: "", - identifier: "", - part: "", - is_of_type: "", - own_section: "", - segment_html_is: "", - segment_epub: "", - segment_epub_is: "", - table_aligns: "", - stow_link: [], - anchor_tags: [], - parent_ocn: 0, - last_descendant_ocn: 0, - ocn_range_first: 0, - ocn_range_last: 0, - children: [], - ancestors: [], - dummy_heading: False, - ocn_off: False, - indent_base: 0, - indent_hang: 0, - bullet: False, - lang: "", - has_links: False, - has_notes_reg: False, - has_notes_star: False, - has_images: False, - segment: "", - segment_prev: "", - segment_next: "", - anchor: "", - heading_lev_anchor: "", - table_cols: 0, - table_widths: "", - table_header: False, - code_syntax: "", - code_linenumbers: False, - heading_ancestors_text: "", - sha256: "", - images: [], - text: "", - ) -} - -/// Whether this object is rendered at all. -/// -/// `is_of_type` "comment" is markup commentary: carried by the abstraction -/// so that nothing is lost, and not put into output. An output that -/// renders it leaks the author's notes into the document, which is what -/// spine itself does not do. -pub fn renders(object: Object) -> Bool { - object.is_of_type != "comment" -} - -/// The section this object belongs to: the block it sits in, unless it -/// says otherwise. -pub fn section_of(object: Object) -> String { - case object.own_section { - "" -> object.section - own -> own - } -} - -/// What the loader met and did not understand. -/// -/// A loader that silently drops what it does not know cannot say whether -/// it understood the document, and so cannot be used to check a -/// specification against an implementation: silence and understanding look -/// alike. -pub type Unknown { - Unknown( - /// `.name` -> times seen - property: Dict(String, Int), - /// `@name` -> times seen - block: Dict(String, Int), - /// first character -> times seen, for lines of no known shape - line: Dict(String, Int), - ) -} - -pub fn new_unknown() -> Unknown { - Unknown(property: dict.new(), block: dict.new(), line: dict.new()) -} - -pub fn any_unknown(unknown: Unknown) -> Bool { - dict.size(unknown.property) > 0 - || dict.size(unknown.block) > 0 - || dict.size(unknown.line) > 0 -} - -pub type Document { - Document( - /// as declared on line 1 of a `.ssp` or in `schema.version`, "" when - /// absent - format_version: String, - source_filename: String, - /// language, languages, digest - source: Dict(String, String), - unknown: Unknown, - metadata: Dict(String, String), - make: Dict(String, String), - doc_has: Dict(String, String), - /// section name -> objects, in document order - sections: Dict(String, List(Object)), - ) -} - -pub fn new_document() -> Document { - Document( - format_version: "", - source_filename: "", - source: dict.new(), - unknown: new_unknown(), - metadata: dict.new(), - make: dict.new(), - doc_has: dict.new(), - sections: dict.new(), - ) -} - -/// The major part of the declared version, "" when none was declared. -pub fn version_major(document: Document) -> String { - case string.split_once(document.format_version, ".") { - Ok(#(major, _)) -> major - Error(_) -> document.format_version - } -} - -pub fn meta(document: Document, key: String) -> String { - case dict.get(document.metadata, key) { - Ok(value) -> value - Error(_) -> "" - } -} - -/// The objects of a named section, empty when the document has none. -pub fn section(document: Document, name: String) -> List(Object) { - case dict.get(document.sections, name) { - Ok(objects) -> objects - Error(_) -> [] - } -} - -/// Is this one of the nine sections of the format? -pub fn is_section(name: String) -> Bool { - list.contains(section_names, name) -} - -/// Is this one of the four preamble blocks? -pub fn is_preamble_block(name: String) -> Bool { - list.contains(preamble_blocks, name) -} -#+END_SRC - -*** what to load -**** inventory - -#+NAME: ocda_inventory -#+HEADER: :no-expand -#+BEGIN_SRC gleam -//// What a source holds, beyond the one document the model is loaded from. -//// -//// The model is one language of one document. Two questions a reader asks -//// are about the file instead: which languages it holds, and, for a -//// database, what else it carries. They are here rather than in a reader -//// because the answer depends on the input form, which is the library's to -//// know, and so that a reader asking them does not learn which form it was -//// handed. A mirror of the D library's `ocda.inventory`. -//// -//// `languages` answers for every form, and answers the same: the languages -//// in the order the manifest gave them (which every `.ssp` records, in -//// `@source languages`), each with the markup file it was made from and the -//// sha256 of its `.ssp`. The database stores that digest; for a `.ssp` and a -//// pod it is taken from the bytes, which are the same bytes. -//// -//// `carried` is the database's: a `.ocda.db` holds the markup, images, -//// configuration and catalogues it was built from, a `.ssp` holds none of -//// them, and a pod's are its archive entries rather than anything the -//// abstraction says. - -import gleam/bit_array -import gleam/dict -import gleam/list -import gleam/result -import gleam/string -import ocda/load -import ocda/load_db -import ocda/load_pod -import ocda/load_ssp -import ocda/zip -import simplifile - -pub type Language { - Language(lang: String, source_filename: String, ssp_digest: String) -} - -pub type Carried { - Carried( - role: String, - compression: String, - count: String, - bytes: String, - stored: String, - ) -} - -pub fn languages(path: String) -> Result(List(Language), String) { - case load.source_of(path) { - load.OcdaDb -> { - use held <- result.try(load_db.languages_of_file(path)) - Ok( - list.map(held, fn(l) { - Language(l.lang, l.source_filename, l.ssp_digest) - }), - ) - } - load.Ssp -> { - use bytes <- result.try( - simplifile.read_bits(path) - |> result.map_error(fn(e) { path <> ": " <> simplifile.describe_error(e) }), - ) - use text <- result.try( - bit_array.to_string(bytes) - |> result.replace_error(path <> ": not valid UTF-8"), - ) - let doc = load_ssp.parse(text) - Ok([ - Language( - source_value(doc.source, "language"), - doc.source_filename, - sha256_hex(bytes), - ), - ]) - } - load.Pod -> { - use archive <- result.try(load_pod.archive_bytes(path)) - use entries <- result.try(zip.entries(archive)) - use found <- result.try( - zip.find_suffix(entries, ".ssp") - |> list.try_map(fn(e) { - use bytes <- result.try(zip.read(archive, e)) - use text <- result.try( - bit_array.to_string(bytes) - |> result.replace_error(e.name <> ": not valid UTF-8"), - ) - let doc = load_ssp.parse(text) - let lang = result.unwrap(load_pod.lang_of(e.name), "") - Ok(#( - Language(lang, doc.source_filename, sha256_hex(bytes)), - string.split(source_value(doc.source, "languages"), " "), - )) - }), - ) - let order = case found { - [#(_, o), ..] -> list.filter(o, fn(l) { l != "" }) - [] -> [] - } - let by_lang = list.map(found, fn(f) { #({ f.0 }.lang, f.0) }) |> dict.from_list - // the manifest's order, and anything it did not name after it - let named = list.filter_map(order, fn(l) { dict.get(by_lang, l) }) - let rest = - list.filter(found, fn(f) { !list.contains(order, { f.0 }.lang) }) - |> list.map(fn(f) { f.0 }) - Ok(list.append(named, rest)) - } - load.Unknown -> Error(path <> ": " <> load.source_name(load.Unknown)) - } -} - -/// Only a `.ocda.db` carries files. -pub fn is_carrier(path: String) -> Bool { - load.source_of(path) == load.OcdaDb -} - -pub fn carried(path: String) -> Result(List(Carried), String) { - case is_carrier(path) { - False -> Ok([]) - True -> { - use rows <- result.try(load_db.carried_of_file(path)) - Ok( - list.filter_map(rows, fn(r) { - case r { - [role, compression, n, bytes, stored] -> - Ok(Carried(role, compression, n, bytes, stored)) - _ -> Error(Nil) - } - }), - ) - } - } -} - -fn source_value(source: dict.Dict(String, String), key: String) -> String { - result.unwrap(dict.get(source, key), "") -} - -@external(erlang, "ocda_ffi", "sha256_hex") -fn sha256_hex(bytes: BitArray) -> String -#+END_SRC - -**** load -***** load - -#+NAME: ocda_load -#+HEADER: :no-expand -#+BEGIN_SRC gleam -//// What a path is, and loading it. -//// -//// One place that decides, so that every reader and every writer accepts -//// the same inputs and a new input form is added once. This mirrors -//// spine's own `ocda/abstraction/load.d`, which names the sources it can -//// enter the pipeline from and tells them apart by suffix. -//// -//// Deliberately *not* here: `.sst` and `.ssm` markup. Reading markup means -//// implementing the parser, which is spine's work and a far larger thing -//// than a reader or a sample writer needs. This library reads -//// abstractions. - -import gleam/list -import gleam/string -import ocda/load_db -import ocda/load_pod -import ocda/load_ssp -import ocda/model.{type Document} - -pub type Source { - /// `.ssp`, the abstraction as text - Ssp - /// `.ocda.db`, the abstraction as sqlite - OcdaDb - /// `.sisupod` or `.zip`, an archive carrying a `.ssp` - Pod - Unknown -} - -/// `.sisupod` is the archive spine writes now; `.zip` is what it wrote -/// before, and every pod already published carries that name, so both are -/// recognised. What is inside decides how it is opened, not the name. -const suffixes = [ - #(".ocda.db", OcdaDb), - #(".ssp", Ssp), - #(".sisupod", Pod), - #(".zip", Pod), - #(".db", OcdaDb), -] - -pub fn source_of(path: String) -> Source { - case list.find(suffixes, fn(s) { string.ends_with(path, s.0) }) { - Ok(#(_, source)) -> source - Error(_) -> Unknown - } -} - -pub fn source_name(source: Source) -> String { - case source { - Ssp -> ".ssp (the abstraction as text)" - OcdaDb -> ".ocda.db (the abstraction as sqlite)" - Pod -> "a pod archive carrying a .ssp" - Unknown -> "not an abstraction this library reads" - } -} - -/// Load whatever the path is into the model. -/// -/// `lang` names which language to take from a `.ocda.db`, which holds -/// every language of its document, and from a pod, which may carry more than -/// one `.ssp`. A `.ssp` is one language by construction and ignores it. -pub fn load( - path: String, - lang: String, - any_version: Bool, -) -> Result(Document, String) { - case source_of(path) { - Ssp -> load_ssp.load_file(path) - OcdaDb -> load_db.load_file(path, lang, any_version) - Pod -> load_pod.load_file(path, lang) - Unknown -> - Error( - path - <> ": " - <> source_name(Unknown) - <> " (expected .ssp, .ocda.db, .sisupod or .zip)", - ) - } -} -#+END_SRC - -***** load pod - -#+NAME: ocda_load_pod -#+HEADER: :no-expand -#+BEGIN_SRC gleam -//// The abstraction out of a pod archive. -//// -//// A pod built with `--pod2` carries the abstraction inside it, at -//// `media/abstraction/<uid>.<lang>.ssp`, and that is what is read. So this is -//// a third input form and *not* a third model: the `.ssp` inside is parsed by -//// `ocda/load_ssp` and comes back as the same value. A writer over the -//// library gains pods without being touched, which is the whole reason the -//// dispatch is in one place. -//// -//// ## The one thing this cannot do -//// -//// A `.sisupod` is **one zstd frame wrapping** the zip. This reads the zip -//// and **not the frame**, because there is no zstd on the BEAM without a NIF, -//// and the one that exists, `ezstd`, clones zstd from github in its build, so -//// it cannot be built in a sandbox without patching it. The D library has -//// both, through a declared `libzstd`. -//// -//// So: a `.zip` pod is read, a `.sisupod` is refused, and the refusal says -//// why rather than pretending the file is malformed. It is the only place the -//// two libraries differ in what they accept, and it is a packaging limit -//// rather than anything about the format. - -import gleam/list -import gleam/result -import gleam/string -import ocda/load_ssp -import ocda/model.{type Document} -import ocda/zip -import simplifile - -/// The magic number of a zstd frame, little endian 0xFD2FB528. -const zstd_magic = <<0x28, 0xB5, 0x2F, 0xFD>> - -/// Which languages of the abstraction the pod carries, sorted. -pub fn languages(path: String) -> Result(List(String), String) { - use bytes <- result.try(archive_bytes(path)) - use entries <- result.try(zip.entries(bytes)) - zip.find_suffix(entries, ".ssp") - |> list.filter_map(fn(e) { lang_of(e.name) }) - |> list.sort(string.compare) - |> Ok -} - -/// Load one language of the abstraction a pod carries. -/// -/// `lang` == "" takes the only one there is, and where there is a choice it -/// says so rather than choosing, as the `.ocda.db` loader does. -pub fn load_file(path: String, lang: String) -> Result(Document, String) { - use bytes <- result.try(archive_bytes(path)) - use entries <- result.try(zip.entries(bytes)) - let ssps = zip.find_suffix(entries, ".ssp") - use chosen <- result.try(case ssps, lang { - [], _ -> - Error( - path - <> ": carries no .ssp. A pod carries the abstraction only when spine" - <> " was asked for it: spine --pod2 puts it in, spine --source does not.", - ) - [only], "" -> Ok(only) - _, "" -> - Error( - path - <> ": holds more than one language (" - <> string.join(langs(ssps), " ") - <> "); name the one wanted", - ) - _, wanted -> - case list.find(ssps, fn(e) { lang_of(e.name) == Ok(wanted) }) { - Ok(found) -> Ok(found) - Error(_) -> - Error( - path - <> ": no language '" - <> wanted - <> "'; it holds: " - <> string.join(langs(ssps), " "), - ) - } - }) - use content <- result.try(zip.read(bytes, chosen)) - case bit_array_to_string(content) { - Ok(text) -> Ok(load_ssp.parse(text)) - Error(_) -> Error(chosen.name <> ": not valid UTF-8") - } -} - -fn langs(entries: List(zip.Entry)) -> List(String) { - list.filter_map(entries, fn(e) { lang_of(e.name) }) -} - -/// `<uid>.<lang>.ssp` -> `<lang>` -pub fn lang_of(name: String) -> Result(String, Nil) { - case string.ends_with(name, ".ssp") { - False -> Error(Nil) - True -> { - let stem = string.drop_end(name, 4) - case string.split(stem, on: ".") |> list.last { - Ok(lang) if lang != stem -> Ok(lang) - _ -> Error(Nil) - } - } - } -} - -/// The bytes decide, not the name: a file misnamed either way is treated by -/// what it holds. -pub fn archive_bytes(path: String) -> Result(BitArray, String) { - case simplifile.read_bits(path) { - Error(e) -> Error(path <> ": " <> simplifile.describe_error(e)) - Ok(bytes) -> - case is_zstd_frame(bytes) { - False -> Ok(bytes) - True -> - Error( - path - <> " is a zstd frame wrapping the archive, and this library has no" - <> " zstd: the only one on the BEAM is a NIF that fetches zstd in" - <> " its own build, so it cannot be built sandboxed. Unwrap it" - <> " first (zstd -d) and read the archive, or use the D library," - <> " which reads both.", - ) - } - } -} - -fn is_zstd_frame(bytes: BitArray) -> Bool { - case bytes { - <<head:bytes-size(4), _:bits>> -> head == zstd_magic - _ -> False - } -} - -@external(erlang, "unicode", "characters_to_binary") -fn bit_array_to_string_ffi(b: BitArray) -> String - -pub fn bit_array_to_string(b: BitArray) -> Result(String, Nil) { - Ok(bit_array_to_string_ffi(b)) -} -#+END_SRC - -***** load ssp - -#+NAME: ocda_load_ssp -#+HEADER: :no-expand -#+BEGIN_SRC gleam -//// Loading a `.ssp` into the model. -//// -//// The format is line oriented and every line is one of eight shapes told -//// apart by its first character, so the loader is a fold over the lines -//// carrying the state the shapes need. -//// -//// Written against `ssp-format-specification.org`. Where it and the D -//// library differ the D library is the bug, or this is: they are meant to -//// give the same model, and `ocda-dump` says whether they do. - -import gleam/dict.{type Dict} -import gleam/int -import gleam/list -import gleam/option -import gleam/result -import gleam/string -import ocda/model.{type Document, type Image, type Object, Image, Object} -import simplifile - -pub fn load_file(path: String) -> Result(Document, String) { - case simplifile.read(path) { - Ok(content) -> Ok(parse(content)) - Error(e) -> Error(path <> ": " <> simplifile.describe_error(e)) - } -} - -type State { - State( - document: Document, - /// the block currently open, "" when none - block: String, - /// the section currently open, "" when in the preamble - section: String, - /// objects of the current section, newest first - objects: List(Object), - /// the object being read, if any - current: Result(Object, Nil), - /// text lines of the object being read, newest first - text: List(String), - /// whether a text line has been seen for the object being read. - /// Not "is the text non-empty": an object whose first text line is - /// blank, which the reference collection has seven of, would then lose - /// it. - seen_text: Bool, - ) -} - -pub fn parse(content: String) -> Document { - content - |> string.split(on: "\n") - |> list.fold(from: initial(), with: line) - |> finish -} - -fn initial() -> State { - State( - document: model.new_document(), - block: "", - section: "", - objects: [], - current: Error(Nil), - text: [], - seen_text: False, - ) -} - -fn line(state: State, raw: String) -> State { - case raw { - "% SiSU Document Abstraction v" <> version -> - case state.document.format_version { - "" -> - State( - ..state, - document: model.Document( - ..state.document, - format_version: string.trim(version), - ), - ) - _ -> state - } - "% Source: " <> filename -> - case state.document.source_filename { - "" -> - State( - ..state, - document: model.Document( - ..state.document, - source_filename: string.trim(filename), - ), - ) - _ -> state - } - "%" <> _ -> state - "@" <> rest -> open_block(state, rest) - "}" <> _ -> close_block(state) - "[" <> rest -> declaration(state, rest) - "." <> rest -> property(state, rest) - "|" <> rest -> text_line(state, rest) - _ -> - case string.trim(raw) { - // a blank line ends the object being read - "" -> commit(state) - _ -> - case model.is_preamble_block(state.block) { - True -> preamble_pair(state, raw) - False -> unknown_line(state, raw) - } - } - } -} - -fn open_block(state: State, rest: String) -> State { - let name = case string.split_once(rest, "{") { - Ok(#(name, _)) -> string.trim(name) - Error(_) -> string.trim(rest) - } - case model.is_preamble_block(name) { - True -> State(..flush(state), block: name) - False -> { - let state = flush(state) - let document = case model.is_section(name) { - True -> state.document - // a block that is neither preamble nor one of the nine sections: - // counted, so that a reader can say the file held something it did - // not expect - False -> - model.Document( - ..state.document, - unknown: model.Unknown( - ..state.document.unknown, - block: bump(state.document.unknown.block, name), - ), - ) - } - State(..state, block: name, section: name, document: document) - } - } -} - -fn close_block(state: State) -> State { - State(..flush(state), block: "") -} - -/// Commit the object being read, then put the section's objects into the -/// document and close the section. -fn flush(state: State) -> State { - let state = commit(state) - case state.section, state.objects { - "", _ -> state - _, [] -> State(..state, section: "", objects: []) - name, objects -> - State( - ..state, - section: "", - objects: [], - document: model.Document( - ..state.document, - sections: add_section( - state.document.sections, - name, - list.reverse(objects), - ), - ), - ) - } -} - -fn preamble_pair(state: State, raw: String) -> State { - case string.split_once(string.trim_start(raw), ":") { - Error(_) -> state - Ok(#(key, value)) -> { - let key = string.trim(key) - let value = string.trim(value) - let document = state.document - let document = case state.block { - "meta" -> - model.Document( - ..document, - metadata: dict.insert(document.metadata, key, value), - ) - "make" -> - model.Document(..document, make: dict.insert(document.make, key, value)) - "source" -> - model.Document( - ..document, - source: dict.insert(document.source, key, value), - ) - _ -> - model.Document( - ..document, - doc_has: dict.insert(document.doc_has, key, value), - ) - } - State(..state, document: document) - } - } -} - -/// `[<ocn>] <kind>`, and for a heading `[<ocn>] heading :<level> -/// [<identifier>]`. -fn declaration(state: State, rest: String) -> State { - let state = commit(state) - case string.split_once(rest, "]") { - Error(_) -> state - Ok(#(ocn_text, tail)) -> { - let ocn = parse_int(string.trim(ocn_text)) - let object = - Object(..model.new_object(), ocn: ocn, section: state.section) - let object = case string.split(string.trim(tail), on: " ") { - ["heading", level, ..identifier] -> - Object( - ..object, - is_a: "heading", - heading_level: string.drop_start(level, 1), - identifier: string.join(identifier, " "), - ) - [""] -> object - [kind, ..] -> Object(..object, is_a: kind) - [] -> object - } - State(..state, current: Ok(object), text: [], seen_text: False) - } - } -} - -fn property(state: State, rest: String) -> State { - case state.current, string.split_once(rest, ":") { - Ok(object), Ok(#(name, raw_value)) -> { - let name = string.trim(name) - let value = string.trim(raw_value) - let #(object, known) = apply_property(object, name, value) - let document = case known { - True -> state.document - False -> - model.Document( - ..state.document, - unknown: model.Unknown( - ..state.document.unknown, - property: bump(state.document.unknown.property, name), - ), - ) - } - State(..state, current: Ok(object), document: document) - } - _, _ -> state - } -} - -/// The properties this library reads, and what each sets. -/// -/// A property it has no use for is counted rather than passed over: -/// ignoring one is conforming within a major version, doing so in silence -/// is what stops a reader being a check on the specification. -fn apply_property( - object: Object, - name: String, - value: String, -) -> #(Object, Bool) { - case name { - "part" -> #(Object(..object, part: value), True) - "is_of_type" -> #(Object(..object, is_of_type: value), True) - "section" -> #(Object(..object, own_section: value), True) - "identifier" -> #(Object(..object, identifier: value), True) - "segment_html_is" -> #(Object(..object, segment_html_is: value), True) - "segment_epub" -> #(Object(..object, segment_epub: value), True) - "segment_epub_is" -> #(Object(..object, segment_epub_is: value), True) - "table_aligns" -> #(Object(..object, table_aligns: value), True) - "stow_link" -> #( - Object(..object, stow_link: list.append(object.stow_link, [value])), - True, - ) - "anchor_tag" -> #( - Object(..object, anchor_tags: list.append(object.anchor_tags, [value])), - True, - ) - "parent" -> #(Object(..object, parent_ocn: parse_int(value)), True) - "last_descendant" -> #( - Object(..object, last_descendant_ocn: parse_int(value)), - True, - ) - "ocn_range" -> - case numbers(value) { - [first, last] -> #( - Object(..object, ocn_range_first: first, ocn_range_last: last), - True, - ) - _ -> #(object, True) - } - "children" -> #(Object(..object, children: numbers(value)), True) - "ancestors" -> #(Object(..object, ancestors: numbers(value)), True) - "dummy" -> #(Object(..object, dummy_heading: value == "true"), True) - "ocn_off" -> #(Object(..object, ocn_off: value == "true"), True) - "indent" -> { - let parts = string.split(value, on: " ") - let base = case parts { - [first, ..] -> parse_int(first) - [] -> 0 - } - let hang = case parts { - [_, second, ..] -> parse_int(second) - _ -> 0 - } - #(Object(..object, indent_base: base, indent_hang: hang), True) - } - "bullet" -> #(Object(..object, bullet: value == "true"), True) - "lang" -> #(Object(..object, lang: value), True) - "has" -> #(has_flags(object, value), True) - "segment" -> #(Object(..object, segment: value), True) - "segment_prev" -> #(Object(..object, segment_prev: value), True) - "segment_next" -> #(Object(..object, segment_next: value), True) - "anchor" -> #(Object(..object, anchor: value), True) - "heading_lev_anchor" -> #( - Object(..object, heading_lev_anchor: value), - True, - ) - "table_cols" -> #(Object(..object, table_cols: parse_int(value)), True) - "table_widths" -> #(Object(..object, table_widths: value), True) - "table_header" -> #(Object(..object, table_header: value == "true"), True) - "code_syntax" -> #(Object(..object, code_syntax: value), True) - "code_linenumbers" -> #( - Object(..object, code_linenumbers: value == "true"), - True, - ) - "heading_ancestors_text" -> #( - Object(..object, heading_ancestors_text: value), - True, - ) - "sha256" -> #(Object(..object, sha256: value), True) - "image" -> #( - Object(..object, images: list.append(object.images, [image(value)])), - True, - ) - _ -> #(object, False) - } -} - -fn has_flags(object: Object, value: String) -> Object { - value - |> string.split(on: " ") - |> list.fold(from: object, with: fn(obj, flag) { - case flag { - "links" -> Object(..obj, has_links: True) - "notes_reg" -> Object(..obj, has_notes_reg: True) - "notes_star" -> Object(..obj, has_notes_star: True) - "images" -> Object(..obj, has_images: True) - _ -> obj - } - }) -} - -/// `<filename> sha256:<HEX> bytes:<n> [px:<w>x<h>] [missing:true] [...]` -/// -/// The filename comes first, the rest are order-independent, and a field -/// this library does not recognise is kept rather than discarded. -fn image(value: String) -> Image { - case string.split(value, on: " ") { - [] -> model.new_image() - [name, ..fields] -> - list.fold(fields, model.new_image(), fn(img, field) { - case string.split_once(field, ":") { - Error(_) -> img - Ok(#(key, val)) -> - case key { - "sha256" -> Image(..img, sha256: val) - "bytes" -> Image(..img, bytes: parse_int(val)) - "px" -> - case string.split_once(val, "x") { - Ok(#(w, h)) -> - Image(..img, width: parse_int(w), height: parse_int(h)) - Error(_) -> img - } - "missing" -> Image(..img, missing: val == "true") - _ -> Image(..img, fields: dict.insert(img.fields, key, val)) - } - } - }) - |> fn(img) { Image(..img, name: name) } - } -} - -/// `| <text>`, and `| ` alone for a blank line. Never a bare `|`: the -/// emitter writes the space. -fn text_line(state: State, rest: String) -> State { - case state.current { - Error(_) -> state - Ok(_) -> { - let content = string.drop_start(rest, 1) - State(..state, text: [content, ..state.text], seen_text: True) - } - } -} - -fn unknown_line(state: State, raw: String) -> State { - let first = string.slice(raw, 0, 1) - State( - ..state, - document: model.Document( - ..state.document, - unknown: model.Unknown( - ..state.document.unknown, - line: bump(state.document.unknown.line, first), - ), - ), - ) -} - -fn commit(state: State) -> State { - case state.current { - Error(_) -> state - Ok(object) -> { - let text = case state.seen_text { - True -> string.join(list.reverse(state.text), "\n") - False -> "" - } - let object = derive_omitted(Object(..object, text: text)) - let objects = case model.is_section(state.section) { - True -> [object, ..state.objects] - False -> state.objects - } - State( - ..state, - current: Error(Nil), - text: [], - seen_text: False, - objects: objects, - ) - } - } -} - -/// Two values the text form leaves out because they can be worked out. -/// -/// `.identifier` is absent when it is simply the ocn, `.anchor` when it is -/// the segment name. The `.ocda.db`, being columns rather than lines, -/// stores both unconditionally. So a loader that takes the text form -/// literally gives a different model from the one the database gives, and -/// over one document that is 1,314 identifiers and 69 anchors. -fn derive_omitted(object: Object) -> Object { - let object = case object.identifier, object.ocn { - "", 0 -> object - "", ocn -> Object(..object, identifier: int.to_string(ocn)) - _, _ -> object - } - case object.anchor, object.segment { - "", "" -> object - "", segment -> Object(..object, anchor: segment) - _, _ -> object - } -} - -fn finish(state: State) -> Document { - flush(state).document -} - -fn add_section( - sections: Dict(String, List(Object)), - name: String, - objects: List(Object), -) -> Dict(String, List(Object)) { - dict.upsert(sections, name, fn(existing) { - case existing { - option.Some(previous) -> list.append(previous, objects) - option.None -> objects - } - }) -} - -fn bump(counts: Dict(String, Int), key: String) -> Dict(String, Int) { - dict.upsert(counts, key, fn(existing) { - case existing { - option.Some(n) -> n + 1 - option.None -> 1 - } - }) -} - -fn parse_int(value: String) -> Int { - value |> string.trim |> int.parse |> result.unwrap(0) -} - -fn numbers(value: String) -> List(Int) { - value - |> string.split(on: " ") - |> list.filter(fn(s) { s != "" }) - |> list.map(parse_int) -} -#+END_SRC - -***** load db - -#+NAME: ocda_load_db -#+HEADER: :no-expand -#+BEGIN_SRC gleam -//// Loading a `.ocda.db` into the model. -//// -//// The point of this module is that it returns *the same value* -//// `ocda/load_ssp` returns from the same document's `.ssp`. A writer is -//// then a function from the model to bytes and cannot tell which -//// serialisation it was given, exactly as spine's own writers cannot. -//// -//// Written against `ocda-db-format-specification.org`, whose section 4 -//// gives the column by column correspondence this module implements. - -import gleam/dict -import gleam/dynamic/decode -import gleam/float -import gleam/int -import gleam/json -import gleam/list -import gleam/option -import gleam/result -import gleam/string -import ocda/model.{type Document, Image, Object} -import sqlight - -/// The schema major version this loader knows. -pub const schema_major_read = "2" - -/// One language of the database, as the manifest gives it. -pub type Language { - Language(doc_id: Int, lang: String, source_filename: String, ssp_digest: String) -} - -/// Load one language of a database as the model. `lang` == "" takes the -/// first language the manifest gives. -pub fn load_file( - path: String, - lang: String, - any_version: Bool, -) -> Result(Document, String) { - // read only: a loader has no business writing, and a read-write open - // *creates* an empty database where a path is mistyped, which then reads - // as a document with nothing in it, a failure that looks like an answer - use conn <- with_connection("file:" <> path <> "?mode=ro", path) - - use _ <- result.try(case - has(conn, "documents") && has(conn, "objects") && has(conn, "metadata") - { - True -> Ok(Nil) - False -> - Error( - path - <> ": no documents/objects/metadata tables; it does not look like a .ocda.db", - ) - }) - - let schema_version = - one(conn, "select value from metadata where doc_id is null - and key = 'schema.version'") - use _ <- result.try(case schema_version, any_version { - "", _ -> Ok(Nil) - _, True -> Ok(Nil) - v, False -> - case major_of(v) == schema_major_read { - True -> Ok(Nil) - False -> - Error( - path - <> " is schema v" - <> v - <> ", this loader knows v" - <> schema_major_read - <> ".x", - ) - } - }) - - let languages = languages(conn) - use chosen <- result.try(case languages, lang { - [], _ -> Error(path <> ": no documents rows") - [first, ..], "" -> Ok(first) - _, wanted -> - case list.find(languages, fn(l) { l.lang == wanted }) { - Ok(found) -> Ok(found) - Error(_) -> - Error( - path - <> ": no language '" - <> wanted - <> "'; it holds: " - <> string.join(list.map(languages, fn(l) { l.lang }), " "), - ) - } - }) - - let source = - dict.new() - |> dict.insert("language", chosen.lang) - |> dict.insert( - "languages", - string.join(list.map(languages, fn(l) { l.lang }), " "), - ) - let digest = meta_value(conn, chosen.doc_id, "source.digest") - let source = case digest { - "" -> source - d -> dict.insert(source, "digest", d) - } - - let document = - model.Document( - ..model.new_document(), - format_version: schema_version, - source_filename: chosen.source_filename, - source: source, - ) - let document = metadata(conn, chosen.doc_id, document) - let document = objects(conn, chosen.doc_id, document) - let document = count_unread(conn, chosen.doc_id, document) - Ok(document) -} - -fn with_connection( - uri: String, - path: String, - then: fn(sqlight.Connection) -> Result(a, String), -) -> Result(a, String) { - case sqlight.open(uri) { - Error(e) -> Error(path <> ": " <> e.message) - Ok(conn) -> { - let outcome = then(conn) - let _ = sqlight.close(conn) - outcome - } - } -} - -/// Every language the database holds, in manifest order. -pub fn languages(conn: sqlight.Connection) -> List(Language) { - let d = { - use id <- decode.field(0, decode.int) - use lang <- decode.field(1, decode.string) - use source <- decode.field(2, decode.string) - use digest <- decode.field(3, decode.string) - decode.success(Language(id, lang, source, digest)) - } - sqlight.query( - "select id, lang, coalesce(source_filename,''), coalesce(ssp_digest,'') - from documents order by id", - on: conn, - with: [], - expecting: d, - ) - |> result.unwrap([]) -} - -/// The three preamble blocks of the text form, out of the one metadata -/// table the database keeps them in. -fn metadata( - conn: sqlight.Connection, - doc_id: Int, - document: Document, -) -> Document { - let d = { - use key <- decode.field(0, decode.string) - use value <- decode.field(1, decode.string) - decode.success(#(key, value)) - } - let rows = - sqlight.query( - "select key, value from metadata where doc_id = " - <> int.to_string(doc_id) - <> " order by key", - on: conn, - with: [], - expecting: d, - ) - |> result.unwrap([]) - - list.fold(rows, document, fn(doc, row) { - let #(key, raw) = row - let value = string.trim(raw) - case key { - // source.* describes the artefact rather than the document, and is - // @source in the text form - "source." <> _ -> doc - // translation.ocn_aligned says whether the languages of this document - // share one numbering. A .ocda.db can know it because it holds every - // language; a .ssp is one language and cannot. It is a fact about the - // file, not the document, and carrying it here would let a writer - // tell which serialisation it was handed. - "translation." <> _ -> doc - // the database tells the three blocks apart by a key prefix. - // Unprefixing them is what makes the two loaders give one model: a - // writer asks make["footer"], never metadata["make.footer"]. - "make." <> rest -> - model.Document(..doc, make: dict.insert(doc.make, rest, value)) - "doc_has." <> rest -> - model.Document(..doc, doc_has: dict.insert(doc.doc_has, rest, value)) - _ -> - model.Document(..doc, metadata: dict.insert(doc.metadata, key, value)) - } - }) -} - -/// The objects, in document order within each section, the sections in the -/// order the format gives rather than the alphabetical one sqlite would. -fn objects( - conn: sqlight.Connection, - doc_id: Int, - document: Document, -) -> Document { - let d = { - use section <- decode.field(0, decode.string) - use ocn <- decode.field(2, decode.int) - use is_a <- decode.field(3, decode.string) - use part <- decode.field(4, decode.string) - use is_of_section <- decode.field(5, decode.string) - use is_of_type <- decode.field(6, decode.string) - use heading_level <- decode.field(7, decode.int) - use identifier <- decode.field(8, decode.string) - use parent <- decode.field(9, decode.int) - use last_descendant <- decode.field(10, decode.int) - use children <- decode.field(11, decode.string) - use ancestors <- decode.field(12, decode.string) - use dummy <- decode.field(13, decode.int) - use ocn_off <- decode.field(14, decode.int) - use indent_base <- decode.field(15, decode.int) - use indent_hang <- decode.field(16, decode.int) - use bullet <- decode.field(17, decode.int) - use lang <- decode.field(18, decode.string) - use has_links <- decode.field(19, decode.int) - use has_notes_reg <- decode.field(20, decode.int) - use has_notes_star <- decode.field(21, decode.int) - use has_images <- decode.field(22, decode.int) - use segment <- decode.field(23, decode.string) - use segment_prev <- decode.field(24, decode.string) - use segment_next <- decode.field(25, decode.string) - use anchor <- decode.field(26, decode.string) - use segment_html_is <- decode.field(27, decode.string) - use table_cols <- decode.field(28, decode.int) - use table_widths <- decode.field(29, decode.string) - use table_aligns <- decode.field(30, decode.string) - use table_header <- decode.field(31, decode.int) - use code_linenumbers <- decode.field(32, decode.int) - use hat <- decode.field(33, decode.string) - use sha256 <- decode.field(34, decode.string) - use text <- decode.field(35, decode.string) - use id <- decode.field(36, decode.int) - use hla <- decode.field(37, decode.string) - use seg_epub <- decode.field(38, decode.string) - use seg_epub_is <- decode.field(39, decode.string) - use range_first <- decode.field(40, decode.int) - use range_last <- decode.field(41, decode.int) - decode.success(#( - id, - Object( - ocn: ocn, - is_a: is_a, - section: section, - heading_level: level_name(heading_level), - identifier: identifier, - part: part, - is_of_type: is_of_type, - own_section: case is_of_section == section { - True -> "" - False -> is_of_section - }, - segment_html_is: segment_html_is, - table_aligns: json_words(table_aligns), - stow_link: [], - anchor_tags: [], - parent_ocn: parent, - last_descendant_ocn: last_descendant, - ocn_range_first: range_first, - ocn_range_last: range_last, - children: json_ints(children), - ancestors: json_ints(ancestors), - dummy_heading: dummy != 0, - ocn_off: ocn_off != 0, - indent_base: indent_base, - indent_hang: indent_hang, - bullet: bullet != 0, - lang: lang, - has_links: has_links != 0, - has_notes_reg: has_notes_reg != 0, - has_notes_star: has_notes_star != 0, - has_images: has_images != 0, - segment: segment, - segment_prev: segment_prev, - segment_next: segment_next, - anchor: anchor, - heading_lev_anchor: hla, - segment_epub: seg_epub, - segment_epub_is: seg_epub_is, - table_cols: table_cols, - table_widths: json_words(table_widths), - table_header: table_header != 0, - code_syntax: "", - code_linenumbers: code_linenumbers != 0, - heading_ancestors_text: json_joined(hat), - sha256: sha256, - images: [], - text: text, - ), - )) - } - let rows = - sqlight.query( - object_sql(doc_id, has_range_columns(conn)), - on: conn, - with: [], - expecting: d, - ) - |> result.unwrap([]) - - // Built newest-first per section and reversed once at the end. Appending - // to the tail of each section instead would be quadratic, and the largest - // document in the reference collection has some thousands of objects in - // one section. - let sections = - list.fold(rows, dict.new(), fn(acc, row) { - let #(id, object) = row - // the four repeatable properties of the text form are child tables - // here, one query each - let object = - Object( - ..object, - images: images(conn, id), - stow_link: links(conn, id), - anchor_tags: anchors(conn, id), - ) - case model.is_section(object.section) { - False -> acc - True -> - dict.upsert(acc, object.section, fn(existing) { - case existing { - option.Some(previous) -> [object, ..previous] - option.None -> [object] - } - }) - } - }) - |> dict.map_values(fn(_, objects) { list.reverse(objects) }) - model.Document(..document, sections: sections) -} - -/// 2.1 added the container range as two columns. A 2.0 database has -/// neither, and reads as having no range, which is what 2.0 meant. -fn has_range_columns(conn: sqlight.Connection) -> Bool { - let d = decode.field(0, decode.int, decode.success) - case - sqlight.query( - "select count(*) from pragma_table_info('objects') - where name = 'ocn_range_first'", - on: conn, - with: [], - expecting: d, - ) - { - Ok([n]) -> n > 0 - _ -> False - } -} - -fn object_sql(doc_id: Int, has_range: Bool) -> String { - let range_cols = case has_range { - True -> "coalesce(ocn_range_first,0), coalesce(ocn_range_last,0)" - False -> "0, 0" - } - "select section, seq, ocn, is_a, coalesce(is_of_part,''), - coalesce(is_of_section,''), coalesce(is_of_type,''), - coalesce(heading_level,-1), coalesce(identifier,''), - coalesce(parent_ocn,0), coalesce(last_descendant_ocn,0), - coalesce(children,''), coalesce(ancestors,''), - coalesce(dummy_heading,0), coalesce(object_number_off,0), - coalesce(indent_base,0), coalesce(indent_hang,0), - coalesce(bullet,0), coalesce(lang,''), - coalesce(has_links,0), coalesce(has_notes_reg,0), - coalesce(has_notes_star,0), coalesce(has_images,0), - coalesce(segment,''), coalesce(segment_prev,''), - coalesce(segment_next,''), coalesce(anchor,''), - coalesce(segment_html_is,''), - coalesce(table_cols,0), coalesce(table_widths,''), - coalesce(table_aligns,''), coalesce(table_header,0), - coalesce(code_linenumbers,0), - coalesce(heading_ancestors_text,''), coalesce(sha256,''), - coalesce(text,''), id, coalesce(heading_lev_anchor,''), - coalesce(segment_epub,''), coalesce(segment_epub_is,''), " <> range_cols <> " - from objects where doc_id = " <> int.to_string(doc_id) <> " - order by case section - when 'head' then 1 when 'toc' then 2 when 'body' then 3 - when 'endnotes' then 4 when 'glossary' then 5 - when 'bibliography' then 6 when 'bookindex' then 7 - when 'blurb' then 8 when 'tail' then 9 else 10 end, seq" -} - -fn images(conn: sqlight.Connection, object_id: Int) -> List(model.Image) { - let d = { - use name <- decode.field(0, decode.string) - use sha256 <- decode.field(1, decode.string) - use bytes <- decode.field(2, decode.int) - use width <- decode.field(3, decode.int) - use height <- decode.field(4, decode.int) - use missing <- decode.field(5, decode.int) - decode.success(Image( - name: name, - sha256: sha256, - bytes: bytes, - width: width, - height: height, - missing: missing != 0, - fields: dict.new(), - )) - } - sqlight.query( - "select name, coalesce(sha256,''), coalesce(bytes,0), - coalesce(width,0), coalesce(height,0), coalesce(missing,0) - from object_images where object_id = " - <> int.to_string(object_id) - <> " order by seq", - on: conn, - with: [], - expecting: d, - ) - |> result.unwrap([]) -} - -fn links(conn: sqlight.Connection, object_id: Int) -> List(String) { - let d = { - use url <- decode.field(0, decode.string) - decode.success(url) - } - sqlight.query( - "select url from object_links where object_id = " - <> int.to_string(object_id) - <> " order by seq", - on: conn, - with: [], - expecting: d, - ) - |> result.unwrap([]) -} - -fn anchors(conn: sqlight.Connection, object_id: Int) -> List(String) { - let d = { - use anchor <- decode.field(0, decode.string) - decode.success(anchor) - } - sqlight.query( - "select anchor from object_anchors where object_id = " - <> int.to_string(object_id) - <> " order by seq", - on: conn, - with: [], - expecting: d, - ) - |> result.unwrap([]) -} - -fn one(conn: sqlight.Connection, sql: String) -> String { - let d = { - use v <- decode.field(0, decode.string) - decode.success(v) - } - case sqlight.query(sql, on: conn, with: [], expecting: d) { - Ok([v, ..]) -> v - _ -> "" - } -} - -fn has(conn: sqlight.Connection, name: String) -> Bool { - one( - conn, - "select cast(count(*) as text) from sqlite_master where name = '" - <> name - <> "'", - ) - != "0" -} - -fn meta_value(conn: sqlight.Connection, doc_id: Int, key: String) -> String { - one( - conn, - "select value from metadata where doc_id = " - <> int.to_string(doc_id) - <> " and key = '" - <> key - <> "'", - ) -} - -fn major_of(version: String) -> String { - case string.split_once(version, ".") { - Ok(#(major, _)) -> major - Error(_) -> version - } -} - -/// The eight heading levels and their indices. -fn level_name(index: Int) -> String { - case index { - 0 -> "A" - 1 -> "B" - 2 -> "C" - 3 -> "D" - 4 -> "1" - 5 -> "2" - 6 -> "3" - 7 -> "4" - _ -> "" - } -} - -/// The JSON columns, read as JSON. -/// -/// Not as a character scan that strips brackets and quotes: that is wrong -/// the moment a value holds an escape, and spine's encoder writes "/" as -/// "\/", so every heading containing a slash comes back with a backslash in -/// it. The D library had exactly that bug and all 36 document-languages -/// differed on heading_ancestors_text because of it. -fn json_strings(value: String) -> List(String) { - case value { - "" -> [] - v -> - case json.parse(v, decode.list(of: loose_string())) { - Ok(items) -> items - Error(_) -> [] - } - } -} - -/// An element that may be a string or a number, which is what these columns -/// hold: children and ancestors are int arrays, table_aligns and -/// heading_ancestors_text string arrays, and table_widths an int array -/// except where the columns divide evenly, when it is [33.3333,...]. One -/// table in the reference collection does that, and without the float -/// branch the widths came back as three empty strings: the fallback turned -/// a value it could not read into a value that looked read. -fn loose_string() -> decode.Decoder(String) { - decode.one_of(decode.string, [ - decode.int |> decode.map(int.to_string), - decode.float |> decode.map(float.to_string), - decode.success(""), - ]) -} - -fn json_ints(value: String) -> List(Int) { - value - |> json_strings - |> list.filter(fn(s) { s != "" }) - |> list.map(fn(s) { s |> int.parse |> result.unwrap(0) }) -} - -/// Space separated in the text form: table_widths, table_aligns. -fn json_words(value: String) -> String { - value |> json_strings |> string.join(" ") -} - -/// heading_ancestors_text is eight slots, joined with "|" in the text form -/// and a JSON array here. The empty slots are significant: they say which -/// levels the heading sits under. -fn json_joined(value: String) -> String { - case json_strings(value) { - [] -> "" - parts -> string.join(parts, "|") - } -} - -/// What the database holds and the model does not. -/// -/// The text loader meets every property line and counts those it does not -/// keep, so a reader can say what was read and not understood. This loader -/// asks for columns by name, and would never see the ones it did not ask -/// for: it would report that it had understood everything, which is the -/// silence the count exists to prevent. So it counts them, on the -/// conditions under which the `.ssp` writes the same property -/// (specification section 6.3), and the report is the same from either -/// form, as the D loader's is. -fn count_unread( - conn: sqlight.Connection, - doc_id: Int, - document: Document, -) -> Document { - let id = int.to_string(doc_id) - let counts = [ - #("ancestors_collapsed", "coalesce(ancestors_collapsed,'') != ''"), - #("dom_status", "coalesce(dom_status,'') != ''"), - #("dom_status_collapsed", "coalesce(dom_status_collapsed,'') != ''"), - #("heading_lev_collapsed", "heading_lev_collapsed is not null"), - #("parent_lev", "coalesce(parent_lev,0) != 0"), - #("segment_lv4_is", "coalesce(segment_lv4_is,'') != ''"), - #("attrib", "coalesce(attrib,'') != ''"), - ] - let property = - list.fold(counts, document.unknown.property, fn(acc, c) { - let n = - count( - conn, - "select count(*) from objects where doc_id = " <> id <> " and " <> c.1, - ) - case n > 0 { - True -> dict.insert(acc, c.0, n) - False -> acc - } - }) - let property = case has(conn, "object_subtoc") { - False -> property - True -> { - let n = - count( - conn, - "select count(*) from object_subtoc s join objects o - on o.id = s.object_id where o.doc_id = " <> id, - ) - case n > 0 { - True -> dict.insert(property, "lev4_subtoc", n) - False -> property - } - } - } - model.Document( - ..document, - unknown: model.Unknown(..document.unknown, property: property), - ) -} - -fn count(conn: sqlight.Connection, sql: String) -> Int { - let d = decode.field(0, decode.int, decode.success) - case sqlight.query(sql, on: conn, with: [], expecting: d) { - Ok([n, ..]) -> n - _ -> 0 - } -} - -/// Every language the database at `path` holds, in manifest order. -pub fn languages_of_file(path: String) -> Result(List(Language), String) { - use conn <- with_connection("file:" <> path <> "?mode=ro", path) - Ok(languages(conn)) -} - -/// What the database at `path` carries besides the abstraction, by role -/// and storage: role, compression, count, original bytes, stored bytes. -pub fn carried_of_file(path: String) -> Result(List(List(String)), String) { - use conn <- with_connection("file:" <> path <> "?mode=ro", path) - case has(conn, "files") { - False -> Ok([]) - True -> { - let d = { - use role <- decode.field(0, decode.string) - use compression <- decode.field(1, decode.string) - use n <- decode.field(2, decode.int) - use bytes <- decode.field(3, decode.int) - use stored <- decode.field(4, decode.int) - decode.success([ - role, - compression, - int.to_string(n), - int.to_string(bytes), - int.to_string(stored), - ]) - } - sqlight.query( - "select role, compression, count(*), sum(bytes), sum(length(data)) - from files group by role, compression order by role", - on: conn, - with: [], - expecting: d, - ) - |> result.map_error(fn(e) { path <> ": " <> e.message }) - } - } -} -#+END_SRC - -**** images - -#+NAME: ocda_images -#+HEADER: :no-expand -#+BEGIN_SRC gleam -//// An image's bytes, from whichever source the document came from. -//// -//// The model *names* images, with digest, byte size and pixel size; it does -//// not hold them. That is right - an abstraction of a document is not a -//// bundle of its assets - but a writer of any format that shows images needs -//// the bytes, so something has to know where they are. This is that -//// something, and it is in the library rather than in a writer because the -//// answer depends on the *input form* and not on the output. -//// -//// | `.ocda.db` | the `files` table, role 'image', by name | -//// | `.zip` | the archive entry whose path ends in `/image/<name>` | -//// | `.sisupod` | the same, if the frame can be unwrapped, which on the BEAM it cannot | -//// | `.ssp` | beside it: a pod puts the abstraction in `media/abstraction/` and its images in `media/image/` | - -import gleam/bit_array -import gleam/dynamic/decode -import gleam/list -import gleam/result -import gleam/string -import ocda/load -import ocda/zip -import simplifile -import sqlight - -/// The bytes, or an error saying why not. -pub fn image_bytes(source: String, name: String) -> Result(BitArray, String) { - case load.source_of(source) { - load.OcdaDb -> from_db(source, name) - load.Pod -> from_archive(source, name) - load.Ssp -> from_beside(source, name) - load.Unknown -> Error(source <> ": not a source this library reads") - } -} - -/// Can this source give images at all? -/// -/// A writer asks before it decides whether to emit them, because an `<img>` -/// pointing at a file it did not include is worse than no image. -pub fn can_give_images(source: String) -> Bool { - case load.source_of(source) { - load.OcdaDb -> True - load.Pod -> zip_readable(source) - load.Ssp -> image_dir_beside(source) != "" - load.Unknown -> False - } -} - -fn zip_readable(path: String) -> Bool { - case simplifile.read_bits(path) { - Error(_) -> False - Ok(bytes) -> - case zip.entries(bytes) { - Ok(_) -> True - Error(_) -> False - } - } -} - -fn from_db(path: String, name: String) -> Result(BitArray, String) { - case sqlight.open("file:" <> path <> "?mode=ro") { - Error(e) -> Error(path <> ": " <> e.message) - Ok(conn) -> { - let out = - sqlight.query( - "select data from files where role = 'image' and name = ?1", - on: conn, - with: [sqlight.text(name)], - expecting: blob(), - ) - |> result.replace_error(path <> ": could not read " <> name) - let _ = sqlight.close(conn) - case out { - Ok([data, ..]) -> Ok(data) - Ok([]) -> Error(name <> ": not in this database") - Error(e) -> Error(e) - } - } - } -} - -/// A blob column, read as bytes rather than as text: an image read as text -/// stops at the first zero byte, and a png's ninth byte is zero. -fn blob() -> decode.Decoder(BitArray) { - use v <- decode.field(0, decode.bit_array) - decode.success(v) -} - -fn from_archive(path: String, name: String) -> Result(BitArray, String) { - use bytes <- result.try( - simplifile.read_bits(path) - |> result.replace_error(path <> ": cannot read"), - ) - use entries <- result.try(zip.entries(bytes)) - case list.find(entries, fn(e) { string.ends_with(e.name, "/image/" <> name) }) { - Error(_) -> Error(name <> ": not in this archive") - Ok(entry) -> zip.read(bytes, entry) - } -} - -fn from_beside(path: String, name: String) -> Result(BitArray, String) { - case image_dir_beside(path) { - "" -> Error(name <> ": a .ssp outside a pod has no image directory beside it") - dir -> - simplifile.read_bits(dir <> "/" <> name) - |> result.replace_error(dir <> "/" <> name <> ": cannot read") - } -} - -/// `media/abstraction/<x>.ssp` -> `media/image/` -fn image_dir_beside(path: String) -> String { - let parts = string.split(path, on: "/") - let dir = case list.reverse(parts) { - [_file, ..rest] -> string.join(list.reverse(rest), "/") - [] -> "" - } - case string.ends_with(dir, "abstraction") { - False -> "" - True -> { - let parent = case list.reverse(string.split(dir, on: "/")) { - [_, ..rest] -> string.join(list.reverse(rest), "/") - [] -> "" - } - let image_dir = parent <> "/image" - case simplifile.is_directory(image_dir) { - Ok(True) -> image_dir - _ -> "" - } - } - } -} - -/// Unused here, kept so that a caller can size a read before making it. -pub fn byte_size(b: BitArray) -> Int { - bit_array.byte_size(b) -} -#+END_SRC - -*** compression - zip - -#+NAME: ocda_zip -#+HEADER: :no-expand -#+BEGIN_SRC gleam -//// Enough of the zip format to find one file in a pod and read it. -//// -//// Not a zip library. A pod archive holds a handful of entries, spine writes -//// them **stored** (uncompressed) so that the outer zstd frame does the -//// compressing once over the whole thing, and what is wanted from it here is -//// one named entry. -//// -//// Deflate is *refused rather than mishandled*. Erlang's `zlib` could -//// inflate it, but a library that says it read an entry it in fact skipped is -//// worse than one that says it cannot, and spine's own archives do not need -//// it. - -import gleam/bit_array -import gleam/int -import gleam/list -import gleam/result -import gleam/string - -pub type Entry { - Entry( - name: String, - /// 0 stored, 8 deflate - method: Int, - compressed: Int, - uncompressed: Int, - local_offset: Int, - ) -} - -const eocd_sig = 0x06054b50 - -const cd_sig = 0x02014b50 - -const local_sig = 0x04034b50 - -/// The entries, from the central directory at the end of the file. -pub fn entries(bytes: BitArray) -> Result(List(Entry), String) { - let size = bit_array.byte_size(bytes) - use _ <- result.try(case size >= 22 { - True -> Ok(Nil) - False -> Error("too short to be a zip archive") - }) - use eocd <- result.try(find_eocd(bytes, size - 22, lowest(size))) - use count <- result.try(u16(bytes, eocd + 10)) - use offset <- result.try(u32(bytes, eocd + 16)) - read_central(bytes, offset, count, []) -} - -/// The end of central directory record is last, and may be followed by a -/// comment of up to 65535 bytes, so it is searched for backwards. -fn find_eocd(bytes: BitArray, at: Int, lowest: Int) -> Result(Int, String) { - case at < lowest { - True -> - Error( - "no end of central directory record; it does not look like a zip archive", - ) - False -> - case u32(bytes, at) { - Ok(sig) if sig == eocd_sig -> Ok(at) - _ -> find_eocd(bytes, at - 1, lowest) - } - } -} - -fn lowest(size: Int) -> Int { - case size > 22 + 65_535 { - True -> size - 22 - 65_535 - False -> 0 - } -} - -fn read_central( - bytes: BitArray, - at: Int, - remaining: Int, - acc: List(Entry), -) -> Result(List(Entry), String) { - case remaining { - 0 -> Ok(list.reverse(acc)) - _ -> { - use sig <- result.try(u32(bytes, at)) - use _ <- result.try(case sig == cd_sig { - True -> Ok(Nil) - False -> - Error( - "central directory entry " - <> int.to_string(list.length(acc)) - <> " is malformed", - ) - }) - use method <- result.try(u16(bytes, at + 10)) - use compressed <- result.try(u32(bytes, at + 20)) - use uncompressed <- result.try(u32(bytes, at + 24)) - use name_len <- result.try(u16(bytes, at + 28)) - use extra_len <- result.try(u16(bytes, at + 30)) - use cmnt_len <- result.try(u16(bytes, at + 32)) - use local_offset <- result.try(u32(bytes, at + 42)) - use name <- result.try(text(bytes, at + 46, name_len)) - let entry = - Entry( - name: name, - method: method, - compressed: compressed, - uncompressed: uncompressed, - local_offset: local_offset, - ) - read_central( - bytes, - at + 46 + name_len + extra_len + cmnt_len, - remaining - 1, - [entry, ..acc], - ) - } - } -} - -/// One entry's bytes, from its local header. -pub fn read(bytes: BitArray, entry: Entry) -> Result(BitArray, String) { - use sig <- result.try(u32(bytes, entry.local_offset)) - use _ <- result.try(case sig == local_sig { - True -> Ok(Nil) - False -> Error(entry.name <> ": local header is malformed") - }) - use name_len <- result.try(u16(bytes, entry.local_offset + 26)) - use extra_len <- result.try(u16(bytes, entry.local_offset + 28)) - use _ <- result.try(case entry.method { - 0 -> Ok(Nil) - m -> - Error( - entry.name - <> ": compression method " - <> int.to_string(m) - <> "; this reader takes only stored entries, which is what spine writes", - ) - }) - let start = entry.local_offset + 30 + name_len + extra_len - case bit_array.slice(bytes, start, entry.compressed) { - Ok(slice) -> Ok(slice) - Error(_) -> Error(entry.name <> ": runs past the end of the archive") - } -} - -/// The entries whose names end this way, in archive order. Used to find -/// media/abstraction/<uid>.<lang>.ssp without knowing the uid. -pub fn find_suffix(entries: List(Entry), suffix: String) -> List(Entry) { - list.filter(entries, fn(e) { string.ends_with(e.name, suffix) }) -} - -fn u16(bytes: BitArray, at: Int) -> Result(Int, String) { - case bit_array.slice(bytes, at, 2) { - Ok(<<v:little-size(16)>>) -> Ok(v) - _ -> Error("read past the end of the archive at " <> int.to_string(at)) - } -} - -fn u32(bytes: BitArray, at: Int) -> Result(Int, String) { - case bit_array.slice(bytes, at, 4) { - Ok(<<v:little-size(32)>>) -> Ok(v) - _ -> Error("read past the end of the archive at " <> int.to_string(at)) - } -} - -fn text(bytes: BitArray, at: Int, len: Int) -> Result(String, String) { - case bit_array.slice(bytes, at, len) { - Ok(slice) -> - bit_array.to_string(slice) - |> result.replace_error("an entry name is not valid UTF-8") - Error(_) -> Error("an entry name runs past the end of the archive") - } -} -#+END_SRC - -*** dump - -#+NAME: ocda_dump -#+HEADER: :no-expand -#+BEGIN_SRC gleam -//// The model, written out canonically. -//// -//// Not an output format: an *oracle*. Two libraries in two languages -//// implement the same model from the same specification, and the cheap way -//// to ask whether they agree is to have each write the whole loaded -//// document as text and diff the two. Byte for byte identical to the D -//// library's `ocda/dump.d`, which is the whole point of it. -//// -//// The rules, which both implementations follow: -//// -//// - every line is `<key> <value>`, the value running to end of line, -//// and only one free-form value per line; -//// - a field at its default is *omitted*, so the output says what the -//// document has rather than what the type declares; -//// - dictionaries come in sorted key order, because neither language's -//// map has an order worth relying on; -//// - sections come in the fixed order of the format. -//// -//// Anything that reads this to reconstruct a document has misunderstood -//// it. Load the `.ssp` or the `.ocda.db`. - -import gleam/dict.{type Dict} -import gleam/int -import gleam/list -import gleam/string -import ocda/model.{type Document, type Object} - -pub fn document(doc: Document) -> String { - let lines = - [] - |> kv("version", doc.format_version) - |> kv("source_filename", doc.source_filename) - |> entries("source", doc.source) - |> entries("meta", doc.metadata) - |> entries("make", doc.make) - |> entries("doc_has", doc.doc_has) - |> counts(doc) - |> objects(doc) - lines |> list.reverse |> join -} - -/// What the loader met and did not understand, sorted. -/// -/// Separate from the document dump: it describes the loader, not the -/// document, and the `.ocda.db` loader has nothing to report where the -/// `.ssp` loader does. -pub fn unknown(doc: Document) -> String { - [] - |> numbered("property", doc.unknown.property) - |> numbered("block", doc.unknown.block) - |> numbered("line", doc.unknown.line) - |> list.reverse - |> join -} - -fn join(lines: List(String)) -> String { - case lines { - [] -> "" - _ -> string.join(lines, "\n") <> "\n" - } -} - -/// Accumulated newest-first and reversed once, so that nothing appends to -/// the tail of a list that grows to a few hundred thousand lines. -fn kv(acc: List(String), key: String, value: String) -> List(String) { - case value { - "" -> acc - v -> [key <> " " <> v, ..acc] - } -} - -fn kn(acc: List(String), key: String, value: Int) -> List(String) { - case value { - 0 -> acc - n -> [key <> " " <> int.to_string(n), ..acc] - } -} - -fn kb(acc: List(String), key: String, value: Bool) -> List(String) { - case value { - False -> acc - True -> [key <> " 1", ..acc] - } -} - -fn entries( - acc: List(String), - prefix: String, - d: Dict(String, String), -) -> List(String) { - d - |> dict.to_list - |> list.sort(fn(a, b) { string.compare(a.0, b.0) }) - |> list.fold(acc, fn(lines, pair) { - [prefix <> " " <> pair.0 <> " " <> pair.1, ..lines] - }) -} - -fn numbered( - acc: List(String), - prefix: String, - d: Dict(String, Int), -) -> List(String) { - d - |> dict.to_list - |> list.sort(fn(a, b) { string.compare(a.0, b.0) }) - |> list.fold(acc, fn(lines, pair) { - [prefix <> " " <> pair.0 <> " " <> int.to_string(pair.1), ..lines] - }) -} - -fn counts(acc: List(String), doc: Document) -> List(String) { - list.fold(model.section_names, acc, fn(lines, name) { - case model.section(doc, name) { - [] -> lines - objects -> - [ - "section " <> name <> " " <> int.to_string(list.length(objects)), - ..lines - ] - } - }) -} - -fn objects(acc: List(String), doc: Document) -> List(String) { - list.fold(model.section_names, acc, fn(lines, name) { - model.section(doc, name) - |> list.index_fold(lines, fn(l, obj, i) { - object(l, name, i, obj) - }) - }) -} - -fn object( - acc: List(String), - section: String, - index: Int, - obj: Object, -) -> List(String) { - let acc = ["object " <> section <> " " <> int.to_string(index), ..acc] - acc - |> kn("ocn", obj.ocn) - |> kv("is_a", obj.is_a) - |> kv("own_section", obj.own_section) - |> kv("level", obj.heading_level) - |> kv("identifier", obj.identifier) - |> kv("part", obj.part) - |> kv("is_of_type", obj.is_of_type) - |> kv("segment_html_is", obj.segment_html_is) - |> kv("segment_epub", obj.segment_epub) - |> kv("segment_epub_is", obj.segment_epub_is) - |> kn("parent", obj.parent_ocn) - |> kn("last_descendant", obj.last_descendant_ocn) - |> kn("ocn_range_first", obj.ocn_range_first) - |> kn("ocn_range_last", obj.ocn_range_last) - |> kv("children", ints(obj.children)) - |> kv("ancestors", ints(obj.ancestors)) - |> kb("dummy", obj.dummy_heading) - |> kb("ocn_off", obj.ocn_off) - |> kn("indent_base", obj.indent_base) - |> kn("indent_hang", obj.indent_hang) - |> kb("bullet", obj.bullet) - |> kv("lang", obj.lang) - |> kb("has_links", obj.has_links) - |> kb("has_notes_reg", obj.has_notes_reg) - |> kb("has_notes_star", obj.has_notes_star) - |> kb("has_images", obj.has_images) - |> kv("segment", obj.segment) - |> kv("segment_prev", obj.segment_prev) - |> kv("segment_next", obj.segment_next) - |> kv("anchor", obj.anchor) - |> kv("heading_lev_anchor", obj.heading_lev_anchor) - |> kn("table_cols", obj.table_cols) - |> kv("table_widths", obj.table_widths) - |> kv("table_aligns", obj.table_aligns) - |> kb("table_header", obj.table_header) - |> kv("code_syntax", obj.code_syntax) - |> kb("code_linenumbers", obj.code_linenumbers) - |> kv("heading_ancestors_text", obj.heading_ancestors_text) - |> kv("sha256", obj.sha256) - |> links(obj) - |> anchor_tags(obj) - |> images(obj) - |> text(obj) -} - -fn links(acc: List(String), obj: Object) -> List(String) { - list.fold(obj.stow_link, acc, fn(lines, url) { kv(lines, "stow_link", url) }) -} - -fn anchor_tags(acc: List(String), obj: Object) -> List(String) { - list.fold(obj.anchor_tags, acc, fn(lines, a) { kv(lines, "anchor_tag", a) }) -} - -fn images(acc: List(String), obj: Object) -> List(String) { - list.fold(obj.images, acc, fn(lines, img) { - // the name last, because a filename may hold a space where none of the - // other fields can - let lines = - kv( - lines, - "image", - img.sha256 - <> " " - <> int.to_string(img.bytes) - <> " " - <> int.to_string(img.width) - <> "x" - <> int.to_string(img.height) - <> " " - <> case img.missing { - True -> "1" - False -> "0" - } - <> " " - <> img.name, - ) - img.fields - |> dict.to_list - |> list.sort(fn(a, b) { string.compare(a.0, b.0) }) - |> list.fold(lines, fn(l, pair) { - kv(l, "image_field", pair.0 <> " " <> pair.1) - }) - }) -} - -/// Text, when there is any. -/// -/// An object whose only text line is blank holds "" here, indistinguishable -/// from no text at all, because the model joins the lines into one string. -/// Both libraries lose the same distinction in the same place, which is -/// what makes them comparable; it is recorded rather than hidden. -fn text(acc: List(String), obj: Object) -> List(String) { - case obj.text { - "" -> acc - t -> { - let lines = string.split(t, on: "\n") - let acc = kv(acc, "text", int.to_string(list.length(lines))) - list.fold(lines, acc, fn(l, line) { ["| " <> line, ..l] }) - } - } -} - -fn ints(xs: List(Int)) -> String { - case xs { - [] -> "" - _ -> xs |> list.map(int.to_string) |> string.join(",") - } -} -#+END_SRC - -*** erlang - -#+NAME: ocda_erlang_ffi -#+HEADER: :no-expand -#+BEGIN_SRC erlang -%% The one thing ocda-lib asks of Erlang directly: a sha256, which OTP's -%% crypto has and the Gleam standard library does not. Upper case hex, as -%% the database stores it and the D library writes it. --module(ocda_ffi). --export([sha256_hex/1]). - -sha256_hex(Bin) -> binary:encode_hex(crypto:hash(sha256, Bin)). -#+END_SRC - -* read -** main - -#+NAME: read -#+HEADER: :no-expand -#+BEGIN_SRC gleam -//// ocda-read, in Gleam: reads the abstraction, in any of the forms the -//// library reads, and says what is in it. -//// -//// ocda-read [--tree|--outline|--languages|--files|--report-unknown] -//// <file>.ssp | <file>.ocda.db | <file>.zip [lang] -//// -//// The modes the Gleam readers had (the summary, the outline, the unknown -//// report, and the database's languages and files), over the library, so -//// that each reads every form. Their output is the D ocda-read's, byte for -//// byte: two readers in unrelated languages, neither written from the other -//// line by line, agreeing is what says each has read the model and not its -//// own idea of it. A `.sisupod` is the one form the Gleam library cannot -//// open (its zstd frame; see `ocda/load_pod`), and it says so. - -import gleam/dict -import gleam/int -import gleam/io -import gleam/list -import gleam/result -import gleam/string -import ocda/inventory.{type Language} -import ocda/load -import ocda/model.{type Document, type Object} - -/// The format major this reader knows. -const major_read = "2" - -type Options { - Options( - tree: Bool, - languages: Bool, - files: Bool, - report_unknown: Bool, - any_version: Bool, - lang: String, - positional: List(String), - ) -} - -/// Run with the arguments after the program name; returns the exit status. -pub fn run(args: List(String)) -> Int { - case parse(args, Options(False, False, False, False, False, "", [])) { - Error(message) -> { - io.println_error("ocda-read: " <> message <> "; --help lists the options") - 2 - } - Ok(Options(positional: [], ..)) -> { - help() - 0 - } - Ok(options) -> read(options) - } -} - -fn parse(args: List(String), o: Options) -> Result(Options, String) { - case args { - [] -> Ok(Options(..o, positional: list.reverse(o.positional))) - ["--help", ..] | ["-h", ..] -> Ok(Options(..o, positional: [])) - ["--tree", ..rest] | ["--outline", ..rest] -> - parse(rest, Options(..o, tree: True)) - ["--languages", ..rest] -> parse(rest, Options(..o, languages: True)) - ["--files", ..rest] -> parse(rest, Options(..o, files: True)) - ["--report-unknown", ..rest] -> - parse(rest, Options(..o, report_unknown: True)) - ["--any-version", ..rest] -> parse(rest, Options(..o, any_version: True)) - ["--lang", code, ..rest] -> parse(rest, Options(..o, lang: code)) - [flag, ..rest] -> - case string.starts_with(flag, "--lang=") { - True -> parse(rest, Options(..o, lang: string.drop_start(flag, 7))) - False -> - case string.starts_with(flag, "-") { - True -> - case - list.contains( - [ - "--html", "--llm-text", "--llm-jsonl", "--images", "--verify", - "--validate", - ], - flag, - ) - { - True -> - Error(flag <> " is a mode of the D ocda-read, not of this one") - False -> Error("Unrecognized option " <> flag) - } - False -> - parse(rest, Options(..o, positional: [flag, ..o.positional])) - } - } - } -} - -fn help() -> Nil { - io.println("ocda-read - reads the document abstraction, in any of its forms") - io.println("") - io.println( - "Usage: ocda-read [options] <file>.ssp|<file>.ocda.db|<file>.zip [lang]", - ) - io.println("") - io.println(" --tree the headings, as a tree") - io.println(" --outline the same as --tree") - io.println(" --languages the languages the file holds") - io.println(" --files what a .ocda.db carries besides the abstraction") - io.println(" --report-unknown what was read and not understood") - io.println(" --lang=<code> the language to read (default: the first)") - io.println(" --any-version read a format major this reader does not know") -} - -fn read(o: Options) -> Int { - let assert [path, ..more] = o.positional - let lang = case o.lang, more { - "", [given, ..] -> given - l, _ -> l - } - let lang_named = lang != "" - let loaded = { - use held <- result.try(inventory.languages(path)) - let lang = case lang, held { - "", [first, ..] -> first.lang - l, _ -> l - } - use _ <- result.try(case - held != [] && !list.any(held, fn(l) { l.lang == lang }) - { - True -> - Error( - "no language '" - <> lang - <> "' in " - <> path - <> "; it holds: " - <> string.join(list.map(held, fn(l) { l.lang }), " "), - ) - False -> Ok(Nil) - }) - use doc <- result.try(load.load(path, lang, o.any_version)) - Ok(#(held, lang, doc)) - } - case loaded { - Error(message) -> { - io.println_error("Error reading " <> path <> ": " <> message) - 2 - } - Ok(#(held, lang, doc)) -> - case gate(path, doc, o.any_version) { - False -> 3 - True -> { - summary(path, doc, held, lang, lang_named) - case o.languages { - True -> print_languages(held) - False -> Nil - } - case o.files { - True -> print_files(path) - False -> Nil - } - case o.tree { - True -> tree(doc) - False -> Nil - } - case o.report_unknown { - True -> report_unknown(doc) - False -> Nil - } - case o.tree || o.languages || o.files || o.report_unknown { - True -> Nil - False -> - io.println( - "\n(this is the summary: --help lists the modes, each adding its output)", - ) - } - 0 - } - } - } -} - -/// The major part, and refuse what is not known. -fn gate(path: String, doc: Document, any_version: Bool) -> Bool { - case doc.format_version, model.version_major(doc) == major_read, any_version { - "", _, _ -> { - io.println_error("warning: " <> path <> " declares no format version") - True - } - _, True, _ -> True - v, False, True -> { - io.println_error( - "warning: " - <> path - <> " is format v" - <> v - <> ", this reader knows v" - <> major_read - <> ".x; reading anyway", - ) - True - } - v, False, False -> { - io.println_error( - "error: " - <> path - <> " is format v" - <> v - <> ", this reader knows v" - <> major_read - <> ".x", - ) - io.println_error( - " a major version differs: a property may have changed meaning or gone away.", - ) - io.println_error(" --any-version reads it regardless.") - False - } - } -} - -const sections = [ - "head", "toc", "body", "endnotes", "glossary", "bibliography", "bookindex", - "blurb", "tail", -] - -/// The sections whose images are counted: every one but the tail, as the D -/// reader counts them. -const image_sections = [ - "head", "toc", "body", "endnotes", "glossary", "bibliography", "bookindex", - "blurb", -] - -fn summary( - path: String, - doc: Document, - held: List(Language), - lang: String, - lang_named: Bool, -) -> Nil { - io.println("Read: " <> path) - case doc.format_version { - "" -> Nil - v -> io.println(" Format: v" <> v) - } - case dict.get(doc.source, "language") { - Error(_) -> Nil - Ok(language) -> { - let of = case dict.get(doc.source, "languages") { - Ok(ls) -> " of " <> ls - Error(_) -> "" - } - let digest = case dict.get(doc.source, "digest") { - Ok(d) -> ", markup digest " <> string.slice(d, 0, 16) <> "..." - Error(_) -> "" - } - io.println(" Source: " <> language <> of <> digest) - } - } - case list.length(held) > 1 && !lang_named { - True -> io.println(" (reading " <> lang <> "; --lang=<code> for another)") - False -> Nil - } - io.println(" Title: " <> model.meta(doc, "title.main")) - io.println(" Author: " <> model.meta(doc, "creator.author")) - io.println( - " Sections: " - <> string.join( - list.map(sections, fn(s) { - s <> "=" <> int.to_string(list.length(model.section(doc, s))) - }), - " ", - ), - ) - let records = - list.fold(image_sections, 0, fn(n, s) { - list.fold(model.section(doc, s), n, fn(n, o) { - n + list.length(o.images) - }) - }) - case records { - 0 -> Nil - 1 -> io.println(" Images: 1 record") - n -> io.println(" Images: " <> int.to_string(n) <> " records") - } -} - -fn print_languages(held: List(Language)) -> Nil { - io.println("") - io.println(" lang source ssp digest") - list.each(held, fn(l) { - let digest = case string.length(l.ssp_digest) > 16 { - True -> string.slice(l.ssp_digest, 0, 16) <> "..." - False -> l.ssp_digest - } - io.println( - " " - <> string.pad_end(l.lang, 7, " ") - <> " " - <> string.pad_end(l.source_filename, 31, " ") - <> " " - <> digest, - ) - }) -} - -fn print_files(path: String) -> Nil { - io.println("") - case inventory.is_carrier(path) { - False -> - io.println( - " carried files: only a .ocda.db carries files; this is " - <> load.source_name(load.source_of(path)), - ) - True -> - case inventory.carried(path) { - Error(message) -> io.println_error("error: " <> message) - Ok([]) -> io.println(" carries no files") - Ok(rows) -> { - io.println(" carried files:") - io.println(" role stored n original in the file") - list.each(rows, fn(r) { - io.println( - " " - <> string.pad_end(r.role, 9, " ") - <> " " - <> string.pad_end(r.compression, 7, " ") - <> " " - <> string.pad_end(r.count, 5, " ") - <> " " - <> string.pad_end(r.bytes, 13, " ") - <> " " - <> r.stored, - ) - }) - } - } - } -} - -/// The headings, as a tree: from the document's root heading, down the -/// children each heading lists, as the D reader walks it. -fn tree(doc: Document) -> Nil { - io.println("Document: " <> model.meta(doc, "title.main")) - case model.meta(doc, "title.subtitle") { - "" -> Nil - sub -> io.println(" " <> sub) - } - io.println("Author: " <> model.meta(doc, "creator.author")) - io.println("") - let headings = - model.section(doc, "body") - |> list.filter(fn(o) { o.is_a == "heading" && o.ocn > 0 }) - |> list.map(fn(o) { #(o.ocn, o) }) - |> dict.from_list - case model.section(doc, "head") { - [root, ..] if root.is_a == "heading" -> { - io.println(heading_line(root, 0)) - walk(root, headings, 1) - } - _ -> Nil - } -} - -fn walk(parent: Object, headings: dict.Dict(Int, Object), depth: Int) -> Nil { - list.each(parent.children, fn(ocn) { - case dict.get(headings, ocn) { - Ok(child) -> { - io.println(heading_line(child, depth)) - walk(child, headings, depth + 1) - } - Error(_) -> Nil - } - }) -} - -fn heading_line(o: Object, depth: Int) -> String { - let level = case o.heading_level { - "" -> "?" - l -> l - } - let ocn = case o.ocn > 0 { - True -> " [" <> int.to_string(o.ocn) <> "]" - False -> "" - } - string.repeat(" ", depth) <> level <> "~" <> ocn <> " " <> strip_markup(o.text) -} - -/// The internal markers a heading's text may carry, removed for display, as -/// the D reader removes them. -fn strip_markup(text: String) -> String { - text - |> string.replace("⑆*", "") - |> string.replace("⑆!", "") - |> string.replace("⑆/", "") - |> string.replace("⑆_", "") - |> string.replace("⑆^", "") - |> string.replace("⑆,", "") - |> string.replace("┨", "") - |> string.replace("┣", "") - |> string.replace("┙", " ") - |> string.replace("┘", "\n") -} - -/// Say what was read and not understood. -fn report_unknown(doc: Document) -> Nil { - let u = doc.unknown - case model.any_unknown(u) { - False -> - io.println( - "\nread and understood: every property, block and line was known", - ) - True -> { - io.println("\nread and NOT understood:") - section_of_unknown(u.property, " properties ignored (", fn(name, n) { - " ." - <> string.pad_end(name, 26, " ") - <> " " - <> string.pad_start(int.to_string(n), 8, " ") - }) - section_of_unknown(u.block, " blocks ignored (", fn(name, n) { - " @" - <> string.pad_end(name, 26, " ") - <> " " - <> string.pad_start(int.to_string(n), 8, " ") - }) - case dict.size(u.line) { - 0 -> Nil - _ -> { - io.println(" lines of no known shape, by first character:") - dict.to_list(u.line) - |> list.sort(fn(a, b) { string.compare(a.0, b.0) }) - |> list.each(fn(e) { - io.println( - " " - <> string.pad_end("\"" <> e.0 <> "\"", 27, " ") - <> " " - <> string.pad_start(int.to_string(e.1), 8, " "), - ) - }) - } - } - io.println( - " (ignoring these is conforming within a major version; the count is so that it is not silent)", - ) - } - } -} - -fn section_of_unknown( - counts: dict.Dict(String, Int), - heading: String, - line: fn(String, Int) -> String, -) -> Nil { - case dict.size(counts) { - 0 -> Nil - n -> { - io.println(heading <> int.to_string(n) <> " distinct):") - dict.to_list(counts) - |> list.sort(fn(a, b) { string.compare(a.0, b.0) }) - |> list.each(fn(e) { io.println(line(e.0, e.1)) }) - } - } -} -#+END_SRC - -* test - -#+NAME: test -#+HEADER: :no-expand -#+BEGIN_SRC gleam -import gleam/dict -import gleam/list -import simplifile -import gleam/string -import gleeunit -import gleeunit/should -import ocda/dump -import ocda/load -import ocda/load_ssp -import ocda/model - -pub fn main() { - gleeunit.main() -} - -const minimal = "% SiSU Document Abstraction v2.0 -% Source: example.sst - -@source { - language: en - languages: en - digest: ABC -} - -@meta { - title.main: An Example -} - -@body { -[1] heading :A the_title -.part: frontmatter -.sha256: DEAD -| An Example - -[2] para -.part: body -.parent: 1 -| First line -| -| Third line, after a blank one - -} -" - -pub fn version_test() { - let doc = load_ssp.parse(minimal) - doc.format_version |> should.equal("2.0") - model.version_major(doc) |> should.equal("2") - doc.source_filename |> should.equal("example.sst") -} - -pub fn preamble_test() { - let doc = load_ssp.parse(minimal) - model.meta(doc, "title.main") |> should.equal("An Example") -} - -pub fn objects_test() { - let doc = load_ssp.parse(minimal) - let body = model.section(doc, "body") - body |> list.length |> should.equal(2) -} - -/// The blank line in the middle of an object's text is kept, and so is the -/// count: three lines, not two. -pub fn blank_text_line_test() { - let doc = load_ssp.parse(minimal) - case model.section(doc, "body") { - [_, second] -> - second.text - |> should.equal("First line\n\nThird line, after a blank one") - _ -> should.fail() - } -} - -/// A heading's identifier comes from its declaration line; an object whose -/// identifier is simply its ocn has it derived, because the text form does -/// not write what it can work out. -pub fn derived_identifier_test() { - let doc = load_ssp.parse(minimal) - case model.section(doc, "body") { - [first, second] -> { - first.identifier |> should.equal("the_title") - second.identifier |> should.equal("2") - } - _ -> should.fail() - } -} - -pub fn source_of_test() { - load.source_of("a/b.ssp") |> should.equal(load.Ssp) - load.source_of("a/b.ocda.db") |> should.equal(load.OcdaDb) - load.source_of("a/b.sst") |> should.equal(load.Unknown) -} - -/// The dump omits a field at its default, so a minimal document gives a -/// short dump and every line in it is a fact. -pub fn dump_test() { - let text = dump.document(load_ssp.parse(minimal)) - text |> string.contains(contain: "version 2.0") |> should.be_true - text |> string.contains(contain: "meta title.main An Example") |> should.be_true - text |> string.contains(contain: "object body 0") |> should.be_true - text |> string.contains(contain: "level A") |> should.be_true - // a false boolean is not written at all - text |> string.contains(contain: "bullet") |> should.be_false -} - - -/// A property is read, which the fixture above had to be corrected to show: -/// a property line is `.name: value` at the start of the line, and an -/// indented one is a line of no known shape. The fixture indented them, so -/// every property in it was being counted as malformed and the tests around -/// it passed for the wrong reason. -pub fn property_test() { - let doc = load_ssp.parse(minimal) - case model.section(doc, "body") { - [first, second] -> { - first.part |> should.equal("frontmatter") - first.sha256 |> should.equal("DEAD") - second.parent_ocn |> should.equal(1) - } - _ -> should.fail() - } - // and nothing in the fixture is malformed any more - dict_size(doc.unknown.line) |> should.equal(0) -} - -@external(erlang, "maps", "size") -fn dict_size(d: dict.Dict(a, b)) -> Int - -/// What a path is, including the two archive forms. -pub fn pod_source_test() { - load.source_of("a/b.sisupod") |> should.equal(load.Pod) - load.source_of("a/b.zip") |> should.equal(load.Pod) -} - -/// A zstd-wrapped archive is refused, and the refusal says why rather than -/// claiming the file is malformed. This is the one place the D and Gleam -/// libraries differ in what they accept, and it is a packaging limit: the -/// only zstd on the BEAM is a NIF that fetches zstd in its own build. -pub fn zstd_refusal_test() { - let path = "test/fixture-zstd-frame.sisupod" - let assert Ok(_) = simplifile.write_bits(path, <<0x28, 0xB5, 0x2F, 0xFD, 0>>) - let assert Error(message) = load.load(path, "", False) - message |> string.contains(contain: "zstd") |> should.be_true - message |> string.contains(contain: "zstd -d") |> should.be_true - let _ = simplifile.delete(path) - Nil -} -#+END_SRC - -* org-tangles -** ocda lib - -#+HEADER: :tangle ../gleam/src/ocda_lib.gleam -#+HEADER: :noweb yes -#+BEGIN_SRC org -////<<doc_header_including_copyright_and_license>> -<<ocda_lib>> -#+END_SRC - -** ocda -*** model - -#+HEADER: :tangle ../gleam/src/ocda/model.gleam -#+HEADER: :noweb yes -#+BEGIN_SRC org -////<<doc_header_including_copyright_and_license>> -<<model>> -#+END_SRC - -*** what to load -**** inventory - -#+HEADER: :tangle ../gleam/src/ocda/inventory.gleam -#+HEADER: :noweb yes -#+BEGIN_SRC org -////<<doc_header_including_copyright_and_license>> -<<ocda_inventory>> -#+END_SRC - -**** load -***** load - -#+HEADER: :tangle ../gleam/src/ocda/load.gleam -#+HEADER: :noweb yes -#+BEGIN_SRC org -////<<doc_header_including_copyright_and_license>> -<<ocda_load>> -#+END_SRC -***** load pod - -#+HEADER: :tangle ../gleam/src/ocda/load_pod.gleam -#+HEADER: :noweb yes -#+BEGIN_SRC org -////<<doc_header_including_copyright_and_license>> -<<ocda_load_pod>> -#+END_SRC - -***** load ssp - -#+HEADER: :tangle ../gleam/src/ocda/load_ssp.gleam -#+HEADER: :noweb yes -#+BEGIN_SRC org -////<<doc_header_including_copyright_and_license>> -<<ocda_load_ssp>> -#+END_SRC - -***** load db - -#+HEADER: :tangle ../gleam/src/ocda/load_db.gleam -#+HEADER: :noweb yes -#+BEGIN_SRC org -////<<doc_header_including_copyright_and_license>> -<<ocda_load_db>> -#+END_SRC - -**** images - -#+HEADER: :tangle ../gleam/src/ocda/images.gleam -#+HEADER: :noweb yes -#+BEGIN_SRC org -////<<doc_header_including_copyright_and_license>> -<<ocda_images>> -#+END_SRC - -*** compression - zip - -#+HEADER: :tangle ../gleam/src/ocda/zip.gleam -#+HEADER: :noweb yes -#+BEGIN_SRC org -////<<doc_header_including_copyright_and_license>> -<<ocda_zip>> -#+END_SRC - -*** dump - -#+HEADER: :tangle ../gleam/src/ocda/dump.gleam -#+HEADER: :noweb yes -#+BEGIN_SRC org -////<<doc_header_including_copyright_and_license>> -<<ocda_dump>> -#+END_SRC - -*** erlang - -#+HEADER: :tangle ../gleam/src/ocda_ffi.erl -#+HEADER: :noweb yes -#+BEGIN_SRC org -%%<<doc_header_including_copyright_and_license>> -<<ocda_erlang_ffi>> -#+END_SRC - -** read -*** main - -#+HEADER: :tangle ../gleam/src/read/main.gleam -#+HEADER: :noweb yes -#+BEGIN_SRC org -////<<doc_header_including_copyright_and_license>> -<<read>> -#+END_SRC - -** test - -#+HEADER: :tangle ../gleam/test/ocda_lib_test.gleam -#+HEADER: :noweb yes -#+BEGIN_SRC org -////<<doc_header_including_copyright_and_license>> -<<test>> -#+END_SRC - -* config -** nix flake.nix :flake: - -#+HEADER: :tangle ../gleam/flake.nix -#+HEADER: :noweb yes -#+BEGIN_SRC nix -{ - description = "ocda-lib: the object-centric document abstraction as a Gleam library"; - inputs.nixpkgs.url = "<<nixpkgs_unstable>>"; - outputs = { self, nixpkgs }: - let - system = "x86_64-linux"; # adjust for aarch64-linux / darwin - pkgs = import nixpkgs { inherit system; }; - - name = "ocda_lib"; - - /* rebar3 with the port compiler already in it. - esqlite builds a NIF, and its rebar3 configuration asks for the - `pc` plugin, which rebar3 would otherwise fetch from Hex at build - time: impossible in a sandbox, and the build fails with - "missing_package pc". nixpkgs carries the plugin, so the answer is - a rebar3 that has it before it starts. - */ - rebar3-pc = pkgs.beamPackages.rebar3WithPlugins { - plugins = [ pkgs.beamPackages.pc ]; - }; - - /* The Hex dependencies, as manifest.toml pins them. - . - There is no Gleam builder in nixpkgs and `gleam build` fetches from - Hex, so a sandboxed build cannot reach them. A fixed output - derivation may, because its result is pinned by hash. - . - When a dependency changes, manifest.toml changes and this hash must - change with it: build, take the hash nix reports, put it here. - */ - deps = pkgs.stdenv.mkDerivation { - pname = "${name}-deps"; - version = "0.1.0"; - src = ./.; - nativeBuildInputs = [ pkgs.gleam pkgs.cacert ]; - buildPhase = '' - export HOME=$TMPDIR - gleam deps download - ''; - installPhase = '' - mkdir -p $out - cp -r build/packages/* $out/ - ''; - dontFixup = true; - outputHashMode = "recursive"; - outputHashAlgo = "sha256"; - outputHash = "sha256-nWU753YRsC28gsaGyy+A3aZaWf15ctULYiUlvsfyB1c="; - }; - in { - packages.${system} = { - default = self.packages.${system}.${name}; - /* Shipped as an Erlang shipment rather than as an escript, because - sqlite on the BEAM is a NIF: esqlite compiles its own sqlite into - a shared object, and a NIF cannot be loaded from inside an - escript archive, the runtime needing a real file on disk. The - shipment keeps priv/ on disk, so it loads. - */ - ${name} = pkgs.stdenv.mkDerivation { - pname = "ocda-lib"; - version = "0.1.0"; - src = ./.; - nativeBuildInputs = [ - pkgs.gleam - pkgs.erlang - rebar3-pc - pkgs.makeWrapper - ]; - buildPhase = '' - runHook preBuild - export HOME=$TMPDIR - # the dependencies, in place of the fetch gleam would do - mkdir -p build/packages - cp -r ${deps}/* build/packages/ - chmod -R u+w build/packages - gleam export erlang-shipment - runHook postBuild - ''; - installPhase = '' - runHook preInstall - mkdir -p $out/lib $out/bin - cp -r build/erlang-shipment $out/lib/${name} - makeWrapper $out/lib/${name}/entrypoint.sh $out/bin/ocda-lib \ - --add-flags run \ - --prefix PATH : ${pkgs.lib.makeBinPath [ pkgs.erlang pkgs.coreutils ]} - # the reader under its own name, as the D flake installs it - makeWrapper $out/lib/${name}/entrypoint.sh $out/bin/ocda-read \ - --add-flags "run read" \ - --prefix PATH : ${pkgs.lib.makeBinPath [ pkgs.erlang pkgs.coreutils ]} - runHook postInstall - ''; - meta = { - description = "the abstraction as a Gleam library, with its reader, dump and agreement check"; - mainProgram = "ocda-lib"; - }; - }; - }; - devShells.${system}.default = pkgs.mkShell { - packages = [ pkgs.gleam pkgs.erlang pkgs.rebar3 pkgs.sqlite ]; - }; - }; -} -#+END_SRC - -** README.md - -#+HEADER: :tangle ../gleam/README.md -#+HEADER: :noweb yes -#+BEGIN_SRC md -# ocda-lib, in Gleam - -The abstraction's model and both its loaders, as a Gleam library, with its -reader and the two programs that put it to the question. - -| | | -|------------------|-----------------------------------------------------------------------------------------| -| `ocda/model` | the document, the object, the image. Depends on nothing but the standard library | -| `ocda/load_ssp` | text -> model | -| `ocda/load_db` | sqlite -> model | -| `ocda/load` | what a path is, and loading it | -| `ocda/inventory` | what a file holds beyond the one document: its languages, and what a `.ocda.db` carries | -| `ocda/dump` | the model written out canonically, as an oracle | -| `read/main` | `ocda-read`, the reader | -| `ocda_ffi.erl` | a sha256, from OTP's crypto, which the standard library lacks | - -## Use - -```gleam -import ocda/load -import ocda/model - -case load.load(path, "en", False) { - Error(why) -> io.println_error(why) - Ok(doc) -> { - // and from here a writer neither knows nor cares which of the two - // files it was handed - list.each(model.section(doc, "body"), fn(obj) { ... }) - } -} -``` - -## The reader - -```sh -nix build -./result/bin/ocda-read [--tree|--outline|--languages|--files|--report-unknown] - <file>.ssp|<file>.ocda.db|<file>.zip [lang] -``` - -`ocda-lib read`, installed also as `ocda-read`. The modes the Gleam readers -had, over the library, so each reads every form this library reads; the -others are the D reader's. **Its output is the D `ocda-read`'s, byte for -byte**, in every mode both have. In `src/read/`, because Gleam compiles -nothing outside `src/`; a program depending on the library compiles it too, -and does not use it. - -A `.sisupod` it refuses, saying why: the zstd frame, below. - -## The two programs - -```sh -nix build -./result/bin/ocda-lib agree <doc>.ssp <doc>.ocda.db [lang] -./result/bin/ocda-lib dump [--unknown] <file>.ssp|<file>.ocda.db [lang] -``` - -`agree` asks whether the two loaders give the same model. `dump` writes -the whole model out canonically, and is the stronger check of the two, -because it is **byte for byte what the D library's `ocda-dump` writes**. -So the four questions can all be asked with `diff`: - -| | | -|-----------------------------|-----------------------------| -| D: `.ssp` == `.ocda.db` | 36 of 36 document-languages | -| Gleam: `.ssp` == `.ocda.db` | 36 of 36 | -| `.ssp`: D == Gleam | 36 of 36 | -| `.ocda.db`: D == Gleam | 36 of 36 | - -## Deliberately a mirror of the D library - -Field for field, name for name, including where that is not the most -Gleamish choice: - -- `text` is one `String` with the lines joined by newline, not a - `List(String)`. So an object whose only text line is blank is - indistinguishable from an object with no text, *in both languages, in the - same place*. A difference that existed in one and not the other would be - worse than a loss both share, because it would show up in every dump and - mean nothing. -- the properties are typed fields rather than a list of pairs. A writer - wants `obj.parent_ocn`, not a lookup that might miss. - -The point is that the two agree. Making them agree is much cheaper when -they are shaped alike, and the shape is the specification's, not either -language's. - -## What the agreement found - -Four disagreements, none of which a comparison of section counts or of a -handful of fields would have shown. All four are recorded in the D -library's history; the short version: - -1. `.identifier` and `.anchor` are omitted by the text form when they can - be worked out (the identifier is the ocn, the anchor is the segment). - Both loaders now derive them. Over one document that is 1,314 - identifiers. -2. `make` and `doc_has` are one metadata table in the database, - distinguished by a key prefix. Both loaders unprefix them. -3. The JSON columns must be *parsed*, not scanned: the encoder writes `/` - as `\/` and every heading with a slash in it came back wrong. -4. `translation.ocn_aligned` is in the database and cannot be in a `.ssp`, - which is one language. It is a fact about the file, so neither loader - puts it in the document. - -## Build - -```sh -nix build # sandboxed, no network -nix develop -c gleam test -``` - -Shipped as an Erlang shipment rather than an escript, because sqlite on the -BEAM is a NIF and a NIF cannot be loaded from inside an escript archive. -The dependency hash in `flake.nix` pins what `manifest.toml` pins; change -one and the other must change with it. - -## Format - -Specified in **sisudoc-abstraction-format**: `ocda-model.org` for the model, -`ssp-format-specification.org` and `ocda-db-format-specification.org` for the -two serialisations. At format **2.x**. -#+END_SRC - -** gleam.toml - -#+HEADER: :tangle ../gleam/gleam.toml -#+HEADER: :noweb yes -#+BEGIN_SRC toml -name = "ocda_lib" -version = "0.1.0" -description = "the object-centric document abstraction as a library: the model, and loaders for its two serialisations" -licences = ["AGPL-3.0-or-later"] - -[dependencies] -gleam_stdlib = ">= 1.0.0 and < 2.0.0" -argv = ">= 1.1.0 and < 2.0.0" -simplifile = ">= 2.7.0 and < 3.0.0" -sqlight = ">= 1.0.0 and < 2.0.0" -gleam_json = ">= 3.0.0 and < 4.0.0" - -[dev-dependencies] -gleeunit = ">= 1.0.0 and < 2.0.0" -#+END_SRC - -** gleam generated manifest.toml - -#+HEADER: :tangle-NO ../gleam/manifest.toml_ -#+HEADER: :noweb yes -#+BEGIN_SRC toml -# Do not manually edit this file, it is managed by Gleam. -# -# This file locks the dependency versions used, to make your build -# deterministic and to prevent unexpected versions from being included -# in your application. -# -# You should check this file into your source control repository. - -packages = [ - { name = "argv", version = "1.1.0", build_tools = ["gleam"], requirements = [], otp_app = "argv", source = "hex", outer_checksum = "3277D100448BDB4A29B6D58C0F36F631CBC349E8BDD09766C6309DF202831140" }, - { name = "esqlite", version = "0.9.0", build_tools = ["rebar3"], requirements = [], otp_app = "esqlite", source = "hex", outer_checksum = "CCF72258A4EE152EC7AD92AA9A03552EB6CA1B06B65C93AD5B6E55C302E05855" }, - { name = "filepath", version = "1.1.2", build_tools = ["gleam"], requirements = ["gleam_stdlib"], otp_app = "filepath", source = "hex", outer_checksum = "B06A9AF0BF10E51401D64B98E4B627F1D2E48C154967DA7AF4D0914780A6D40A" }, - { name = "gleam_json", version = "3.1.0", build_tools = ["gleam"], requirements = ["gleam_stdlib"], otp_app = "gleam_json", source = "hex", outer_checksum = "44FDAA8847BE8FC48CA7A1C089706BD54BADCC4C45B237A992EDDF9F2CDB2836" }, - { name = "gleam_stdlib", version = "1.0.5", build_tools = ["gleam"], requirements = [], otp_app = "gleam_stdlib", source = "hex", outer_checksum = "CEE5B6C076A85B45F60C585F4316C63EC8B7127C119D5738C3958A9C4D50404E" }, - { name = "gleeunit", version = "1.11.0", build_tools = ["gleam"], requirements = ["gleam_stdlib"], otp_app = "gleeunit", source = "hex", outer_checksum = "EC31ABA74256AEA531EDF8169931D775BBB384FED0A8A1BDC4DD9354E3E21826" }, - { name = "simplifile", version = "2.7.0", build_tools = ["gleam"], requirements = ["filepath", "gleam_stdlib"], otp_app = "simplifile", source = "hex", outer_checksum = "A2727627B063E87351934C7F7F008F2D1FDB16F6DE0B8C79F9E46459CFC9C164" }, - { name = "sqlight", version = "1.2.0", build_tools = ["gleam"], requirements = ["esqlite", "gleam_stdlib"], otp_app = "sqlight", source = "hex", outer_checksum = "841768D0A81107EE2DB46B949354F284A68B2E8098315C28A1FE3C07D206C17A" }, -] - -[requirements] -argv = { version = ">= 1.1.0 and < 2.0.0" } -gleam_json = { version = ">= 3.0.0 and < 4.0.0" } -gleam_stdlib = { version = ">= 1.0.0 and < 2.0.0" } -gleeunit = { version = ">= 1.0.0 and < 2.0.0" } -simplifile = { version = ">= 2.7.0 and < 3.0.0" } -sqlight = { version = ">= 1.0.0 and < 2.0.0" } -#+END_SRC - -* org includes -** project version - -#+NAME: spine_version -#+HEADER: :noweb yes -#+BEGIN_SRC emacs-lisp -<<./sisudoc_spine_version_info_and_doc_header_including_copyright_and_license.org:spine_project_version()>> -#+END_SRC - -** year - -#+NAME: year -#+HEADER: :noweb yes -#+BEGIN_SRC emacs-lisp -<<./sisudoc_spine_version_info_and_doc_header_including_copyright_and_license.org:year()>> -#+END_SRC - -** document header including copyright & license - -#+NAME: doc_header_including_copyright_and_license -#+HEADER: :noweb yes -#+BEGIN_SRC emacs-lisp -<<./sisudoc_spine_version_info_and_doc_header_including_copyright_and_license.org:spine_doc_header_including_copyright_and_license()>> -#+END_SRC - -* nixpkgs source - -#+NAME: nixpkgs_unstable -#+HEADER: :noweb yes -#+BEGIN_SRC emacs-lisp -<<./sisudoc_spine_version_info_and_doc_header_including_copyright_and_license.org:nixpkgs_unstable()>> -#+END_SRC - -* __END__ |
