format 2.0: sample writers over the abstraction, in D and in Gleam

Writers over sisudoc-ocda-lib: functions from the document model to
bytes, in five formats, one program, ocda-write, in each language.
Begun 2026-09-25; this is the writer as it stood at format 2.0.

  ocda-write --text|--md|--html|--epub|--typst <file> [lang]

They are samples. Spine is the output pipeline and nothing here
replaces it. What they show is that the abstraction is sufficient:
given a .ssp, a .ocda.db or a .sisupod and nothing else, a few hundred
lines per format make a complete document. A writer is never told
which form it was given, and every writer is run against every form
of every document.

Each format is held to an oracle that can say no:

  --text   the words equal the html writer's objects, as a multiset;
           every object number exactly once
  --md     CommonMark; every (#x) resolves to an <a id=x>, no id twice
  --html   it parses; every internal link resolves; the objects' text
           equals spine's scroll html
  --epub   epubcheck
  --typst  typst itself: a link to a missing label does not compile

and in every format the same bytes from every input form, and the D
and Gleam writers byte for byte alike. Over the markup document
samples every check holds for every document-language. The two
languages' writers were not written from each other, and their
agreement is the strongest check: twice it caught what nothing else
did (an ampersand in an attribute mangled by Erlang's replacement
string; typst label names built from bytes in one language and
graphemes in the other).

Byte identity with spine is not the aim, by decision. The text writer
began byte identical to spine --text, and was rewritten: a note is
written once, markup comments and dummy headings are not written, an
image keeps its caption, a table is padded to display width. The html
writer is valid html5 rather than a copy of spine's (h1..h6 where
spine writes <h0>; one charset; no placeholder metadata; an id for
every anchor tag; self-contained). Each difference is a better
presentation, and the code says why at the point of choice.

The typst writer puts the object number in the margin as the label
every citation points at, a note as a real footnote with a link back,
and a table of contents that locates by object number rather than by
page, as the markdown writer's does: a page number is a fact about one
typesetting, the object number is not. The markdown writer is
CommonMark with yaml front matter; a note is at its reference, with
the document's own number, and --image-prefix gives the image path,
which the model rightly does not know.

Holding the writers to these checks found defects in spine, several
since fixed there, and spine's typst and markdown writers were ported
from these.

Use: nix build, in dlang/ or in gleam/. Each flake takes the library
from git.sisudoc.org, and flake.lock pins the commit of it these were
built and checked against. gleam/ builds with nix-gleam's builder,
kept in gleam/nix-gleam/ (MIT); in a working tree, gleam.toml reads
the library through the symlink ./ocda-lib.

See README.md, dlang/README.md and gleam/README.md, and pipeline/
for building everything and running it over a directory of artefacts.

(assisted by Claude-Code)

3 files changed, 333 insertions, 0 deletions

diff --git a/gleam/nix-gleam/LICENSE b/gleam/nix-gleam/LICENSE
new file mode 100644
index 0000000..c35ad09
--- /dev/null
+++ b/gleam/nix-gleam/LICENSE
@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2023 Arnar Gauti Ingason
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
diff --git a/gleam/nix-gleam/README.md b/gleam/nix-gleam/README.md
new file mode 100644
index 0000000..39bd1b6
--- /dev/null
+++ b/gleam/nix-gleam/README.md
@@ -0,0 +1,32 @@
+# nix-gleam, kept here
+
+`default.nix` is the builder of **nix-gleam**, by Arnar Gauti Ingason,
+copied unchanged from
+
+- https://github.com/arnarg/nix-gleam
+- `builder/default.nix` at commit `1f38154` (2026-10-06)
+
+under the MIT licence, which is in `LICENSE` beside it.
+
+It is kept here rather than taken as a flake input so that the build does
+not depend on a third repository being reachable, and so that what builds
+this package is in the package.
+
+## Why it is used
+
+There is no Gleam builder in nixpkgs, and a sandboxed build cannot fetch
+from Hex. The usual answer, one fixed output derivation that runs
+`gleam deps download` and is pinned by a hash of everything it fetched, has
+two faults: the hash has to be updated by hand whenever a dependency
+changes, and gleam writes `packages.toml` in a different order on each run,
+so the same dependencies do not always give the same hash.
+
+nix-gleam fetches each Hex package on its own with `fetchHex`, using the
+checksum gleam has already written for it into `manifest.toml`, and writes
+`packages.toml` itself. So there is no hash to maintain, and nothing a run
+of gleam could make come out differently.
+
+## Updating it
+
+Copy `builder/default.nix` from a newer commit over `default.nix`, update
+the commit above, and build.
diff --git a/gleam/nix-gleam/default.nix b/gleam/nix-gleam/default.nix
new file mode 100644
index 0000000..d074263
--- /dev/null
+++ b/gleam/nix-gleam/default.nix
@@ -0,0 +1,280 @@
+{
+  stdenv,
+  lib,
+  git,
+  xxhash,
+  fetchHex,
+  gleam,
+  beamPackages,
+  rsync,
+  nodejs,
+}: let
+  inherit (builtins) fromTOML readFile;
+in {
+  buildGleamApplication = {
+    src,
+    nativeBuildInputs ? [],
+    localPackages ? [],
+    erlangPackage ? beamPackages.erlang,
+    rebar3Package ? beamPackages.rebar3,
+    ...
+  } @ attrs: let
+    # gleam.toml contains an application name and version.
+    gleamToml = lib.importTOML (src + "/gleam.toml");
+
+    # manifest.toml contains a list of required packages including a sha256 checksum
+    # that can be used by nix fetchHex fetcher.
+    manifestToml = lib.importTOML (src + "/manifest.toml");
+
+    # Specify which target to build for.
+    buildTarget = attrs.target or gleamToml.target or "erlang";
+
+    # Generates a packages.toml expected by gleam compiler.
+    packagesTOML = with lib;
+      concatStringsSep "\n" (
+        ["[packages]"]
+        ++ (map
+          (p: "${p.name} = \"${p.version}\"")
+          manifestToml.packages)
+        ++ (lib.optional (gitRecordsTOML != "") "")
+        ++ (lib.optional (gitRecordsTOML != "") gitRecordsTOML)
+      );
+
+    # Gleam records a git dependency twice: once in `[packages]`,
+    # and once in a `[git.<name>]` table with the commit.
+    # Without the second, Gleam will try to download it.
+    gitRecordsTOML = with lib;
+      concatMapStringsSep "\n" (
+        p: "[git.${p.name}]\ncommit = \"${p.commit}\""
+      ) (filterPackagesBySource "git" manifestToml.packages);
+
+    # Helper function to filter manifest.toml packages
+    filterPackagesBySource = type: packages: lib.lists.filter (p: p.source == type) packages;
+
+    gitDerivs =
+      map
+      (p: {
+        name = p.name;
+        derivation = fetchGit {
+          url = p.repo;
+          rev = p.commit;
+        };
+      })
+      (filterPackagesBySource "git" manifestToml.packages);
+
+    # Fetch all dependencies
+    depsDerivs =
+      map
+      (p: {
+        name = p.name;
+        derivation = fetchHex {
+          inherit (p) version;
+          pkg = p.name;
+          sha256 = p.outer_checksum;
+        };
+      })
+      (filterPackagesBySource "hex" manifestToml.packages);
+
+    # Find replacement paths for `local` package dependencies
+    # from `localPackages` list.
+    localDeps = let
+      # Build a lookup attrset for local packages.
+      localDerivs = lib.mergeAttrsList (map (
+          p: let
+            name = (fromTOML (readFile (p + "/gleam.toml"))).name;
+          in {
+            "${name}" = p;
+          }
+        )
+        localPackages);
+    in
+      map (
+        p: {
+          inherit (p) name path;
+          localSrc =
+            if localDerivs ? "${p.name}"
+            then localDerivs.${p.name}
+            else throw "Local dependency \"${p.name}\" not found in `localPackages`.";
+          # Keep local packages in a writable location during build.
+          newPath = p.path;
+        }
+      ) (filterPackagesBySource "local" manifestToml.packages);
+
+    # Check if elixir is needed in nativeBuildInputs by checking if "mix" is in
+    # required build_tools.
+    isElixirProject = with lib; p: any (t: t == "mix") p.build_tools;
+    needsElixir = with lib; any isElixirProject manifestToml.packages;
+
+    # nativeBuildInputs needed for both targets.
+    defaultNativeBuildInputs = [gleam beamPackages.hex rsync git xxhash];
+  in
+    # Base common mkDerivation attributes
+    stdenv.mkDerivation (attrs
+      // {
+        pname = attrs.pname or gleamToml.name;
+        version = attrs.version or gleamToml.version;
+
+        src = lib.cleanSource attrs.src;
+
+        # Here we must copy the dependencies into the right spot and
+        # create a packages.toml file so the gleam compiler does not
+        # attempt to pull the dependencies from the internet.
+        configurePhase =
+          attrs.configurePhase
+          or ''
+            runHook preConfigure
+
+            mkdir -p build/packages
+
+            # Write the packages.toml file
+            cat <<EOF > build/packages/packages.toml
+            ${packagesTOML}
+            EOF
+
+            ${
+              # Gleam writes build output into the local dependency's own source
+              # directory, so it cannot be the read-only store path.
+              lib.concatStringsSep "\n" (
+                lib.forEach localDeps (
+                  d: ''
+                    mkdir -p "${d.newPath}"
+                    rsync --chmod=Du=rwx,Dg=rx,Do=rx,Fu=rw,Fg=r,Fo=r -r ${d.localSrc}/* "${d.newPath}/"
+                  ''
+                )
+              )
+            }
+
+            ${
+              lib.concatStringsSep "\n" (
+                lib.forEach gitDerivs (
+                  d: ''
+                    rsync --chmod=Du=rwx,Dg=rx,Do=rx,Fu=rw,Fg=r,Fo=r -r ${d.derivation}/* build/packages/${d.name}/
+                  ''
+                )
+              )
+            }
+
+            # Copy all the dependencies into place
+            ${lib.concatStringsSep "\n" (
+              lib.forEach depsDerivs (
+                # gleam outputs files inside the dependency's source directory
+                # and therefor it needs to have permissive permissions.
+                d: ''
+                  rsync --chmod=Du=rwx,Dg=rx,Do=rx,Fu=rw,Fg=r,Fo=r -r ${d.derivation}/* build/packages/${d.name}/
+                ''
+              )
+            )}
+
+            # To prevent dependency resolution in Gleam 1.15+, local packages
+            # need to have their fingerprint up-to-date.
+            ${lib.concatStringsSep "\n" (
+              lib.forEach localDeps (d: ''
+                printf "%u" 0x$(xxhsum -H3 ${d.newPath}/gleam.toml | cut -d' ' -f1 | cut -d '_' -f2) > build/packages/${d.name}.config_fingerprint
+              '')
+            )}
+
+            ${
+              # Prime local package dependency caches so they do not try to fetch.
+              #
+              # Gleam compiles a local dependency inside that dependency's own
+              # directory, using the package cache found there.
+              lib.concatStringsSep "\n" (
+                lib.forEach localDeps (
+                  d: ''
+                    mkdir -p "${d.newPath}/build/packages"
+                    cp build/packages/packages.toml "${d.newPath}/build/packages/packages.toml"
+                    rsync --chmod=Du=rwx,Dg=rx,Do=rx,Fu=rw,Fg=r,Fo=r -r build/packages/* "${d.newPath}/build/packages/"
+                  ''
+                )
+              )
+            }
+
+            runHook postConfigure
+          '';
+      }
+      # When the build target is erlang
+      // lib.optionalAttrs (buildTarget == "erlang") {
+        nativeBuildInputs =
+          defaultNativeBuildInputs
+          ++ [erlangPackage rebar3Package]
+          ++ (lib.optional needsElixir beamPackages.elixir)
+          ++ nativeBuildInputs;
+
+        # The gleam compiler has a nice export function for erlang shipment.
+        buildPhase =
+          attrs.buildPhase
+          or ''
+            runHook preBuild
+
+            export REBAR_CACHE_DIR="$TMP/.rebar-cache"
+
+            gleam export erlang-shipment
+
+            runHook postBuild
+          '';
+
+        # Install all built packages into lib and create an entrypoint script
+        # that starts the application.
+        installPhase =
+          attrs.installPhase
+          or ''
+            runHook preInstall
+
+            mkdir -p $out/{bin,lib}
+
+            rsync --exclude=entrypoint.sh -r build/erlang-shipment/* $out/lib/
+
+            cat <<EOF > $out/bin/${gleamToml.name}
+            #!/usr/bin/env sh
+            ${erlangPackage}/bin/erl \
+              -pa $out/lib/*/ebin \
+              -eval "${gleamToml.name}@@main:run(${gleamToml.name})" \
+              -noshell \
+              -extra "\$@"
+            EOF
+            chmod +x $out/bin/${gleamToml.name}
+
+            runHook postInstall
+          '';
+      }
+      # When the build target is javascript
+      // lib.optionalAttrs (buildTarget == "javascript") {
+        nativeBuildInputs = defaultNativeBuildInputs ++ nativeBuildInputs;
+
+        # The gleam compiler doesn't provide an export mechanism for javascript target.
+        buildPhase =
+          attrs.buildPhase
+          or ''
+            runHook prebuild
+
+            gleam build --target javascript
+
+            runHook postBuild
+          '';
+
+        # Install all built packages into lib and create an entrypoint script
+        # that starts the application.
+        installPhase =
+          attrs.installPhase
+          or ''
+            runHook preInstall
+
+            mkdir -p $out/{bin,lib}
+
+            rsync --exclude=gleam.lock --exclude=gleam_version -r build/dev/javascript/* $out/lib/
+
+            cat <<EOF > $out/lib/${gleamToml.name}/main.mjs
+            import { main } from "./${gleamToml.name}.mjs";
+            main();
+            EOF
+
+            cat <<EOF > $out/bin/${gleamToml.name}
+            #!/usr/bin/env sh
+            ${nodejs}/bin/node $out/lib/${gleamToml.name}/main.mjs "\$@"
+            EOF
+            chmod +x $out/bin/${gleamToml.name}
+
+            runHook postInstall
+          '';
+      });
+}