gleamunison

# gleamunison

**Content-addressed language runtime on the BEAM, built in Gleam.**

A running prototype of a Unison-style content-addressed programming language
that compiles to BEAM bytecode and loads dynamically into the Erlang VM.

## Rationale & Gap Analysis

### Why Gleamunison?
Gleamunison combines the type-safe concurrency of the BEAM (via Gleam) with Unison's content-addressed codebase and algebraic effects, enabling zero-downtime hot upgrades and dynamic sandboxing.

### Feature Set Differences (Gleamunison vs Unison)

| Feature | Unison | Gleamunison | Trade-off / Benefit |
|---|---|---|---|
| **Identity** | SHA3-512 (Term+Type) | SHA256 (Term+Type) | SHA256 is native on BEAM; less hash size overhead. |
| **Primitives** | `##` Prefix Namespace | Genesis Block (Hash space) | Genesis eliminates dual-identity complexity. |
| **Effect Model** | Explicit continuation `k` | Implicit stack-based frame | Stack-based is simpler; lacks explicit `k` resume. |
| **Codebase Store** | SQLite / Event Log | DETS / ETS Storage | DETS is native and lightweight on BEAM. |
| **Namespaces** | Hierarchical Projects | Flat namespace mapping | Flat is simpler; hierarchy can be layered. |

### Complexity vs. Utility

| Element | Complexity | Utility | Recommendation |
|---|---|---|---|
| Genesis Primitives | Low | High | **Adopted**: Kept hash-space uniform. |
| Stack-based Effects | Medium | High | **Adopted**: Simpler runtime implementation. |
| Unique Type GUIDs | Low | Medium | **Adopted**: Prevents structural hash collisions. |
| Remote Ability | High | Low | **Out-of-Scope**: Rely on BEAM distribution instead. |

## How Hashes Act as Executables

In `gleamunison`, a program is not a monolithic binary file. It is a **Merkle Directed Acyclic Graph (Merkle DAG)** of content-addressed AST definitions stored in a database.

### The Merkle DAG Code Structure
Each function definition is parsed into an AST and hashed based on its structure, stripping variable names to ensure α-equivalence. If a function references other functions, it references them strictly by their hashes (`RefTo` AST node).

```
                  ┌───────────────────────┐
                  │   start-server Hash   │  ◄─── This single Root Hash IS the executable
                  │       [#ff45e8]       │
                  └───────────┬───────────┘
                              │
             ┌────────────────┴────────────────┐
             ▼                                 ▼
 ┌───────────────────────┐         ┌───────────────────────┐
 │  page-template Hash   │         │    query-db Hash      │
 │       [#ac88b2]       │         │       [#3b12ef]       │
 └───────────────────────┘         └───────────┬───────────┘
                                               │
                                               ▼
                                   ┌───────────────────────┐
                                   │  Datalog-Transact Jet │
                                   │  [#00000064] (Builtin)│
                                   └───────────────────────┘
```

The entire application (CMS, database, or GUI) is represented by a single **root hash** (e.g. `#ff45e8`).

### Step-by-Step Execution
When running a program by its hash:
1. **DAG Resolution**: The runtime looks up the root hash in its database, transitively walks the dependency graph, and fetches any missing dependency hashes from peers via the **Pull Sync Protocol**.
2. **On-the-Fly Compilation**: The compiler translates the resolved AST nodes into Erlang BEAM bytecode binary arrays named after their hashes (e.g. `m_ff45e8.beam`).
3. **Dynamic Loading & Run**: The VM dynamically loads the bytecode (`code:load_binary/3`) and calls the module's entrypoint function `'m_ff45e8':'$eval'()`.

This enables zero-downtime hot upgrades (running new logic by swapping to a new root hash) and conflict-free dependency resolution.

## Unique Usecases (Impossible on Gleam or Unison Alone)

Gleamunison combines the scheduling, distribution, and runtime efficiency of the BEAM with the content-addressing and algebraic effects constraints of Unison:

### Hot-Upgrades & Evolution
1. **Zero-Downtime Stateful Actor Upgrades**: Hot-swapping active actor code on-the-fly without state loss. Standard Gleam module updates clash on name collisions; Gleamunison addresses this by compiling into hash-named modules.
2. **Stateful Chatbot Hot-Upgrades**: Preserve active user conversation states in Erlang actors. Swaps the actor message-handling loop to a new hash definition on the fly without state or connection loss.
3. **IoT Firmware Hot-Patching**: IoT devices download modular function hashes instead of heavy firmware images, updating local logic dynamically without device resets.
4. **Dynamic API Gateways**: Route HTTP requests based on endpoint definition hashes, dynamically compiling and loading handlers on demand.

### Secure Sandboxing & Multitenancy
5. **Decoupled Multi-tenant Sandboxing**: Run untrusted plugins concurrently. Process boundaries isolate resource usage, while algebraic effects intercept and sandbox system actions (file, network).
6. **Zero-Trust Serverless Executions**: Execute third-party thunks securely. The host caps execution time via process CPU schedulers and restricts access using custom effect handlers.
7. **Capabilities-as-Code**: Database and network handles are represented strictly as abilities. User code only typechecks if the required abilities match their group access privileges.
8. **Sandbox Game Modding**: Run game modding scripts in isolated BEAM processes. Mod APIs are exposed as abilities, preventing malicious access to the host filesystem.

### Distributed Compute & Edge topographies
9. **Resilient P2P Job Stealing**: Edge nodes dynamically pull, structurally verify, locally compile, and run job definitions by hash.
10. **Live Process Migration**: Serialize a running actor's continuation closure, ship it to a remote node, sync missing code dependencies via pull protocol, and resume execution.
11. **Zero-Config Clustered Map-Reduce**: Parallelize map-reduce workflows. Code dependencies are automatically resolved and shipped by the runtime on target locations using Merkle sync.
12. **Edge-Cloud Compute Offloading**: IoT devices offload heavy compute thunks to BEAM cloud nodes, verifying code integrity by hash to prevent remote exploits.
13. **Content-Addressable CDN Handlers**: CDNs compile custom request handlers, push them to edge nodes by hash, and process CDN requests concurrently on edge processes.
14. **P2P Software Distribution**: Sync codebases incrementally. Nodes exchange root hashes and request only missing modules, reducing patching bandwidth.

### Determinism, Auditing & Tools
15. **Time-Traveling Replay Debugging**: Capture trace logs of execution. Replay the exact execution path deterministically using mock clock/random effect handlers.
16. **Distributed Event Sourcing with Code Auditing**: Event stores record event payloads alongside the handler's hash, allowing historic events to be replayed with the exact code version.
17. **Smart Contract Workflows**: Execute decentralized workflows. All state mutations and payments are modeled as abilities, sandboxed by host-defined contract handlers.
18. **Multi-Tenant Concurrent Parsers**: Compile user-provided parser grammar thunks. Preemptive BEAM scheduling prevents a single bad parsing loop from blocking others.
19. **Self-Documenting Code Registries**: Code definitions are hashed and immutable. Documentation and tests are linked directly to hashes; renaming never breaks documentation.
20. **Immutable Cloud Shell**: Run interactive REPL sessions where every expression is compiled and stored. Keeps old module versions in memory for historic comparisons.
21. **Content-Addressable Microservices**: Services call others by passing function hashes over RPC. The runtime resolves, syncs, and loads the code dynamically.
22. **Decentralized Knowledge Graph**: A wiki-like graph where nodes are content-addressed definitions and links are type-safe references forming a Merkle DAG.
23. **Reproducible Monte Carlo Simulations**: Replay complex stochastic simulations by mock-handling random generator and timer abilities using fixed seeds.


## Project State

**Production-grade runtime (Phases 0–14 complete).** All components are implemented and verified. The runtime is **fully playbook-certified**, passing all 5696 playbook conformance levels and 54 unit test suites (all passed). **v3.11.0** includes native content-addressed Datalog database engine with temporal indexing, graph algorithms, BM25, and vector search, alongside safe FFI deserialization, localhost endpoint restrictions, Ranch port monitoring safeguards, modular genesis builtin extraction, and a data-driven test runner framework.

| Step | Status |
|---|---|
| AST → Hash (SHA256) | ✓ Content-addressed identity |
| Codebase insert with hash verification | ✓ DETS/ETS persistence, dedup |
| Compile to BEAM binary (all Term variants) | ✓ Int/Float/Text/List/Lambda/Apply/Let/Match |
| Load into VM (`code:load_binary/3`) | ✓ OTP 29 compatible |
| Type inference (Int/Float/Text/List) | ✓ Hindley-Milner style |
| Elaboration (Surface → Core) | ✓ Two-phase with name resolution |
| Effects runtime (process dict stack) | ✓ do_/handle_/push_frame/pop_frame |
| Sync protocol (pull-based) | ✓ Types + Erlang distribution FFI + TCP sync |
| Native Datalog Engine | ✓ Pure S-expression EAVT/AVET, temporal, rules, BM25, vector |
| escript standalone binary | ✓ ~1.2 MB, no Gleam dependency at runtime |

## Conformance Tests

To execute the unit and integration test suites:
```sh
gleam test                          # Runs all 54 unit test suites including native Datalog
bb scripts/run_playbook_tests.clj   # Runs playbook conformance suite
```

## Why the escript is only 1.2 MB

The standalone binary (`gleamunison_escript`) contains the full content-addressed runtime — parser, elaborator, typechecker, compiler, loader, codebase, effects system, web server, REPL, 52 genesis modules, and all stdlib dependencies. At ~1.2 MB, it's compact because:

**BEAM bytecode is dense.** The compiled `.beam` files are ~2.4 MB uncompressed; zip compression brings that to ~1.2 MB.

**No VM bundled.** Unlike Go or Rust binaries that statically link a runtime, the escript relies on the system's Erlang/OTP installation (~150 MB, installed once). The escript itself is just a zip archive with a 50-byte launcher header.

| Format | Size | Dependencies |
|---|---|---|
| gleamunison escript | **1.2 MB** | Erlang/OTP |
| Go binary | 10–20 MB | None |
| Rust binary | 5–15 MB | None |
| Node.js app + deps | 100–500 MB | Node.js |

If you already have Erlang installed, this is as close to a zero-install language runtime as it gets.

## Modules (44 Gleam source modules, Erlang FFI, 52 genesis modules, 13 Datalog engine modules)

| Module | Concern | Status |
|---|---|---|
| `gleamunison/identity` | Opaque Hash, DefinitionRef, LocalVar | Real |
| `gleamunison/ast` | Core AST: Term (15 variants), Type, Definition, Unit | Real |
| `gleamunison/types` | Core type definitions | Real |
| `gleamunison/typecheck` | Type checker | Real |
| `gleamunison/inference` | Type inference engine (Hindley-Milner) | Real |
| `gleamunison/infer_helper` | Type inference helpers (alpha-equivalence, substitution) | Real |
| `gleamunison/codebase` | Content-addressed Merkle store | Real |
| `gleamunison/elaborate` | Surface → Core elaboration orchestration | Real |
| `gleamunison/elab_def` | Definition elaboration (term, type, ability) | Real |
| `gleamunison/elab_pat` | Pattern elaboration | Real |
| `gleamunison/elab_term` | Term elaboration | Real |
| `gleamunison/elab_types` | Type elaboration | Real |
| `gleamunison/elab_ctx` | Elaboration context | Real |
| `gleamunison/lower` | AST lowering / IR transformations | Real |
| `gleamunison/parser` | S-expression parser & tokenizer | Real |
| `gleamunison/lexer` | Lexer / tokenizer | Real |
| `gleamunison/type_pretty` | Pretty-printer for types | Real |
| `gleamunison/compile` | AST → Erlang source → BEAM binary | Real |
| `gleamunison/loader` | Dynamic module loading into VM | Real |
| `gleamunison/effects` | Algebraic effect types + Erlang runtime | Real |
| `gleamunison/storage` | ETS, DETS, Partitioned DETS, and Mnesia storage adapters | Real |
| `gleamunison/repl` | REPL entry point and loop orchestrator | Real |
| `gleamunison/repl_eval` | REPL evaluation and definition compiler pipeline | Real |
| `gleamunison/repl_io` | REPL bracket counter and line accumulator | Real |
| `gleamunison/sync` | Pull-based sync protocol | Real |
| `gleamunison/sync_types` | Sync protocol type definitions | Real |
| `gleamunison/http` | Web server entry point | Real |
| `gleamunison/http_client` | Typed HTTP client (get/post/put/delete) | Real |
| `gleamunison/json` | JSON encode/decode | Real |
| `gleamunison/datetime` | Opaque DateTime, ISO 8601, arithmetic | Real |
| `gleamunison/filepath` | Opaque Path manipulation | Real |
| `gleamunison/crypto` | Hash, HMAC, random bytes | Real |
| `gleamunison/template` | {{var}} string interpolation | Real |
| `gleamunison/log` | Structured logging (debug/info/warn/error) | Real |
| `gleamunison/config` | Configuration management (env/TOML/CLI) | Real |
| `gleamunison/health` | Health checks and readiness probes | Real |
| `gleamunison/metrics` | Counter/gauge/histogram with telemetry | Real |
| `gleamunison/pipeline` | Factored pipeline phases (parse_only, elaborate_only, compile_only, load_and_eval) | Real |
| `gleamunison/jets` | FFI Jet compiler interception registry | Real |
| `gleamunison/bootstraps` | Cycle-free genesis bootstrap definitions | Real |
| `gleamunison/genesis` | Genesis block constants and hashes | Real |
| `gleamunison/util` | Shared utilities (e.g. range helper) | Real |
| `gleamunison/verify` | Interactive and file-based verification runner | Real |
| `gleamunison/dogfood_runner` | Data-driven dynamic VM test runner | Real |
| `gleamunison_datalog/src/` | 13-module S-expression Datalog database engine | Real |
| `gleamunison_ffi.erl` | FFI: hashing, compilation, loading, process dict | Real |
| `gleamunison_effets.erl` | Effects runtime: push/pop/find_frame, do_op/handle_comp | Real |
| `gleamunison_storage.erl` | ETS/DETS/Mnesia storage backend with SHA256 atom safety | Real |
| `gleamunison_tcp_sync.erl` | TCP sync server with safe deserialization & length limit | Real |
| `gleamunison_http.erl` | HTTP server with trace capture, SSE, and health routes | Real |
| `gleamunison_http_routes.erl` | Route handlers: eval, define, browse, traces, logs, modules | Real |
| `gleamunison_http_util.erl` | HTTP utilities: JSON, MIME, URL decode, SSE broadcast | Real |
| `gleamunison_sup.erl` | OTP Supervisor tree | Real |
| `gleamunison_repl_ffi.erl` | REPL FFI bridge | Real |
| `gleamunison_trace.erl` | Request trace capture (DETS) for Darklang-style development | Real |
| `gleamunison_adapters.erl` | Lazy CAS type adapters for schema migration | Real |
| `gleamunison_log.erl` | Structured log ETS backend | Real |
| `gleamunison_config.erl` | Environment variable config loader | Real |
| `gleamunison_health.erl` | Node health status (memory, modules) | Real |
| `gleamunison_metrics.erl` | Counter/gauge/histogram with telemetry | Real |
| `gleamunison_crypto.erl` | SHA256/512, HMAC, random bytes backend | Real |
| `gleamunison_datetime.erl` | ISO 8601 parse/format backend | Real |
| `gleamunison_template.erl` | String interpolation with HTML escaping | Real |
| `gleamunison_json.erl` | JSON encode/decode backend | Real |
| `gleamunison_http_client.erl` | HTTP client wrapping httpc | Real |
| `gleamunison_property.erl` | Property-based testing framework | Real |
| `m_*.erl` (52 files) | Content-addressed genesis modules | Real |

## Quick start

```sh
cd ~/Desktop/gleamunison_dogfood/gleamunison_repo
gleam run -- all             # Run all dogfooding levels via dynamic VM runner
gleam test                  # Run unit tests (54 test suites)
./gleamunison_escript repl   # Start interactive REPL via standalone escript (after ./build_escript.sh)
```

## Documentation

- [Playbook](docs/PLAYBOOK.md) — how to work on this project
- [Architecture](docs/ARCHITECTURE.md) — deep dive into the design
- [Roadmap](docs/ROADMAP.md) — 14-phase plan from spec to production
- [Operations Runbook](docs/OPERATIONS.md) — deploy, configure, monitor, upgrade, troubleshoot
- [Learnings](docs/LEARNINGS.md) — architectural insights discovered
- [Patterns](docs/PATTERNS.md) — design patterns used
- [Reference Manual](docs/MANUAL.md) — complete reference user guide
- [ADRs](docs/adr/) — Architecture Decision Records (62 records)
- [Standard Library](docs/stdlib/index.html) — module reference

## Runtime output

```
=== Gleamunison ===
Int(42)           ✓ Hash → Compile → Load
Lambda(id)        ✓ Hash → Compile → Load
Apply(id, 99)     ✓ Hash → Compile → Load
Let(V0=42, V0)    ✓ Hash → Compile → Load
Text(hello)       ✓ Hash → Compile → Load
List([1,2,3])     ✓ Hash → Compile → Load
Match(42, cases)  ✓ Hash → Compile → Load
Type Inference    ✓ Int/Float/Text/List
Elaboration       ✓ Surface → typed
Effects           ✓ RuntimeConfig
Sync              ✓ PeerId/SyncState
```

## License

MIT