iconvex

# Iconvex

**2,105 codecs. Ten runtime packages. Pure Elixir/BEAM. Zero NIFs, ports,
operating-system `iconv`, or external processes.**

[![Hex.pm](https://img.shields.io/hexpm/v/iconvex.svg)](https://hex.pm/packages/iconvex)
[![HexDocs](https://img.shields.io/badge/hex-docs-8e7ce6.svg)](https://hexdocs.pm/iconvex/)
[![CI](https://github.com/edescourtis/iconvex/actions/workflows/iconvex.yml/badge.svg)](https://github.com/edescourtis/iconvex/actions/workflows/iconvex.yml)
[![License](https://img.shields.io/hexpm/l/iconvex.svg)](LICENSE)

Iconvex begins as a native BEAM port of GNU libiconv 1.19, including every
codec selected by `--enable-extra-encodings`. Then it wanders happily into the
rest of character-set history: GSM septets and national shift tables, Morse
and Baudot, UTF-9 nonets, UTF-18 words, Punycode, SCSU, MARC-8, PETSCII,
ATASCII, DEC SIXBIT and RADIX-50, UNIVAC FIELDATA, punched cards, TeX fonts,
vendor compatibility pages, current ICU converters, and 1,050 pinned
historical ICU revisions.

Yes: 1,050 historical ICU revisions, each source-filename-qualified instead of
silently collapsing incompatible versions behind one friendly alias.

GNU libiconv 1.19 fixed-codec parity remains **198/198**: 112 default codecs in
Core and 86 optional codecs in Extras. The **2,105 unique canonical codecs**
number is the larger Iconvex ecosystem, not a redefinition of GNU's surface.

The ordinary API stays ordinary:

```elixir
{:ok, "café"} =
  Iconvex.convert(<<0x63, 0x61, 0x66, 0xE9>>, "CP1252", "UTF-8")
```

When eight-bit binaries would be a lie, package APIs return exact Elixir
bitstrings or self-describing LSB containers.

## Pick your package

| Package | Canonical codecs | The rabbit hole |
|---|---:|---|
| `iconvex` | **112** | GNU default: Unicode, CJK, ISO-2022, HZ, Windows/Mac/ISO pages, C99/JAVA, transliteration, generic 1–8-bit packing |
| `iconvex_extras` | **86** | Complete GNU `--enable-extra-encodings`: AIX, DOS, OSF/1, z/OS EBCDIC, Atari ST, RISC OS, DEC Asian, JIS X 0213 |
| `iconvex_telecom` | **61** | GSM 03.38 national tables, Morse, Baudot/ITA, SITOR/NAVTEX, AIS6, TBCD, SIM alphabets, broadcast and teleprinter formats |
| `iconvex_unicode` | **17** | BOCU-1, CESU-8, SCSU, UTF-EBCDIC, UTF-1/5/6, WTF-8, signature profiles, plus native UTF-9 and UTF-18 APIs |
| `iconvex_standards` | **248** | RFC 1345, Punycode, IMAP UTF-7, ISCII, ISO-IR, ECMA, PCL, Unihan and standards-defined transforms |
| `iconvex_bibliographic` | **10** | MARC-8, ANSEL, Cork/T1, TeX OT1/OML/OMS, PostScript and glyph-vector mappings |
| `iconvex_retro` | **192** | PETSCII, ATASCII, Apple/DEC/CDC/UNIVAC, terminals, calculators, punched cards and 57 packed profiles |
| `iconvex_platforms` | **117** | IBM, Microsoft, .NET, Java, CPython, glibc, Kermit, Adobe, Mac and other vendor-defined compatibility surfaces |
| `iconvex_icu` | **212** | Current ICU-derived SBCS/MBCS/stateful tables, LMBCS variants, Unicode profiles and compound text |
| `iconvex_icu_archive` | **1,050** | Source-filename-qualified historical ICU revisions, backed by six byte-balanced data packages |

Every package is independently installable and depends only on Core, except
the Archive facade, which pulls its six exact-version provider-only data
packages. Starting a leaf atomically registers its complete codec set. Install
only the families the application actually uses.

`iconvex_specs` is an unpublished generation, research, conformance, and
integration workspace. It owns no runtime codec and must not be an application
dependency.

For the whole map, open the generated
[codec-to-package index](https://github.com/edescourtis/iconvex/blob/v0.1.1/ICONVEX_CODEC_PACKAGE_INDEX.md)
or its machine-readable
[`ICONVEX_CODEC_PACKAGE_INDEX.csv`](https://github.com/edescourtis/iconvex/blob/v0.1.1/ICONVEX_CODEC_PACKAGE_INDEX.csv).
The [full GNU comparison](https://github.com/edescourtis/iconvex/blob/v0.1.1/ICONVEX_FULL_STACK_SUPPORT.md)
lists all 2,105 names with ownership and GNU status. The
[package architecture](https://github.com/edescourtis/iconvex/blob/v0.1.1/PACKAGE_ARCHITECTURE.md)
documents dependencies, lifecycle, artifacts, and release order.

## Installation

Core only:

```elixir
def deps do
  [{:iconvex, "~> 0.1"}]
end
```

Exact GNU fixed-codec surface:

```elixir
def deps do
  [
    {:iconvex, "~> 0.1"},
    {:iconvex_extras, "~> 0.1"}
  ]
end
```

Choose any leaves:

```elixir
def deps do
  [
    {:iconvex, "~> 0.1"},
    {:iconvex_telecom, "~> 0.1"},
    {:iconvex_unicode, "~> 0.1"},
    {:iconvex_standards, "~> 0.1"},
    {:iconvex_bibliographic, "~> 0.1"},
    {:iconvex_retro, "~> 0.1"},
    {:iconvex_platforms, "~> 0.1"},
    {:iconvex_icu, "~> 0.1"}
  ]
end
```

For the historical archive, add only the facade:

```elixir
{:iconvex_icu_archive, "~> 0.1"}
```

Mix fetches `iconvex_icu_archive_data_01` through
`iconvex_icu_archive_data_06` transitively. Those packages own table storage,
not additional codec names.

## Encoding gallery

These examples are executable integration tests, not aspirational snippets.

### Core: escapes, state machines, and Unicode

Core owns exactly GNU's 112 default codecs and accepts 446 normalized names.
C99/JAVA source escapes are codecs, and HZ remains a state machine:

```elixir
{:ok, ~S(\u00e9\U0001f600)} =
  Iconvex.convert("é😀", "UTF-8", "C99")

{:ok, "~{VP~}~~{ND~}"} =
  Iconvex.convert("中~文", "UTF-8", "HZ")

[<<"caf", 0xC3>>, <<0xA9>>]
|> Iconvex.stream!("UTF-8", "CP1252")
|> Enum.join()
#=> <<0x63, 0x61, 0x66, 0xE9>>
```

Highlights include UTF-7/8/16/32, UCS-2/4, GB18030:2005 and :2022,
EUC-JP/KR/TW, Big5/HKSCS, Shift_JIS, JOHAB, TCVN, VISCII, and the complete
ISO-2022-JP/JP-1/JP-2/JP-MS/KR/CN/CN-EXT family.

### Extras: GNU's whole weird attic

Extras is the exact 86-codec complement behind GNU's optional definition
families. It includes DOS pages, AIX, OSF/1, 53 z/OS pages, `ATARIST`,
`RISCOS-LATIN1`, `DEC-HANYU`, `BIG5-2003`, JIS X 0213 and
`ISO-2022-JP-3`.

The z/OS UNIX surface makes EBCDIC newline policy explicit:

```elixir
{:ok, ebcdic} =
  Iconvex.convert("hello\n", "ASCII", "IBM-1047/ZOS_UNIX")

<<0x88, 0x85, 0x93, 0x93, 0x96, 0x15>> = ebcdic
{:ok, "hello\n"} = Iconvex.convert(ebcdic, "IBM-1047/ZOS_UNIX", "ASCII")
```

IBM-423 is here too: the Greek EBCDIC codec grounded in the original glibc
implementation.

### Telecom: septets, Baudot, Morse, and radio alphabets

Telecom owns 61 byte codecs and 53 packed profiles spanning 3, 5, 6, and 7
bits. International Morse follows ITU-R M.1677:

```elixir
{:ok, "... --- ..."} = Iconvex.convert("SOS", "UTF-8", "MORSE-CODE")
{:ok, "SOS"} = Iconvex.convert("... --- ...", "MORSE-CODE", "UTF-8")
# MORSE-CODE resolves to MORSE-ITU-M1677.
```

GSM 03.38's registry surface uses one septet per octet, exactly as SMPP does.
The high bit stays clear:

```elixir
{:ok, <<0x04, 0x0B>>} = Iconvex.convert("€Ğ", "UTF-8", "GSM0338-TURKISH")
{:ok, "€Ğ"} = Iconvex.convert(<<0x04, 0x0B>>, "GSM0338-TURKISH", "UTF-8")
```

All national locking and single-shift tables are available. They can also be
paired independently:

```elixir
alias Iconvex.Telecom.GSM0338

options = [locking: :portuguese, single_shift: :bengali]
{:ok, [0x00EA, 0x09E6]} = GSM0338.decode(<<0x04, 0x1B, 0x19>>, options)
{:ok, <<0x04, 0x1B, 0x19>>} = GSM0338.encode([0x00EA, 0x09E6], options)
182 = length(GSM0338.valid_pairs())
```

TPDU packing produces the actual contiguous wire septets:

```elixir
alias Iconvex.Telecom.GSM0338.Packing

{:ok, packed} = Packing.pack("hellohello")
"E8329BFD4697D9EC37" = Base.encode16(packed)
{:ok, "hellohello"} = Packing.unpack(packed, 10)
```

The same package covers ITA1/ITA2, Murray 1905, Wheatstone/Hausmann,
CCIR476/SITOR/NAVTEX, AIS6, TBCD, SIM/USIM alphabets, ARIB and three
Unicode 17 carrier-emoji profiles.

### Unicode: nonets, 18-bit words, and compression

Unicode owns BOCU-1, CESU-8, FSS-UTF, SCSU, UTF-EBCDIC, UTF-1, UTF-5, UTF-6,
WTF-8 and explicit signature profiles. UTF-9 and UTF-18 are logical non-octet
codecs, so their native APIs preserve the real width:

```elixir
{:ok, utf9} = Iconvex.Unicode.UTF9.encode_packed([?A, 0x0391, 0x10330])
54 = bit_size(utf9)
{:ok, [?A, 0x0391, 0x10330]} = Iconvex.Unicode.UTF9.decode_packed(utf9)

{:ok, utf18} = Iconvex.Unicode.UTF18.encode_packed([?A, 0x10330])
36 = bit_size(utf18)
{:ok, [?A, 0x10330]} = Iconvex.Unicode.UTF18.decode_packed(utf18)
```

Byte protocols use `UTF-9-16BE`, `UTF-9-16LE`, `UTF-18-24BE`, or
`UTF-18-24LE`. Padding and endianness are part of the name instead of ambient
machine state.

### Standards: RFCs, ISO registers, and raw Punycode

Punycode is the raw RFC 3492 transform; it deliberately does not invent an
`xn--` DNS prefix:

```elixir
{:ok, "ihqwcrb4cv8a8dqg056pqjye"} =
  Iconvex.convert("他们为什么不说中文", "UTF-8", "PUNYCODE")

{:ok, "他们为什么不说中文"} =
  Iconvex.convert("ihqwcrb4cv8a8dqg056pqjye", "PUNYCODE", "UTF-8")
```

This is also where RFC 1345 identities, IMAP UTF-7, ISCII, KOI variants,
ECMA-1/44, PCL symbol sets, ISO-IR registers and property-token mappings live.
GNU spellings that collide with historical RFC 1345 names keep the GNU route;
the historical identity remains explicit as `RFC1345:IBM037`, and so on.

### Bibliographic: MARC-8, ANSEL, and TeX machinery

MARC-8 puts combining marks before the base letter. Unicode does the opposite.
The codec knows:

```elixir
{:ok, "e\u0301"} = Iconvex.convert(<<0xE2, ?e>>, "MARC-8", "UTF-8")
```

The package also owns ANSEL, Cork/T1, OT1, OML, OMS, Adobe PostScript and AGL
glyph-vector mappings, including explicit packed seven-bit math-font profiles.

### Retro: six-bit machines, punched cards, and calculator glyphs

The Apple-1 Signetics 2513 address order is `@`–`_`, then space–`?`. The
registry exposes one unit per byte; `Iconvex.Retro.Packed` preserves six bits:

```elixir
{:ok, <<0, 1, 2, 3, 32, 63>>} =
  Iconvex.convert("@ABC ?", "UTF-8", "APPLE-1-SIGNETICS-2513-CM2140-6BIT")

{:ok, packed} =
  Iconvex.Retro.Packed.encode_from_utf8(
    "@ABC ?",
    Iconvex.Retro.Apple1Signetics2513,
    :msb
  )

36 = bit_size(packed)

{:ok, "@ABC ?"} =
  Iconvex.Retro.Packed.decode_to_utf8(
    packed,
    Iconvex.Retro.Apple1Signetics2513,
    :msb
  )
```

DEC RADIX-50 really is a base-40 word:

```elixir
{:ok, <<0o115402::16-little>>} =
  Iconvex.convert("X2B", "UTF-8", "DEC-RADIX-50")
```

Other delights include `PETSCII`, `ATASCII`, Apple II, ZX80/81/Spectrum,
Amstrad, MSX, TRS-80, TI calculators, `DEC-SIXBIT`,
`FIELDATA-UNIVAC-1100`, CDC display codes, Hollerith and 12-bit punched cards.

### Platforms: vendor reality is also a specification

Platforms preserves source-qualified vendor behavior instead of flattening
near-matches into a convenient fiction:

```elixir
{:ok, "\u1E83"} = Iconvex.convert(<<0xFF>>, "MacCeltic", "UTF-8")
```

IBM-423 and GNU's optional IBM pages remain in Extras; additional IBM CCSIDs,
Windows best-fit maps, Java modified UTF-8, .NET/CPython profiles, glibc
charmaps, Adobe mappings and Mac variants live here.

### ICU: current converter profiles

Current ICU data and algorithm profiles are separate from historical revisions.
LMBCS optimization groups remain distinct codecs:

```elixir
{:ok, <<0xD5>>} = Iconvex.convert("Ω", "UTF-8", "LMBCS-2")
{:ok, "Ω"} = Iconvex.convert(<<0xD5>>, "LMBCS-2", "UTF-8")
```

The 212-codec package includes 191 generated table providers plus compound
text, JIS, LMBCS and Unicode variant implementations.

### ICU Archive: reproducible historical behavior

The Archive owns 1,050 source-filename-qualified revisions. Six provider-only
packages keep artifacts manageable without adding duplicate codec identities.
Here is a stateful IBM-930 revision, including SO/SI:

```elixir
codec = Iconvex.ICUArchive.Codecs.C674
bytes = <<0xC1, 0x0E, 0x44, 0x81, 0x44, 0x82, 0x0F, 0xC2>>

"ICU-ARCHIVE-ibm-930_P120-1999" = codec.canonical_name()
{:ok, ^bytes} = codec.encode(~c"AあいB")
{:ok, ~c"AあいB"} = codec.decode(bytes)
```

The facade manifest, shard assignment, every table digest and every generated
decoder/encoder mapping are exhaustively checked.

## Daily conversion API

Bang functions return the binary directly. Suffixes and explicit options share
one implementation:

```elixir
<<"caf", 0xE9>> = Iconvex.convert!("café", "UTF-8", "CP1252")
{:ok, "AB"} = Iconvex.convert("A😀B", "UTF-8", "ASCII//NON_IDENTICAL_DISCARD")
```

Options can also be passed explicitly:

```elixir
Iconvex.convert(input, from, to,
  invalid: :discard,
  unrepresentable: :discard,
  transliterate: true,
  byte_substitute: "<0x%02x>",
  unicode_substitute: "<U+%04X>",
  on_invalid_byte: fn event -> {:replace, "<byte-#{event.byte}>"} end
)
```

The `byte_substitute` option replaces malformed source bytes; `unicode_substitute`
replaces characters unavailable in the destination. The formats accept GNU-style
`%x`/`%X` fields, including zero-padded widths. With `iconvex_extras` installed,
IBM-1047 newline behavior can be selected by name or explicitly:

Default destination handling mirrors GNU libiconv. An ordinary character that
the target cannot represent is an error. Extended UCS values that reach a
Unicode-family target use U+FFFD when that target defines and can encode the
fallback; `unrepresentable: :discard`, transliteration, or
`unicode_substitute` takes precedence. JAVA and UTF-7 intentionally preserve
isolated 16-bit surrogate units when the source codec exposes them. Generic
UTF-16 and UTF-32 emit a BOM only when at least one target character is emitted.
ISO-2022-JP-2 also implements GNU's case-insensitive `ja`, `ko`, and `zh`
Unicode language tags, including cancellation and streaming state.

```elixir
Iconvex.convert(input, "IBM-1047/ZOS_UNIX", "UTF-8")
Iconvex.convert(input, "UTF-8", "IBM-1047", to_surface: :zos_unix)
```

`on_invalid_byte` handles source bytes a codec cannot decode. The callback gets
an `Iconvex.InvalidByte` containing canonical encoding, error kind, absolute
offset, current byte, and codec error sequence. It returns `:default`, `:error`,
`:discard`, or `{:replace, utf8_or_codepoints}`. Iconvex does not label an
unmapped byte as a display command, corruption, or application control. That
decision belongs to the file format and caller; the same callback works in
one-shot and lazy Stream conversion.

External fixed-width codecs can declare multi-byte recovery consumption, and
whole-string transforms can declare stop-on-error recovery so callbacks never
restart a desynchronized suffix. See [EXTENDING.md](EXTENDING.md).

Errors contain the source byte offset or unrepresentable Unicode code point:

```elixir
{:error, %Iconvex.Error{kind: :invalid_sequence, offset: 3}} =
  Iconvex.convert(input, "UTF-8", "CP1252")
```

For lazy bounded-memory conversion, input enumerable emits binaries and output
is an Elixir Stream of converted binaries:

```elixir
output =
  File.stream!(path, [], 64 * 1024)
  |> Iconvex.stream!("PETSCII", "UTF-8", on_invalid_byte: controls)
  |> Enum.into("")
```

Multibyte lookahead, multi-code-point target mappings, source offsets, UTF BOM
state, and built-in HZ/ISO-2022/UTF-7 state survive chunk boundaries. A UTF-7
shifted run is committed only after its base64 padding and complete UTF-16
payload have been validated; a plus inside an active shift remains in that
Base64 run, and byte-at-a-time shifted input is processed in linear time.
ISO-2022 diagnostic frames and JP-MS SO/SI state are likewise split-invariant.
Output chunks need not align with input chunks. Conversion failures raise
`Iconvex.Error` during enumeration because earlier output may already have
been consumed. External codecs use the incremental callbacks documented in
[EXTENDING.md](EXTENDING.md).

For a buffered chunked transaction:

```elixir
{:ok, converter} = Iconvex.new("UTF-8", "SHIFT_JIS")
{:ok, <<>>, converter} = Iconvex.feed(converter, chunk1)
{:ok, <<>>, converter} = Iconvex.feed(converter, chunk2)
{:ok, output, finished} = Iconvex.finish_with_state(converter)
{:error, :already_finished} = Iconvex.feed(finished, chunk3)
```

Every source is buffered until `finish/1` or `finish_with_state/1`. Each feed is
O(1), and finalization performs one conversion over the accumulated iodata.
This contract makes chunked output byte-identical to one-shot output across
every split, including stateful designations, malformed-input policies,
Vietnamese composition, multi-code-point destination mappings, and UTF BOMs.
Errors use offsets from the beginning of the complete stream. `finish/1` keeps
the original compatibility return shape; use `finish_with_state/1` when the
terminal converter state and repeated-finish rejection are required.

Use `Iconvex.encodings/0` for canonical names and
`Iconvex.canonical_name/1` to resolve an alias.

## Non-octet packed transports

Codecs that expose one complete signal unit per octet can be packed without
copying their mapping logic:

```elixir
{:ok, packed} = Iconvex.Packed.encode_from_utf8("Hello", "ASCII", 7)
{:ok, "Hello"} = Iconvex.Packed.decode_to_utf8(packed, "ASCII", 7)

{:ok, lsb} = Iconvex.Packed.encode_from_utf8_lsb("hellohello", "ASCII", 7)
"E8329BFD4697D9EC37" = Base.encode16(lsb.data)
{:ok, "hellohello"} = Iconvex.Packed.decode_to_utf8_lsb(lsb, "ASCII")
```

`pack/2` returns an exact MSB-first Elixir bitstring. `pack_lsb/2` returns an
`Iconvex.Packed.LSB` value carrying the byte data, exact meaningful bit count,
and unit width; this prevents final-octet padding from becoming ambiguous.
If unpacking succeeds but the named codec rejects one or more logical units,
the returned `Iconvex.Error` uses `offset_unit: :bit` and a physical bit
offset. MSB errors retain the exact offending bitstring; LSB errors retain a
self-describing `Iconvex.Packed.LSB` fragment rather than exposing temporary
one-byte-per-unit coordinates.
Every unit value and width 1–8 is exhaustively tested in both orders. Wider
formats with codec-specific scalar rules live in their owning leaves:
`iconvex_unicode` owns UTF-9/UTF-18, while Standards, Bibliographic, Retro, and
Telecom publish their own exact profile catalogs.

## External codec libraries

External packages implement `Iconvex.Codec`, then atomically register a module
at startup:

```elixir
defmodule MyLibrary.Codec do
  use Iconvex.Codec

  def canonical_name, do: "X-MY-CODEC"
  def decode(binary), do: MyLibrary.Decoder.decode(binary)
  def decode_discard(binary), do: MyLibrary.Decoder.decode_discard(binary)
  def encode(codepoints), do: MyLibrary.Encoder.encode(codepoints)
  def encode_discard(codepoints), do: MyLibrary.Encoder.encode_discard(codepoints)
  def encode_substitute(codepoints, replacer),
    do: MyLibrary.Encoder.encode_substitute(codepoints, replacer)
end

{:ok, ownership} = Iconvex.register_codec_if_absent(MyLibrary.Codec)
{:ok, output} = Iconvex.convert(input, "X-MY-CODEC", "UTF-8")
```

If `ownership` is a reference, retain it and conditionally unregister with
`Iconvex.unregister_codec(MyLibrary.Codec, ownership)` on shutdown. The
`:existing` result means another caller owns the registration and it must be
left in place. Versioned recovery snapshots preserve registrations and exact
ownership tokens across consecutive supervised registry-worker crashes while
the `:iconvex` application remains running. A clean application stop erases
that recovery state, so registrations never cross a deliberate stop/start
boundary. Replacement publishes at one commit point, so concurrent lookups
resolve the complete old registration or the complete new registration.

Strict per-commit durability is deliberate. Every successful registry
mutation writes a synchronous recovery snapshot of the complete registry to
`:persistent_term`; replacing that term can schedule VM-wide garbage
collection. Keep registration on the control plane rather than a request hot
path. Package authors with many codecs should use
`Iconvex.Extension.register/2`, which wraps the internal `register_set`
transaction, publishes the complete set atomically, and amortizes the snapshot
to one checkpoint. Debouncing this write would weaken the documented
consecutive worker-crash guarantee.

Mapping tables and their dense/sparse decode accelerators are also lazy
`:persistent_term` entries. The shipped resident set is bounded by the finite
codec inventories of the applications actually loaded; each table identity
has one base term and at most one term for each of the three derived cache
shapes. Unregistering a table provider erases its base term and all three
derived keys under their build locks, so repeated external-package
load/use/unload cycles do not accumulate provider cache generations. Provider
startup and shutdown therefore belong on the control plane as well; arbitrary
hot-path LRU replacement would impose the VM-wide cost that these immutable
caches are designed to avoid.

If a module's declared name collides with another source, register it under an
explicit source-qualified name. The declared canonical name is replaced for
that registration; module aliases and option aliases are still registered and
collision-checked:

```elixir
:ok =
  Iconvex.register_codec(MyLibrary.Codec,
    canonical: "MY-SOURCE-X-MY-CODEC",
    aliases: ["MY-SOURCE-MYCODEC"]
  )
```

Consumer configuration can load modules automatically with
`config :iconvex, external_codecs: [MyLibrary.Codec]`. Tuple entries accept the
same `:canonical` and `:aliases` options. Registered codecs appear
in `Iconvex.encodings/0`; built-in names cannot be overridden.

External OTP applications that own a complete codec inventory and mapping
tables can publish both with one public package transaction:

```elixir
{:ok, token} =
  Iconvex.Extension.register(
    [MyLibrary.Codec, MyLibrary.OtherCodec],
    owner: :my_library,
    priority: 10,
    table_providers: [:my_mapping]
  )

# Keep the opaque token in Application state.
:ok = Iconvex.Extension.unregister(token)
```

Registration and removal are atomic to concurrent lookups, failures roll back
newly acquired providers, supervised worker restarts preserve the exact token,
and stale tokens cannot remove replacements. See
[EXTENDING.md](EXTENDING.md) for callback/error contracts, aliases, stateful
streaming, package priorities and collisions, required linear substitution,
optional zero-copy UTF-8 callbacks, performance results, and an external-library
test checklist. `iconvex_extras` is the reference external codec package: it
shares core conversion engines while keeping every optional mapping table and
codec module outside the core artifact.

Each of the nine extension facades installs one atomic managed registration
set. The integration contract exercises forward, reverse, and every cyclic
start boundary and requires the same 2,105-name registry. Canonical claims rank
ahead of aliases, then by fixed package priority; stopping a winner exposes a
retained fallback without a transient lookup gap. Individual third-party
registrations through `register_codec*` remain strict: any distinct module
claiming an existing name gets a `{:name_conflict, name}` error rather than
silently joining a managed set.

The generated
[full-stack codec comparison](https://github.com/edescourtis/iconvex/blob/v0.1.1/ICONVEX_FULL_STACK_SUPPORT.md)
records the owning package and GNU libiconv status for every canonical name.
The richer
[codec-to-package index](https://github.com/edescourtis/iconvex/blob/v0.1.1/ICONVEX_CODEC_PACKAGE_INDEX.md)
adds modules, aliases, sources, surface kinds, packed profiles, provenance
records, and install snippets.

## Exhaustive means exhaustive

Counts in this README come from checked-in generated inventories, not hand-kept
marketing lists:

- Core registers 112/112 GNU-default codecs.
- Core + Extras register all 198 GNU libiconv 1.19 codecs, including every
  codec behind `--enable-extra-encodings`.
- Telecom registers 61 codecs and publishes 53 fixed-width packed profiles.
- Unicode registers 17 byte codecs and exposes two additional logical
  non-octet APIs.
- Standards, Bibliographic, Retro, Platforms, and current ICU register
  248 + 10 + 192 + 117 + 212 codecs.
- The ICU Archive facade registers 1,050 historical revisions; its six
  provider packages add storage, not codec identities.
- The ten runtime packages expose exactly 2,105 unique canonical names.

The GNU differential walks one sequential UCS-4BE corpus containing all
1,114,112 code points from U+0000 through U+10FFFF—including scalar values and
all 2,048 surrogate code points—in forward, reverse, and both cross-decoder
directions. The combined performance gate takes the fastest of three isolated
samples and rejects any supported GNU codec slower than 30x GNU libiconv.

Every leaf adds source-pinned vectors, full encode/decode table walks where
applicable, malformed-input policies, every-split streaming checks, packed
bit-order round trips, lifecycle tests, and clean artifact contracts. The
README gallery itself is executable integration-test input.

The current research projection tracks 385 direct Wikipedia-sourced clusters
absent from GNU libiconv 1.19. Of those, 249 are implemented by Iconvex. The
remaining 136 are audited instead of guessed: 12 are actionable codec gaps, 96
still require exact public mapping/specification work, and 28 are families,
mapping notations, repertoire abstractions/profiles, or a withdrawn unassigned
part rather than standalone byte codecs. See the
[generated disposition report](https://github.com/edescourtis/iconvex/blob/v0.1.1/iconvex/research/WIKIPEDIA_MISSING_FROM_GNU.md).

## Operational and trust boundaries

- One-shot conversion and the buffered `new/feed/finish` API materialize the
  complete source and, on general paths, a Unicode code-point list. Use
  `stream/4` or `stream!/4` for lazy bounded-memory conversion. Applications
  using the materializing APIs must enforce an input-size limit derived from
  their own memory budget; Iconvex deliberately does not choose a universal cap.
- Runtime mapping tables load lazily. A process using every core table should
  budget more than the measured 62.9 MiB flat decoded-table size, plus BEAM map,
  code, output, and temporary-list overhead. Typical workloads load only touched
  tables; optional mappings remain in `iconvex_extras`/other external packages.
- Substitution printf widths are validated and capped at 65,536 digits. Unknown,
  duplicate, or malformed options return typed request errors.
- Shipped table/transliteration ETF assets use safe decoding, versioned caches,
  and serialized cold loads. External codec modules remain trusted application
  code: callback exceptions may propagate. A malformed `encode_substitute/2`
  result returns a typed `{:invalid_codec_callback_return, ...}` request error.

## Verification

```sh
mix test
MIX_ENV=prod mix run bench/benchmark.exs
MIX_ENV=prod mix run bench/external_codec_benchmark.exs
GNU_ICONV=/path/to/gnu-libiconv-1.19/bin/iconv \
  MIX_ENV=prod mix run tools/exhaustive_unicode_differential.exs
```

The public source monorepo is
[`edescourtis/iconvex`](https://github.com/edescourtis/iconvex). ExDoc source
links are pinned to the matching `v0.1.1` tag and the `iconvex/` subdirectory,
so published documentation resolves to the exact release source.

The repository carries
267 byte-exact upstream files plus one derived configured `Makefile` in its
268-file GNU libiconv 1.19 test-fixture set (9.0 MB). Separate
and combined aggregate SHA-256 values verify their provenance boundary. The set
is intentionally excluded from the consumer Hex artifact; generated runtime
tables and public conformance documents remain in that artifact. Core ExUnit
runs the applicable default-codec cases; `iconvex_extras` carries and runs the
92 extra charmap/inverse fixtures plus ISO-2022-JP-3 and IBM-1047 behavioral
cases.
Together the suites port the 164 stateless charmap/inverse checks, all 27 snippet
round-trips, the complete GB18030 supplementary range,
transliteration/substitution/EBCDIC/discard checks, and the four C regression
tests. The z/OS filesystem-tag check remains platform-only, just as upstream
skips it outside z/OS. See
[UPSTREAM_TEST_COVERAGE.md](UPSTREAM_TEST_COVERAGE.md) for the machine-checked
traceability matrix. [SUPPORTED_ENCODINGS.md](SUPPORTED_ENCODINGS.md) lists all
198 codecs side-by-side with package ownership and GNU definition/default-build
status. Also see
[EXHAUSTIVE_UNICODE_DIFFERENTIAL.md](EXHAUSTIVE_UNICODE_DIFFERENTIAL.md) for the
112-codec core run, and the extras package's report for the combined 198-codec,
1,114,112-code-point forward/reverse/cross-decode differential against GNU
libiconv 1.19. The sequential UCS-4BE corpus contains all 1,112,064 Unicode
scalar values plus all 2,048 surrogate code points; surrogates are code points,
not scalar values. The combined directional performance gate records the
fastest of three isolated samples and enforces a 30x-GNU ceiling. Also see
[BENCHMARKS.md](BENCHMARKS.md) and
[TDD_LOG.md](TDD_LOG.md).
The disposition of every independent review finding is in
[DEEP_DIVE_REMEDIATION.md](DEEP_DIVE_REMEDIATION.md).

`elixir tools/release_check.exs` builds docs, unpacks the exact Hex artifact into
a temporary directory, and verifies compilation and representative conversions
from a clean consumer project.

To regenerate the registry, mappings, extensions, and transliteration table from
an unpacked GNU libiconv 1.19 source tree:

```sh
elixir tools/generate_registry.exs /path/to/libiconv-1.19
mix format
```

The generator writes the 112 default codecs into this package and the exact
86-codec complement into the sibling `iconvex_extras` package.

## License

Iconvex and its derived GNU mapping/transliteration data are distributed under
LGPL-2.1-or-later. The repository-only upstream test mirror includes GNU's
test-only `uniq-u.c`, licensed GPL-3.0-only; that source and its license text are
excluded from the Hex artifact. See [LICENSE](LICENSE),
[GNU GPL 3.0](https://www.gnu.org/licenses/gpl-3.0.html), and [NOTICE](NOTICE).