iconvex

# Iconvex

**2,105 codecs. Ten runtime packages. Pure Elixir/BEAM. Zero NIFs, ports,
operating-system `iconv`, or external processes.**

[![Hex.pm](https://img.shields.io/hexpm/v/iconvex.svg)](https://hex.pm/packages/iconvex)
[![HexDocs](https://img.shields.io/badge/hex-docs-8e7ce6.svg)](https://hexdocs.pm/iconvex/)
[![CI](https://github.com/edescourtis/iconvex/actions/workflows/iconvex.yml/badge.svg)](https://github.com/edescourtis/iconvex/actions/workflows/iconvex.yml)
[![License](https://img.shields.io/hexpm/l/iconvex.svg)](LICENSE)

Iconvex provides a native BEAM port of GNU libiconv 1.19.
It includes every codec that `--enable-extra-encodings` selects.

The wider collection covers GSM septets, Morse, Baudot, UTF-9, UTF-18, Punycode,
SCSU, MARC-8, PETSCII, ATASCII, DEC formats, and UNIVAC FIELDATA.
It also covers punched cards, TeX fonts, vendor pages, current ICU converters,
and 1,050 historical ICU revisions.

Each historical ICU name includes its source filename.
This rule keeps incompatible revisions separate.

Core and Extras match all **198** GNU libiconv 1.19 fixed codecs.
Core supplies 112 default codecs. Extras supplies 86 optional codecs.
The wider Iconvex ecosystem supplies **2,105 unique canonical codecs**.

Use the standard conversion API:

```elixir
{:ok, "café"} =
  Iconvex.convert(<<0x63, 0x61, 0x66, 0xE9>>, "CP1252", "UTF-8")
```

Package APIs return exact Elixir bitstrings for non-octet formats.
They use self-describing LSB containers when the bit order requires one.

## Package selection

| Package | Canonical codecs | Coverage |
|---|---:|---|
| [`iconvex`](https://hex.pm/packages/iconvex) | **112** | GNU default: Unicode, CJK, ISO-2022, HZ, Windows/Mac/ISO pages, C99/JAVA, transliteration, generic 1–8-bit packing |
| [`iconvex_extras`](https://hex.pm/packages/iconvex_extras) | **86** | Complete GNU `--enable-extra-encodings`: AIX, DOS, OSF/1, z/OS EBCDIC, Atari ST, RISC OS, DEC Asian, JIS X 0213 |
| [`iconvex_telecom`](https://hex.pm/packages/iconvex_telecom) | **61** | GSM 03.38 national tables, Morse, Baudot/ITA, SITOR/NAVTEX, AIS6, TBCD, SIM alphabets, broadcast and teleprinter formats |
| [`iconvex_unicode`](https://hex.pm/packages/iconvex_unicode) | **17** | BOCU-1, CESU-8, SCSU, UTF-EBCDIC, UTF-1/5/6, WTF-8, signature profiles, plus native UTF-9 and UTF-18 APIs |
| [`iconvex_standards`](https://hex.pm/packages/iconvex_standards) | **248** | RFC 1345, Punycode, IMAP UTF-7, ISCII, ISO-IR, ECMA, PCL, Unihan and standards-defined transforms |
| [`iconvex_bibliographic`](https://hex.pm/packages/iconvex_bibliographic) | **10** | MARC-8, ANSEL, Cork/T1, TeX OT1/OML/OMS, PostScript and glyph-vector mappings |
| [`iconvex_retro`](https://hex.pm/packages/iconvex_retro) | **192** | PETSCII, ATASCII, Apple/DEC/CDC/UNIVAC, terminals, calculators, punched cards and 57 packed profiles |
| [`iconvex_platforms`](https://hex.pm/packages/iconvex_platforms) | **117** | IBM, Microsoft, .NET, Java, CPython, glibc, Kermit, Adobe, Mac and other vendor-defined compatibility surfaces |
| [`iconvex_icu`](https://hex.pm/packages/iconvex_icu) | **212** | Current ICU-derived SBCS/MBCS/stateful tables, LMBCS variants, Unicode profiles and compound text |
| [`iconvex_icu_archive`](https://hex.pm/packages/iconvex_icu_archive) | **1,050** | Source-filename-qualified historical ICU revisions, backed by six byte-balanced data packages |

You can install each package separately.
Each package depends only on Core.
The Archive facade depends on six exact-version provider-only data packages.
A leaf registers its complete codec set in one transaction.
Install only the families that your application uses.

`iconvex_specs` is an unpublished generation workspace.
It supports research, conformance, and integration.
It owns no runtime codec.
Do not add it as an application dependency.

The repository root contains `ICONVEX_CODEC_PACKAGE_INDEX.md` and
`ICONVEX_CODEC_PACKAGE_INDEX.csv`.
These generated files map the complete codec set.
`ICONVEX_FULL_STACK_SUPPORT.md` lists ownership and GNU status for all 2,105
names.
`PACKAGE_ARCHITECTURE.md` describes dependencies, lifecycle, artifacts, and
release order.

## Installation

Core only:

```elixir
def deps do
  [{:iconvex, "~> 0.1"}]
end
```

Exact GNU fixed-codec surface:

```elixir
def deps do
  [
    {:iconvex, "~> 0.1"},
    {:iconvex_extras, "~> 0.1"}
  ]
end
```

Choose any leaves:

```elixir
def deps do
  [
    {:iconvex, "~> 0.1"},
    {:iconvex_telecom, "~> 0.1"},
    {:iconvex_unicode, "~> 0.1"},
    {:iconvex_standards, "~> 0.1"},
    {:iconvex_bibliographic, "~> 0.1"},
    {:iconvex_retro, "~> 0.1"},
    {:iconvex_platforms, "~> 0.1"},
    {:iconvex_icu, "~> 0.1"}
  ]
end
```

For the historical archive, add only the facade:

```elixir
{:iconvex_icu_archive, "~> 0.1"}
```

Mix fetches `iconvex_icu_archive_data_01` through
`iconvex_icu_archive_data_06` transitively. Those packages own table storage,
not additional codec names.

## Encoding examples

Integration tests execute each example.

### Core: escapes, state machines, and Unicode

Core owns GNU's 112 default codecs and accepts 446 normalized names.
C99 and JAVA source escapes are codecs. HZ uses a state machine.

```elixir
{:ok, ~S(\u00e9\U0001f600)} =
  Iconvex.convert("é😀", "UTF-8", "C99")

{:ok, "~{VP~}~~{ND~}"} =
  Iconvex.convert("中~文", "UTF-8", "HZ")

[<<"caf", 0xC3>>, <<0xA9>>]
|> Iconvex.stream!("UTF-8", "CP1252")
|> Enum.join()
#=> <<0x63, 0x61, 0x66, 0xE9>>
```

Core includes UTF-7/8/16/32, UCS-2/4, GB18030:2005, GB18030:2022,
EUC-JP/KR/TW, Big5/HKSCS, Shift_JIS, JOHAB, TCVN, and VISCII.
It also includes the complete ISO-2022-JP/JP-1/JP-2/JP-MS/KR/CN/CN-EXT family.

### Extras: GNU optional encodings

Extras is the exact 86-codec complement behind GNU's optional definition
families. It includes DOS pages, AIX, OSF/1, 53 z/OS pages, `ATARIST`,
`RISCOS-LATIN1`, `DEC-HANYU`, `BIG5-2003`, JIS X 0213 and
`ISO-2022-JP-3`.

The z/OS UNIX surface makes EBCDIC newline policy explicit:

```elixir
{:ok, ebcdic} =
  Iconvex.convert("hello\n", "ASCII", "IBM-1047/ZOS_UNIX")

<<0x88, 0x85, 0x93, 0x93, 0x96, 0x15>> = ebcdic
{:ok, "hello\n"} = Iconvex.convert(ebcdic, "IBM-1047/ZOS_UNIX", "ASCII")
```

Extras provides IBM-423, the Greek EBCDIC codec from the original glibc
implementation.

### Telecom: septets, Baudot, Morse, and radio alphabets

Telecom owns 61 byte codecs and 53 packed profiles spanning 3, 5, 6, and 7
bits. International Morse follows ITU-R M.1677:

```elixir
{:ok, "... --- ..."} = Iconvex.convert("SOS", "UTF-8", "MORSE-CODE")
{:ok, "SOS"} = Iconvex.convert("... --- ...", "MORSE-CODE", "UTF-8")
# MORSE-CODE resolves to MORSE-ITU-M1677.
```

GSM 03.38's registry surface uses one septet per octet, exactly as SMPP does.
The high bit stays clear:

```elixir
{:ok, <<0x04, 0x0B>>} = Iconvex.convert("€Ğ", "UTF-8", "GSM0338-TURKISH")
{:ok, "€Ğ"} = Iconvex.convert(<<0x04, 0x0B>>, "GSM0338-TURKISH", "UTF-8")
```

All national locking and single-shift tables are available.
You can pair these tables independently:

```elixir
alias Iconvex.Telecom.GSM0338

options = [locking: :portuguese, single_shift: :bengali]
{:ok, [0x00EA, 0x09E6]} = GSM0338.decode(<<0x04, 0x1B, 0x19>>, options)
{:ok, <<0x04, 0x1B, 0x19>>} = GSM0338.encode([0x00EA, 0x09E6], options)
182 = length(GSM0338.valid_pairs())
```

TPDU packing produces contiguous wire septets:

```elixir
alias Iconvex.Telecom.GSM0338.Packing

{:ok, packed} = Packing.pack("hellohello")
"E8329BFD4697D9EC37" = Base.encode16(packed)
{:ok, "hellohello"} = Packing.unpack(packed, 10)
```

The same package covers ITA1/ITA2, Murray 1905, Wheatstone/Hausmann,
CCIR476/SITOR/NAVTEX, AIS6, TBCD, SIM/USIM alphabets, ARIB and three
Unicode 17 carrier-emoji profiles.

### Unicode: nonets, 18-bit words, and compression

Unicode owns BOCU-1, CESU-8, FSS-UTF, SCSU, UTF-EBCDIC, UTF-1, UTF-5, UTF-6,
WTF-8 and explicit signature profiles. UTF-9 and UTF-18 are logical non-octet
codecs, so their native APIs preserve the specified bit width:

```elixir
{:ok, utf9} = Iconvex.Unicode.UTF9.encode_packed([?A, 0x0391, 0x10330])
54 = bit_size(utf9)
{:ok, [?A, 0x0391, 0x10330]} = Iconvex.Unicode.UTF9.decode_packed(utf9)

{:ok, utf18} = Iconvex.Unicode.UTF18.encode_packed([?A, 0x10330])
36 = bit_size(utf18)
{:ok, [?A, 0x10330]} = Iconvex.Unicode.UTF18.decode_packed(utf18)
```

Byte protocols use `UTF-9-16BE`, `UTF-9-16LE`, `UTF-18-24BE`, or
`UTF-18-24LE`. Padding and endianness are part of the name instead of ambient
machine state.

### Standards: RFCs, ISO registers, and raw Punycode

Punycode implements the raw RFC 3492 transform.
It does not add an `xn--` DNS prefix.

```elixir
{:ok, "ihqwcrb4cv8a8dqg056pqjye"} =
  Iconvex.convert("他们为什么不说中文", "UTF-8", "PUNYCODE")

{:ok, "他们为什么不说中文"} =
  Iconvex.convert("ihqwcrb4cv8a8dqg056pqjye", "PUNYCODE", "UTF-8")
```

Standards also supplies RFC 1345 identities, IMAP UTF-7, ISCII, KOI variants,
ECMA-1/44, PCL sets, ISO-IR registers, and property-token mappings.
GNU spellings keep the GNU route when they collide with historical RFC 1345 names.
The historical identity keeps an explicit name such as `RFC1345:IBM037`.

### Bibliographic: MARC-8, ANSEL, and TeX machinery

MARC-8 puts combining marks before the base letter. Unicode does the opposite.
The codec restores Unicode mark order:

```elixir
{:ok, "e\u0301"} = Iconvex.convert(<<0xE2, ?e>>, "MARC-8", "UTF-8")
```

The package also owns ANSEL, Cork/T1, OT1, OML, OMS, Adobe PostScript and AGL
glyph-vector mappings, including explicit packed seven-bit math-font profiles.

### Retro: six-bit machines, punched cards, and calculator glyphs

The Apple-1 Signetics 2513 address order is `@`–`_`, then space–`?`.
The registry exposes one unit per byte.
`Iconvex.Retro.Packed` preserves six bits.

```elixir
{:ok, <<0, 1, 2, 3, 32, 63>>} =
  Iconvex.convert("@ABC ?", "UTF-8", "APPLE-1-SIGNETICS-2513-CM2140-6BIT")

{:ok, packed} =
  Iconvex.Retro.Packed.encode_from_utf8(
    "@ABC ?",
    Iconvex.Retro.Apple1Signetics2513,
    :msb
  )

36 = bit_size(packed)

{:ok, "@ABC ?"} =
  Iconvex.Retro.Packed.decode_to_utf8(
    packed,
    Iconvex.Retro.Apple1Signetics2513,
    :msb
  )
```

DEC RADIX-50 uses a base-40 word:

```elixir
{:ok, <<0o115402::16-little>>} =
  Iconvex.convert("X2B", "UTF-8", "DEC-RADIX-50")
```

Other codecs include `PETSCII`, `ATASCII`, Apple II, ZX80/81/Spectrum,
Amstrad, MSX, TRS-80, TI calculators, `DEC-SIXBIT`,
`FIELDATA-UNIVAC-1100`, CDC display codes, Hollerith and 12-bit punched cards.

### Platforms: vendor-qualified profiles

Platforms preserves source-qualified vendor behavior.
It does not merge mappings that only appear similar.

```elixir
{:ok, "\u1E83"} = Iconvex.convert(<<0xFF>>, "MacCeltic", "UTF-8")
```

Extras owns IBM-423 and GNU's optional IBM pages.
Platforms owns additional IBM CCSIDs, Windows best-fit maps, Java modified
UTF-8, and .NET or CPython profiles.
It also owns glibc charmaps, Adobe mappings, and Mac variants.

### ICU: current converter profiles

Current ICU data and algorithm profiles are separate from historical revisions.
LMBCS optimization groups remain distinct codecs:

```elixir
{:ok, <<0xD5>>} = Iconvex.convert("Ω", "UTF-8", "LMBCS-2")
{:ok, "Ω"} = Iconvex.convert(<<0xD5>>, "LMBCS-2", "UTF-8")
```

The 212-codec package includes 191 generated table providers plus compound
text, JIS, LMBCS and Unicode variant implementations.

### ICU Archive: reproducible historical behavior

The Archive owns 1,050 source-filename-qualified revisions. Six provider-only
packages limit artifact size without adding duplicate codec identities.
The following example uses a stateful IBM-930 revision with SO/SI:

```elixir
codec = Iconvex.ICUArchive.Codecs.C674
bytes = <<0xC1, 0x0E, 0x44, 0x81, 0x44, 0x82, 0x0F, 0xC2>>

"ICU-ARCHIVE-ibm-930_P120-1999" = codec.canonical_name()
{:ok, ^bytes} = codec.encode(~c"AあいB")
{:ok, ~c"AあいB"} = codec.decode(bytes)
```

Tests exhaustively check the facade manifest and shard assignment.
They also check every table digest and generated decoder/encoder mapping.

## Conversion API

Bang functions return the binary directly. Suffixes and explicit options share
one implementation:

```elixir
<<"caf", 0xE9>> = Iconvex.convert!("café", "UTF-8", "CP1252")
{:ok, "AB"} = Iconvex.convert("A😀B", "UTF-8", "ASCII//NON_IDENTICAL_DISCARD")
```

You can also pass options explicitly:

```elixir
Iconvex.convert(input, from, to,
  invalid: :discard,
  unrepresentable: :discard,
  transliterate: true,
  byte_substitute: "<0x%02x>",
  unicode_substitute: "<U+%04X>",
  on_invalid_byte: fn event -> {:replace, "<byte-#{event.byte}>"} end
)
```

The `byte_substitute` option replaces malformed source bytes.
The `unicode_substitute` option replaces unsupported destination characters.
Both formats accept GNU-style `%x` and `%X` fields with zero-padded widths.
With `iconvex_extras`, you can select IBM-1047 newline behavior by name or option.

Default destination handling matches GNU libiconv.
Any character causes an error when the target cannot represent it.
Unicode-family targets use U+FFFD for supported fallback values.
Discard, transliteration, or `unicode_substitute` takes priority over that fallback.

JAVA and UTF-7 preserve isolated 16-bit surrogate units from the source codec.

Generic UTF-16 and UTF-32 emit a BOM after they emit a target character.

ISO-2022-JP-2 implements GNU's case-insensitive `ja`, `ko`, and `zh` Unicode tags.
It supports cancellation and streaming state.

```elixir
Iconvex.convert(input, "IBM-1047/ZOS_UNIX", "UTF-8")
Iconvex.convert(input, "UTF-8", "IBM-1047", to_surface: :zos_unix)
```

`on_invalid_byte` handles source bytes that a codec cannot decode.
The callback receives an `Iconvex.InvalidByte` value.
The value contains the canonical encoding, error kind, absolute offset, current byte,
and codec error sequence.
The callback returns `:default`, `:error`, `:discard`, or `{:replace, utf8_or_codepoints}`.

Iconvex does not classify an unmapped byte as a display command, corruption,
or application control.
The file format and caller make that decision.
The same callback works with one-shot and lazy Stream conversion.

External fixed-width codecs can declare multi-byte recovery consumption, and
whole-string transforms can declare stop-on-error recovery so callbacks never
restart a desynchronized suffix. See [EXTENDING.md](EXTENDING.md).

Errors contain the source byte offset or unrepresentable Unicode code point:

```elixir
{:error, %Iconvex.Error{kind: :invalid_sequence, offset: 3}} =
  Iconvex.convert(input, "UTF-8", "CP1252")
```

For lazy bounded-memory conversion, input enumerable emits binaries and output
is an Elixir Stream of converted binaries:

```elixir
output =
  File.stream!(path, [], 64 * 1024)
  |> Iconvex.stream!("PETSCII", "UTF-8", on_invalid_byte: controls)
  |> Enum.into("")
```

The stream preserves lookahead, source offsets, UTF BOM state, and built-in
HZ, ISO-2022, and UTF-7 state across chunk boundaries.
It also preserves multi-code-point target mappings.

The UTF-7 decoder commits a shifted run after it validates the padding and UTF-16 payload.
A plus sign inside an active shift stays in that Base64 run.
The decoder processes byte-at-a-time shifted input in linear time.

ISO-2022 diagnostic frames and JP-MS SO/SI state do not depend on split locations.
Output chunks do not need to align with input chunks.
Conversion failures raise `Iconvex.Error` during enumeration because the caller might have consumed earlier output.
External codecs use the incremental callbacks in [EXTENDING.md](EXTENDING.md).

For a buffered chunked transaction:

```elixir
{:ok, converter} = Iconvex.new("UTF-8", "SHIFT_JIS")
{:ok, <<>>, converter} = Iconvex.feed(converter, chunk1)
{:ok, <<>>, converter} = Iconvex.feed(converter, chunk2)
{:ok, output, finished} = Iconvex.finish_with_state(converter)
{:error, :already_finished} = Iconvex.feed(finished, chunk3)
```

Iconvex buffers each source until `finish/1` or `finish_with_state/1`.
Each feed takes O(1) time.
Finalization converts the accumulated iodata once.

This contract makes chunked output byte-identical to one-shot output for every split.
It covers stateful designations, malformed-input policies, Vietnamese composition,
multi-code-point mappings, and UTF BOMs.
Errors use offsets from the start of the complete stream.

`finish/1` keeps the original return shape.
Use `finish_with_state/1` to get terminal state and repeated-finish rejection.

Use `Iconvex.encodings/0` for canonical names and
`Iconvex.canonical_name/1` to resolve an alias.

## Non-octet packed transports

You can pack codecs that expose one complete signal unit per octet.
The operation does not copy their mapping logic.

```elixir
{:ok, packed} = Iconvex.Packed.encode_from_utf8("Hello", "ASCII", 7)
{:ok, "Hello"} = Iconvex.Packed.decode_to_utf8(packed, "ASCII", 7)

{:ok, lsb} = Iconvex.Packed.encode_from_utf8_lsb("hellohello", "ASCII", 7)
"E8329BFD4697D9EC37" = Base.encode16(lsb.data)
{:ok, "hellohello"} = Iconvex.Packed.decode_to_utf8_lsb(lsb, "ASCII")
```

`pack/2` returns an exact MSB-first Elixir bitstring.
`pack_lsb/2` returns an `Iconvex.Packed.LSB` value.
The value contains byte data, meaningful bit count, and unit width.
This metadata removes ambiguity from final-octet padding.

When a codec rejects an unpacked unit, `Iconvex.Error` uses a physical bit offset.
It sets `offset_unit: :bit`.
MSB errors retain the offending bitstring.
LSB errors retain a self-describing `Iconvex.Packed.LSB` fragment.

Tests cover every unit value and each width from 1 through 8 in both orders.

Owning leaves implement wider formats that have codec-specific scalar rules.
Unicode owns UTF-9 and UTF-18.
Standards, Bibliographic, Retro, and Telecom publish their exact profile catalogs.

## External codec libraries

External packages implement `Iconvex.Codec`, then atomically register a module
at startup:

```elixir
defmodule MyLibrary.Codec do
  use Iconvex.Codec

  def canonical_name, do: "X-MY-CODEC"
  def decode(binary), do: MyLibrary.Decoder.decode(binary)
  def decode_discard(binary), do: MyLibrary.Decoder.decode_discard(binary)
  def encode(codepoints), do: MyLibrary.Encoder.encode(codepoints)
  def encode_discard(codepoints), do: MyLibrary.Encoder.encode_discard(codepoints)
  def encode_substitute(codepoints, replacer),
    do: MyLibrary.Encoder.encode_substitute(codepoints, replacer)
end

{:ok, ownership} = Iconvex.register_codec_if_absent(MyLibrary.Codec)
{:ok, output} = Iconvex.convert(input, "X-MY-CODEC", "UTF-8")
```

If `ownership` is a reference, retain it.
On shutdown, pass it to `Iconvex.unregister_codec(MyLibrary.Codec, ownership)`.
The `:existing` result means that another caller owns the registration.
Leave that registration in place.

Versioned snapshots preserve registrations and ownership tokens across consecutive registry-worker crashes.
The `:iconvex` application must stay active during recovery.
A clean application stop erases the recovery state.
Therefore, registrations do not cross a deliberate stop and start boundary.
Replacement uses one commit point.
Concurrent lookups resolve either the complete old registration or the complete new registration.

Strict per-commit durability writes a synchronous recovery snapshot to `:persistent_term`.
Replacing that term can schedule VM-wide garbage collection.
Keep registration on the control plane and outside the request-processing path.

Packages with many codecs should use `Iconvex.Extension.register/2`.
It wraps the internal `register_set` transaction.
It publishes the complete set in one transaction and writes one snapshot.
A delayed write would weaken the consecutive worker-crash guarantee.

Mapping tables and their decode accelerators use lazy `:persistent_term` entries.
Loaded application inventories set the maximum resident set.
Each table identity has one base term.
It also has at most one term for each of three derived cache shapes.

Provider removal erases the base term and all derived keys under their build locks.
Repeated package cycles do not accumulate provider cache generations.
Keep provider startup and shutdown on the control plane.
LRU replacement in the request-processing path would add a VM-wide cost to these immutable caches.

If a declared name collides with another source, use an explicit source-qualified name.
That registration uses the replacement canonical name.
Iconvex still registers and checks the module aliases and option aliases.

```elixir
:ok =
  Iconvex.register_codec(MyLibrary.Codec,
    canonical: "MY-SOURCE-X-MY-CODEC",
    aliases: ["MY-SOURCE-MYCODEC"]
  )
```

Consumer configuration can load modules with
`config :iconvex, external_codecs: [MyLibrary.Codec]`.
Tuple entries accept the same `:canonical` and `:aliases` options.
`Iconvex.encodings/0` includes the registered codecs.
External packages cannot override built-in names.

External OTP applications that own a complete codec inventory and mapping
tables can publish both with one public package transaction:

```elixir
{:ok, token} =
  Iconvex.Extension.register(
    [MyLibrary.Codec, MyLibrary.OtherCodec],
    owner: :my_library,
    priority: 10,
    table_providers: [:my_mapping]
  )

# Keep the opaque token in Application state.
:ok = Iconvex.Extension.unregister(token)
```

Registration and removal use atomic transactions.
Failures roll back newly acquired providers.
Supervised worker restarts preserve the exact token.
Stale tokens cannot remove replacements.

[EXTENDING.md](EXTENDING.md) documents:

- Callback contracts and errors.
- Aliases, package priorities, and collisions.
- Stateful streaming and UTF-8 callbacks.
- Substitution behavior.
- Performance results and tests.

`iconvex_extras` provides the reference external codec package.
It shares Core engines but keeps optional tables and modules outside Core.

Each of the nine extension facades installs one managed registration set.
The integration contract tests forward, reverse, and every cyclic start boundary.
Each path produces the same 2,105-name registry.

Canonical claims rank ahead of aliases and then by fixed package priority.
Stopping a winner exposes a retained fallback without a lookup gap.
Individual third-party registrations through `register_codec*` remain strict.
A distinct module that claims an existing name gets a `{:name_conflict, name}` error.

The generated `ICONVEX_FULL_STACK_SUPPORT.md` records the owning package and GNU
libiconv status for every canonical name.
The repository `ICONVEX_CODEC_PACKAGE_INDEX.md` document also includes:

- Modules and aliases.
- Sources and surface kinds.
- Packed profiles.
- Provenance records.
- Installation examples.

## Coverage evidence

Checked-in generated inventories supply the counts in this README.

- Core registers 112/112 GNU-default codecs.
- Core + Extras register all 198 GNU libiconv 1.19 codecs, including every
  codec behind `--enable-extra-encodings`.
- Telecom registers 61 codecs and publishes 53 fixed-width packed profiles.
- Unicode registers 17 byte codecs and exposes two additional logical
  non-octet APIs.
- Standards, Bibliographic, Retro, Platforms, and current ICU register
  248 + 10 + 192 + 117 + 212 codecs.
- The ICU Archive facade registers 1,050 historical revisions.
  Its six provider packages add storage but no codec identities.
- The ten runtime packages expose exactly 2,105 unique canonical names.

The GNU differential walks one sequential UCS-4BE corpus in four directions.
The corpus contains all 1,114,112 code points from U+0000 through U+10FFFF.
It includes all scalar values and all 2,048 surrogate code points.
The performance gate uses the fastest of three isolated samples.
It rejects a supported GNU codec when its result exceeds 30x GNU libiconv.

Each leaf adds source-pinned vectors and full table walks where applicable.
It also adds malformed-input policies, streaming checks, packed round trips,
lifecycle tests, and artifact contracts.
Integration tests execute the README gallery.

The research projection tracks 385 direct Wikipedia-sourced clusters that GNU libiconv 1.19 lacks.
Iconvex implements 249 of them.

The audit identifies 12 actionable codec gaps.
Another 96 clusters still need exact public mappings or specifications.
The final 28 are families, mapping notations, repertoire profiles, or a withdrawn part.
They are not standalone byte codecs.

See the generated `iconvex/research/WIKIPEDIA_MISSING_FROM_GNU.md` disposition report.

## Operational and trust boundaries

- One-shot conversion and the buffered `new/feed/finish` API materialize the
  complete source. General paths also materialize a Unicode code-point list.
  Applications must enforce an input-size limit for each materializing API.
  Iconvex does not choose a universal limit because application memory budgets differ.
  Use `stream/4` or `stream!/4` for lazy bounded-memory conversion.
- Runtime mapping tables load lazily. A process using every core table should
  budget more than the measured 62.9 MiB flat decoded-table size.
  Also budget for BEAM maps, code, output, and temporary lists.
  Typical workloads load only the tables that they use.
  Optional mappings stay in external packages such as `iconvex_extras`.
- Iconvex validates substitution printf widths and caps them at 65,536 digits.
  Unknown, duplicate, or malformed options return typed request errors.
- Shipped table/transliteration ETF assets use safe decoding, versioned caches,
  and serialized cold loads.
  External codec modules remain trusted application code.
  Callback exceptions can propagate.
  A malformed `encode_substitute/2` result returns a typed request error.

## Verification

```sh
mix test
MIX_ENV=prod mix run bench/benchmark.exs
MIX_ENV=prod mix run bench/external_codec_benchmark.exs
GNU_ICONV=/path/to/gnu-libiconv-1.19/bin/iconv \
  MIX_ENV=prod mix run tools/exhaustive_unicode_differential.exs
```

The public source monorepo is
[`edescourtis/iconvex`](https://github.com/edescourtis/iconvex).
ExDoc source links use the durable `main` branch and the `iconvex/` subdirectory.
These links work before a matching release tag exists.

The repository test-fixture set contains 268 GNU libiconv 1.19 files.
It contains 267 byte-exact upstream files plus one derived configured `Makefile`.
The set uses 9.0 MB.
Separate and combined SHA-256 values verify its provenance boundary.

The consumer Hex artifact excludes the test-fixture set.
It includes generated runtime tables and public conformance documents.
Core ExUnit runs the applicable default-codec cases.
`iconvex_extras` runs 92 extra fixtures plus ISO-2022-JP-3 and IBM-1047 cases.

Together, the suites port 164 stateless checks, 27 snippet round trips,
the GB18030 supplementary range, policy checks, and four C regression tests.
The z/OS filesystem-tag check stays platform-only because upstream skips it outside z/OS.

See [UPSTREAM_TEST_COVERAGE.md](UPSTREAM_TEST_COVERAGE.md) for the checked traceability matrix.
[SUPPORTED_ENCODINGS.md](SUPPORTED_ENCODINGS.md) lists all 198 codecs with package ownership and GNU status.

[EXHAUSTIVE_UNICODE_DIFFERENTIAL.md](EXHAUSTIVE_UNICODE_DIFFERENTIAL.md) describes the 112-codec Core run.
The Extras report describes the combined 198-codec differential against GNU libiconv 1.19.
The sequential UCS-4BE corpus contains all 1,112,064 scalar values and 2,048 surrogate code points.
Surrogates are code points, not scalar values.
The performance gate uses the fastest of three samples and enforces a 30x-GNU limit.

See [BENCHMARKS.md](BENCHMARKS.md) and [TDD_LOG.md](TDD_LOG.md).
[DEEP_DIVE_REMEDIATION.md](DEEP_DIVE_REMEDIATION.md) records each independent review finding.

`elixir tools/release_check.exs` builds the documentation.
It unpacks the exact Hex artifact into a temporary directory.
It verifies compilation and representative conversions from a clean consumer project.

To regenerate the registry, mappings, extensions, and transliteration table from
an unpacked GNU libiconv 1.19 source tree:

```sh
elixir tools/generate_registry.exs /path/to/libiconv-1.19
mix format
```

The generator writes the 112 default codecs into this package and the exact
86-codec complement into the sibling `iconvex_extras` package.

## License

LGPL-2.1-or-later covers Iconvex and its derived GNU mapping data.
The repository-only test mirror includes GNU's test-only `uniq-u.c`.
GPL-3.0-only covers that source.
The Hex artifact excludes the source and its license text.
See [LICENSE](LICENSE), [GNU GPL 3.0](https://www.gnu.org/licenses/gpl-3.0.html),
and [NOTICE](NOTICE).

<!-- iconvex-codec-catalog:start -->
## Complete codec catalog

The generator builds this table from the release inventories.
Do not edit this section by hand.
Codec names ignore ASCII letter case.
You can use each alias to look up its codec.
Aliases do not increase the canonical codec count.

**112 canonical codecs · 334 aliases · 446 accepted names.**

### Registered codec names

| Canonical codec | Accepted aliases | Stateful |
|---|---|:---:|
| `ARMSCII-8` | — | No |
| `BIG5` | `BIG-5`<br>`BIG-FIVE`<br>`BIGFIVE`<br>`CN-BIG5`<br>`CSBIG5` | No |
| `BIG5-HKSCS` | `BIG5-HKSCS:2008`<br>`BIG5HKSCS` | No |
| `BIG5-HKSCS:1999` | — | No |
| `BIG5-HKSCS:2001` | — | No |
| `BIG5-HKSCS:2004` | — | No |
| `C99` | — | No |
| `CP1131` | `IBM-1131` | No |
| `CP1133` | `IBM-CP1133` | No |
| `CP1250` | `IBM-1250`<br>`MS-EE`<br>`WINDOWS-1250` | No |
| `CP1251` | `ANSI-1251`<br>`IBM-1251`<br>`MS-CYRL`<br>`WINDOWS-1251` | No |
| `CP1252` | `IBM-1004`<br>`IBM-1252`<br>`MS-ANSI`<br>`WINDOWS-1252` | No |
| `CP1253` | `IBM-1253`<br>`MS-GREEK`<br>`WINDOWS-1253` | No |
| `CP1254` | `IBM-1254`<br>`MS-TURK`<br>`WINDOWS-1254` | No |
| `CP1255` | `IBM-1255`<br>`MS-HEBR`<br>`WINDOWS-1255` | No |
| `CP1256` | `IBM-1256`<br>`MS-ARAB`<br>`WINDOWS-1256` | No |
| `CP1257` | `IBM-1257`<br>`WINBALTRIM`<br>`WINDOWS-1257` | No |
| `CP1258` | `WINDOWS-1258` | No |
| `CP850` | `850`<br>`CSPC850MULTILINGUAL`<br>`IBM-4946`<br>`IBM-850`<br>`IBM850` | No |
| `CP862` | `862`<br>`CSPC862LATINHEBREW`<br>`IBM-862`<br>`IBM862` | No |
| `CP866` | `866`<br>`CSIBM866`<br>`IBM-866`<br>`IBM866` | No |
| `CP874` | `IBM-874`<br>`WINDOWS-874` | No |
| `CP932` | `IBM-932` | No |
| `CP936` | `MS936`<br>`WINDOWS-936` | No |
| `CP949` | `IBM-949`<br>`KSC5601`<br>`UHC` | No |
| `CP950` | `IBM-950` | No |
| `EUC-CN` | `CN-GB`<br>`CSGB2312`<br>`DECHANZI`<br>`EUCCN`<br>`GB2312`<br>`HP15CN`<br>`IBM-1381`<br>`IBM-1383`<br>`IBM-EUCCN` | No |
| `EUC-JP` | `CSEUCPKDFMTJAPANESE`<br>`EUCJP`<br>`EXTENDED_UNIX_CODE_PACKED_FORMAT_FOR_JAPANESE`<br>`IBM-3372`<br>`IBM-33722`<br>`IBM-954`<br>`IBM-EUCJP`<br>`SDECKANJI` | No |
| `EUC-KR` | `5601`<br>`CSEUCKR`<br>`DECKOREAN`<br>`EUCKR`<br>`IBM-970`<br>`IBM-EUCKR` | No |
| `EUC-TW` | `CNS11643`<br>`CSEUCTW`<br>`EUCTW`<br>`EUCTW-1993`<br>`IBM-964`<br>`IBM-EUCTW` | No |
| `GB18030` | `GB18030:2005` | No |
| `GB18030:2022` | — | No |
| `GBK` | `IBM-1386` | No |
| `GB_1988-80` | `CN`<br>`CSISO57GB1988`<br>`ISO-IR-57`<br>`ISO646-CN` | No |
| `GB_2312-80` | `CHINESE`<br>`CSISO58GB231280`<br>`ISO-IR-58` | No |
| `GEORGIAN-ACADEMY` | — | No |
| `GEORGIAN-PS` | — | No |
| `HP-ROMAN8` | `CSHPROMAN8`<br>`R8`<br>`ROMAN8` | No |
| `HZ` | `HZ-GB-2312` | Yes |
| `ISO-2022-CN` | `CSISO2022CN` | Yes |
| `ISO-2022-CN-EXT` | — | Yes |
| `ISO-2022-JP` | `CSISO2022JP` | Yes |
| `ISO-2022-JP-1` | `CSJISENCODING`<br>`IBM-5054`<br>`JIS_ENCODING` | Yes |
| `ISO-2022-JP-2` | `CSISO2022JP2` | Yes |
| `ISO-2022-JP-MS` | `CP50221` | Yes |
| `ISO-2022-KR` | `CSISO2022KR` | Yes |
| `ISO-8859-1` | `CP819`<br>`CSISOLATIN1`<br>`IBM-819`<br>`IBM819`<br>`ISO-IR-100`<br>`ISO8859-1`<br>`ISO88591`<br>`ISO_8859-1`<br>`ISO_8859-1:1987`<br>`L1`<br>`LATIN1` | No |
| `ISO-8859-10` | `CSISOLATIN6`<br>`ISO-IR-157`<br>`ISO8859-10`<br>`ISO_8859-10`<br>`ISO_8859-10:1992`<br>`L6`<br>`LATIN6` | No |
| `ISO-8859-11` | `ISO8859-11`<br>`ISO_8859-11` | No |
| `ISO-8859-13` | `IBM-921`<br>`ISO-IR-179`<br>`ISO8859-13`<br>`ISO_8859-13`<br>`L7`<br>`LATIN7` | No |
| `ISO-8859-14` | `ISO-CELTIC`<br>`ISO-IR-199`<br>`ISO8859-14`<br>`ISO_8859-14`<br>`ISO_8859-14:1998`<br>`L8`<br>`LATIN8` | No |
| `ISO-8859-15` | `IBM-923`<br>`ISO-IR-203`<br>`ISO8859-15`<br>`ISO885915`<br>`ISO_8859-15`<br>`ISO_8859-15:1998`<br>`LATIN-9` | No |
| `ISO-8859-16` | `ISO-IR-226`<br>`ISO8859-16`<br>`ISO_8859-16`<br>`ISO_8859-16:2001`<br>`L10`<br>`LATIN10` | No |
| `ISO-8859-2` | `CSISOLATIN2`<br>`IBM-912`<br>`ISO-IR-101`<br>`ISO8859-2`<br>`ISO88592`<br>`ISO_8859-2`<br>`ISO_8859-2:1987`<br>`L2`<br>`LATIN2` | No |
| `ISO-8859-3` | `CSISOLATIN3`<br>`IBM-913`<br>`ISO-IR-109`<br>`ISO8859-3`<br>`ISO_8859-3`<br>`ISO_8859-3:1988`<br>`L3`<br>`LATIN3` | No |
| `ISO-8859-4` | `CSISOLATIN4`<br>`IBM-914`<br>`ISO-IR-110`<br>`ISO8859-4`<br>`ISO_8859-4`<br>`ISO_8859-4:1988`<br>`L4`<br>`LATIN4` | No |
| `ISO-8859-5` | `CSISOLATINCYRILLIC`<br>`CYRILLIC`<br>`IBM-915`<br>`ISO-IR-144`<br>`ISO8859-5`<br>`ISO88595`<br>`ISO_8859-5`<br>`ISO_8859-5:1988` | No |
| `ISO-8859-6` | `ARABIC`<br>`ASMO-708`<br>`CSISOLATINARABIC`<br>`ECMA-114`<br>`IBM-1089`<br>`ISO-8859-6-E`<br>`ISO-8859-6-I`<br>`ISO-IR-127`<br>`ISO8859-6`<br>`ISO88596`<br>`ISO_8859-6`<br>`ISO_8859-6:1987` | No |
| `ISO-8859-7` | `CSISOLATINGREEK`<br>`ECMA-118`<br>`ELOT_928`<br>`GREEK`<br>`GREEK8`<br>`IBM-813`<br>`ISO-IR-126`<br>`ISO-IR-227`<br>`ISO8859-7`<br>`ISO88597`<br>`ISO_8859-7`<br>`ISO_8859-7:1987`<br>`ISO_8859-7:2003` | No |
| `ISO-8859-8` | `CSISOLATINHEBREW`<br>`ECMA-121`<br>`HEBREW`<br>`IBM-916`<br>`ISO-8859-8-E`<br>`ISO-8859-8-I`<br>`ISO-IR-138`<br>`ISO8859-8`<br>`ISO88598`<br>`ISO_8859-8`<br>`ISO_8859-8:1988` | No |
| `ISO-8859-9` | `CSISOLATIN5`<br>`IBM-920`<br>`ISO-IR-148`<br>`ISO8859-9`<br>`ISO88599`<br>`ISO_8859-9`<br>`ISO_8859-9:1989`<br>`L5`<br>`LATIN5` | No |
| `ISO-IR-165` | `CN-GB-ISOIR165` | No |
| `JAVA` | — | No |
| `JIS_C6220-1969-RO` | `CSISO14JISC6220RO`<br>`ISO-IR-14`<br>`ISO646-JP`<br>`JP` | No |
| `JIS_X0201` | `CSHALFWIDTHKATAKANA`<br>`JISX0201-1976`<br>`X0201` | No |
| `JIS_X0208` | `CSISO87JISX0208`<br>`ISO-IR-168`<br>`ISO-IR-87`<br>`JIS0208`<br>`JIS_C6226-1983`<br>`JIS_X0208-1983`<br>`JIS_X0208-1990`<br>`X0208` | No |
| `JIS_X0212` | `CSISO159JISX02121990`<br>`ISO-IR-159`<br>`JIS_X0212-1990`<br>`JIS_X0212.1990-0`<br>`X0212` | No |
| `JOHAB` | `CP1361`<br>`KO_KR.JOHAP92` | No |
| `KOI8-R` | `CSKOI8R`<br>`IBM-878` | No |
| `KOI8-RU` | — | No |
| `KOI8-T` | — | No |
| `KOI8-U` | — | No |
| `KSC_5601` | `CSKSC56011987`<br>`ISO-IR-149`<br>`KOREAN`<br>`KS_C_5601-1987`<br>`KS_C_5601-1989` | No |
| `MacArabic` | — | No |
| `MacCentralEurope` | — | No |
| `MacCroatian` | — | No |
| `MacCyrillic` | — | No |
| `MacGreek` | — | No |
| `MacHebrew` | — | No |
| `MacIceland` | — | No |
| `MacRoman` | `CSMACINTOSH`<br>`MAC`<br>`MACINTOSH` | No |
| `MacRomania` | — | No |
| `MacThai` | — | No |
| `MacTurkish` | — | No |
| `MacUkraine` | — | No |
| `MULELAO-1` | — | No |
| `NEXTSTEP` | — | No |
| `PT154` | `CP154`<br>`CSPTCP154`<br>`CYRILLIC-ASIAN`<br>`PTCP154` | No |
| `RK1048` | `CSKZ1048`<br>`KZ-1048`<br>`STRK1048-2002` | No |
| `SHIFT_JIS` | `CSSHIFTJIS`<br>`MS_KANJI`<br>`PCK`<br>`SHIFT-JIS`<br>`SJIS` | No |
| `TCVN` | `TCVN-5712`<br>`TCVN5712-1`<br>`TCVN5712-1:1993` | No |
| `TIS-620` | `ISO-IR-166`<br>`TACTIS`<br>`TIS620`<br>`TIS620-0`<br>`TIS620.2529-1`<br>`TIS620.2533`<br>`TIS620.2533-0`<br>`TIS620.2533-1` | No |
| `UCS-2` | `CSUNICODE`<br>`ISO-10646-UCS-2` | No |
| `UCS-2-INTERNAL` | — | No |
| `UCS-2-SWAPPED` | — | No |
| `UCS-2BE` | `CSUNICODE11`<br>`ISO-IR-162`<br>`ISO-IR-174`<br>`ISO-IR-176`<br>`UNICODE-1-1`<br>`UNICODEBIG` | No |
| `UCS-2LE` | `UNICODELITTLE` | No |
| `UCS-4` | `CSUCS4`<br>`ISO-10646-UCS-4` | No |
| `UCS-4-INTERNAL` | — | No |
| `UCS-4-SWAPPED` | — | No |
| `UCS-4BE` | `ISO-IR-163`<br>`ISO-IR-175`<br>`ISO-IR-177` | No |
| `UCS-4LE` | — | No |
| `US-ASCII` | `646`<br>`ANSI_X3.4-1968`<br>`ANSI_X3.4-1986`<br>`ASCII`<br>`CP367`<br>`CSASCII`<br>`IBM-367`<br>`IBM367`<br>`ISO-IR-6`<br>`ISO646-US`<br>`ISO_646.IRV:1991`<br>`US` | No |
| `UTF-16` | `ISO-IR-193`<br>`ISO-IR-194`<br>`ISO-IR-195` | No |
| `UTF-16BE` | `CP1201`<br>`UNICODEFFFE`<br>`WINDOWS-1201` | No |
| `UTF-16LE` | `CP1200`<br>`WINDOWS-1200` | No |
| `UTF-32` | — | No |
| `UTF-32BE` | — | No |
| `UTF-32LE` | `CP12000`<br>`WINDOWS-12000` | No |
| `UTF-7` | `CSUNICODE11UTF7`<br>`UNICODE-1-1-UTF-7` | Yes |
| `UTF-8` | `IBM-1208`<br>`ISO-IR-190`<br>`ISO-IR-191`<br>`ISO-IR-192`<br>`ISO-IR-196`<br>`UTF8` | No |
| `VISCII` | `CSVISCII`<br>`VISCII1.1-1` | No |
<!-- iconvex-codec-catalog:end -->