iconvex_standards

# Iconvex Standards

`iconvex_standards` adds the standards, registry, and archival codecs that do
not belong in the small Iconvex core. It is pure Elixir: no NIF, port, or
system `iconv` process is used.

Starting the OTP application atomically installs 248 collision-reviewed byte
codecs and 205 table providers into Iconvex. Stopping it removes exactly that
extension. A failed or conflicting installation is rolled back in full.

## Install

```elixir
def deps do
  [
    {:iconvex, "~> 0.1"},
    {:iconvex_standards, "~> 0.1"}
  ]
end
```

The application starts automatically:

```elixir
{:ok, koi8} = Iconvex.convert("Привет", "UTF-8", "KOI8-F")
{:ok, "Привет"} = Iconvex.convert(koi8, "KOI8-F", "UTF-8")

{:ok, bootstring} =
  Iconvex.convert("Καλημέρα 日本語", "UTF-8", "PUNYCODE")

{:ok, "Καλημέρα 日本語"} =
  Iconvex.convert(bootstring, "PUNYCODE", "UTF-8")

{:ok, mailbox_name} =
  Iconvex.convert("旅行/日本", "UTF-8", "UTF-7-IMAP")
```

All normal Iconvex conversion policies remain available, including strict
errors, discard, byte substitution, and Unicode substitution.

## What is in the package?

The frozen package manifest contains:

| Surface | Count | Highlights |
| --- | ---: | --- |
| Registered byte codecs | 248 | 143 RFC 1345 sets, ISO-IR, ISCII, Punycode, IMAP UTF-7, VIQR, VSCII-2, PASCII, KOI, Kermit, TACE-16 |
| Table providers | 205 | Package-local, immutable ETF lookup tables |
| Packed profiles | 5 | ECMA-1 six-bit plus four seven-bit profiles, each MSB- and LSB-first |
| Property-token mappings | 4 | Unicode 17 mainland/Taiwan telegraph and kGB3 |
| Raw transports | 2 | ECMA-44 seven- and eight-bit punched-card combinations |
| Catalogued quarantine | 2 | Fully mapped RFC 1345 tables intentionally not registered |

The larger generated families are 143 registered RFC 1345 codecs, 14 IANA PCL
symbol sets, 12 modern ISO-IR sets, seven CNS 11643 sets, six historical
graphic sets, five mosaic/technical sets, four Unicode legacy mappings, three
JIS X 0213 sets, and two KPS 9566-97 profiles.

Notable algorithmic and source-qualified codecs include:

- `PUNYCODE`, `UTF-7-IMAP`, `VIQR`, `MNEMONIC`, and `MNEM`
- all ten ISCII 1991 scripts
- `KOI7-switched`, `KOI8-F`, draft KOI8-C, and `SHORT-KOI`
- ABICOMP, BRASCII, VSCII-2, and four explicit PASCII policies
- ECMA-1, ISO 10585, ISO-IR 42, ISO-IR 169, and fixed-width Japanese EUC
- three Kermit ISO profiles and three commit-qualified Lithuanian standards
- TACE-16 in both big- and little-endian word transports

`Iconvex.Standards.encodings/0` returns the exact registered canonical names.
`Iconvex.Standards.registrations/0` also exposes aliases and source provenance.
The machine-readable ownership authority is
[`SURFACE_MANIFEST.tsv`](SURFACE_MANIFEST.tsv).

## Odd-width transports without pretending bits are bytes

The byte codec API represents one six- or seven-bit unit per octet. For wire
formats, use the explicit packed facade:

```elixir
alias Iconvex.Standards.Packed

{:ok, bits} =
  Packed.encode_from_utf8("ABC", "ECMA-1-PACKED-MSB")

{:ok, "ABC"} =
  Packed.decode_to_utf8(bits, "ECMA-1-PACKED-MSB")

{:ok, lsb_transport} =
  Packed.encode_from_utf8("Привет", "SHORT-KOI-PACKED-LSB")

{:ok, "Привет"} =
  Packed.decode_to_utf8(lsb_transport, "SHORT-KOI-PACKED-LSB")
```

LSB-first data is returned as `Iconvex.Packed.LSB`, retaining the exact bit
length, unit width, and bit order. Passing an explicitly named MSB profile with
`:lsb` (or vice versa) is rejected instead of silently changing the transport.

## Unicode property tokens are not byte encodings

The Unihan telegraph and kGB3 values are single property tokens. They therefore
implement `Iconvex.Standards.PropertyTokenMapping`, not `Iconvex.Codec`:

```elixir
alias Iconvex.Standards.Unihan17MainlandTelegraphDecimalToken, as: Telegraph

{:ok, scalar} = Telegraph.decode_token("0001")
{:ok, token} = Telegraph.encode_scalar(scalar)
```

Taiwan telegraph offers separate readable and lossless-VPUA reverse policies.
The distinction prevents duplicate and normalization-sensitive property values
from being presented as a fictitious concatenated byte-stream encoding.

## Verification

The package's RED/GREEN suite includes the 197 migrated source tests plus
package-boundary tests. It exhaustively traverses every decode and canonical
encode entry in all 205 provider tables, all single-byte inputs, all 13,069
ISCII oracle vectors, every TACE-16 word in both byte orders, every packed
source unit in both bit orders, and all 7,236 kGB3 property/GL rows. Lifecycle
tests force a provider collision and verify complete rollback.

Run it locally:

```console
ICONVEX_PATH=../iconvex mix test --warnings-as-errors
elixir tools/generate_surface_module.exs --check
```

Performance and scaling gates live in [`bench/`](bench/) and are documented in
[`BENCHMARKS.md`](BENCHMARKS.md). Test chronology is retained in
[`TDD_LOG.md`](TDD_LOG.md).

## License and provenance

Original Iconvex code is LGPL-2.1-or-later, matching GNU libiconv. Packaged
upstream evidence retains its own Unicode, BSD-3-Clause, MIT, or PSF terms.
See [`NOTICE`](NOTICE) and the accompanying license files.