Current section
Files
Jump to
Current section
Files
erlang_python
CHANGELOG.md
CHANGELOG.md
# Changelog
## 2.2.0 (2026-03-24)
### Added
- **OWN_GIL Mode** - True parallel Python execution with Python 3.14+ subinterpreters
- Each subinterpreter runs with its own GIL (`Py_GIL_OWN`) in a dedicated thread
- Full isolation between interpreters (separate namespaces, modules, state)
- `py_context:start_link(N, owngil)` to create OWN_GIL contexts
- Enables true parallelism for CPU-bound Python workloads
- See [OWN_GIL Internals](docs/owngil_internals.md) for architecture details
- **Process-Bound Python Environments** - Per-Erlang-process Python namespaces
- Each Erlang process gets isolated Python globals/locals
- State persists across calls within the same process
- Automatic cleanup when Erlang process terminates
- See [Process-Bound Environments](docs/process-bound-envs.md) for details
- **Event Loop Pool** - Process affinity for parallel async execution
- `py_event_loop_pool` distributes async tasks across multiple event loops
- Scheduler-affinity routing for cache-friendly execution
- Supports worker, subinterp, and owngil modes
- **ByteChannel API** - Raw byte streaming without term serialization
- `py_byte_channel:new/0,1` - Create byte channels
- `py_byte_channel:send/2` - Send raw bytes
- `py_byte_channel:recv/1,2` - Receive bytes
- Python `ByteChannel` class with sync/async iteration
- Ideal for HTTP bodies, file streaming, binary protocols
- **PyBuffer API** - Zero-copy buffer for WSGI input streams
- `py_buffer:new/0,1` - Create buffers with optional max size
- `py_buffer:write/2` - Write data to buffer
- Python `PyBuffer` class with file-like interface (`read`, `readline`, `readlines`)
- Non-blocking reads for async I/O patterns
- See [Buffer API](docs/buffer.md) for details
- **True streaming API** - New `py:stream_start/3,4` and `py:stream_cancel/1` functions
for event-driven streaming from Python generators. Unlike `py:stream/3,4` which
collects all values at once, `stream_start` sends `{py_stream, Ref, {data, Value}}`
messages as values are yielded. Supports both sync and async generators. Useful for
LLM token streaming, real-time data feeds, and processing large sequences incrementally.
- **`erlang.whereis(name)`** - Lookup registered Erlang process PIDs from Python
- Returns `erlang.Pid` object or `None` if not registered
- Enables Python code to discover and message named processes
- **`erlang.schedule_inline(callback)`** - Inline continuation scheduling
- Release dirty scheduler and continue with callback in same context
- Preserves globals/locals across the continuation
- Useful for cooperative long-running tasks
- **`py:spawn_call/3,4,5`** - Fire-and-forget with result delivery
- Executes Python call asynchronously
- Sends `{py_result, Ref, Result}` to caller when complete
- Non-blocking alternative to `py:call` for async patterns
- **Explicit bytes conversion** - `{bytes, Binary}` tuple for round-trip safety
- Erlang binaries convert to Python `str` by default
- Use `{bytes, Binary}` to force Python `bytes` type
- Ensures correct handling for binary protocols
- **Import caching API** - Lazy module import with caching
- `py:import/1,2` - Import and cache modules
- `py:add_import/1,2` - Register imports applied to all contexts
- `py:add_path/1` - Add to sys.path across all contexts
- Per-interpreter caching with generation tracking
- **Per-interpreter preload code** - Execute code in new interpreters
- Configure via `{erlang_python, [{preload_code, <<"import mylib">>}]}`
- Code runs with inherited globals from main interpreter
- Useful for initializing common imports/state
### Fixed
- **Channel notification for create_task** - Fixed async channel receive hanging when using
`py_event_loop:create_task`. The `event_loop_add_pending()` now sends `task_ready` to the
worker, not just `pthread_cond_signal`. Also fixed Python 3.9 compatibility in ByteChannel
(`Optional[bytes]` instead of `bytes | None`)
- **Channel waiter race condition** - Fixed `waiter_exists` errors during fast async iteration.
Waiter state is now cleared before releasing mutex, preventing race where callback fires
before `channel_send` clears `has_waiter`
- **Event Loop Isolation and Resource Safety** - Three fixes for event loop and atom handling
- **Single-loop-per-interpreter enforcement** - Prevents multiple `ErlangEventLoop` instances
from causing event confusion. Added `_has_loop_ref()` check that detects running loops;
attempting to create a second loop while one is running raises `RuntimeError`
- **Atom creation safety** - Added Python-level caching with configurable limit (10000 default,
`ERLANG_PYTHON_MAX_ATOMS` env var) to prevent BEAM atom table exhaustion from untrusted code.
The `erlang.atom()` API now goes through the cached wrapper; internal `_atom()` NIF still available
- **Global capsule resource leak** - Added `global_loop_capsule_destructor` that properly calls
`enif_release_resource()` when capsule is garbage collected. Previously NULL destructor caused
reference leaks on each `ErlangEventLoop` creation
- **Python 3.14 venv activation** - Fixed `.pth` file processing in subinterpreters. Python 3.14
stricter module isolation prevented `sys._venv_site_packages` from persisting across eval/exec calls.
Now embeds site-packages path directly in the exec code string
- **OWN_GIL Safety Fixes** - Critical fixes for OWN_GIL subinterpreter mode
- **Mutex leak in erlang module** - `async_futures_mutex` now always destroyed in
`erlang_module_free()` regardless of `pipe_initialized` flag
- **ABBA deadlock prevention** - Fixed lock ordering in `event_loop_down()` and
`event_loop_destructor()` to acquire GIL before `namespaces_mutex`, matching the
normal execution path and preventing deadlocks
- **Dangling env pointer detection** - Added `interp_id` validation in
`owngil_execute_*_with_env()` functions to detect and reject env resources
created by a different interpreter, returning `{error, env_wrong_interpreter}`
- **OWN_GIL callback documentation** - Documented that `erlang.call()` from OWN_GIL
contexts uses `thread_worker_call()` rather than suspension/resume protocol;
re-entrant calls to the same OWN_GIL context are not supported
### Changed
- **`py:cast` is now fire-and-forget** - `py:cast/3,4,5` no longer returns a reference.
For async calls with result delivery, use the new `py:spawn_call/3,4,5` instead.
- **OWN_GIL requires Python 3.14+** - The OWN_GIL subinterpreter mode requires Python 3.14
or later due to C extension compatibility issues in earlier versions. Use `worker` or
`subinterp` modes for Python 3.12-3.13.
- **Removed auto-started io pool** - The io pool is no longer started automatically at
application startup to reduce memory usage. Users who need a dedicated I/O pool can
create one manually via `py_context_router:start_pool(io, 10, worker)`. The configuration
options `io_pool_size` and `io_pool_mode` have been removed.
- **Removed py_event_router** - Removed legacy `py_event_router` module. The `py_event_worker`
now handles all event loop functionality including FD events, timers, and task processing.
This simplifies the architecture by consolidating event handling into a single worker process.
The `py_nif:set_shared_router/1` function has been removed.
- **Config-based initialization** - Import and path configuration via application environment
- Configure imports: `{erlang_python, [{imports, [{json, dumps}]}]}`
- Configure paths: `{erlang_python, [{paths, ["/path/to/modules"]}]}`
- Applied immediately to all running interpreters
- See [Imports documentation](docs/imports.md) for details
### Performance
- **Direct NIF channel operations** - Channel send/receive bypass `erlang.call()` overhead
for up to 1760x speedup in raw throughput benchmarks
- **nif_process_ready_tasks optimization** - ~15% improvement in async task processing
- Replace `asyncio.iscoroutine()` with `PyCoro_CheckExact` C API
- Use stack buffers for module/func strings
- Cache `asyncio.events` module
- Pool `ErlNifEnv` allocations with mutex protection
## 2.1.0 (2026-03-12)
### Added
- **Async Task API** - uvloop-inspired task submission from Erlang
- `py_event_loop:run/3,4` - Blocking run of async Python functions
- `py_event_loop:create_task/3,4` - Non-blocking task submission with reference
- `py_event_loop:await/1,2` - Wait for task result with timeout
- `py_event_loop:spawn_task/3,4` - Fire-and-forget task execution
- Thread-safe submission via `enif_send` (works from dirty schedulers)
- Message-based result delivery via `{async_result, Ref, Result}`
- See [Async Task API docs](docs/asyncio.md#async-task-api-erlang) for details
- **`erlang.spawn_task(coro)`** - Spawn async tasks from both sync and async contexts
- Works in sync code called by Erlang (where `asyncio.get_running_loop()` fails)
- Returns `asyncio.Task` for optional await/cancel (fire-and-forget pattern)
- Automatically wakes up the event loop in sync context
- **Explicit Scheduling API** - Control dirty scheduler release from Python
- `erlang.schedule(callback, *args)` - Release scheduler, continue via Erlang callback
- `erlang.schedule_py(module, func, args, kwargs)` - Release scheduler, continue in Python
- `erlang.consume_time_slice(percent)` - Check if NIF time slice exhausted
- `ScheduleMarker` type for cooperative long-running tasks
- See [Scheduling API docs](docs/asyncio.md#explicit-scheduling-api)
- **Distributed Python Execution** - Documentation and Docker demo
- Run Python across Erlang nodes using `rpc:call`
- Docker Compose setup for testing distributed patterns
- See [Distributed Execution docs](docs/distributed.md)
### Changed
- **Event Loop Performance Optimizations**
- Growable pending queue with capacity doubling (256 to 16384)
- Snapshot-detach pattern to reduce mutex contention
- Callable cache (64 slots) avoids PyImport/GetAttr per task
- Task wakeup coalescing with atomic flag
- Drain-until-empty loop for faster task processing
### Fixed
- `ensure_venv` now always installs dependencies, even if venv exists
- `erlang.sleep()` timing in sync context
- `time()` returns fresh value when loop not running
- Handle pooling bugs in ErlangEventLoop
- Task wakeup race causing batch task stalls
## 2.0.0 (2026-03-09)
### Added
- **Virtual Environment Management** - Automatic venv creation and activation
- `py:ensure_venv/2,3` - Create venv if missing, then activate
- Automatically detects Python executable
- Supports pip install of dependencies
- **File Descriptor Duplication** - Safe socket handoff from Erlang to Python
- `py:dup_fd/1` - Duplicate fd for independent ownership
- Prevents double-close issues when passing sockets to Python reactor
- **Custom Pool Support** - Create pools on demand for CPU-bound and I/O-bound operations
- `default` pool - Automatically started, sized to number of schedulers
- `py_context_router:start_pool/2,3` - Start named pools programmatically
- `py_context_router:stop_pool/1` - Stop a named pool
- `py_context_router:pool_started/1` - Check if a pool is running
- `py_context_router:get_context(Pool)` - Get context from a named pool
- `py_context_router:num_contexts(Pool)` - Get pool size
- `py_context_router:contexts(Pool)` - Get all contexts in a pool
- `py_context_router:lookup_pool(Module, Func)` - Query pool routing
- `py:call(PoolName, Module, Func, Args)` - Execute on a specific pool
- Registration-based routing (no call site changes needed):
- `py:register_pool(io, requests)` - Route all `requests.*` calls to io pool
- `py:register_pool(io, {aiohttp, get})` - Route specific function to io pool
- `py:unregister_pool(Module)` - Remove module registration
- `py:unregister_pool({Module, Func})` - Remove function registration
- Automatic routing: `py:call(requests, get, [Url])` goes to io pool when registered
- Backward compatible: existing code using `py:call/3,4,5` works unchanged
- New test suite: `test/py_pool_SUITE.erl`
- **Channel API** - Bidirectional message passing between Erlang and Python
- `py_channel:new/0,1` - Create channels with optional backpressure (`max_size`)
- `py_channel:send/2` - Send Erlang terms to Python (returns `busy` on backpressure)
- `py_channel:close/1` - Close channel, signals `StopIteration` to Python
- Python `Channel` class with sync and async interfaces:
- `channel.receive()` - Blocking receive (suspends Python, yields to Erlang)
- `channel.try_receive()` - Non-blocking receive
- `await channel.async_receive()` - Asyncio-compatible receive
- `for msg in channel:` - Sync iteration
- `async for msg in channel:` - Async iteration
- `erlang.channel.reply(pid, term)` - Send messages to Erlang processes
- Zero-copy IOQueue buffering via `enif_ioq`
- 8x faster than Reactor for small messages, 2x faster for 16KB messages
- **OWN_GIL Subinterpreter Thread Pool** - True parallelism with Python 3.12+ subinterpreters
- Each subinterpreter runs in its own thread with its own GIL (`Py_GIL_OWN`)
- Thread pool manages N subinterpreters for parallel Python execution
- `py:context(N)` returns the Nth context PID for explicit context selection
- `py_context_router` provides scheduler-affinity routing for automatic distribution
- Cast operations are 25-30% faster compared to worker mode
- Full isolation between subinterpreters (separate namespaces, modules, state)
- New C files: `py_subinterp_pool.c`, `py_subinterp_pool.h`
- **`erlang.reactor` module** - FD-based protocol handling for building custom servers
- `reactor.Protocol` - Base class for implementing protocols
- `reactor.serve(sock, protocol_factory)` - Serve connections using a protocol
- `reactor.run_fd(fd, protocol_factory)` - Handle a single FD with a protocol
- Integrates with Erlang's `enif_select` for efficient I/O multiplexing
- Zero-copy buffer management for high-throughput scenarios
- Supports SHARED_GIL subinterpreters via `py_reactor_context`
- Each reactor context has isolated protocol factory when using `mode=subinterp`
- **ETF encoding for PIDs and References** - Full Erlang term format support
- Erlang PIDs encode/decode properly in ETF binary format
- Erlang References encode/decode properly in ETF binary format
- Enables proper serialization for distributed Erlang communication
- **PID serialization** - Erlang PIDs now convert to `erlang.Pid` objects in Python
and back to real PIDs when returned to Erlang. Previously, PIDs fell through to
`None` (Erlang→Python) or string representation (Python→Erlang).
- **`erlang.send(pid, term)`** - Fire-and-forget message passing from Python to
Erlang processes. Uses `enif_send()` directly with no suspension or blocking.
Raises `erlang.ProcessError` if the target process is dead.
- **`erlang.ProcessError`** - New exception for dead/unreachable process errors.
Subclass of `Exception`, so it's catchable with `except Exception` or
`except erlang.ProcessError`.
- **Audit hook sandbox** - Block dangerous operations when running inside Erlang VM
- Uses Python's `sys.addaudithook()` (PEP 578) for low-level blocking
- Blocks: `os.fork`, `os.system`, `os.popen`, `os.exec*`, `os.spawn*`, `subprocess.Popen`
- Raises `RuntimeError` with clear message about using Erlang ports instead
- Automatically installed when `py_event_loop` NIF is available
- **Process-per-context architecture** - Each Python context runs in dedicated process
- `py_context_process` - Gen_server managing a single Python context
- `py_context_sup` - Supervisor for context processes
- `py_context_router` - Routes calls to appropriate context process
- Improved isolation between contexts
- Better crash recovery and resource management
- **Worker thread pool** - High-throughput Python operations
- Configurable pool size for parallel execution
- Efficient work distribution across threads
- **`py:contexts_started/0`** - Helper to check if contexts are ready
### Changed
- **`py:call_async` renamed to `py:cast`** - Follows gen_server convention where
`call` is synchronous and `cast` is asynchronous. The semantics are identical,
only the name changed.
- **Unified `erlang` Python module** - Consolidated callback and event loop APIs
- `erlang.run(coro)` - Run coroutine with ErlangEventLoop (like uvloop.run)
- `erlang.new_event_loop()` - Create new ErlangEventLoop instance
- `erlang.install()` - Install ErlangEventLoopPolicy (deprecated in 3.12+)
- `erlang.EventLoopPolicy` - Alias for ErlangEventLoopPolicy
- Removed separate `erlang_asyncio` module - all functionality now in `erlang`
- **Async worker backend replaced with event loop model** - The pthread+usleep
polling async workers have been replaced with an event-driven model using
`py_event_loop` and `enif_select`:
- Removed `py_async_worker.erl` and `py_async_worker_sup.erl`
- Removed `py_async_worker_t` and `async_pending_t` structs from C code
- Deprecated `async_worker_new`, `async_call`, `async_gather`, `async_stream` NIFs
- Added `py_event_loop_pool.erl` for managing event loop-based async execution
- Added `py_event_loop:run_async/2` for submitting coroutines to event loops
- Added `nif_event_loop_run_async` NIF for direct coroutine submission
- Added `_run_and_send` wrapper in Python for result delivery via `erlang.send()`
- **Internal change**: `py:async_call/3,4` and `py:await/1,2` API unchanged
- **`SuspensionRequired` base class** - Now inherits from `BaseException` instead
of `Exception`. This prevents ASGI/WSGI middleware `except Exception` handlers
from intercepting the suspension control flow used by `erlang.call()`.
- **Per-interpreter isolation in py_event_loop.c** - Removed global state for
proper subinterpreter support. Each interpreter now has isolated event loop state.
- **ErlangEventLoopPolicy always returns ErlangEventLoop** - Previously only
returned ErlangEventLoop for main thread; now consistent across all threads.
### Deprecated
- **`py_asgi` module** - Deprecated in favor of the Channel API (`py_channel`)
or Reactor API (`erlang.reactor`). The module still works but will be removed
in a future release.
- **`py_wsgi` module** - Deprecated in favor of the Channel API (`py_channel`)
or Reactor API (`erlang.reactor`). The module still works but will be removed
in a future release.
### Removed
- **Context affinity functions** - Removed `py:bind`, `py:unbind`, `py:is_bound`,
`py:with_context`, and `py:ctx_*` functions. The new `py_context_router` provides
automatic scheduler-affinity routing. For explicit context control, use
`py_context_router:bind_context/1` and `py_context:call/5`.
- **Signal handling support** - Removed `add_signal_handler`/`remove_signal_handler`
from ErlangEventLoop. Signal handling should be done at the Erlang VM level.
Methods now raise `NotImplementedError` with guidance.
- **Subprocess support** - ErlangEventLoop raises `NotImplementedError` for
`subprocess_shell` and `subprocess_exec`. Use Erlang ports (`open_port/2`)
for subprocess management instead.
### Fixed
- **`py_reactor_context` now extends erlang module in subinterpreters** - Previously,
`py_reactor_context` with `mode=subinterp` would fail to import `erlang.reactor`
because the erlang module extension was not applied. Now calls
`py_context:extend_erlang_module_in_context/1` after context creation.
- **FD stealing and UDP connected socket issues** - Fixed file descriptor handling
for UDP sockets in connected mode
- **Context test expectations** - Updated tests for Python contextvars behavior
- **Unawaited coroutine warnings** - Fixed warnings in test suite
- **Timer scheduling for standalone ErlangEventLoop** - Fixed timer callbacks not
firing for loops created outside the main event loop infrastructure
- **Subinterpreter cleanup and thread worker re-registration** - Fixed cleanup
issues when subinterpreters are destroyed and recreated
- **ProcessError exception class identity in subinterpreters** - Fixed exception
class mismatch when raising `erlang.ProcessError` in subinterpreter contexts.
The exception class is now looked up from the current interpreter's `erlang`
module at runtime instead of using a global variable.
- **Thread worker handlers not re-registering after app restart** - Workers now
properly re-register when application restarts
- **Timeout handling** - Improved timeout handling across the codebase
- **Eval locals_term initialization** - Fixed uninitialized variable in eval
- **Two race conditions in worker pool** - Fixed concurrent access issues
- **`activate_venv/1` now processes `.pth` files** - Uses `site.addsitedir()` instead of
`sys.path.insert()` so that editable installs (uv, pip -e, poetry) work correctly.
New paths are moved to the front of `sys.path` for proper priority.
- **`deactivate_venv/0` now restores `sys.path`** - The previous implementation used
`py:eval` with semicolon-separated statements which silently failed (eval only accepts
expressions). Switched to `py:exec` for correct statement execution.
### Performance
- **Async coroutine latency reduced from ~10-20ms to <1ms** - The event loop model
eliminates pthread polling overhead
- **Zero CPU usage when idle** - Event-driven instead of usleep-based polling
- **No extra threads** - Coroutines run on the existing event loop infrastructure
## 1.8.1 (2026-02-25)
### Fixed
- **ASGI scope caching bug** - HTTP method was not treated as a dynamic field in the
scope template cache. This caused incorrect method values when the same path was
accessed with different HTTP methods (e.g., GET /path followed by POST /path would
return method="GET" for both requests).
## 1.8.0 (2026-02-25)
### Added
- **ASGI NIF Optimizations** - Six optimizations for high-performance ASGI request handling
- **Direct Response Tuple Extraction** - Extract `(status, headers, body)` directly without generic conversion
- **Pre-Interned Header Names** - 16 common HTTP headers cached as PyBytes objects
- **Cached Status Code Integers** - 14 common HTTP status codes cached as PyLong objects
- **Zero-Copy Request Body** - Large bodies (≥1KB) use buffer protocol for zero-copy access
- **Scope Template Caching** - Thread-local cache of 64 scope templates keyed by path hash
- **Lazy Header Conversion** - Headers converted on-demand for requests with ≥4 headers
- **erlang_asyncio Module** - Asyncio-compatible primitives using Erlang's native scheduler
- `erlang_asyncio.sleep(delay, result=None)` - Sleep using Erlang's `erlang:send_after/3`
- `erlang_asyncio.run(coro)` - Run coroutine with ErlangEventLoop
- `erlang_asyncio.gather(*coros)` - Run coroutines concurrently
- `erlang_asyncio.wait_for(coro, timeout)` - Wait with timeout
- `erlang_asyncio.wait(fs, timeout, return_when)` - Wait for multiple futures
- `erlang_asyncio.create_task(coro)` - Create background task
- `erlang_asyncio.ensure_future(coro)` - Wrap coroutine in Future
- `erlang_asyncio.shield(arg)` - Protect from cancellation
- `erlang_asyncio.timeout` - Context manager for timeouts
- Event loop functions: `get_event_loop()`, `new_event_loop()`, `set_event_loop()`, `get_running_loop()`
- Re-exports: `TimeoutError`, `CancelledError`, `ALL_COMPLETED`, `FIRST_COMPLETED`, `FIRST_EXCEPTION`
- **Erlang Sleep NIF** - Synchronous sleep primitive for Python
- `py_event_loop._erlang_sleep(delay_ms)` - Sleep using Erlang timer
- Releases GIL during sleep, no Python event loop overhead
- Uses pthread condition variables for efficient blocking
- `py_nif:dispatch_sleep_complete/2` - NIF to signal sleep completion
- **Scalable I/O Model** - Worker-per-context architecture
- `py_event_worker` - Dedicated worker process per Python context
- Combined FD event dispatch and reselect via `handle_fd_event_and_reselect` NIF
- Sleep tracking with `sleeps` map in worker state
- **New Test Suite** - `test/py_erlang_sleep_SUITE.erl` with 8 tests
- `test_erlang_sleep_available` - Verify NIF is exposed
- `test_erlang_sleep_basic` - Basic functionality
- `test_erlang_sleep_zero` - Zero delay returns immediately
- `test_erlang_sleep_accuracy` - Timing accuracy
- `test_erlang_asyncio_module` - Module functions present
- `test_erlang_asyncio_gather` - Concurrent execution
- `test_erlang_asyncio_wait_for` - Timeout support
- `test_erlang_asyncio_create_task` - Background tasks
### Performance
- **ASGI marshalling optimizations** - 40-60% improvement for typical ASGI workloads
- Direct response extraction: 5-10% improvement
- Pre-interned headers: 3-5% improvement
- Cached status codes: 1-2% improvement
- Zero-copy body buffers: 10-15% for large bodies (≥1KB)
- Scope template caching: 15-20% for repeated paths
- Lazy header conversion: 5-10% for apps accessing few headers
- **Eliminates event loop overhead** for sleep operations (~0.5-1ms saved per call)
- **Sub-millisecond timer precision** via BEAM scheduler (vs 10ms asyncio polling)
- **Zero CPU when idle** - event-driven, no polling
## 1.7.1 (2026-02-23)
### Fixed
- **Hex package missing priv directory** - Added explicit `files` configuration to include
`priv/erlang_loop.py` and other necessary files in the hex.pm package
## 1.7.0 (2026-02-23)
### Added
- **Shared Router Architecture for Event Loops**
- Single `py_event_router` process handles all event loops
- Timer and FD messages include loop identity for correct dispatch
- Eliminates need for per-loop router processes
- Handle-based Python C API using PyCapsule for loop references
- **Per-Loop Capsule Architecture** - Each `ErlangEventLoop` instance has its own isolated capsule
- Dedicated pending queue per loop for proper event routing
- Full asyncio support (timers, FD operations) with correct loop isolation
- Safe for multi-threaded Python applications where each thread needs its own loop
- See `docs/asyncio.md` for usage and architecture details
## 1.6.1 (2026-02-22)
### Fixed
- **ASGI headers now correctly use bytes instead of str** - Fixed ASGI spec compliance
issue where headers were being converted to Python `str` objects instead of `bytes`.
The ASGI specification requires headers to be `list[tuple[bytes, bytes]]`. This was
causing authentication failures and form parsing issues with frameworks like Starlette
and FastAPI, which search for headers using bytes keys (e.g., `b"content-type"`).
- Added explicit header handling in `asgi_scope_from_map()` to bypass generic conversion
- Headers are now correctly converted using `PyBytes_FromStringAndSize()`
- Supports both list `[name, value]` and tuple `{name, value}` header formats from Erlang
- Fixes GitHub issue #1
## 1.6.0 (2026-02-22)
### Added
- **Python Logging Integration** - Forward Python's `logging` module to Erlang's `logger`
- `py:configure_logging/0,1` - Setup Python logging to forward to Erlang
- `erlang.ErlangHandler` - Python logging handler that sends to Erlang
- `erlang.setup_logging(level, format)` - Configure logging from Python
- Fire-and-forget architecture using `enif_send()` for non-blocking messaging
- Level filtering at NIF level for performance (skip message creation for filtered logs)
- Log metadata includes module, line number, and function name
- Thread-safe - works from any Python thread
- **Distributed Tracing** - Collect trace spans from Python code
- `py:enable_tracing/0`, `py:disable_tracing/0` - Enable/disable span collection
- `py:get_traces/0` - Retrieve collected spans
- `py:clear_traces/0` - Clear collected spans
- `erlang.Span(name, **attrs)` - Context manager for creating spans
- `erlang.trace(name)` - Decorator for tracing functions
- Span events via `span.event(name, **attrs)`
- Automatic parent/child span linking via thread-local storage
- Error status capture with exception details
- Duration tracking in microseconds
- **New Erlang modules**
- `py_logger` - gen_server receiving log messages from Python workers
- `py_tracer` - gen_server collecting and managing trace spans
- **New C source**
- `c_src/py_logging.c` - NIF implementations for logging and tracing
- **Documentation and examples**
- `docs/logging.md` - Logging and tracing documentation
- `examples/logging_example.erl` - Working escript example
- Updated `docs/getting-started.md` with logging/tracing section
- **New test suite**
- `test/py_logging_SUITE.erl` - 9 tests for logging and tracing
- `ATOM_NIL` for Elixir `nil` compatibility in type conversions
### Performance
- **Type conversion optimizations** - Faster Python ↔ Erlang marshalling
- Use `enif_is_identical` for atom comparison instead of `strcmp`
- Use `PyLong_AsLongLongAndOverflow` to avoid exception machinery
- Cache `numpy.ndarray` type at init for fast isinstance checks
- Stack allocate small tuples/maps (≤16 elements) to avoid heap allocation
- Use `enif_make_map_from_arrays` for O(n) map building vs O(n²) puts
- Reorder type checks for web workloads (strings/dicts first)
- UTF-8 decode with bytes fallback for invalid sequences
- **Fire-and-forget NIF architecture** - Log and trace calls never block Python execution
- Uses `enif_send()` to dispatch messages asynchronously to Erlang processes
- Python code continues immediately after sending, no round-trip wait
- **NIF-level log filtering** - Messages below threshold are discarded before term creation
- Volatile bool flags for O(1) receiver availability checks
- Level threshold stored in C global, no Erlang callback needed
- **Minimal term allocation** - Direct Erlang term building without intermediate structures
- Timestamps captured at NIF level using `enif_monotonic_time()`
### Fixed
- **Python 3.12+ event loop thread isolation** - Fixed asyncio timeouts on Python 3.12+
- `ErlangEventLoop` now only used for main thread; worker threads get `SelectorEventLoop`
- Async worker threads bypass the policy to create `SelectorEventLoop` directly
- Per-call `ErlNifEnv` for thread-safe timer scheduling in free-threaded mode
- Fail-fast error handling in `erlang_loop.py` instead of silent hangs
- Added `gil_acquire()`/`gil_release()` helpers to avoid GIL double-acquisition
## 1.5.0 (2026-02-18)
### Added
- **`py_asgi` module** - Optimized ASGI request handling with:
- Pre-interned Python string keys (15+ ASGI scope keys)
- Cached constant values (http type, HTTP versions, methods, schemes)
- Thread-local response pooling (16 slots per thread, 4KB initial buffer)
- Direct NIF path bypassing generic py:call()
- ~60-80% throughput improvement over py:call()
- Configurable runner module via `runner` option
- Sub-interpreter and free-threading (Python 3.13+) support
- **`py_wsgi` module** - Optimized WSGI request handling with:
- Pre-interned WSGI environ keys
- Direct NIF path for marshalling
- ~60-80% throughput improvement over py:call()
- Sub-interpreter and free-threading support
- **Web frameworks documentation** - New documentation at `docs/web-frameworks.md`
## 1.4.0 (2026-02-18)
### Added
- **Erlang-native asyncio event loop** - Custom asyncio event loop backed by Erlang's scheduler
- `ErlangEventLoop` class in `priv/erlang_loop.py`
- Sub-millisecond latency via Erlang's `enif_select` (vs 10ms polling)
- Zero CPU usage when idle - no busy-waiting or polling overhead
- Full GIL release during waits for better concurrency
- Native Erlang scheduler integration for I/O events
- Event loop policy via `get_event_loop_policy()`
- **TCP support for asyncio event loop**
- `create_connection()` - TCP client connections
- `create_server()` - TCP server with accept loop
- `_ErlangSocketTransport` - Non-blocking socket transport with write buffering
- `_ErlangServer` - TCP server with `serve_forever()` support
- **UDP/datagram support for asyncio event loop**
- `create_datagram_endpoint()` - Create UDP endpoints with full parameter support
- `_ErlangDatagramTransport` - Datagram transport implementation
- Parameters: `local_addr`, `remote_addr`, `reuse_address`, `reuse_port`, `allow_broadcast`
- `DatagramProtocol` callbacks: `datagram_received()`, `error_received()`
- Support for both connected and unconnected UDP
- New NIF helpers: `create_test_udp_socket`, `sendto_test_udp`, `recvfrom_test_udp`, `set_udp_broadcast`
- New test suite: `test/py_udp_e2e_SUITE.erl`
- **Asyncio event loop documentation**
- New documentation: `docs/asyncio.md`
- Updated `docs/getting-started.md` with link to asyncio documentation
### Performance
- **Event loop optimizations**
- Fixed `run_until_complete` callback removal bug (was using two different lambda references)
- Cached `ast.literal_eval` lookup at module initialization (avoids import per callback)
- O(1) timer cancellation via handle-to-callback_id reverse map (was O(n) iteration)
- Detach pending queue under mutex, build Erlang terms outside lock (reduced contention)
- O(1) duplicate event detection using hash set (was O(n) linear scan)
- Added `PERF_BUILD` cmake option for aggressive optimizations (-O3, LTO, -march=native)
## 1.3.2 (2026-02-17)
### Fixed
- **torch/PyTorch introspection compatibility** - Fixed `AttributeError: 'erlang.Function'
object has no attribute 'endswith'` when importing torch or sentence_transformers in
contexts where erlang_python callbacks are registered.
- Root cause: torch does dynamic introspection during import, iterating through Python's
namespace and calling `.endswith()` on objects. The `erlang` module's `__getattr__` was
returning `ErlangFunction` wrappers for *any* attribute access.
- Solution: Added C-side callback name registry. Now `__getattr__` only returns
`ErlangFunction` wrappers for actually registered callbacks. Unregistered attributes
raise `AttributeError` (normal Python behavior).
- New test: `test_callback_name_registry` in `py_reentrant_SUITE.erl`
## 1.3.1 (2026-02-16)
### Fixed
- **Hex.pm packaging** - Added `files` section to app.src to include build scripts
(`do_cmake.sh`, `do_build.sh`) and other necessary files in the hex.pm package
## 1.3.0 (2026-02-16)
### Added
- **Asyncio Support** - New `erlang.async_call()` for asyncio-compatible callbacks
- `await erlang.async_call('func', arg1, arg2)` - Call Erlang from async Python code
- Integrates with asyncio event loop via `add_reader()`
- No exceptions raised for control flow (unlike `erlang.call()`)
- Releases dirty NIF thread while waiting (non-blocking)
- Works with FastAPI, Starlette, aiohttp, and other ASGI frameworks
- Supports concurrent calls via `asyncio.gather()`
- New test: `test_async_call` in `py_reentrant_SUITE.erl`
- New test module: `test/py_test_async.py`
- Updated documentation: `docs/threading.md` - Added Asyncio Support section
### Fixed
- **Flag-based callback detection in replay path** - Fixed SuspensionRequired exceptions
leaking when ASGI middleware catches and re-raises exceptions. The replay path in
`nif_resume_callback_dirty` now uses flag-based detection (checking `tl_pending_callback`)
instead of exception-type detection.
### Changed
- **C code optimizations and refactoring**
- **Thread safety fixes**: Used `pthread_once` for async callback initialization,
fixed mutex held during Python calls in async event loop thread
- **Timeout handling**: Added `read_with_timeout()` and `read_length_prefixed_data()`
helpers with proper timeouts on all blocking pipe reads (30s for callbacks, 10s for spawns)
- **Code deduplication**: Merged `create_suspended_state()` and
`create_suspended_state_from_existing()` into unified `create_suspended_state_ex()`,
extracted `build_pending_callback_exc_args()` and `build_suspended_result()` helpers
- **Performance**: Optimized list conversion using `enif_make_list_cell()` to build
lists directly without temporary array allocation
- Removed unused `make_suspended_term()` function
## 1.2.0 (2026-02-15)
### Added
- **Context Affinity** - Bind Erlang processes to dedicated Python workers for state persistence
- `py:bind()` / `py:unbind()` - Bind current process to a worker, preserving Python state
- `py:bind(new)` - Create explicit context handles for multiple contexts per process
- `py:with_context(Fun)` - Scoped helper with automatic bind/unbind
- Context-aware functions: `py:ctx_call/4-6`, `py:ctx_eval/2-4`, `py:ctx_exec/2`
- Automatic cleanup via process monitors when bound processes die
- O(1) ETS-based binding lookup for minimal overhead
- New test suite: `test/py_context_SUITE.erl`
- **Python Thread Support** - Any spawned Python thread can now call `erlang.call()` without blocking
- Supports `threading.Thread`, `concurrent.futures.ThreadPoolExecutor`, and any other Python threads
- Each spawned thread lazily acquires a dedicated "thread worker" channel
- One lightweight Erlang process per Python thread handles callbacks
- Automatic cleanup when Python thread exits via `pthread_key_t` destructor
- New module: `py_thread_handler.erl` - Coordinator and per-thread handlers
- New C file: `py_thread_worker.c` - Thread worker pool management
- New test suite: `test/py_thread_callback_SUITE.erl`
- New documentation: `docs/threading.md` - Threading support guide
- **Reentrant Callbacks** - Python→Erlang→Python callback chains without deadlocks
- Exception-based suspension mechanism interrupts Python execution cleanly
- Callbacks execute in separate processes to prevent worker pool exhaustion
- Supports arbitrarily deep nesting (tested up to 10+ levels)
- Transparent to users - `erlang.call()` works the same, just without deadlocks
- New test suite: `test/py_reentrant_SUITE.erl`
- New examples: `examples/reentrant_demo.erl` and `examples/reentrant_demo.py`
### Changed
- Callback handlers now spawn separate processes for execution, allowing workers
to remain available for nested `py:eval`/`py:call` operations
- **Modular C code structure** - Split monolithic `py_nif.c` (4,335 lines) into
logical modules for better maintainability:
- `py_nif.h` - Shared header with types, macros, and declarations
- `py_convert.c` - Bidirectional type conversion (Python ↔ Erlang)
- `py_exec.c` - Python execution engine and GIL management
- `py_callback.c` - Erlang callback support and asyncio integration
- Uses `#include` approach for single compilation unit (no build changes needed)
### Fixed
- **Multiple sequential erlang.call()** - Fixed infinite loop when Python code makes
multiple sequential `erlang.call()` invocations in the same function. The replay
mechanism now falls back to blocking pipe behavior for subsequent calls after the
first suspension, preventing the infinite replay loop.
- **Memory safety in C NIF** - Fixed memory leaks and added NULL checks
- `nif_async_worker_new`: msg_env now freed on pipe/thread creation failure
- `multi_executor_stop`: shutdown requests now properly freed after join
- `create_suspended_state`: binary allocations cleaned up on failure paths
- Added NULL checks on all `enif_alloc_resource` and `enif_alloc_env` calls
- **Dialyzer warnings** - Added `{suspended, ...}` return type to NIF specs for
`worker_call`, `worker_eval`, and `resume_callback` functions
- **Dead code removal** - Cleaned up unused code discovered during code review:
- Removed `execute_direct()` function in `py_exec.c` (duplicated inline logic)
- Removed unused `ref` field from `async_pending_t` struct in `py_nif.h`
- Removed `worker_recv/2` from `py_nif.erl` (declared but never implemented in C)
### Documentation
- **Doxygen-style C documentation** - Added documentation to all C source files:
- Architecture overview with execution mode diagrams
- Type mapping tables for conversions
- GIL management patterns and best practices
- Suspension/resume flow diagrams for callbacks
- Function-level `@param`, `@return`, `@pre`, `@warning`, `@see` annotations
## 1.1.0 (2026-02-15)
### Added
- **Shared State API** - ETS-backed storage for sharing data between Python workers
- `state_set/get/delete/keys/clear` accessible from Python via `from erlang import ...`
- `py:state_store/fetch/remove/keys/clear` from Erlang
- Atomic counters with `state_incr/decr` (Python) and `py:state_incr/decr` (Erlang)
- New example: `examples/shared_state_example.erl`
- **Native Python Import Syntax** for Erlang callbacks
- `from erlang import my_func; my_func(args)` - most Pythonic
- `erlang.my_func(args)` - attribute-style access
- `erlang.call('my_func', args)` - legacy syntax still works
- **Module Reload** - Reload Python modules across all workers during development
- `py:reload(module)` uses `importlib.reload()` to refresh modules from disk
- `py_pool:broadcast` for sending requests to all workers
- **Documentation improvements**
- Added shared state section to getting-started, scalability, and ai-integration guides
- Added embedding caching example using shared state
- Added hex.pm badges to README
### Fixed
- **Memory safety** - Added NULL checks to all `enif_alloc()` calls in NIF code
- **Worker resilience** - Fixed crash in `py_subinterp_pool:terminate` when workers undefined
- **Streaming example** - Fixed to work with worker pool design (workers don't share namespace)
- **ETS table ownership** - Moved `py_callbacks` table creation to supervisor for resilience
### Changed
- Created `py_util` module to consolidate duplicate code (`to_binary/1`, `send_response/3`, `normalize_timeout/1-2`)
- Consolidated `async_await/2` to call `await/2` reducing duplication
## 1.0.0 (2026-02-14)
Initial release of erlang_python - Execute Python from Erlang/Elixir using dirty NIFs.
### Features
- **Python Integration**
- Call Python functions with `py:call/3-5`
- Evaluate expressions with `py:eval/1-3`
- Execute statements with `py:exec/1-2`
- Stream from Python generators with `py:stream/3-4`
- **Multiple Execution Modes** (auto-detected)
- Free-threaded Python 3.13+ (no GIL, true parallelism)
- Sub-interpreters Python 3.12+ (per-interpreter GIL)
- Multi-executor for older Python versions
- **Worker Pools**
- Main worker pool for synchronous calls
- Async worker pool for asyncio coroutines
- Sub-interpreter pool for parallel execution
- **Erlang/Elixir Callbacks**
- Register functions callable from Python via `py:register_function/2-3`
- Python code calls back with `erlang.call('name', args...)`
- **Virtual Environment Support**
- Activate venvs with `py:activate_venv/1`
- Use isolated package dependencies
- **Rate Limiting**
- ETS-based semaphore prevents overload
- Configurable max concurrent operations
- **Type Conversion**
- Automatic conversion between Erlang and Python types
- Integers, floats, strings, lists, tuples, maps/dicts, booleans
- **Memory Management**
- Access Python GC stats with `py:memory_stats/0`
- Force garbage collection with `py:gc/0-1`
- Memory tracing with `py:tracemalloc_start/stop`
### Examples
- `semantic_search.erl` - Text embeddings and similarity search
- `rag_example.erl` - Retrieval-Augmented Generation with Ollama
- `ai_chat.erl` - Interactive LLM chat
- `erlang_concurrency.erl` - 10x speedup with BEAM processes
- `elixir_example.exs` - Full Elixir integration demo
### Documentation
- Getting Started guide
- AI Integration guide
- Type Conversion reference
- Scalability and performance tuning
- Streaming with generators