erlang_python

# State machines

The states each long-lived thing in erlang_python moves through, what moves
it, and what is allowed in each state. Read this before changing a loop,
a shutdown path or a restart policy; the invariants at the end are the ones
a change must keep. Messages and frames are in [protocols](protocols.md).

## The Python runtime (C)

`g_runtime_state` in `c_src/py_nif.h`, moved with compare-and-swap so only
one thread wins each transition.

```
UNINIT --init--> INITING --ok--> RUNNING --finalize--> SHUTTING_DOWN --> STOPPED
                    |                                                      ^
                    +--------------------- failure ------------------------+
```

- `runtime_is_running()` gates every NIF that touches Python; a NIF that
  finds `SHUTTING_DOWN` or `STOPPED` returns `not_running` or an error.
- `STOPPED` may re-enter `INITING`: the runtime can be finalized and
  initialized again in one VM (the suites do this).

## An embedded context (`py_context` process + context thread)

Two cooperating machines: the Erlang process and the pthread in C.

Context thread (`ctx_thread_main_worker`, `ctx_thread_main_owngil`
in `c_src/py_nif.c`):

```
starting --namespaces created--> waiting --dequeue--> executing --reply--> waiting
    |                                |                                       
    +-- init_error -----> exited     +-- shutdown_requested ----> exited
```

- `waiting` blocks on `queue_not_empty` under `queue_mutex`.
- `executing` is bracketed by `py_context_exec_enter` / `exec_leave`
  (interrupt bookkeeping) around the GIL; the request mirror on
  `py_context_t` is valid only here.
- `exited` sets `thread_running = false`; `nif_context_destroy` joins with a
  timeout and, if the join fails, marks the context `leaked` and pins the
  resource instead of freeing it.

Erlang process (`loop/1` in `py_context_embedded`):

```
idle --{call|eval|exec|submit}--> in_request --{py_result}--> idle
 |                                   |
 |                                   +--{suspended, ...}--> in_callback --resume--> in_request
 |
 +--{start_loop}--> loop_running --{py_result, LoopReq}--> idle
                        |
                        +--{stop_loop, GraceMs}--> stopping --grace--> interrupt --deadline--> idle
```

- `in_request` is a blocking receive for the reply; nested callbacks
  arrive as `{erlang_callback, ...}` (pipe) or `{suspended, ...}` and are
  served inline, so the process never deadlocks with its own thread.
- In `loop_running` the loop request `LoopReq` occupies the thread: `call`,
  `eval`, `exec` and `call_method` answer `{error, loop_running}`; `stop`
  first stops the loop. The owner is monitored and its `DOWN` stops the loop.
- `stopping` arms `loop_stop_deadline` (cooperative) then
  `loop_interrupt_deadline`; the loop exit is the `py_result` for `LoopReq`,
  and the owner gets `{py_loop_exit, Ctx, Result}`.

## A request (`ctx_request_t`)

```
created (refcount 1) --enqueue--> queued (2) --dequeue--> running --done--> completed
                                      |                                        |
                                      +-- cancelled (destroy, timeout) -------+
                                                                               v
                                                              freed when refcount hits 0
```

The queue and the caller each hold one reference; whoever releases last
frees. `cancelled` is checked by the thread before running, so a request
cancelled while queued is answered `{error, cancelled}` without touching
Python.

## An isolated context (`py_isolated`, `gen_statem`)

States are the `state()` type in `src/py_isolated.erl`; `sys:get_state/1`
shows the current one and `sys:trace/2` prints transitions.

```
            start_child + handshake
                     |
                     v
   +--------------> idle <----------------------------------+
   |                 |                                      |
   |  main request   |  start_loop                          |
   |                 v                                      |
   |           {busy, Id} ---reply Id---> idle              |
   |                 |                                      |
   |                 |          looping --stop_loop-----> stopping_loop
   |                 |             |         (grace: interrupt, then kill)
   |                 |             +--loop_exit event--------+
   |                 |
   +-- child exit / kill / socket error --> {restarting, Reason} --new child--> idle
                                                     |
                                                     +-- budget exhausted --> stop
```

Per state:

| State | Main requests (`call`, `eval`, `exec`, `start_loop`) | Other requests | Timers armed |
|---|---|---|---|
| `idle` | dispatched, go to `{busy, Id}` (`start_loop` goes to `looping`) | dispatched | none |
| `{busy, Id}` | postponed, unless from a process running a callback for this context (nested, dispatched) | dispatched | `{timeout, kill}` bound to `Id` once an interrupt was sent |
| `looping` | `{error, loop_running}` | dispatched (`submit`, `pass_fd`, ...) | none |
| `stopping_loop` | postponed | dispatched | `state_timeout` for the interrupt, then `{timeout, kill}` bound to `loop` |
| `{restarting, R}` | postponed | postponed | `state_timeout` waiting for the port's `exit_status` |

Transitions and their triggers:

- `{busy, Id}` to `idle`: a status-1/2 frame with `Id`. Frames for other
  ids (nested requests) do not change state.
- Anything to `{restarting, Reason}`: `{Port, {exit_status, S}}`, a socket
  `abort`, a `{memory_limit, Rss}` event, `kill/1`, or a request the state
  machine cannot deliver. In-flight requests, submitted tasks and a running
  loop fail with `Reason` (`fail_pending/2`); postponed requests are kept
  and served by the next child.
- `{restarting, _}` to `idle`: the port reported the exit and a new child
  passed the handshake. `restart_allowed/1` counts restarts in
  `restart_period`; over `max_restarts` (or with `restart => false`) the
  process stops with `{child_exited, Reason}`.
- `looping` to `stopping_loop`: `stop_loop/2` or the owner's `DOWN`.
  `stopping_loop` to `idle`: the `{loop_exit, R}` event. The interrupt
  `state_timeout` and the kill backstop escalate if the loop does not exit.

Interrupt timing: `interrupt/1` in `{busy, Id}` sends `{interrupt, Id}` and
arms `{{timeout, kill}, KillAfter, Id}`. The reply for `Id` cancels the
timer; if it fires, the child gets SIGKILL and the machine goes to
`{restarting, killed}`. An interrupt in any other state answers
`not_running`.

## The child (`_isolated.py`)

The main thread runs one request at a time but can nest:

```
idle --request--> executing [stack: Id1]
                     |
                     +-- erlang.call --> waiting for reply, serving nested requests [Id1, Id2] ...
                     |
                     +-- SIGUSR1 while running --> _Interrupted raised in the request on top
```

- `_exec_stack` holds the ids being executed, innermost last. An
  `{interrupt, Target}` control is honoured only if `Target` is the top of
  the stack; otherwise it is stale and dropped. The signal handler raises
  only while `running` is true, so an interrupt between requests cannot
  leak into the next one.
- The reader thread never runs Python code: it parses frames, resolves
  waiters, and pushes requests and interrupts to the main thread's inbox.
- `broken` (EOF or a hard error on the socket) is terminal: the reader
  calls `os._exit`, since nothing useful can happen in the process any more
  and a main thread stuck in a C call must not keep it alive. Erlang sees
  the port's `exit_status` and runs the restart policy above.
- With a loop: `start_loop` runs `run_forever` on the main thread; requests
  that need the main thread are refused with `loop_running`, `submit` goes
  through `call_soon_threadsafe`, and `stop_loop` calls `loop.stop()` from
  the reader thread.

## A shared region (`py_shm`)

```
new --> open --close/1 or owner DOWN--> closed (file unlinked, handle closed)
```

Mappings in Python outlive `closed` until the wrapper is closed or
collected; a later access raises `ValueError`, never a fault. A shared
buffer adds `closed = true` in its header at `py_buffer:close/1`, and
readers waiting in `_py_buffer_wait` are answered with the closed flag.

## Invariants

- A context executes one top-level request at a time, in every mode.
  Embedded: one thread, one dequeue. Isolated: `{busy, Id}` plus `postpone`.
- Nested requests only come from a process serving a callback of the same
  context. Anything else waits.
- A restart never loses a queued request, only the ones in flight, and
  callers of those get an error naming the cause.
- Interrupts target the request executing now; a stale interrupt is
  dropped on both sides (kill timer bound to the id in Erlang, stack check
  in the child, `interrupt_pending` cleared in `exec_leave` for embedded
  contexts).
- Shutdown never frees memory a thread may still use: embedded contexts
  leak on a failed join, the child is reaped through the port.