Current section

Files

Jump to
snakepit docs PRODUCTION_DEPLOYMENT_CHECKLIST.md
Raw

docs/PRODUCTION_DEPLOYMENT_CHECKLIST.md

# Production Deployment Checklist (Snakepit v0.7.4)
Use this checklist before shipping Snakepit-backed services to production. It
covers the 0.7.4 runtime upgrades (zero-copy, crash barrier, hermetic Python,
exception translation) plus operational readiness.
---
## Pre-Deployment
### 1) Runtime Selection (Hermetic Python)
- [ ] Decide Python strategy:
- `:uv` managed runtime (recommended for reproducibility)
- `:system` + `SNAKEPIT_PYTHON` when using an external interpreter
- [ ] If using uv:
```elixir
config :snakepit, :python,
strategy: :uv,
managed: true,
python_version: "3.12.3",
runtime_dir: "priv/snakepit/python"
```
- [ ] Run `mix snakepit.setup` and `mix snakepit.doctor` on the build host.
- [ ] Confirm runtime identity is present in environment metadata:
`SNAKEPIT_PYTHON_RUNTIME_HASH`, `SNAKEPIT_PYTHON_VERSION`, `SNAKEPIT_PYTHON_PLATFORM`.
### 2) Crash Barrier & Idempotency
- [ ] Configure crash barrier to match your reliability policy:
```elixir
config :snakepit, :crash_barrier,
enabled: true,
retry: :idempotent,
max_retries: 1,
taint_ms: 5_000
```
- [ ] Ensure calls that are safe to retry include `idempotent: true` in payloads
(or use SnakeBridge wrappers that include it automatically).
- [ ] Confirm exit-code classifications if you rely on hardware-specific exits
(OOM, SIGKILL, etc.).
### 3) Zero-Copy Readiness (Optional)
- [ ] Enable zero-copy only when your adapter/runtime supports it:
```elixir
config :snakepit, :zero_copy, enabled: true
```
- [ ] Validate DLPack/Arrow support in Python (`torch`, `pyarrow`, or equivalent).
- [ ] Confirm fallback telemetry is acceptable when zero-copy is unavailable.
### 4) Telemetry & Logging
- [ ] Attach telemetry handlers for pool lifecycle, crash barrier, zero-copy,
and exception translation events.
- [ ] Optional: enable OpenTelemetry exporter for distributed tracing.
- [ ] Set Snakepit log level for production:
```elixir
config :snakepit, log_level: :warning
```
- [ ] Ensure log redaction is enabled for payloads (default behaviour).
### 5) Pool Sizing & Resource Limits
- [ ] Review pool size and startup batch settings.
- [ ] Configure Python thread limits to prevent fork storms:
```elixir
config :snakepit, :python_thread_limits, %{openblas: 1, omp: 1, mkl: 1}
```
- [ ] Configure worker TTL or max requests if you need periodic recycling.
### 6) Validation & Smoke Tests
- [ ] `mix test`
- [ ] `mix dialyzer`
- [ ] `mix credo --strict`
- [ ] `mix format --check-formatted`
- [ ] `./test_python.sh`
- [ ] Run an example smoke test: `mix run --no-start examples/grpc_basic.exs`
---
## Post-Deployment
### Health Checks
- [ ] Monitor `:snakepit, :pool, :status` for queue depth and availability.
- [ ] Monitor crash barrier events (`:worker, :crash`, `:worker, :tainted`).
- [ ] Validate exception translation rates (`:python, :exception, :mapped`).
### Alerts
1. Crash barrier taint rate above threshold.
2. Queue depth > capacity for sustained window.
3. Repeated worker restarts per minute.
4. Zero-copy fallbacks spiking unexpectedly.
---
## Rollback Plan
If instability appears:
1. Disable crash barrier retries (set `retry: :never` or `max_retries: 0`).
2. Disable zero-copy (`zero_copy: [enabled: false]`).
3. Switch to system Python if uv runtime integrity is suspect.
4. Roll back to the previous Snakepit release once traffic is drained.