Packages
snakepit
0.11.1
0.13.0
0.12.0
0.11.1
0.11.0
0.10.1
0.10.0
0.9.1
0.9.0
0.8.9
0.8.8
0.8.7
0.8.6
0.8.5
0.8.4
0.8.3
0.8.2
0.8.1
0.8.0
0.7.7
0.7.6
0.7.5
0.7.4
0.7.3
0.7.2
0.7.1
0.7.0
0.6.11
0.6.10
0.6.9
0.6.8
0.6.7
0.6.6
0.6.5
0.6.4
0.6.3
0.6.2
0.6.1
0.6.0
0.5.1
0.5.0
0.4.3
0.4.2
0.4.1
0.4.0
0.3.3
0.3.2
0.3.1
0.3.0
0.2.1
0.2.0
0.1.2
0.1.1
0.1.0
High-performance pooler and session manager for external language integrations. Supports Python, Node.js, Ruby, and more with gRPC streaming, session management, and production-ready process cleanup.
Current section
Files
Jump to
Current section
Files
lib/snakepit/serialization.ex
defmodule Snakepit.Serialization do
@moduledoc """
Utilities for working with Snakepit's serialization layer.
When Python returns data to Elixir, non-JSON-serializable objects (like
`datetime.datetime`, custom classes, or library-specific objects) are
handled gracefully by Snakepit's serialization layer:
1. Objects with conversion methods (`model_dump`, `to_dict`, `_asdict`,
`tolist`, `isoformat`) are automatically converted.
2. Objects that cannot be converted are replaced with an "unserializable marker"
containing type information.
This module provides utilities for detecting and inspecting these markers.
## Marker Format
Unserializable markers are maps with the following structure:
%{
"__ffi_unserializable__" => true,
"__type__" => "module.ClassName" # Full Python type path
}
By default, the marker only contains type information (safe for production).
The `__repr__` field is only included when explicitly enabled via environment
variables.
## Environment Variables
The following environment variables control marker detail level. These are
set on the Python worker processes:
- `SNAKEPIT_UNSERIALIZABLE_DETAIL` - Controls what information is included:
- `none` (default) - Only type, no repr (safe for production)
- `type` - Placeholder string with type name
- `repr_truncated` - Include truncated repr (may leak secrets)
- `repr_redacted_truncated` - Truncated repr with common secrets redacted
- `SNAKEPIT_UNSERIALIZABLE_REPR_MAXLEN` - Maximum repr length (default: 500, max: 2000)
## Configuring Worker Environment
Python workers inherit environment from the BEAM VM. Set these before
starting Snakepit:
# In your application startup or config/runtime.exs
System.put_env("SNAKEPIT_UNSERIALIZABLE_DETAIL", "repr_redacted_truncated")
System.put_env("SNAKEPIT_UNSERIALIZABLE_REPR_MAXLEN", "200")
Or via shell before starting the application:
SNAKEPIT_UNSERIALIZABLE_DETAIL=repr_redacted_truncated mix run ...
For per-pool configuration, use the `:env` option in pool config (if supported
by your adapter).
## Operational Guidance
- **Production**: Use default (`none`) or omit the env vars entirely
- **Development/Debugging**: Use `repr_redacted_truncated` with small maxlen
- **Never in production**: `repr_truncated` without redaction
## Example Usage
# Check if a value is an unserializable marker
if Snakepit.Serialization.unserializable?(value) do
{:ok, info} = Snakepit.Serialization.unserializable_info(value)
IO.puts("Cannot serialize: \#{info.type}")
end
## Security Note
When repr is enabled, the `__repr__` field may contain sensitive information
(API keys, auth tokens, etc.) that was present in the Python object's string
representation. The `repr_redacted_truncated` mode applies best-effort
redaction of common secret patterns but is NOT a security boundary.
"""
@marker_key "__ffi_unserializable__"
@doc """
Checks if a value is an unserializable marker.
Returns `true` if the value is a map with `"__ffi_unserializable__" => true`,
indicating it represents a Python object that could not be serialized to JSON.
## Examples
iex> Snakepit.Serialization.unserializable?(%{"__ffi_unserializable__" => true})
true
iex> Snakepit.Serialization.unserializable?(%{"key" => "value"})
false
iex> Snakepit.Serialization.unserializable?(nil)
false
"""
@spec unserializable?(term()) :: boolean()
def unserializable?(%{@marker_key => true}), do: true
def unserializable?(_), do: false
@doc """
Extracts information from an unserializable marker.
Returns `{:ok, info}` if the value is a valid unserializable marker,
where `info` is a map with `:type` and `:repr` keys. Returns `:error`
for non-marker values.
## Examples
iex> marker = %{
...> "__ffi_unserializable__" => true,
...> "__type__" => "datetime.datetime",
...> "__repr__" => "datetime.datetime(2024, 1, 11, 10, 30)"
...> }
iex> {:ok, info} = Snakepit.Serialization.unserializable_info(marker)
iex> info.type
"datetime.datetime"
iex> Snakepit.Serialization.unserializable_info(%{"key" => "value"})
:error
## Return Value
On success, returns `{:ok, %{type: String.t() | nil, repr: String.t() | nil}}`.
The `:type` and `:repr` fields may be `nil` if not present in the marker.
"""
@spec unserializable_info(term()) ::
{:ok, %{type: String.t() | nil, repr: String.t() | nil}} | :error
def unserializable_info(%{@marker_key => true} = marker) do
{:ok,
%{
type: Map.get(marker, "__type__"),
repr: Map.get(marker, "__repr__")
}}
end
def unserializable_info(_), do: :error
end