snakepit

# Snakepit Test Architecture with Supertester Integration

## Overview

This document outlines the comprehensive test architecture for Snakepit, integrating Supertester's OTP-compliant testing patterns to eliminate race conditions, enable true async testing, and ensure deterministic test execution.

## Core Principles

1. **Zero Process.sleep** - All synchronization must be deterministic
2. **Full Isolation** - Every test runs in complete process/ETS isolation
3. **Async by Default** - Tests must run concurrently without conflicts
4. **Test the Public API** - Focus on user-facing behavior, not implementation
5. **Supervision Testing** - Explicitly test fault tolerance and recovery

## Directory Structure

```
test/
├── unit/                           # Isolated component tests
│   ├── bridge/
│   │   ├── session_store_test.exs
│   │   ├── serialization_test.exs
│   │   └── variables_test.exs
│   ├── grpc/
│   │   ├── grpc_worker_test.exs
│   │   ├── bridge_server_test.exs
│   │   └── client_test.exs
│   └── pool/
│       ├── pool_test.exs
│       ├── registry_test.exs
│       └── worker_supervisor_test.exs
├── integration/                    # Cross-component tests
│   ├── grpc_bridge_test.exs
│   ├── pool_lifecycle_test.exs
│   ├── session_workflow_test.exs
│   └── streaming_test.exs
├── performance/                    # Load and benchmark tests
│   ├── pool_saturation_test.exs
│   ├── grpc_throughput_test.exs
│   └── concurrent_sessions_test.exs
├── chaos/                         # Resilience testing
│   ├── worker_crash_test.exs
│   ├── network_failure_test.exs
│   └── supervisor_recovery_test.exs
└── support/
    ├── test_helpers.ex
    ├── grpc_test_adapter.ex
    └── mock_python_server.ex
```

## Supertester Integration

### 1. Base Test Module

```elixir
defmodule Snakepit.TestCase do
  @moduledoc """
  Base test case for all Snakepit tests.
  Provides Supertester integration and common helpers.
  """
  
  defmacro __using__(opts \\ []) do
    quote do
      use ExUnit.Case, async: unquote(Keyword.get(opts, :async, true))
      use Supertester.UnifiedTestFoundation, isolation: :full_isolation
      
      import Supertester.OTPHelpers
      import Supertester.GenServerHelpers
      import Supertester.Assertions
      import Snakepit.TestHelpers
      
      # Ensure clean application state
      setup_all do
        Application.ensure_all_started(:snakepit)
        on_exit(fn -> cleanup_all_sessions() end)
      end
    end
  end
end
```

### 2. gRPC Worker Testing Pattern

```elixir
defmodule Snakepit.GRPCWorkerTest do
  use Snakepit.TestCase
  import Supertester.GenServerHelpers
  
  describe "gRPC worker lifecycle" do
    setup context do
      # Isolated worker with unique port
      port = 50100 + :rand.uniform(1000)
      worker_id = "test_worker_#{context.test}"
      
      {:ok, worker} = setup_isolated_genserver(
        Snakepit.GRPCWorker,
        worker_id,
        adapter: MockGRPCAdapter,
        port: port,
        id: worker_id
      )
      
      # Wait for server ready without sleep
      assert_receive {:grpc_ready, ^port}, 5_000
      
      %{worker: worker, port: port, worker_id: worker_id}
    end
    
    test "worker starts gRPC server and connects", %{worker: worker, port: port} do
      assert_genserver_responsive(worker)
      
      # Verify server is listening
      {:ok, channel} = GRPC.Stub.connect("localhost:#{port}")
      assert channel != nil
    end
    
    test "worker handles commands", %{worker: worker} do
      # Use synchronous call for deterministic behavior
      {:ok, result} = call_with_timeout(worker, {:execute, "ping", %{}, 5_000})
      assert result["status"] == "pong"
    end
    
    test "worker recovers from server crash", %{worker: worker, port: port} do
      # Monitor the worker
      monitor_ref = Process.monitor(worker)
      
      # Simulate server crash
      send(worker, {:EXIT, :simulated_crash})
      
      # Worker should restart
      assert_process_restarted(monitor_ref, worker, 5_000)
      assert_receive {:grpc_ready, ^port}, 5_000
    end
  end
end
```

### 3. Pool Testing Pattern

```elixir
defmodule Snakepit.PoolTest do
  use Snakepit.TestCase
  import Supertester.SupervisorHelpers
  
  describe "pool management" do
    setup do
      pool_name = :"test_pool_#{System.unique_integer()}"
      
      {:ok, pool} = setup_isolated_supervisor(
        Snakepit.Pool,
        pool_name,
        pool_size: 2,
        worker_module: MockWorker
      )
      
      %{pool: pool, pool_name: pool_name}
    end
    
    test "pool starts configured number of workers", %{pool: pool} do
      assert_child_count(pool, 2)
      assert_all_children_alive(pool)
    end
    
    test "pool handles concurrent requests", %{pool_name: pool_name} do
      # Start 10 concurrent tasks
      tasks = for i <- 1..10 do
        Task.async(fn ->
          Snakepit.Pool.execute(pool_name, "echo", %{id: i})
        end)
      end
      
      # All should complete without error
      results = Task.await_many(tasks, 5_000)
      assert length(results) == 10
      assert Enum.all?(results, &match?({:ok, _}, &1))
    end
  end
end
```

### 4. Integration Testing Pattern

```elixir
defmodule Snakepit.GRPCBridgeIntegrationTest do
  use Snakepit.TestCase, async: false
  import Supertester.MessageHelpers
  
  describe "full gRPC bridge integration" do
    setup do
      # Start isolated application instance
      app_name = :"snakepit_test_#{System.unique_integer()}"
      
      # Override configuration for this test
      Application.put_env(app_name, :pooling_enabled, true)
      Application.put_env(app_name, :pool_config, %{pool_size: 2})
      Application.put_env(app_name, :grpc_port, 51000 + :rand.uniform(1000))
      
      {:ok, _} = Application.ensure_all_started(app_name)
      
      on_exit(fn -> Application.stop(app_name) end)
      
      %{app: app_name}
    end
    
    test "end-to-end session workflow", %{app: app} do
      session_id = "integration_#{System.unique_integer()}"
      
      # Initialize session
      {:ok, _} = Snakepit.execute_in_session(session_id, "initialize_session", %{})
      
      # Register variable with proper synchronization
      {:ok, var_id} = Snakepit.execute_in_session(
        session_id,
        "register_variable",
        %{name: "counter", type: "integer", initial_value: 0}
      )
      
      # Concurrent updates
      tasks = for i <- 1..10 do
        Task.async(fn ->
          Snakepit.execute_in_session(
            session_id,
            "set_variable",
            %{name: "counter", value: i}
          )
        end)
      end
      
      Task.await_many(tasks, 5_000)
      
      # Verify final state
      {:ok, var} = Snakepit.execute_in_session(
        session_id,
        "get_variable",
        %{name: "counter"}
      )
      
      assert var["value"] in 1..10
    end
  end
end
```

## Key Testing Patterns

### 1. Synchronization Without Sleep

```elixir
# BAD - Race condition prone
GenServer.cast(worker, :start_server)
Process.sleep(1000)  # Arbitrary wait
assert server_started?()

# GOOD - Deterministic synchronization
ref = monitor_genserver_call(worker, :start_server)
assert_receive {:server_started, port}, 5_000
assert_receive {:DOWN, ^ref, :process, _, :normal}
```

### 2. Port Allocation for gRPC

```elixir
defmodule Snakepit.TestHelpers do
  @base_test_port 52000
  @port_range 1000
  
  def allocate_test_port do
    # Use test number to ensure unique ports
    test_number = Process.get(:test_number, 0)
    Process.put(:test_number, test_number + 1)
    
    @base_test_port + rem(test_number, @port_range)
  end
end
```

### 3. Mock Adapters for Testing

```elixir
defmodule MockGRPCAdapter do
  @behaviour Snakepit.Adapter
  
  def executable_path, do: "echo"
  def script_path, do: "/dev/null"
  def script_args, do: []
  
  def init_grpc_connection(port) do
    # Simulate server startup
    send(self(), {:grpc_ready, port})
    {:ok, %{channel: :mock_channel, port: port}}
  end
  
  def grpc_execute(_conn, "ping", _args, _timeout) do
    {:ok, %{"status" => "pong"}}
  end
end
```

### 4. Chaos Testing Patterns

```elixir
defmodule Snakepit.ChaosTest do
  use Snakepit.TestCase
  import Supertester.ChaosHelpers
  
  test "pool recovers from worker failures" do
    {:ok, pool} = start_supervised_pool(size: 4)
    
    # Inject failures
    inject_random_failures(pool, 
      failure_rate: 0.5,
      failure_types: [:exit, :timeout, :disconnect]
    )
    
    # System should still process requests
    results = for _ <- 1..20 do
      Task.async(fn -> Snakepit.execute("ping", %{}) end)
    end
    |> Task.await_many(10_000)
    
    successful = Enum.count(results, &match?({:ok, _}, &1))
    assert successful >= 15  # At least 75% success rate
  end
end
```

## Supertester Gaps & Enhancements Needed

### What Supertester Provides
- Process isolation mechanisms
- Deterministic synchronization helpers  
- GenServer and Supervisor test utilities
- Message tracing and assertions
- Chaos engineering framework

### What's Missing for Snakepit
1. **gRPC-specific helpers** - Waiting for server ready, port management
2. **Python process helpers** - Starting/stopping external processes
3. **Streaming assertions** - Testing gRPC streams
4. **Session cleanup utilities** - Bulk session management

### Proposed Snakepit Test Extensions

```elixir
defmodule Snakepit.TestHelpers do
  @moduledoc "Snakepit-specific test helpers extending Supertester"
  
  def assert_grpc_server_ready(port, timeout \\ 5_000) do
    assert_receive {:grpc_ready, ^port}, timeout
  end
  
  def with_python_server(port, fun) do
    # Start Python gRPC server
    server = start_python_server(port)
    try do
      assert_grpc_server_ready(port)
      fun.()
    after
      stop_python_server(server)
    end
  end
  
  def assert_streaming_response(stream_ref, expected_chunks) do
    Enum.each(expected_chunks, fn expected ->
      assert_receive {:stream_chunk, ^stream_ref, chunk}, 1_000
      assert chunk == expected
    end)
    assert_receive {:stream_end, ^stream_ref}, 1_000
  end
end
```

## Migration Strategy

### Phase 1: Foundation (Week 1)
1. Add Supertester dependency
2. Create base test case module
3. Set up test directory structure
4. Create mock adapters

### Phase 2: Unit Tests (Week 2)
1. Migrate SessionStore tests
2. Migrate GRPCWorker tests
3. Migrate Pool tests
4. Remove all Process.sleep calls

### Phase 3: Integration Tests (Week 3)
1. Rewrite gRPC bridge tests
2. Add end-to-end workflow tests
3. Add streaming tests
4. Enable async: true everywhere

### Phase 4: Advanced Testing (Week 4)
1. Add performance benchmarks
2. Add chaos tests
3. Add supervisor recovery tests
4. Document testing patterns

## Success Metrics

1. **Zero Process.sleep** in test suite
2. **100% async: true** tests
3. **No test interdependence** - random test order passes
4. **< 30s total test time** with parallelization
5. **Zero flaky tests** over 100 runs

## Conclusion

By integrating Supertester and following these patterns, Snakepit will have:
- Deterministic, fast tests
- True isolation enabling parallel execution
- Proper OTP testing patterns
- Comprehensive coverage including chaos scenarios
- A foundation for continuous deployment confidence

The key is treating tests as first-class citizens that demonstrate proper usage patterns while ensuring reliability through deterministic execution.