Current section
Files
Jump to
Current section
Files
README.md
# CDPEx
[](https://hex.pm/packages/cdp_ex)
[](https://hexdocs.pm/cdp_ex)
[](https://github.com/patrols/cdp_ex/actions/workflows/ci.yml)
[](https://github.com/patrols/cdp_ex/blob/main/LICENSE)
OTP-native [Chrome DevTools Protocol](https://chromedevtools.github.io/devtools-protocol/)
browser automation for Elixir. Launch headless Chrome and drive it directly over a
`Mint.WebSocket` connection. **No ChromeDriver, no Node.js.**
```elixir
CDPEx.with_page([], fn page ->
{:ok, _} = CDPEx.Page.navigate(page, "https://example.com")
CDPEx.Page.html(page)
end)
#=> {:ok, "<html>…</html>"}
```
## Why CDPEx?
It drives Chrome over CDP the way Puppeteer and Playwright do, but in pure
Elixir: the browser and each page's CDP connection are **supervised OTP
processes** (a page is a lightweight handle over its connection). A Chrome crash
or a dropped socket surfaces to the caller as `{:error, reason}` instead of a
hung session, and `terminate/2` guarantees the OS process is reaped (no zombie
Chromes).
It's production-tested: it runs as the sole browser engine for a JavaScript-heavy
scraper, where it replaced a Wallaby/ChromeDriver setup.
| | CDPEx | chrome_remote_interface | ChromicPDF | Wallaby |
|---|---|---|---|---|
| Transport | CDP (WebSocket) | CDP (WebSocket) | CDP (WebSocket) | WebDriver / ChromeDriver |
| Runtime deps | `mint_web_socket`, `jason`, `telemetry` | `hackney` + others | a few | ChromeDriver process |
| Supervised lifecycle | ✅ | — | ✅ (PDF pool) | partial |
| Scope | general automation | low-level client | PDF / screenshots | testing |
If you want a small, dependency-light CDP client with proper OTP supervision, and
you'd rather not run a ChromeDriver process (like Wallaby) or a Node sidecar (like
Playwright or Puppeteer), that's the gap CDPEx fills.
> #### Status {: .info}
>
> **Transports:** pages default to one WebSocket each (strong crash isolation).
> Opt into `sessionId` multiplexing (many pages over the one browser socket)
> with `new_page(browser, transport: :session)`; the trade-off is shared fate (a
> dropped browser connection drops all of its session pages).
>
> **Connect to a running Chrome:** attach to a Chrome you didn't launch — local,
> a sidecar, or a remote/cloud `wss://` endpoint — with `connect/2` (`:session`
> transport; it never reaps a process it didn't start). See below.
>
> Stealth / anti-fingerprinting presets remain out of scope for now (evidence-gated).
## Installation
Add `cdp_ex` to your deps in `mix.exs`:
```elixir
def deps do
[
{:cdp_ex, "~> 0.12.0"}
]
end
```
You also need Chrome or Chromium installed. CDPEx finds it via, in order: the
`:chrome_binary` option, `CDP_EX_CHROME_BINARY`, `CHROME_BINARY`, then an OS default.
For reproducible setups, point it at a
[Chrome for Testing](https://googlechromelabs.github.io/chrome-for-testing/) binary.
> #### Sandbox {: .warning}
>
> CDPEx launches Chrome with `--no-sandbox` by default, since the sandbox can't
> start in many CI/container environments. If you run as root or drive untrusted
> pages, re-enable it by overriding `:args` — see `CDPEx.Chrome`.
## Running in containers
CDPEx is validated on macOS, but its defaults already target Linux containers:
`--no-sandbox`, `--disable-dev-shm-usage`, and `--disable-setuid-sandbox` are on
by default, so headless Chrome starts out of the box on a constrained host (e.g.
2 vCPU / 2 GB). A few things smooth the path further:
- **Tune `:launch_timeout` for cold starts.** Chrome's first launch in a fresh
container is slower than on a warm dev machine. `:launch_timeout` is a *ceiling*,
not a fixed wait (readiness is polled and returns as soon as Chrome is
reachable), so a generous value costs nothing on a fast launch:
```elixir
CDPEx.launch(launch_timeout: 30_000)
```
- **Fresh-profile cost.** Each launch creates a new `--user-data-dir` (removed on
stop), so there's no warm disk cache between launches and the *first* navigation
pays a cold-start cost. For throughput, prefer one long-lived browser (`launch/1`
+ reuse) over a throwaway `with_page([...])` per request, or pass a persistent
`:user_data_dir`.
- **`/dev/shm` sizing.** Docker defaults `/dev/shm` to 64 MB, which Chrome can
exhaust (crashing tabs). CDPEx ships `--disable-dev-shm-usage` by default (Chrome
writes to `/tmp` instead), so it works on a small `/dev/shm` as-is. To size it up
instead (`--shm-size=1g`), drop that flag via a custom `:args`.
- **Memory.** On a 2 GB host a single browser with a few pages is comfortable; each
open page and large screenshots/PDFs add transient memory. Close pages promptly,
or use `transport: :session` to cut per-page socket/process overhead when you
don't need crash isolation.
- **`--remote-allow-origins`.** Some Chrome builds enforce an `Origin` check on the
DevTools WebSocket upgrade and may reject a CDP client with a 403. CDPEx doesn't
set this by default; if you hit `{:error, {:ws_upgrade, _}}` at connect, add it:
```elixir
CDPEx.launch(launch_timeout: 30_000, extra_args: ["--remote-allow-origins=*"])
```
## Usage
### Resource-safe (recommended)
`with_page/3` opens a page, runs your function, and always tears everything down —
even if the function raises:
```elixir
# Throwaway browser + page for one job:
{:ok, title} =
CDPEx.with_page([], fn page ->
{:ok, _} = CDPEx.Page.navigate(page, "https://example.com")
CDPEx.Page.evaluate(page, "document.title")
end)
```
Per-operation timeouts stack, so a slow page can hold a browser far longer than any
one of them. `deadline:` bounds the whole call, launch included: when it runs out the
browser is torn down underneath `fun` (only the page, for a browser pid you pass in)
and `with_page` returns `{:error, :deadline_exceeded}`. `fun` keeps running in your
process; see `CDPEx.with_page/3` for what can and can't be interrupted.
```elixir
CDPEx.with_page([], &CDPEx.Page.html/1, deadline: 20_000)
```
### Explicit lifecycle
```elixir
{:ok, browser} = CDPEx.launch(headless: true)
{:ok, page} = CDPEx.new_page(browser)
{:ok, _page} = CDPEx.Page.navigate(page, "https://example.com")
:ok = CDPEx.Page.wait_for_selector(page, "h1")
{:ok, html} = CDPEx.Page.html(page)
{:ok, "Example Domain"} = CDPEx.Page.evaluate(page, "document.querySelector('h1').textContent")
{:ok, _png} = CDPEx.Page.screenshot(page, path: "example.png")
:ok = CDPEx.close_page(browser, page)
:ok = CDPEx.stop(browser)
```
### Connecting to an existing Chrome
Drive a Chrome that CDPEx didn't launch — one you started yourself, a sidecar
container, or a cloud browser provider — with `connect/2`:
```elixir
# An http(s):// base URL is discovered via GET /json/version:
{:ok, browser} = CDPEx.connect("http://localhost:9222")
# …or pass the browser WebSocket URL directly (ws:// or, for a TLS endpoint, wss://):
{:ok, browser} = CDPEx.connect("wss://chrome.example.com/cdp?token=…")
{:ok, page} = CDPEx.new_page(browser, transport: :session)
{:ok, _} = CDPEx.Page.navigate(page, "https://example.com")
# stop/1 closes the pages YOU opened and disconnects — it never kills that Chrome.
:ok = CDPEx.stop(browser)
```
A connected browser uses `:session` transport (many pages over the one socket);
`:dedicated` over a connection isn't supported yet. `wss://` is verified against
the OS trust store — pass `:cacertfile` for a private CA, or `insecure: true` to
skip verification. Raise `:discovery_timeout` (default `5_000` ms) for a slow
remote `/json/version`. `with_page([connect: "http://localhost:9222"], fun)` is the
one-shot form.
> **`http(s)://` discovery is IP/localhost only.** Chrome's DevTools HTTP endpoint
> returns **403** for a non-IP/non-`localhost` `Host` (DNS-rebinding protection), so
> for a **named** remote/sidecar/cloud Chrome pass the `ws(s)://` browser URL
> directly rather than the `http(s)://` discovery form.
### Under your supervision tree
Because `terminate/2` reaps Chrome, supervise the browser with a `:shutdown`
timeout (not `:brutal_kill`):
```elixir
children = [
{CDPEx.Browser, name: MyBrowser, headless: true}
]
Supervisor.start_link(children, strategy: :one_for_one)
```
### Pooling
Reuse warm browsers instead of paying a cold launch per job with `CDPEx.Pool`:
```elixir
children = [
{CDPEx.Pool, name: MyPool, size: 4, launch_opts: [headless: true]}
]
Supervisor.start_link(children, strategy: :one_for_one)
# Borrow a warm browser for one fetch (a pooled drop-in for with_page/3):
CDPEx.Pool.with_page(MyPool, fn page ->
{:ok, _} = CDPEx.Page.navigate(page, "https://example.com")
CDPEx.Page.html(page)
end)
```
Browsers launch lazily up to `:size` and are reused; `checkout/2` blocks (up to
`:checkout_timeout`) when all are busy. Launches are asynchronous, so the pool
stays responsive during a cold start and warms multiple browsers concurrently
under load. A caller that crashes returns its browser automatically, and a
crashed browser is relaunched on demand.
## Page operations
| Function | Description |
|---|---|
| `navigate/3` | Go to a URL, waiting for `networkAlmostIdle` (configurable) |
| `wait_for_selector/3` | Poll until a CSS selector matches |
| `wait_for_response/3` | Block until a network response URL matches (fn / `Regex` / substring) |
| `wait_for_network_idle/2` | Block until the network settles (Puppeteer "networkidle") |
| `evaluate/3` | Run JS and return the value (`returnByValue`) |
| `click/3` | Real trusted click on the first match (synthetic via `trusted: false`); `wait:` retries until it exists and has a box |
| `click_text/3` | Click an element by its text content (whitespace-normalized, innermost match) |
| `type/4` | Focus an element and enter text (`Input.insertText`) |
| `press/4` | Press a named key — `Enter`, `Tab`, arrows, … — with real key events |
| `scroll/3` | Scroll the window to `:top`, `:bottom`, or `{x, y}` |
| `html/2` | Full serialized DOM (`document.documentElement.outerHTML`) |
| `screenshot/2` | PNG bytes, or write to `:path` |
| `pdf/2` | Render the page to PDF bytes, or write to `:path` |
| `set_user_agent/3` | Override the UA string, with optional Client-Hints metadata + `Accept-Language` |
| `observe_network/2` | Stream `Network` request/response events to the caller |
| `response_body/3` | Fetch a response body by requestId (`Network.getResponseBody`) |
| `enable_request_interception/2` | Pause matching requests for the caller to resolve |
| `continue_request/3` / `fulfill_request/3` / `fail_request/3` | Resolve a paused request (proceed / synthetic response / fail) |
| `authenticate/4` | Answer a proxy / HTTP Basic auth challenge (call before `navigate/3`) |
Full API: [hexdocs.pm/cdp_ex](https://hexdocs.pm/cdp_ex).
## Recipes
End-to-end patterns for the event-driven features. Each `CDPEx.Page` function is
documented in full on [hexdocs](https://hexdocs.pm/cdp_ex/CDPEx.Page.html).
### Block or rewrite requests (interception)
Interception is event-driven: the process that enables it receives one
`Fetch.requestPaused` event per request and must resolve every one — with
`continue_request/3`, `fulfill_request/3`, or `fail_request/3`. Drive it from a single
long-lived process (an unresolved request stalls the page).
```elixir
defmodule ImageBlocker do
# Drops image requests; lets everything else through.
def run(page) do
:ok = CDPEx.Page.enable_request_interception(page)
loop(page)
end
defp loop(page) do
receive do
{:cdp_event, _conn, "Fetch.requestPaused", %{"requestId" => id, "request" => request}, _sid} ->
if String.ends_with?(request["url"], [".png", ".jpg", ".webp"]) do
CDPEx.Page.fail_request(page, id, reason: :blocked_by_client)
else
CDPEx.Page.continue_request(page, id)
end
loop(page)
after
1_000 -> :ok
end
end
end
```
### Grab an API response triggered by a click (SPA)
Arm the waiter *before* the action that fires the request. The matcher is a substring,
a `Regex`, or a `(url -> boolean)` function.
```elixir
CDPEx.with_page([], fn page ->
{:ok, _} = CDPEx.Page.navigate(page, "https://example.com/app")
waiter = Task.async(fn -> CDPEx.Page.wait_for_response(page, ~r{/api/items}) end)
:ok = CDPEx.Page.click(page, "#load-more")
{:ok, %{"requestId" => id, "response" => %{"status" => 200}}} = Task.await(waiter)
CDPEx.Page.response_body(page, id)
end)
```
When you just need the page to settle after hydration (no specific response to match),
use `wait_for_network_idle/2` instead.
### Detect a 403 wall / 404 / login redirect
`navigate/3` returns `{:ok, page}` even for a 403 or a redirect-to-login. Pass
`response: true` to also get the main document's HTTP status and final (post-redirect)
URL:
```elixir
case CDPEx.Page.navigate(page, url, response: true) do
{:ok, _page, %{status: 200, url: final_url}} -> {:landed, final_url}
{:ok, _page, %{status: status}} -> {:blocked, status}
{:error, reason} -> {:error, reason}
end
```
### Authenticated proxy
Pass `:proxy` — CDPEx sets `--proxy-server` and answers the proxy's auth challenge on
each page automatically, so you just navigate:
```elixir
{:ok, browser} = CDPEx.launch(proxy: "http://user:pass@proxy.example.com:8080")
# keyword form avoids percent-encoding a special-char password:
# CDPEx.launch(proxy: [server: "proxy.example.com:8080", username: "u", password: "p@ss"])
CDPEx.with_page(browser, fn page ->
{:ok, _} = CDPEx.Page.navigate(page, "https://example.com")
CDPEx.Page.html(page)
end)
```
For a one-off HTTP Basic challenge on an *origin* (not a proxy), arm it per page with
`authenticate/4` before navigating:
`CDPEx.Page.authenticate(page, "user", "pass", source: :server)`.
## Error handling
Operations return `{:error, reason}` on failure, runtime failures included: Chrome
that can't be spawned, a browser, page or pool that is gone or dies mid-call, a call
that can't be served in time. None of them raises or exits in your process, so
`with_page/3` needs no `rescue`/`catch :exit` around it. Invalid arguments still
raise, and so does anything your own `with_page` function raises, throws or exits
with.
That covers return values, not links. A browser started with `CDPEx.launch/1` or
`CDPEx.connect/2` is linked to the process that started it (the link is what reaps
Chrome when that process dies), so if the browser stops on its own, for example
because Chrome died, the exit signal still reaches a starter that doesn't trap exits.
Run a long-lived browser under your own supervisor. `with_page/3` with launch options
contains this for its throwaway browser.
Rather than hard-code the reason shapes, classify them to drive retries:
```elixir
case CDPEx.Page.navigate(page, url) do
{:ok, page} ->
{:ok, page}
{:error, reason} ->
if CDPEx.transient?(reason), do: retry(), else: {:error, reason}
end
```
`CDPEx.classify_error/1` buckets a reason as `:transient` (connection dropped or
couldn't be established, the page closed, timeout, Chrome died or was slow to start, an internal helper
crashed, or a connection-layer `net::ERR_*` navigation error, so a fresh attempt may
succeed), `:terminal` (selector miss, JS exception, usage/validation error, which a
retry won't fix), or `:unknown` (payload-dependent, e.g. an ambiguous `net::ERR_*`
navigation error or a CDP error code; you decide). The library tracks the error surface, so the
transient/terminal decision lives in one place instead of drifting across callers.
The reason shapes are documented as
[`t:CDPEx.error_reason/0`](https://hexdocs.pm/cdp_ex/CDPEx.html#t:error_reason/0).
Retries are yours to bound: cap attempts, back off, and on a `:transient` result
re-establish the resource (open a fresh page/browser) rather than reusing a dead
handle — a dead page keeps returning `:noproc`, or `{:session_closed, session_id}` for a
`:session` page (the default on a `connect/2` browser).
## Telemetry
CDPEx emits [`:telemetry`](https://hexdocs.pm/telemetry) events and attaches no
handlers; attach your own to record them (emitting with nothing attached is a no-op).
Events: `[:cdp_ex, :launch, …]`, `[:cdp_ex, :connect, …]`, and
`[:cdp_ex, :navigate, …]` spans, `[:cdp_ex, :page, :start | :stop]`, and
`[:cdp_ex, :error]`. See
[`CDPEx.Telemetry`](https://hexdocs.pm/cdp_ex/CDPEx.Telemetry.html) for the full
taxonomy (measurements + metadata).
```elixir
:telemetry.attach(
"cdp-nav",
[:cdp_ex, :navigate, :stop],
fn _event, %{duration: d}, %{url: url, status: status}, _config ->
ms = System.convert_time_unit(d, :native, :millisecond)
IO.puts("#{url} -> #{inspect(status)} in #{ms}ms")
end,
nil
)
```
`status` (and the post-redirect `final_url`) are `nil` unless the navigation used
`response: true` — see `CDPEx.Page.navigate/3`.
## Process labels
Every GenServer and task CDPEx starts sets a [process
label](https://hexdocs.pm/elixir/Process.html#set_label/1), so crash reports,
`:observer`, and Phoenix LiveDashboard name it instead of showing a bare pid.
Nothing to configure; labels are set on Elixir 1.17+ and read by OTP 27+. (The
`Task.Supervisor` backing `CDPEx.Pool`'s launches is the one exception — OTP
offers no way to label a supervisor from outside it.)
| Label | Process |
|-------|---------|
| `{:cdp_browser, os_pid}` | `CDPEx.Browser` — `os_pid` is the Chrome process it launched (`:starting` until Chrome is up, `:connected` in connect-mode, `:launched` if the OS pid could not be read) |
| `{:cdp_connection, "/devtools/browser/…"}` | the browser WebSocket |
| `{:cdp_connection, "/devtools/page/<targetId>"}` | a `:dedicated` page WebSocket |
| `{:cdp_page_helper, :idle_wait \| :capture, target_id}` | the short-lived process behind `wait_for_network_idle/2` and `navigate/3` with `response: true` |
| `{:cdp_page_helper, :detach_watch, target_id}` | the short-lived process that watches a `:session` page's detach during `navigate/3` and `wait_for_navigation/2` |
| `{:cdp_authenticate, target_id}` | the short-lived relay behind `Page.authenticate/4` |
| `{:cdp_fetch_auth, source}` | the `Fetch` auth handler (`:proxy`, `:server`, or `:any`) |
| `{:cdp_fetch_disable, conn}` | the throwaway task that releases interception after its owner dies |
| `{:cdp_pool, size}` / `{:cdp_pool_launch, pool}` | `CDPEx.Pool` and its browser-launch tasks |
| `{:cdp_deadline, ms}` | the watchdog behind `with_page/3`'s `:deadline` |
The page-helper label is the one to reach for when a scrape appears hung: it
names the target that is still waiting, and the browser label's `os_pid` ties a
suspect BEAM process to a row in `ps`.
Connection labels carry the DevTools path without its query string: the target
can hold an auth token for a remote endpoint, and labels are read far more
widely than internal state.
## Development
```bash
mix deps.get
mix test # unit tests (no Chrome needed)
mix test --include integration # real-Chrome tests (set CDP_EX_CHROME_BINARY)
mix ci # the full gate: format, deps audit, docs, credo, dialyzer, unit tests
```
Integration tests are tagged `:integration` and excluded by default; they launch a
real Chrome and drive it against a local fixture HTTP server.
## Acknowledgements
Built on [`mint_web_socket`](https://hex.pm/packages/mint_web_socket). Inspired by
the production CDP work in [ChromicPDF](https://github.com/bitcrowd/chromic_pdf) and
by Puppeteer's protocol layer.
## License
MIT. See [LICENSE](LICENSE).