Packages
bb_camera_avf
0.1.0
Beam Bots camera driver for macOS, capturing JPEG via ffmpeg's AVFoundation input.
Current section
Files
Jump to
Current section
Files
bb_camera_avf
README.md
README.md
<!--
SPDX-FileCopyrightText: 2026 James Harton
SPDX-License-Identifier: Apache-2.0
-->
<img src="https://github.com/beam-bots/bb/blob/main/logos/beam_bots_logo.png?raw=true" alt="Beam Bots Logo" width="250" />
# bb_camera_avf
[](https://github.com/beam-bots/bb_camera_avf/actions/workflows/ci.yml)
[](https://opensource.org/licenses/Apache-2.0)
[](https://hex.pm/packages/bb_camera_avf)
[](https://hexdocs.pm/bb_camera_avf)
[](https://api.reuse.software/info/github.com/beam-bots/bb_camera_avf)
[](https://deepwiki.com/beam-bots/bb_camera_avf)
A [Beam Bots](https://github.com/beam-bots/bb) camera driver for macOS —
built-in FaceTime cameras, USB webcams, Continuity Camera — capturing JPEG
through ffmpeg's AVFoundation input in a supervised process.
It exists mostly so that a robot developed on a Mac can see through a real
camera before it is deployed to Linux, where
[`bb_camera_v4l`](https://github.com/beam-bots/bb_camera_v4l) takes over
behind the same interface.
It is a `BB.Sensor` and a `BB.Camera`, declared like any other sensor:
```elixir
topology do
link :head do
sensor :eye, {BB.Camera.AVF, device: "FaceTime HD Camera", width: 1280, height: 720}
end
end
```
Consumers go through [`bb_camera`](https://github.com/beam-bots/bb_camera),
which shares one capture between all of them. Two browser tabs previewing the
same camera produce one ffmpeg, not two.
**Frames never travel on the robot's pubsub.** That carries capture lifecycle
only — `BeginCapture` and `EndCapture`. Frame bytes go over `bb_camera`'s
bounded media interface.
## Installation
```elixir
def deps do
[
{:bb, "~> 0.33"},
{:bb_camera, "~> 0.1"},
{:bb_camera_avf, "~> 0.1"}
]
end
```
Working on this driver from a `bb_workspace` checkout instead? `BB_VERSION=local`
resolves `bb`, `bb_camera` and `bb_camera_ffmpeg` from the sibling checkouts
rather than from Hex.
## Requirements
- **macOS.** This driver speaks AVFoundation through ffmpeg.
- **ffmpeg** on `$PATH` — `brew install ffmpeg`.
- **The Xcode command line tools**, to build MuonTrap's wrapper program — see
[`bb_camera_ffmpeg`](https://github.com/beam-bots/bb_camera_ffmpeg).
- **Camera access for the app that starts the VM.** See below; this is the
thing most likely to go wrong.
## Camera access
macOS grants camera access to an *app*, not to a process. An `iex` started in
Terminal, iTerm or Ghostty captures with *that terminal's* permission, and the
first capture from a terminal that has never asked raises the system prompt.
A VM started by something with no user interface — a launchd job, a daemon,
an agent runner — may never be asked. It gets no prompt and no frames, and
nothing says why: ffmpeg opens the camera and then waits for a first frame
that never comes. In that wait it ignores both `q` and `SIGTERM`.
So capture gives up after `:start_timeout_ms` (10 seconds) without a frame,
stops ffmpeg — with `SIGKILL` after `:stop_timeout_ms`, since it ignores
`SIGTERM` — and fails with `{:no_frames, 10000}` as the reason. If the VM
stops first, ffmpeg goes with it: it runs under MuonTrap, which kills it when
the VM's end of the pipe closes. The warning
on the console points here. The timeout cannot tell missing permission from a
camera that sends nothing, so the error names what happened rather than a
cause.
To check or grant access, open **System Settings > Privacy & Security >
Camera** and look for the app you start the VM from.
## Smoke test on a Mac
This walks from nothing to a frame from a real camera. The first half needs no
camera at all, which is deliberate: if the synthetic source works and the real
one does not, the problem is the camera or its permission, and you have
narrowed it down without guessing.
### 1. Install ffmpeg
```sh
brew install ffmpeg
```
### 2. Prove the pipeline works without a camera
From the package directory:
```sh
BB_VERSION=local mix deps.get
BB_VERSION=local iex -S mix
```
`BB_VERSION=local` is needed on **both** commands. It is read each time mix
runs, not stored in the lockfile.
Paste this robot into the session. It captures one of ffmpeg's own synthetic
sources: real ffmpeg, real JPEG encoder, real pipe, real framing — and no
hardware.
```elixir
defmodule SynthCam do
use BB
settings do
name :synth_cam
end
topology do
link :head do
sensor :eye, {BB.Camera.AVF, lavfi: "testsrc2", width: 320, height: 240, frame_rate: 10}
end
end
end
{:ok, _pid} = SynthCam.start_link([])
{:ok, session} = BB.Camera.Media.open(SynthCam, :eye, format: :jpeg)
{:ok, frame} = BB.Camera.Media.next(session, 10_000)
{frame.width, frame.height, byte_size(frame.data)}
#=> {320, 240, 8620}
File.write!("/tmp/frame.jpg", frame.data)
BB.Camera.Media.close(session)
```
```sh
open /tmp/frame.jpg
```
If that failed, nothing below will work, and the problem is ffmpeg rather than
your camera.
### 3. Find the camera
```sh
BB_VERSION=local mix bb_camera_avf.scan
```
```
Cameras:
[0] HD Pro Webcam C920
[1] FaceTime HD Camera
Use the name, not the number - numbers move when cameras are plugged in:
sensor :eye, {BB.Camera.AVF, device: "HD Pro Webcam C920"}
```
It asks ffmpeg, so the names are spelled the way ffmpeg will match them, and
it opens no camera, so it needs no camera access.
Use the **name**. The number is a position in this list and moves when a
camera is plugged in. Numbers past the last camera are *screens*, so a stale
index can capture a display. Two identical webcams share one name, and this
driver cannot tell them apart.
To find the sizes and rates a camera supports, ask for one it does not:
```sh
ffmpeg -hide_banner -f avfoundation -framerate 1 -video_size 640x480 \
-i "HD Pro Webcam C920" -frames:v 1 -f null -
```
ffmpeg refuses, and prints every mode the camera lists:
```
Selected framerate (1.000000) is not supported by the device.
Supported modes:
1280x720@[30.000030 30.000030]fps
1920x1080@[30.000030 30.000030]fps
...
```
A rate written as `30.000030` is asked for as `30`: ffmpeg matches rates to
within 0.01.
### 4. Capture from the real camera
Run this from a terminal app, so that macOS has someone to ask. The first
time, it shows a permission prompt; answer it within the 10-second timeout,
or open the session again afterwards.
```elixir
defmodule MyCam do
use BB
settings do
name :my_cam
end
topology do
link :head do
sensor :eye, {BB.Camera.AVF, device: "FaceTime HD Camera", width: 1280, height: 720, frame_rate: 30}
end
end
end
{:ok, _pid} = MyCam.start_link([])
BB.subscribe(MyCam, [:sensor, :head, :eye])
{:ok, session} = BB.Camera.Media.open(MyCam, :eye, format: :jpeg)
{:ok, frame} = BB.Camera.Media.next(session, 15_000)
{frame.width, frame.height}
File.write!("/tmp/camera.jpg", frame.data)
```
If `next/2` returns `CaptureFailed` with `{:no_frames, 10000}`, see
[Camera access](#camera-access).
`frame.width` and `frame.height` are read out of the JPEG that was actually
encoded. If ffmpeg could not configure the camera as asked, it says
`Configuration of video device failed, falling back to default` and captures
at the camera's default size instead; this is where you find out.
### 5. Shut it down
```elixir
BB.Camera.Media.close(session)
flush()
```
```sh
pgrep -lx ffmpeg
```
Nothing. Capture stops when the last consumer leaves, and the camera's light
goes out.
## A preview in a browser
`bb_liveview` mounts a dashboard that previews any camera a robot declares,
this one included, as MJPEG. Declare the camera in your robot and point the
dashboard at it; there is nothing driver-specific to wire up.
## Options
| Option | Default | |
|---|---|---|
| `:device` | — | Camera name, `"default"`, or index. Required unless `:lavfi` is given |
| `:lavfi` | — | An ffmpeg synthetic source such as `"testsrc2"`, instead of a camera |
| `:width`, `:height` | `640`, `480` | Must be a size the camera lists |
| `:frame_rate` | `30` | A hertz unit, a number, or `{numerator, denominator}` |
| `:input_format` | `:nv12` | Pixel format to ask AVFoundation for |
| `:quality` | `5` | ffmpeg's `-q:v`, 1 (best) to 31 |
| `:ffmpeg` | `$PATH` | Path to the ffmpeg executable |
| `:start_timeout_ms` | `10000` | Fail capture when no frame has arrived by then |
| `:max_frame_bytes` | 4 MiB | Largest frame accepted before the stream is resynchronised |
| `:drop_queue_len` | `8` | Shed frames while more reads than this are waiting |
| `:stdio_window` | 256 KiB | Bytes of output ffmpeg may have in flight unacknowledged |
| `:stop_timeout_ms` | `2000` | How long ffmpeg gets after `SIGTERM` before `SIGKILL` |
## How it holds together
```
MuonTrap ──> ffmpeg ──pipe──> Capture ──frames──> BB.Camera.Hub ──> consumers
│
└─ lifecycle ──> BB.PubSub
```
- **`BB.Camera.AVF.Command`** builds the argument vector and validates it.
No shell ever parses a camera name.
- **[`bb_camera_ffmpeg`](https://github.com/beam-bots/bb_camera_ffmpeg)** does
the rest, shared with `bb_camera_v4l`: it runs ffmpeg under MuonTrap, so
ffmpeg cannot outlive the capture or the VM; it flow-controls ffmpeg's
output; it frames the JPEG stream structurally; and it holds the first frame
until the hub has been told capture started.
## Limitations
- **macOS only.**
- **Every frame is encoded by ffmpeg.** AVFoundation hands over decoded
buffers, even from a webcam that sends MJPEG, so there is no passthrough.
- **Cameras are chosen by name or index.** ffmpeg 9.0 cannot select one by
its unique ID, so two identical webcams cannot be told apart.
- **Camera access is the app's, not the VM's**, and a missing grant looks
exactly like a camera that sends nothing. See
[Camera access](#camera-access).
- **Timestamps are arrival times, not exposure times, and are not suitable for
synchronisation.** The MJPEG stream ffmpeg writes has no container and no
timestamps; a frame is stamped when the read completing it reached the
BEAM.
- **The frame rate is reported as requested, not negotiated.** What the frames
actually did is measured by `BB.Camera.Hub` and reported in
`BB.Message.Camera.CaptureState`.
- **No capability enumeration.** ffmpeg prints a camera's modes when asked for
one it does not have; see the smoke test.
- **A failed capture is not retried.** Reopening is the consumer's decision.
- **Changing an option that reaches ffmpeg restarts it**, with `EndCapture`
and `BeginCapture` around the change.
- **ffmpeg's diagnostics go to the VM's console**, not into the error.
- **JPEG only.** Interframe codecs are refused rather than carried; see
`bb_camera` for why.
## Testing
```sh
BB_VERSION=local mix deps.get
BB_VERSION=local mix test # no camera, no ffmpeg needed
BB_VERSION=local mix test --include ffmpeg # also runs ffmpeg for real
BB_VERSION=local mix check --no-retry
```
The default suite drives the driver through `bb_camera_ffmpeg`'s stub
backend, so every lifecycle case — a capture that fails, a stop racing a
start, a parameter change mid-stream — is a function call rather than
something to wait for. No test sleeps. Framing, flow control and process
cleanup are tested in `bb_camera_ffmpeg`.
The `:ffmpeg` tagged tests run this driver's own argument vector through real
ffmpeg against a synthetic source. They need ffmpeg, but no camera. The one
also tagged `:macos` needs AVFoundation and is skipped elsewhere.
## Licence
Apache-2.0. See `LICENSES/`.