wsdataselect

[![coverage report](https://gricad-gitlab.univ-grenoble-alpes.fr/OSUG/RESIF/wsdataselect/badges/main/coverage.svg)](https://gricad-gitlab.univ-grenoble-alpes.fr/OSUG/RESIF/wsdataselect/-/commits/main)

# Wsdataselect

Implementation of the [FDSN dataselect webservice](https://www.fdsn.org/webservices/) for the [EPOS-France seismological datacenter](https://seismology.resif.fr). Serves seismological miniSEED data by querying the [sigma](https://github.com/OSUG/RESIF/sigma) inventory database and shelling out to the [`dataselect`](https://github.com/EarthScope/dataselect) binary to extract and trim data files.

**Stack:** Elixir 1.19 / Erlang 28 / Phoenix 1.8 / PostgreSQL / Bandit

## Architecture

```
                         Client (HTTP)
                              │
┌─────────────────────────────▼─────────────────────────────────────────────┐
│                          Phoenix Endpoint                                 │
│                                                                           │
│  ┌────────┐    ┌──────────┐    ┌────────┐    ┌──────────┐               │
│  │ Health │───►│ Traffic  │───►│ Version│───►│  Router  │               │
│  │        │    │  Drain   │    │        │    │          │               │
│  └────────┘    └──────────┘    └────────┘    └────┬─────┘               │
│                                                    │                     │
│        (post-response)                             │                     │
│  ┌──────────────────┐   ┌──────────────────────────▼──────────────┐      │
│  │  DeliveryMetrics │   │  /query, /queryauth     /openapi        │      │
│  └────────┬─────────┘   │  pipeline: :fdsn       pipeline:openapi│      │
│           │             └──────────────────────────┬──────────────┘      │
│  ┌────────▼─────────┐                              │                     │
│  │    Cleanup       │   ┌──────────────────────────▼──────────────┐      │
│  └──────────────────┘   │          FdsnDataselectPlugs            │      │
│                         │    (parse & validate FDSN parameters)   │      │
│                         └──────────────────────────┬──────────────┘      │
│                                                    │                     │
│                         ┌──────────────────────────▼──────────────┐      │
│                         │           QueryController               │      │
│                         │  ┌────────┐ ┌───────────┐ ┌────────┐   │      │
│                         │  │Authent.│ │Authoriz.  │ │Filters │   │      │
│                         │  └───┬────┘ └─────┬─────┘ └───┬────┘   │      │
│                         │      │            │           │        │      │
│                         │  ┌───▼────────────▼───────────▼────┐   │      │
│                         │  │        evaluate_size            │   │      │
│                         │  └─────────────────────────────────┘   │      │
│                         └──────────────┬────────────────────────┘      │
│                                        │                               │
└────────────────────────────────────────┼───────────────────────────────┘
                                         │
    ┌──────────┐              ┌──────────▼──────────┐    ┌───────────────┐
    │  Auth DB │              │    Dataselect        │    │   Inventory   │
    │ (Digest) │              │      binary          │    │      DB       │
    └──────────┘              └──────────┬───────────┘    └───────┬───────┘
                                         │                       │
                                  ┌──────▼───────┐        ┌──────▼───────┐
                                  │  temp workdir│        │  Data files  │
                                  │   (trimmed)  │        │  (archive)   │
                                  └──────┬───────┘        └──────────────┘
                                         │
                                  ┌──────▼───────┐
                                  │   chunked    │
                                  │   response   │
                                  │  (miniSEED)  │
                                  └──────────────┘
```

## Quick start

### Prerequisites

- Elixir 1.19+ and Erlang 28+ (use [mise](https://mise.jdx.dev/) or [asdf](https://asdf-vm.com/))
- PostgreSQL 13+
- The [`dataselect`](https://github.com/EarthScope/dataselect) binary (for data extraction)

### Setup

```bash
git clone https://gricad-gitlab.univ-grenoble-alpes.fr/OSUG/RESIF/wsdataselect.git
cd wsdataselect
mix setup          # deps.get + ecto.create + migrate + seed
MIX_ENV=dev mix phx.server
```

The server starts at `http://localhost:4000`. The Swagger UI is served at `/`.

### Run tests

Tests require a PostgreSQL instance on port 5432 with trust auth.

```bash
# Option 1: auto-start a Postgres container via podman
mise run test

# Option 2: manually
podman run --rm -d -p 5432:5432 -e POSTGRES_HOST_AUTH_METHOD=trust docker.io/postgres:13.22-trixie
MIX_ENV=test mix test
```

A pre-built `dataselect` binary for tests is at `test/bin/dataselect`.

## Request flow

```
 Client                Endpoint              Controller            DB / Binary
   │                      │                      │                     │
   │  HTTP GET /query     │                      │                     │
   ├─────────────────────►│                      │                     │
   │                      │  FdsnDataselectPlugs │                     │
   │                      ├─────────────────────►│                     │
   │                      │  parse FDSN params   │                     │
   │                      │◄─────────────────────┤                     │
   │                      │                      │                     │
   │                      │  Authentication      │                     │
   │                      ├─────────────────────►│  query Auth DB      │
   │                      │                      ├────────────────────►│
   │                      │                      │◄────────────────────┤
   │                      │                      │                     │
   │                      │  Authorization       │                     │
   │                      ├─────────────────────►│  get closed nets    │
   │                      │                      ├────────────────────►│
   │                      │                      │◄────────────────────┤
   │                      │                      │                     │
   │                      │  Filters             │                     │
   │                      ├─────────────────────►│                     │
   │                      │                      │  query inventory    │
   │                      │                      ├────────────────────►│
   │                      │                      │  (data files list)  │
   │                      │                      │◄────────────────────┤
   │                      │                      │                     │
   │                      │                      │  evaluate_size      │
   │                      │                      │  (too large? → 413) │
   │                      │                      │                     │
   │                      │                      │  dataselect binary  │
   │                      │                      ├────────────────────►│
   │                      │                      │  (read & trim MSEED)│
   │                      │                      │◄────────────────────┤
   │                      │                      │                     │
   │                      │  chunked response    │                     │
   │◄─────────────────────┤◄─────────────────────┤                     │
   │                      │                      │                     │
   │                      │  DeliveryMetrics     │                     │
   │                      ├─────────────────────►│  write metrics DB   │
   │                      │                      ├────────────────────►│
   │                      │                      │                     │
   │                      │  Cleanup             │                     │
   │                      ├─────────────────────►│  rm temp workdir    │
   │                      │                      │                     │
```

1. **FdsnDataselectPlugs** (external [`fdsn_plugs`](https://hex.pm/packages/fdsn_plugs) library) parses FDSN query parameters into a structured `fdsn_parameters` map.
2. **Authentication** — `/queryauth` uses HTTP Digest ([RFC 2617](https://datatracker.ietf.org/doc/html/rfc2617)); `/query` sets the user as `anonymous`.
3. **Authorization** — closed networks are checked against the user. Access can be granted via the `WSDATASELECT_AUTHORIZATIONS` env var.
4. **Filters** — source identifiers are converted to `Filter` structs for the database query.
5. **Inventory query** — `Repo.get_contents/1` fetches matching data files from the inventory database.
6. **Size evaluation** — if the total response exceeds `WSDATASELECT_MAX_RESPONSE_SIZE`, a 413 is returned.
7. **Dataselect binary** — reads and trims miniSEED files in parallel (up to `WSDATASELECT_MAX_CONCURRENCY`), writing output to a temporary workdir.
8. **Chunked response** — the trimmed miniSEED data is streamed back to the client.
9. **DeliveryMetrics** — computes usage statistics per source identifier and writes them to the metrics database.
10. **Cleanup** — removes the temporary workdir.

## Authentication

### HTTP Digest (`/queryauth`)

Implements [RFC 2617](https://datatracker.ietf.org/doc/html/rfc2617). The username (or `anonymous` if unknown) is added to the data request structure.

The default realm is `"FDSN"`, hardcoded in the credential hashes at RESIF. Override at compile time:

```bash
AUTH_REALM="MyRealm" mix compile
```

### JWT

Planned but **not yet implemented**. The JOSE library is available in deps for future JWT-based authentication.

## Configuration

Configuration is done with environment variables, at runtime.

### Application

| Variable | Default | Description |
|---|---|---|
| `PHX_SERVER` | _(unset)_ | Set to any value to enable the Phoenix endpoint server |
| `PHX_HOST` | `"example.com"` | Hostname for URL generation |
| `PORT` | `"4000"` | HTTP listen port |
| `SECRET_KEY_BASE` | **required** | Phoenix secret for signing/encryption |
| `WSDATASELECT_URL_PREFIX` | `"/fdsnws/dataselect/1/"` | URL prefix where the service is accessible |
| `WSDATASELECT_WORKDIR` | `"/tmp/dataselect"` | Temporary directory where dataselect writes output |
| `DNS_CLUSTER_QUERY` | _(none)_ | DNS cluster query for Phoenix DNSCluster |

### Dataselect binary

| Variable | Default | Description |
|---|---|---|
| `WSDATASELECT_DATASELECT_PATH` | `"/usr/local/bin/dataselect"` | Path to the dataselect binary |
| `WSDATASELECT_DATASELECT_TIMEOUT` | `"5000"` | Timeout for reading data with the dataselect binary (ms) |
| `WSDATASELECT_MAX_CONCURRENCY` | `"8"` | Number of dataselect processes to start simultaneously |
| `WSDATASELECT_MAX_RESPONSE_SIZE` | `"2000000000"` | Maximum response size in bytes (~2 GB) |
| `WSDATASELECT_REPOSITORIES_ROOT` | `"/data"` | Root mountpoint of the data repositories |

### Database

| Variable | Default | Description |
|---|---|---|
| `DATABASE_URL` | **required** (prod/dev/staging) | Main inventory Postgres URL |
| `AUTH_DATABASE_URL` | **required** (prod/dev/staging) | Authentication Postgres URL |
| `METRICS_DATABASE_URL` | **required** (if metrics enabled) | Delivery metrics Postgres URL |
| `WSDATASELECT_POOL_SIZE` | `"10"` | Ecto repo pool size |
| `WSDATASELECT_POOL_COUNT` | `"1"` | Ecto repo pool count |
| `WSDATASELECT_REPO_TIMEOUT` | `"15000"` | Ecto repo query timeout (ms) |
| `ECTO_IPV6` | _(unset)_ | Set to `"true"` or `"1"` to enable IPv6 for DB connections |

### Delivery metrics

| Variable | Default | Description |
|---|---|---|
| `WSDATASELECT_DELIVERY_METRICS` | `"true"` | Set `"false"` to disable delivery metrics |
| `WSDATASELECT_CLEANUP_INTERVAL` | `"360000"` | Vacuum cleaner interval (ms) |
| `WSDATASELECT_AUTHORIZATIONS` | `"{}"` | JSON-encoded authorization overrides |
| `WSDATASELECT_AUTHORIZATIONS_CACHE_TTL` | `"3600"` | Authorization cache TTL (seconds) |
| `MAXMIND_LICENSE_KEY` | `nil` | MaxMind GeoIP license key |

### S3 / Presigned URLs

| Variable | Default | Description |
|---|---|---|
| `AWS_ACCESS_KEY_ID` | _(none)_ | AWS/S3 access key |
| `AWS_SECRET_ACCESS_KEY` | _(none)_ | AWS/S3 secret key |
| `WSDATASELECT_S3_PUBLIC_HOST` | `"s3.seisdata.epos-france.fr"` | S3 public host for data access |
| `WSDATASELECT_S3_BUCKET` | `"sandbox"` | S3 bucket name |

### Observability (Sentry)

| Variable | Default | Description |
|---|---|---|
| `SENTRY_DSN` | _(none)_ | Sentry DSN for error tracking (read by the Sentry library) |
| `SENTRY_ENVIRONMENT` | _(none)_ | Sentry environment name (read by the Sentry library) |
| `SENTRY_RELEASE` | _(set at build)_ | Sentry release identifier |
| `SENTRY_TRACES_SAMPLE_RATE` | `"0.001"` | Sentry traces sample rate (compile-time) |

### Authentication & Networking

| Variable | Default | Description |
|---|---|---|
| `AUTH_REALM` | `"FDSN"` | HTTP Digest auth realm (compile-time: `AUTH_REALM="MyRealm" mix compile`) |
| `WSDATASELECT_PROXIES` | `""` | Space-separated list of trusted proxy IPs (compile-time) |

### Test

| Variable | Default | Description |
|---|---|---|
| `MIX_TEST_DBHOST` | `"localhost"` | Test DB hostname |
| `MIX_TEST_PARTITION` | `""` | Test DB partition suffix |

## Deployment

### Prerequisites

- PostgreSQL databases: inventory (sigma), authentication, and optionally delivery metrics
- Data archives mounted at `WSDATASELECT_REPOSITORIES_ROOT`, coherent with the `repositories` table in the inventory DB
- [`dataselect`](https://github.com/EarthScope/dataselect) binary compiled and present at `WSDATASELECT_DATASELECT_PATH`

### Containers

Pre-built containers are available in the [Gricad GitLab forge](https://gricad-gitlab.univ-grenoble-alpes.fr/OSUG/RESIF/wsdataselect/container_registry/931).

The Dockerfile builds an Elixir release and copies the `dataselect` binary from a dedicated container image.

### CI/CD

GitLab CI pipeline with stages: `test` → `build` → `publish` → `deploy`.

- **Staging** deploys on every commit to the default branch
- **Production** deploys on version tags (`v*`)
- Deployments target a Kubernetes cluster via `kubectl rollout restart`

## Test

Tests use `ex_machina` for factories, `excoveralls` for coverage, and `junit_formatter` for CI reports.

```bash
# Run all tests
MIX_ENV=test mix test

# Run a single test file
mix test test/path/to/file_test.exs

# Run previously failed tests
mix test --failed

# Full precommit check (compile + lint + format + test)
mix precommit
```

CI runs `mix coveralls.multiple --type html --type cobertura` with the `dataselect` binary at `test/bin/dataselect`.