queue_keeper_check

# Changelog

All notable changes to this project are documented here. The format follows
[Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and the project uses
[Semantic Versioning](https://semver.org/).

## [Unreleased]

## [0.4.1] - 2026-11-11

### Fixed

- Ledger `fetch/2` no longer aborts a Postgres read-only transaction when the
  optional ledger tables are missing (`ERROR 25P02 in_failed_sql_transaction`).
  Existence is probed first (`to_regclass` on Postgres; catalog tables on
  SQLite and MySQL). If either table is absent, the check falls back to
  `oban_jobs`. Genuine permission or connectivity errors are still raised.
- `:backlog` severity now follows the strongest breached dimension. A queue
  whose depth is critical is no longer downgraded to a warning just because
  latency also exceeds its (lower) warning threshold.
- `:stalled` no longer treats a cancelled or discarded job as evidence that a
  consumer is running unless that job was also attempted inside the lookback
  window. Cancelling an unattempted job, or a stale attempt with a recent
  cancel/discard, does not hide a stalled queue.
- The opt-in ledger records telemetry only for the configured repository.
  Events from another repo that share the same prefix, or events with missing
  or malformed repo metadata, are ignored. `:oban_name :all` still records
  every Oban name on the intended repo.
- `:dead_worker_candidates` honors a shorter per-worker `overrides` window
  even when the default `window_days` is longer than observed history. A
  longer override is still insufficient when history does not cover it.
  Insufficient-history findings are now emitted per worker rather than as a
  single global finding that hid shorter overrides.
- Explicitly requested `beam_roots` or `beam_paths` that do not exist make
  static-scan coverage incomplete (`requested_missing`), so they cannot produce
  a `:high_confidence` combined finding. Default compile directories that are
  absent are not counted.
- `:hanging` still reports over-budget executing jobs when `:orphaned` is not
  in the run (`only: [:hanging]`). The orphan-age upper bound is kept only as
  a dedup when both checks run.

### Changed

- Removed a dead `Config.validate!/3` clause and a redundant type check that
  Elixir 1.20 warned about. The supported Elixir baseline remains `~> 1.15`.
- The documented Hex dependency version is `~> 0.4.1`. The Hex package itself
  will be published after this GitHub tag is tested.

## [0.4.0] - 2026-11-10

### Added

- `:dead_worker_candidates` (opt-in, report-only): combined review of enqueue
  inactivity and static reachability. A high-confidence finding requires no
  enqueue for at least `window_days` (inclusive `age >= window`; default 7
  days when you only set `known` or `discover: true`) with sufficient history,
  no code or cron reference, a complete static scan, and no leftover pending or
  future-scheduled jobs. It never claims a worker is definitely dead.
  Insufficient history is reported as `:unknown`, not as inactivity. Reasons:
  `:high_confidence`, `:referenced`, `:pending_jobs`, `:incomplete_analysis`,
  `:insufficient_history`, `:unknown`. Reuses the existing `:dead_workers`
  activity and pending-job SELECTs plus `DeadCode.scan_report/1`.

### Changed

- The implicit `window_days` default for `:dead_workers` when you set `known` or
  `discover: true` without `window_days` is now **7** (was 14 in 0.3.0). The
  same 7-day implicit default applies to `:dead_worker_candidates`. The window
  remains any positive integer; set `window_days: 14` to keep the previous
  implicit window. The standalone `:dead_workers` comparison is still a strict
  `age > window` (exactly N days does not fire). This is a breaking change to
  that implicit default, so this is a minor (0.4.0) bump rather than a patch.
- The documented Hex dependency version is `~> 0.4.0`.

## [0.3.0] - 2026-11-10

### Added

- `:dead_workers` (opt-in): report workers that have had no job inserted within a
  window of any positive integer number of days (`window_days`). 7, 14 and 30 are
  documented examples, not a restricted set; if you set `known` or `discover: true`
  without `window_days`, the window is 14. Findings are always `:warning`.
  * Last activity is `max(inserted_at)` per worker in any state (last enqueue, not last
    execution), so a worker that is still being inserted is not treated as dead code.
  * If observable finished-job history (`min(inserted_at)` among `completed` /
    `cancelled` / `discarded` rows) is shorter than a worker's window, that worker is
    not flagged, to avoid calling a recently used worker dead after the Pruner deleted
    its rows. Set `report_short_history: true` to emit a warning about the short history
    instead of staying silent.
  * `ignore` skips named workers, `overrides` sets a per-worker window in days, and
    `known` lists extra workers to check even when they have no rows at all.
  * Optional `discover: true` also checks `Oban.Worker` modules in loaded non-OTP
    applications; `mix queue_keeper.check` and `QueueKeeperCheck.Release` do not start
    the host app, so they will not see that app's modules.
  * Workers listed in `cron_missing.expected` are skipped so a late cron job is not
    also reported as dead.
  * Like `:cron_missing`, the `oban_jobs` fallback is a sequential scan (no index on
    `worker` / `inserted_at`), bounded by `:statement_timeout`.
  * Opt-in last-seen ledger (`QueueKeeperCheck.Ledger`): a tiny `(prefix, worker)`
    table written by a telemetry handler on `[:oban, :engine, :insert_job, :stop]`
    and `:insert_all_jobs`. Sightings are coalesced in ETS and flushed every few
    seconds with `GREATEST`, so aggressive pruning does not require a longer
    Pruner `max_age`. The check reads the ledger when it is present and stays
    silent until the ledger itself has been recording for the window. Writing is
    explicit (`Migration` + `attach`); `QueueKeeperCheck.run/2` remains read-only.
    Optional one-time `backfill` from current `oban_jobs`.
  * `mix queue_keeper.dead_code`: lists compiled `Oban.Worker` modules with no
    static call sites and not in Oban/Pro crontab config as **review candidates**,
    not proven dead code. Dynamic dispatch, missing BEAM debug info, and
    runtime-only cron are documented false positives. `scan_report/1` exposes
    coverage. Needs the project compiled; does not start the app.
  * `:persisted_jobs` (opt-in, report-only): bounded read-only comparison of
    nonterminal jobs (`available`, `scheduled` including future, `retryable`,
    `executing`, and core `suspended`) against a trusted inventory of deployed
    workers (`workers`, `previous_workers`, optional `discover`) and effective
    queues. Findings are always `:warning` and carry a `:reason` of
    `:missing_worker`, `:unconfigured_queue`, `:paused_queue`,
    `:no_active_consumer`, or `:insufficient_evidence`. Pause and consumer
    state are taken only from options you pass; the check never infers
    deployment-wide absence from one node, and never creates atoms from database
    worker or queue strings. One `run/2` covers one Oban prefix. Oban Pro
    support beyond core `suspended` is untested and unverified (no Pro license).

### Changed

- **Migration (`DeadCode` status atom):** `QueueKeeperCheck.DeadCode.scan/1`
  now returns `status: :review_candidate` instead of `status: :dead`. Treat this
  as a breaking change if you matched on `:dead`. `review_candidates/1` is the
  preferred helper; `dead_workers/1` still returns the same names. JSON from
  `mix queue_keeper.dead_code --json` adds `review_candidates` and `coverage`
  and keeps `dead` as a deprecated alias of the same list. Exit codes are
  unchanged.
- `:dead_workers` findings no longer say "this may be dead code". A bounded
  `state IN (...)` query surfaces leftover nonterminal and future-scheduled
  jobs. No recent enqueue with pending work is reported as inactivity, not
  unused code. Short-history messages say "insufficient history". Do not raise
  Pruner `max_age` for this check. Caller lookup in the static scan no longer
  creates atoms from worker-name strings.

- The documented Hex dependency version is `~> 0.3.0`.

## [0.2.1] - 2026-10-22

First release published to Hex.

### Changed

- `:retry_storm` now honours its window for `retryable` jobs too: a `retryable` job counts
  only if its last failed attempt started within the window (`attempted_at >= now - window`,
  with a redundant `scheduled_at >= now - window` bound so the query is an index range scan
  on Oban's `(state, queue, priority, scheduled_at, id)` index). Up to v0.2.0 every
  `retryable` job counted regardless of the window, so a worker with many jobs waiting out
  long backoffs from failures hours ago could fire a "retry storm in the last 300s". Expect
  fewer `:retry_storm` findings after upgrading.
- `:orphaned` findings and docs now say what is measured: execution age only. The message
  reads "executing for up to N s (limit L s), far beyond any expected runtime; possibly
  orphaned by a node crash or shutdown, or genuinely still running" instead of implying the
  node died. The check name is unchanged.

### Documentation

- README: "Piloting safely" (report-only rollout, what to measure, tuning, retention for
  history-based checks); `:stalled` also fires for queues whose slots are all held by
  long-running jobs.
- `docs/OBAN_PRO_COMPATIBILITY.md` and an Oban Pro row under "Engine support": likely
  compatible, unverified and untested, with the Pro features known to skew results.
- `docs/SECURITY_REVIEW.md`: updated `:retry_storm` SQL listing.

## [0.2.0] - 2026-10-22

First public release, tagged on GitHub. (An earlier version of this entry said it was
published to Hex; it has not been published to Hex yet.) 0.1.0 (below) was an
internal milestone: it was never tagged, released or published, and it had the
production-safety problems listed under "Fixed" (see the findings QK-01 to QK-11 in
`docs/SECURITY_REVIEW.md`). If you pinned a pre-release commit from GitHub, read
"Changed" before upgrading.

### Fixed

- `mix queue_keeper.check` no longer starts the host application (and therefore Oban,
  its queues and plugins); it loads config and starts only the repo (QK-01).
- Queries that could take minutes on large tables: sample queries no longer
  `ORDER BY id`; `:orphaned`/`:hanging` aggregate in SQL; `:stalled` probes only late
  queues with index-bounded `LIMIT 1` queries; sample queries capped at 100 per run (QK-02, QK-07).
- Text output replaces control/bidi characters; Slack payloads escape `&`, `<` and `>` (QK-05).
- `sample_ids` no longer render as a charlist (QK-08).

### Added

- `QueueKeeperCheck.Guard`: on Postgres every check runs in a rolled-back transaction with
  `transaction_read_only`, `statement_timeout`, `lock_timeout` and
  `idle_in_transaction_session_timeout`; on MySQL `max_execution_time` (QK-03).
- Options `:statement_timeout` (5000), `:lock_timeout` (1000), `:check_timeout` (30000),
  `:read_only` (true); `stalled: [lookback: 3600]`; `retry_storm: [include_completed: false]`.
- `report.errors`, status `:error` and exit code 3; per-check error isolation; remaining
  checks are skipped after a connection error (QK-04).
- Strict option validation, including an identifier allow-list for `:prefix` (QK-06).
- `SECURITY.md`, `docs/SECURITY_REVIEW.md`, benchmark and verification scripts.
- CI on PostgreSQL 17, SQLite and MySQL 8.4, with actions pinned to commit SHAs and
  `mix hex.audit` (QK-09).

### Changed

- `:retry_storm` counts completed retried jobs only with `include_completed: true`
  (that part needs a full table scan).
- `Release.check/2` never raises, starts the repo with a pool of 1 and stops it afterwards.
- Running `QueueKeeperCheck.run/2` inside an open transaction on Postgres is refused.
- Errors (bad options, database unreachable, timeouts) now exit with 3 instead of 1, so
  "could not check" is no longer reported as "warning" (QK-04).
- Unknown or invalid options raise `ArgumentError` instead of being ignored (QK-06).

### Known limitations

- `:cron_missing` (opt-in) needs a sequential scan; it is bounded by `:statement_timeout` (QK-10).
- On MySQL and SQLite, read-only enforcement depends on a SELECT-only database user or a
  read-only connection; the library only issues `SELECT`s (QK-11).

## [0.1.0] - 2026-10-22 (never published)

Initial implementation. Not tagged, released or published; superseded by 0.2.0.


### Added

- `QueueKeeperCheck.run/2`: read-only checks on `oban_jobs` for `:orphaned`, `:hanging`,
  `:backlog`, `:stalled`, `:retry_storm`, `:discarded`, `:cancelled` and `:cron_missing`,
  with configurable thresholds (`QueueKeeperCheck.Config`).
- `QueueKeeperCheck.format/1` (plain text) and `QueueKeeperCheck.slack_payload/1`
  (Slack incoming-webhook body; never sent by the library).
- `mix queue_keeper.check` task with exit codes 0 (ok), 1 (warning), 2 (critical).
- `QueueKeeperCheck.Release.main/2` / `check/2` for running from a release via `eval`,
  outside Oban (e.g. system cron).
- Support for Oban's Basic (PostgreSQL), Lite (SQLite3) and Dolphin (MySQL 8.4+) engines.

[Unreleased]: https://github.com/TitaniumMonkey/Queue-keeper-check/compare/v0.4.1...HEAD
[0.4.1]: https://github.com/TitaniumMonkey/Queue-keeper-check/compare/v0.4.0...v0.4.1
[0.4.0]: https://github.com/TitaniumMonkey/Queue-keeper-check/compare/v0.3.0...v0.4.0
[0.3.0]: https://github.com/TitaniumMonkey/Queue-keeper-check/compare/v0.2.1...v0.3.0
[0.2.1]: https://github.com/TitaniumMonkey/Queue-keeper-check/releases/tag/v0.2.1
[0.2.0]: https://github.com/TitaniumMonkey/Queue-keeper-check/releases/tag/v0.2.0
[0.1.0]: https://github.com/TitaniumMonkey/Queue-keeper-check/commit/48b75a1