2025-07-30/31 Hex Outage

Between approximately 2025-07-30 21:27 UTC and 2025-07-31 07:27 UTC hex.pm, diff.hex.pm, and preview.hex.pm were running with expired SSL certificates and couldn’t be accessed over secure connections.

The Hex website and API, at hex.pm, Hex Diff service, and Hex Preview service were affected during the outage. All of Hex API features, including publishing packages, searching packages with filters, managing accounts and organizations were affected. Hosting related to private packages was affected, due to its reliance on the Hex API for authentication, this included the private package registry and package tarballs, at repo.hex.pm used by mix deps.get, and private package documentation at hexdocs.pm.

Other Hex services like hosting public registry and package tarballs, at repo.hex.pm and used by mix deps.get, and public documentation at hexdocs.pm, were unaffected.

Hex infrastructure uses SSL certificates from letsencrypt.org that need to be periodically renewed. Our certificate renewal process relied on an external service that has been discontinued.

The delay in incident response was due to PagerDuty being misconfigured on a newly set up phone and only having one person on-call. In addition, the PagerDuty account had “International Phone Call & SMS Notifications” disabled, which prevented escalation from app notifications to SMS or phone calls.

To prevent similar issues in the future, we’ve made a few changes along with more long term improvements:

1. An internal periodic job has been added that renews the certificates and alerts Hex team on any errors.
2. A daily check has been added to verify that all Hex domains have valid certificates with at least two weeks before expiration.
3. We have enabled “International Phone Call & SMS Notifications” for our PagerDuty account.
4. PagerDuty incidents are now more visible to the Hex team and although we still only have one person on-call, we are working on adding more people to be on-call.
5. The reliance on the Hex API will be removed for private Hex repositories, which will improve reliability by ensuring that package delivery has no external dependencies, other than AWS S3 and the CDN itself.

Hex provides core infrastructure services that are relied on by many. Such outages are unacceptable and we apologize to everyone affected.