Permit io - API service unavailable due to performance issues under unusual traffic load – Incident details

API service unavailable due to performance issues under unusual traffic load

ResolvedMajor outage36 minutes

Affected

Permit.io Backend

Permit.io OPAL

Major outage from 11:34 AM to 12:10 PM

OPAL US East

Major outage from 11:34 AM to 12:10 PM

  • ResolvedUTC

    An unusual spike in traffic caused some performance bottlenecks in our API service. PDPs were not affected in their read paths, which means permit.check() calls continued working normally.

    Some customers could not create new data in the system (new users / new environments).

    We added caching to fix the bottlenecks we discovered today.

  • InvestigatingUTC

    We are experiencing an unusual load on our API service (api.permit.io) at the moment, which causes slowness of the dashboard and some of our api endpoints.