Permit io - API service unavailable due to performance issues under unusual traffic load – Incident details

All systems operational

API service unavailable due to performance issues under unusual traffic load

Resolved
Major outage
Started about 2 years agoLasted 36 minutes

Affected

Permit.io Backend

Permit.io OPAL

Major outage from 11:34 AM to 12:10 PM

OPAL US East

Major outage from 11:34 AM to 12:10 PM

Updates
  • Resolved
    Resolved

    An unusual spike in traffic caused some performance bottlenecks in our API service. PDPs were not affected in their read paths, which means permit.check() calls continued working normally.

    Some customers could not create new data in the system (new users / new environments).

    We added caching to fix the bottlenecks we discovered today.

  • Investigating
    Investigating

    We are experiencing an unusual load on our API service (api.permit.io) at the moment, which causes slowness of the dashboard and some of our api endpoints.