Do Son 
Code hosting platform GitHub experienced a large-scale, prolonged outage this week, disrupting normal service for a substantial number of users. Following its standard practice, GitHub published a detailed postmortem report once the issue was resolved to maintain transparency. According to that incident report, the outage was triggered by unusually heavy external traffic, compounded by several underlying flaws within GitHub’s internal infrastructure – a combination of factors that together produced the extended disruption.
Network Congestion Began at a U.S. Central Data Center
Portions of GitHub’s services run atop an Istio service mesh, with each application container paired alongside a sidecar responsible for handling network communication. At the onset of the incident, several of these sidecars had already reached their maximum concurrent processing capacity. However, GitHub’s autoscaling policy primarily evaluated whether to scale based on the application itself, without adequately accounting for the connection and concurrency state of the sidecars.
As a result, even as the network proxies approached their processing limits, the system failed to provision additional instances in time. As certain nodes became unable to process traffic normally, requests began shifting to other nodes. This, in turn, caused additional high-availability load balancer nodes to gradually reach their own capacity ceilings, ultimately creating severe congestion along the network paths responsible for critical services such as authentication.
Because GitHub.com, the GitHub API, GitHub Actions, and numerous other services all depend on this authentication and gateway infrastructure, the severe congestion along that critical path rapidly expanded the scope of the outage.
Automatic Retries Triggered a Cascading Retry Storm
Under normal circumstances, automatically retrying requests when a service briefly returns an error can improve overall request success rates. In this incident, however, the automatic retry mechanism instead compounded the failure. As authentication requests began timing out, a large number of clients and internal services continuously issued new requests, sending even more traffic toward load balancers that were already overloaded – a phenomenon known as a retry storm.
In simple terms, what began as a subset of failed requests triggered automatic resending by the system in an effort to improve success rates. This, in turn, placed even greater strain on backend services and produced additional timeouts, which then triggered further rounds of retries – forming a vicious cycle. GitHub ultimately restored its primary services by pausing several malfunctioning load balancer nodes and redirecting a portion of traffic to other data centers.
A Serious Bug Inside GitHub Copilot
GitHub Copilot proved to be the service with the longest recovery time during this incident. Investigation revealed an abnormal retry issue within the GitHub Copilot extension for Microsoft Visual Studio Code: when the service responsible for issuing authentication tokens responded slowly or failed outright, certain clients continued repeatedly requesting new authentication tokens without pause.
Under normal conditions, the authentication token issuance service handles between 7,000 and 9,000 requests per second. During the incident, however, that request volume surged to between 70,000 and 100,000 requests per second – roughly ten times the normal baseline. These additional requests continued piling pressure onto the authentication infrastructure, which explains why GitHub Copilot remained non-functional even after other services had already recovered.
Heavy Crawler Traffic Compounded Infrastructure Strain
In its postmortem, GitHub also noted that during the recovery period, its Codeload code download service faced several waves of large-scale automated crawler traffic. While this abnormal traffic was not the root cause of the outage, it added further strain to already-taxed infrastructure during the recovery window and made resolving the incident considerably more difficult.
GitHub’s Planned Remediation
In response to the issues this incident exposed, GitHub stated that it will revise its autoscaling policies and conduct a thorough review of retry behavior across both its internal gateways and client-side integrations, in order to prevent a surge of automatic retries from overwhelming backend services following a future failure.
Support Our Threat Intelligence
If you find our CVE report and cybersecurity news helpful, consider supporting our work.
Buy Me a Coffee
PayPal
