Bango Incident Report

On two occasions last week, Bango observed very high levels of requests to the Bango.net servers from a range of IP addresses, presenting unusual device headers.  Due to the unusual nature of this traffic, our load balancing systems routed traffic to our “bango.net” servers for processing in a way that caused some servers to get overloaded and unable to respond quickly enough to the requests.
For approximately 2hrs from 18.48 UTC on Wednesday 16th September and 2.5hrs from 14.13 UTC on Saturday 19th September any traffic routed to these affected servers would have been either ignored or timed-out, causing failure of that request, whether for Analytics or Payment.

On the first occasion the problem was resolved by assigning more servers from our reserve pool to bango.net to handle the unusual traffic.

On the second occasion, further servers were brought into the pool, and changes made to capture, re-route and analyze this anomalous traffic for investigation by dedicated servers were put in place.

The unusual traffic appeared again on Sunday 20th, but the traffic was processed by dedicated servers and the bango.net service was unaffected.

Other impacts

Approximately 12.5% to 37.5% of bango.net http requests would have failed or “timed out” over the period affected by the unusual traffic type.

Remedial action

Bango is now monitoring and tracking this anomalous traffic to determine its source to make sure that there will be no further re-occurrence to impact production systems in future.

This entry was posted in Analytics, Bango News, Bango service updates, System Fixes, System Updates and tagged , . Bookmark the permalink.