Due to a slower than expected replication of data between two of our data-centres over the weekend, you may have seen live reports not giving fully up to date data during 8th and 9th December.
We are pleased to report that real-time reports are now fully operational again.
As previously notified at https://bangoserviceblog.wordpress.com/2008/12/02/datacenter-maintenance-notification/ we switched traffic from our Main Datacenter to Datacenter South from Friday 5th December at 10:00UTC until Monday 8th December 10:00UTC.
This ensured that payment and analytics transactions were unaffected by a 10-15 minute planned maintenance requirement by our hosting facilities supplier, and ensured we maintain our high levels of service availability.
When the Main Datacenter was re-instated as the primary server on Monday morning, we became aware of delays in replication of some analytics records captured while Datacenter South was primary. Ordinarily the replication between datacentres happens near real-time, but we observed that replication during the weekend had slowed down considerably. The slow replication problem was addressed by our technical team, and the replication “caught up” by late on Tuesday 8th December.
No data was lost during this process. It was just that real-time reporting on the primary data-centre did not have access to all the data that had been captured in the alternate data-centre until it had replicated across.
Having identified and corrected the reasons for the slowed replication, we do not believe this situation will happen in future.