On January 24th Google had some problems with a few of its services. Gmail users and people who used various other Google services were impacted just as the Google Reliability Team was to take part in an Ask Me Anything on Reddit. Everything seemed to be resolved and back up within an hour. The Official Google Blog had a short note about what happened
from Ben Treynor, a VP of Engineering. According to the blog post it
appears that the outage was caused by a bug that caused a system that
creates configurations to send a bad one to various 'live services.' An
internal monitoring system noticed the problem a short time later and
caused a new configuration to be spread around the services. Ben had
this to say of it on the Google Blog, 'Engineers were still debugging 12
minutes later when the same system, having automatically cleared the
original error, generated a new correct configuration at 11:14 a.m. and
began sending it; errors subsided rapidly starting at this time. By
11:30 a.m. the correct configuration was live everywhere and almost all
users' service was restored.
