A Rube Goldberg chain of failures led to breach of Microsoft-hosted government emails

In the first half of July, Microsoft disclosed that the Chinese hacking group Storm-0558 had gained access to emails from around 25 organizations, including agencies in the US government. Today, the company is explaining how that happened thanks to a series of internal errors while sharply underscoring just how serious a responsibility it is to maintain massive, growing software infrastructure in an increasingly digitally insecure world.

According to Microsoft’s investigation summary, Storm-0558 was able to gain access to corporate and government emails by obtaining a “Microsoft account consumer key,” which let them create access tokens to their targets’ accounts.

Storm-0558 obtained the key after a Rube Goldberg machine-style series of events put the key somewhere it should never have been in the first place. The company writes that when the system made a debugging snapshot of a process that had crashed, it didn’t strip, as it should have, the so-called “crash dump” of all sensitive information, leaving the key in.

Microsoft’s systems still should have detected the “key material” in the crash dump, but apparently, they didn’t. So when company engineers found the dump, they assumed it was free of sensitive data and transferred it, key and all, from the “isolated production network” to the company’s debugging environment.

Then another fail-safe — a credential scan that should have also caught the key — missed that the key was there. The final gate fell when Storm-0558 managed to compromise a Microsoft engineer’s corporate account, giving the hackers access to the very debugging environment that never should have had the key to begin with.

Microsoft writes that it has no logs showing evidence this is how the key was shuffled out of its systems but says it’s the “most probable” route the hackers took.

There’s one final kicker: this was a consumer key, but it let threat actors get into enterprise Microsoft accounts. Microsoft says it began using common key metadata publishing in 2018 in response to demand for support software that worked across both consumer and enterprise accounts.

The company added that support, but it failed to make the proper updates to the systems used to authenticate keys — that is, determine whether they’re consumer or enterprise keys. Mail system engineers, assuming the updates had been made, built in no additional authentication, leaving the mail system blind to what sort of key was used.

In short, had those libraries been updated properly, even given all the other failure points, Storm-0558 hackers might not have been able to access the enterprise email accounts used by the corporations they targeted.

Microsoft says it has corrected all of the issues above, including the error that sent the signing key to the crash dump in the first place. The company adds in its post that it is “continuously hardening systems.” Microsoft has increasingly come under fire for its security practices, which both Senator Ron Wyden (D-OR) and Tenable CEO Amit Yoran have called “negligent,” with Yoran accusing Microsoft of being too slow to react to its security flaws.