jacks-reid.xyz

Reflecting on Google Cloud’s Asset Key Thief Vulnerability

In the first half of 2023 I discovered and disclosed a Google Cloud service account private key exposure vulnerability. I dubbed it ‘Asset Key Thief’ (as cloud vulnerabilities do not receive CVEs). In a nutshell, private keys that allow for the assumption of service account principals, and whatever permissions those service accounts are permitted in a Google Cloud environment, could be exfiltrated out of an unrelated asset inventory API. This private key material was never supposed to be stored outside of the initial key provisioning process. Any average joe with access to the asset inventory API could also steal the permissions of any service account who had a private key created in the past 24 hours.

This vulnerability disclosure process took a significant toll on my wellbeing. Inspired by Michael Bargury’s blog post, I want to reflect on the trying phases of a high severity vulnerability disclosure process as an outsider.

The initial days of finding the vulnerability were filled with anxiety. I still couldn’t believe what I stumbled upon. It seemed like I had to be wrong somewhere. Service account private keys are so closely guarded that it was unimaginable to me to find them sitting out in the open during my daily work. I cobbled together a report to Google’s Vulnerability Reward Program as fast as I could, submitted, and then waited. Nervous pings to coworkers did their best to calm my nerves as hours dragged into days.

My initial vulnerability disclosure report was then repeatedly closed three times over the course of a few weeks. Imposter syndrome started to kick in - did I really know what I was talking about? Thankfully I stuck to it, rang every possible alarm bell I knew of at Google Cloud, and got the right pair of eyes on the report. Google’s famous “Nice catch!” response to my report acknowledging the vulnerability brought a wave of relief and I was optimistic that the light was at the end of the tunnel.

However the light was not at the end of tunnel. I quickly learned that my role as security researcher did not grant me any special privileges into the remediation process of the vulnerability I reported. I spent the next two months waiting for Google Cloud to tell me the vulnerability was remediated. Every week I’d end Friday recreating the vulnerability and getting bummed to see it was still active. I felt guilty - was I doing enough? Finally, I got notice that the vulnerability was fixed by the product team. My final proof-of-concept recreation at the terminal ended with a smile - private key material was no longer accessible.

I then found myself at a crossroads. Would I consider this journey done or would I push forward with a public vulnerability disclosure? This decision was difficult given my ties to Google Cloud. I was working at a Google Cloud partner. I was a Google Cloud Fellow. Much of my career was tied to Google Cloud succeeding. Did I really need to air Google Cloud’s dirty laundry to the world? Deep down did I truly believe this was the best thing to do or was I excited about the street cred?

I spent a lot of time talking to mentors. Folks I trusted to give me objective advice. They emboldened me to push forward. The nature of vulnerability had far reaching consequences if exploited to its full potential. I had a responsibility to see it through.

Pushing forward with a public disclosure was by far the most difficult part of the journey. My technical writeup had to be unimpeachable. I had to provide a complete breakdown of remediation guidance (which I usually wouldn’t complain about, however there were no ‘smoking guns’ audit logs screaming ’this was exploited!’ with the vulnerability). I had to convince marketing and public relations folks that disclosures were commonplace for security. I ended up having a lot of amicable, but intense, conversations with product teams. Once I had enough of the back and forths, I set a disclosure date and blocked my calendar. This was something that needed to happen and I was never going to get everyone to agree with me.

Ultimately the disclosure day came it went. I had my 15 minutes of fame. The vulnerability makes a buzz in the sort of communities that would care about a Google Cloud vulnerability. Google Cloud ends up sending its own “Advisory Notification” (read: fire alarm) directly to customers notifying them of ‘at-risk’ service account private keys that should probably be rotated. I never got the inside word on what that investigation evaluated and probably never will. In the long run the vulnerability got its spotlight and then the world moved on.

Was it all worth it? It was. I believe it was best for Google Cloud customers. My close hacker friends supported me. My presentation on the vulnerability at BSides Atlanta 2023 was a full house. I’ve been invited on podcasts to talk about the journey. Government agencies at Google Cloud Next told me they had to respond to the vulnerability report! I’d do it again in a heartbeat.

I’ve since moved away from daily work with Google Cloud and I feel reflective about an era closing. I cannot close this chapter without thanking again the army involved in this journey. Friends, coworkers, mentors, and public relations folks - all who were so integral to getting the disclosure out the door. I am so grateful.

My Guidelines for Security Research Disclosures

If I could do things differently, I would. Here are guidelines I recommend to researchers navigating vulnerability disclosures:

  • You can only publish what you observe. Security research is oftentimes blackbox research. You don’t have access to the code on the other side. Don’t let anyone try to convince you to doubt what you’ve seen or report anything other than what you have seen.
  • Stick to working with a company’s vulnerability disclosure team. You are under no obligation to spread the word within their organization.
  • Hold up your end of the bargain. Disclosure the vulnerability, behave professionally, and do everything in your power to protect customers. You are under no obligation, nor do you have the capability, to drive mass remediation and retrospectives across an entire ecosystem. That’s on the provider.
  • Don’t let anyone bully you. A wide variety of folks did not think I should go through with the disclosure. Some folks didn’t consider the vulnerability severe. Stick to your guns and don’t be afraid of being told that you are wrong.
  • Prioritize the little guy. During this process I determined my north star would be Google Cloud security engineers at small companies. How do I do best by them, not necessarily best by Google Cloud or my organization?

Press References

I never took the time to compile all the press references on AKT for my own records. These will be fun to reread in the future.