By Yonatan Khen, Threat Hunting Expert at Team Axon
Over the past few years, there has been a notable shift towards cloud technology, with the development of cloud and SaaS tools designed to boost efficiency by integrating features that simplify work processes. One of these features is Google's Domain-Wide delegation. This feature permits a comprehensive delegation between Google Cloud Platform (GCP) identity objects and Google Workspace (GWS) applications. In other words, it enables GCP identities to execute tasks on Google SaaS applications, such as Gmail, Google Calendar, Google Drive, and more, on behalf of other Workspace users.
During 2023, Team Axon conducted extensive research on the Domain-Wide delegation feature and its potential implications for organizations. The research primarily focused on gaining a thorough understanding of the relevant attack surface, the service internals, and, most importantly, how it can be detected effectively.
This paper aims to illustrate how threat actors with varying privileges in a target GCP environment can potentially abuse Domain-Wide delegation. In addition, we’ll also introduce a new proof-of-concept tool that allows for a full takeover of the Google Workspace domain using relevant GCP role permissions. With this tool, red teams, pentesters and security researchers can evaluate their security risks and improve the posture of their Workspace and GCP environments.
The final sections of the blog post offer actionable insights on digital forensics, threat-hunting methodologies, and detection engineering best practices based on telemetry logs from Google Cloud Platform and Google Workspace.
Before delving into the feature internals, let’s first discuss the relationship between GWS and GCP.
Domain-Wide delegation is based on the systemic connection between Google Cloud and Workspace, but aside from both being developed by Google, what is the connection between the two?
The Google ecosystem offers a robust suite of tools and services to meet the diverse needs of organizations. Two of these services - Google Cloud and Google Workspace (or Cloud Identity) - have distinctive roles but are intricately linked, particularly when we examine authentication.
The concept of integrating identity is key to setting up user authentication and access control for GCP services. While GCP IAM is capable of managing control and visibility for resources internally (for example, projects or individual compute resources), it does not dictate who can access the GCP itself.
When a business chooses to use Google Cloud, it needs a central 'hub' for managing all users, groups, settings, and data. This 'hub' (or ‘directory’) can either be Google Workspace or a Google Cloud Identity account. Google categorizes those with the term ‘IDaaS’ or identity-as-a-service.
Though Google Workspace is commonly known for its productivity tools like Gmail, it also has a crucial role in managing user identities across Google Cloud services. Google Cloud Identity plays as the cheaper option for organizations who like to use GCP, but do not necessarily need the productive applications offered by Workspace (for example organizations that are using O365).
Interestingly, even for organizations who are using third party identity providers (IdP), such as Okta or Azure AD, to manage their users and identities, Google still requires authentication through their IDaaS mechanisms. This means organizations that want to use GCP services will still need to sync their third party IdP with either GWS or Cloud Identity directories.
Google's IDaaS concept
It's crucial to understand the interconnected nature of Google Cloud and Workspace, as this understanding underscores how the misuse of one could potentially impact the other.
Google Workspace's Domain-Wide delegation allows an identity object, either an external app from Google Workspace Marketplace or an internal GCP Service Account, to access data across the Workspace on behalf of users. This feature, which is crucial for apps interacting with Google APIs or services needing user impersonation, enhances efficiency and minimizes human error by automating tasks. Using OAuth 2.0, app developers and administrators can give these service accounts access to user data without individual user consent.
Google Workspace allows the creation of two main types of global delegated object identities:
The next step is to understand how it works under the hood when leveraging a GCP service account private key pair to initiate a request to Google APIs on behalf of other identities in Google Workspace.
In terms of implementing a delegated authorization, Google aligns with other well-known Cloud providers by utilizing OAuth 2.0 RFC 6749. The fundamental concept here involves allowing an identity to grant permission to different Workspace REST API applications, without the need to expose his credentials.
Let’s see how it works. The following diagram illustrates the steps created by the delegated identity to access Google API.
OAuth 2.0 High-Level diagram
1. Identity creates a JWT: The Identity uses the service account's private key (part of the JSON key pair file) to sign a JWT. This JWT contains claims about the service account, the target user to impersonate, and the OAuth scopes of access to the REST API which is being requested.
2. The Identity uses the JWT to request an access token: The application/user uses the JWT to request an access token from Google's OAuth 2.0 service. The request also includes the target user to impersonate (the user's Workspace email), and the scopes for which access is requested.
3. Google's OAuth 2.0 service returns an access token: The access token represents the service account's authority to act on behalf of the user for the specified scopes. This token is typically short-lived and must be refreshed periodically (per the application's need). It's essential to understand that the OAuth scopes specified in the JWT token have validity and impact on the resultant access token. For instance, access tokens possessing multiple scopes will hold validity for numerous REST API applications.
4. The Identity uses the access token to call Google APIs: Now with a relevant access token, the service can access the required REST API. The application uses this access token in the "Authorization" header of its HTTP requests destined for Google APIs. These APIs utilize the token to verify the impersonated identity and confirm it has the necessary authorization.
5. Google APIs return the requested data: If the access token is valid and the service account has appropriate authorization, the Google APIs return the requested data. For example, in the following picture, we’ve leveraged the users.messages.list method to list all the Gmail message IDs associated with a target Workspace user.
Up until now, we’ve discussed the feature internals, but have not touched on how it could potentially be abused. Let’s review two scenarios that include different permissions of the target identity within GCP and Google Workspace.
The first scenario is a typical abuse of Domain-Wide delegation as an impactful post-exploitation method of creating a new delegation after gaining access to a Super Admin privilege on the target Workspace environment. This technique has been observed in the wild by Team Axon and has been known to be exploited by threat actors in recent years.
The second scenario is a new method to abuse existing delegations rather than creating a new one. Instead of requiring Super Admin privilege on the Workspace environment, the method requires less privileged access on the relevant GCP projects in order to enumerate successful combinations of service account keys and OAuth scopes. The concept will be detailed in the “DeleFriend״ section.
In the first scenario, an actor typically gains initial access to an IAM identity, with the ability to create service accounts in a GCP project. In addition, after extensive work in the target domain, he now holds a super admin privilege to GWS, and is looking for smart options to achieve strong persistence and exfiltration capabilities.
Attack steps:
The default filename given by Google upon downloading the keys composed from a fixed naming format: <gcp_project_id>-<private_key_id{12}>.json. This format makes these files particularly attractive to red team members and potential attackers once they've gained access to a workstation or storage. We'll delve into methods for finding these files in the Threat Hunting Posture section.
2. Creation of new delegation: After having a relevant identity object and a related private key that enables authentication with Google APIs, we need to establish a new delegation rule for the service account resource within Google Workspace. This delegation rule will enable us to perform the Google APIs activity on Workspace REST API applications. It's important to understand that only the Super Admin role possesses the capability to set up global Domain-Wide delegation in Google Workspace.
Furthermore, Domain-Wide delegation cannot be set up programmatically. It can only be created and adjusted manually through the Google Workspace console. Throughout our research, we tried both documented and undocumented methods of doing this, without any success. Interestingly, even admins who manage subdomains within Google Workspace cannot delegate permissions to applications and service accounts.
The creation of the rule can be found under the page API controls → Manage Domain-Wide delegation in Google Workspace Admin console.
3. Attaching OAuth scopes privilege: When configuring a new delegation, Google requires only 2 parameters, the Client ID, which is the OAuth ID of the GCP Service Account resource, and OAuth scopes that define what API calls the delegation requires.
OAuth scopes is a concept in Google APIs that limits an application's access to a user's account. Each scope represents a specific permission granted by the user to the application.
For example, if an application requests access to a user's Google Calendar data, the scope would be https://www.googleapis.com/auth/calendar. If the application also requested access to Google Drive, another scope https://www.googleapis.com/auth/drive would be needed.
Each scope corresponds to a specific set of permissions, such as read, write, or delete. For instance, granting a read scope allows the application to view data but not modify it. Conversely, granting a write scope allows the application to modify data.
It's important to note that, for security purposes, scopes should be as limited as possible. The best practice of least privilege always should be applied, where an application only asks for the permissions it absolutely needs to function. We’ll revisit that in the threat hunting section when speaking about posture best practices.
It is important to note that the delegation is attached to the service account identity itself in the form of the OAuth Client ID, and NOT for a specific key. This is important as we’ll discuss how it can be abused in the next exploitation method.
4. Acting on behalf of the target identity: At this point, we have a functioning delegated object in GWS. Now, using the GCP Service Account private key, we can perform API calls (in the scope defined in the OAuth scope parameter) to trigger it and act on behalf of any identity that exists in Google Workspace. As we learned, the service account will generate access tokens per its needs and according to the permission he has to REST API applications.
In the given example, we utilize the users.messages.list API to iterate through and list the message IDs of emails in the target inbox. We then use users.messages.get to fetch the entire email content associated with the GWS Identity of a user named Eli Ohana.
Obviously, the example demonstrates requests to the Gmail REST API, but in theory, it can be used on any other Workspace applications based on the OAuth scopes attached to the object.
Bonus: Cross-Organizational delegation
While this scenario pointed to an adversary with relevant permissions to create a new service account in GCP, our research observed that OAuth SA ID is global and can be used for cross-organizational delegation. There has been no restriction implemented to prevent cross-global delegation. In simple terms, service accounts from different GCP organizations can be used to configure domain-wide delegation on other Workspace organizations. This would result in only needing Super Admin access to Workspace, and not access to the same GCP account, as the adversary can create Service Accounts and private keys on his personally controlled GCP account.
We're uncertain if this functionality was an intentional design by Google, but it certainly elevates the risk of creating more “invisible” domain-wide delegations if an adversary creates the service accounts outside of the target organization’s visibility.
In the first scenario, we discussed a situation where the actor already gained privileged access to the Super Admin account in GWS. While this isn’t uncommon in breaches, we researched further possibilities for abusing GWS delegation objects using lower privileges. We tried to look for ways to pivot from limited GCP permissions to Workspace, without necessarily holding a Super Admin privilege.
We discovered a way to do exactly this.
There is one questionable decision Google made when designing the feature. When the delegation configuration is created, it is defined by the service account resource identification (the OAuth ID) and not by the specific private key/s attached to the identity. This means that in case we have access to a relevant GCP service account resource with existing Domain-Wide delegation within the IAM Policy, nothing stops us from creating a new key, enumerating all the existing JWT possibilities and using that existing delegation to perform API calls to Google Workspace on behalf of other identities in the domain.
Now, understanding the theory behind this is one thing, but we want to give you a little bit more. That’s why we have written a proof-of-concept tool that can help the security community increase awareness around Domain-Wide delegation attacks, and improve the security posture of their environments.
Introducing DeleFriend POC - a new tool to automate, find, and abuse existing delegation between GCP and GWS.
Let’s see how the POC tool works:
Validating access token combinations using tokeninfo
|
You can find the source code in our GitHub repository: https://github.com/axon-git/DeleFriend |
After discovering DeleFriend and completing our tool's research and development, we came across Chris Moberly’s research, “GCP Privilege Escalation," which briefly mentioned the exploitation of DWD through a GCP UI console access, allowing the creation of a Super Admin account. This insightful blog could have greatly expedited our research process, as it depicts the same concept that we have identified. However, we're pleased that our in-depth exploration of the DWD feature allowed us to develop a programmatic and systematic method to identify Service Accounts with delegation. After looking into Moberly’s research, we could not find any parameter in the UI that hints at whether a service account enables DWD (it is possible this feature has been deprecated). But, Chris’s blog is a highly recommended read for valuable insights for GCP red teaming and we would like to take the opportunity to credit him for being the first that publish materials around DWD exploitation.
Cloud vulnerabilities, or “attacking abuse techniques,” have a completely different concept from old school on-premise vulnerabilities we’ve been used to. Most of them are based on a design flaw of a feature, rather than a logical code error in the application itself. The same is with DeleFriend.
Since the root cause is part of the design of the feature, we put together a couple of suggestions for improving the feature, which were also introduced to Google as part of our vulnerability bug report. We think this section is particularly interesting, as it shows the difficulty of fixing the “new era” of over-permissive features and the tradeoff that sometimes comes with it. Please note that these recommendations are related to the design of the feature, and not practical recommendations for users and organizations. (see the “Postures & Hygiene” section for that)
Our recommendation would be to consider updating the delegation requirements to be based on a specific predefined private key(s), and not for the whole service account id. We understand this design was probably meant to provide flexibility with dynamic key creation to the service accounts, however, it opens a door for lateral movement attack methods from GCP to Workspace, and abuse of existing delegations without the need for Super Admin on the domain. Granting GCP IAMs with the singular permission of serviceAccountKeys.create, which is also included in common roles like Editor and Owner, can enable the takeover of existing delegations and, consequently, control over the entire Workspace. This effectively equates these roles to having Super Admin permissions within Workspace, the highest level of permission available. This is a flaw in the permission structure that needs to be rectified.
While disabling the whole design of delegation per OAuth ID might impact the feature capabilities, we suggest implementing at least an optional requirement to configure Domain-Wide delegation by a specific private key. We expect the optional requirement feature will be highly used by organizations as our research found that organizations mostly don’t generate keys dynamically for delegations but use the same private key for the whole lifetime of the service/application.
While generating a new JWT for each OAuth scope is designed to control different REST API accesses, we recommend implementing a limit on the number of JWT requests within a short time frame. This would curtail or at least make it more challenging to identify authorized Domain-Wide delegation combinations by a newly generated or stolen private key pair.
We suggest considering the removal of the iam.serviceAccountKeys.create permission from the Editor basic role in GCP. While the Editor role is meant for high privileges within the project, this permission can potentially lead to Domain-Wide delegation abuse. Even though setting up Domain-Wide delegation needs Super Admin rights, it equates the Editor role to the same level.
Domain-wide delegation abuse is a powerful attack vector, so let’s list the advantages that this backdoor brings to attackers.
Before delving into threat-hunting examples, it's important to first review the GCP and GWS data sets to determine what elements are key for our examination. Understanding the structure of the logs, as well as how GCP and GWS are interconnected, is vital for effectively carrying out threat hunting and posture assessment.
When an application or a delegated service account performs an API call on behalf of another user, an application event of the type token, will be audited. This event represents OAuth token activity made by identity objects in the Workspace domain. When the application needs to authorize in order to receive a temporary access token, an EVENT_NAME value of authorize will be logged, while actual API calls will be logged with the value activity.
For each of the token events, the target user on which the activity was performed on their behalf, will be logged under the attribute ACTOR_EMAIL. This can be highly confusing for security investigators, as without knowing what token events are, they might think the activity was done by the user value in ACTOR_EMAIL although the activity was made by delegated OAuth context.
So, how to identify what is the identity object which initiated the API call? A hidden attribute inside the EVENT_PARAMTERS JSON name app_name will include the delegated object which performed the action on behalf of the target identity. Alternatively, the attribute client_id can be used either in cases where the application is configured with an OAuth name.
The value admin in ID_APPLICATION_NAME, as the name suggests, represents events made in the context of GWS administration. It can be either for the activity made from the Admin Console interactively or via the admin API.
When a new global delegation is created or modified, an event name AUTHORIZE_API_CLIENT_ACCESS will be logged under the admin log source. This event represents global OAuth access to an identity object. Global means this event won’t be created for OAuth consent of a specific user (for example, a user that installed a specific application and consented to his account), but for domain delegation which affects the entire domain and the identities within it.
As we’ve discussed, Google allows two main options for identity objects: GWS Marketplace applications and GCP Service accounts.
So how do we identify them in the logs?
Under the EVENT:PARAMTERS JSON, there is a key name API_CLIENT_NAME. This value represents the delegated object name that is assigned to the domain.
From here, we’ve identified two patterns that are matching two identity objects we’ve presented above.
1. GWS Marketplace applications will be declared with a naming convention ending with apps.googleusercontent.com. This domain convention is associated with GWS marketplace applications and serving as a unique identifier for applications.
GWS Marketplace Application audited
In case you would like to understand what the exact application is, the first 12 integers in the UID, can be correlated with the last directory within the relevant Marketplace application URI, for instance, 7081045131.apps.googleusercontent.com.
Application identifier from the Marketplace URI
2. GCP Service Accounts are declared with the OAuth Service Account ID, which is made up of 21 integer values. It is important to note that the Service Account ID is the only identifier that exists in this particular event, as the IAM email address isn’t available.
Interestingly, during our research, we found that Google doesn’t provide the initiating caller IP of Domain-Wide delegation events for GCP service account objects. The IP value is usually audited under the attribute IP_ADDRESS, which exists in almost all of the events in the Workspace schema. As we couldn’t think of any reasonable reason for that, we addressed it to Google. This forensic insight is highly important, as the IP address is one of the most valuable attributes for investigation purposes, especially in this kind of event, as we want to understand the characteristics of the identity that was created by the global delegation.
Global delegation events with GCP SA identity objects, audited without initiating IP caller value
As we’ve examined in the GWS logs, the events that represent activity made by GCP Service Accounts included only a 21-integer number. During threat hunting or incident investigations, it is highly important to understand forensic data point connections between the data sources, as at first impression, this integer represents nothing to us.
First, it is important to understand what this 21-integer number means. This number represents the OAuth ID of the target Service Account. A mandatory and globally unique value that Google attached to every Service Account that is created in the GCP domain.
In order to pivot and find the SA IAM identity behind the OAuth ID, we can do a couple of steps by examining the GCP audit logs, and by that getting a full picture of the potential event or incident. The main purpose is to find important information that isn’t included in the Workspace log, in order to enrich the investigation and provide more context.
The most recommended method to understand more about the SA is to pivot to the relevant GCP audit logs, specifically for private key creation to the relevant service account. Those logs can be identified under the GCP Event Name CreateServiceAccountKey.
The following diagram represents the connection between the two attributes in the different data sources.
Overview
The thesis looks for the creation or update of domain delegation event AUTHORIZE_API_CLIENT_ACCESS in Google Workspace admin logs and focusing on GCP service account identities by identifying the target application as an integer.
Given that the AUTHORIZE_API_CLIENT_ACCESS event in Google Workspace doesn't contain details on the target service account rather than the OAuth client ID, the thesis performs a cross-correlation with the related GCP events to gather information on the relevant GCP Service Account details, such as the related GCP project, and private key creation events on the target identity object.
Relevant Data Sources
Blind Spots
Investigation Flow
→ SQL Query can be found here.
Overview
The thesis looks for suspicious OAuth API calls. Namely, detecting anomalous token events with a combination of an unseen API call and a target Workspace user made by a GCP Service Account Identity object. Please note that it is recommended that this thesis be used with UEBA or ML algorithms- the provided is a proof of concept implementation using SQL.
Relevant Data Sources
Blind Spots
Investigation Flow
→ SQL Query can be found here.
Overview
The thesis looks for extensive service account private key creations over a short period of time which can indicate an automatic attacking activity to find Service Accounts with DWD functionality. This thesis is specifically based on Delefriend characteristics.
Relevant Data Sources
Blind Spots
Investigation Flow
→ SQL Query can be found here.
The importance of managing inactive delegated service accounts in GCP has been underscored in this paper, particularly after demonstrating how a single dormant service account can lead to a complete takeover of the whole Workspace domain.
So, how can we prevent this? To help identify inactive delegated service account entities in your GWS environment, we've developed a posture query that analyzes Workspace audit logs.
The logic will work as follows:
Guidelines
In the explanatory sections, we've studied the naming convention for GCP SA private keys upon creation. Once a key pair has been created, it is automatically downloaded and will not be retrievable afterward. Neglecting a GCP SA key pair can create a significant security posture gap. Hence, it is advised to actively search for and expose any unused pairs that might be stored insecurely in workstations and servers.
The naming structure of the GCP SA key pair includes the GCP Project ID that the Service Account is associated with, and a randomly generated string of 12 characters:
<gcp_project_id>-<private_key_id{12}>.json
By cleverly utilizing and cross-correlating GCP audit logs and EDR data, we are able to reliably hunt for file events associated with GCP SA key pairs. We’ll leverage the project names from the GCP audit logs in order to make the search as efficient and accurate as possible and search them in the form of the naming convention in the EDR data.
→ The query can be found here.
Guidelines:
In today’s blog post, we drew attention to the design vulnerability in Google Workspace. The concern was rooted in an overly permissive design of the Domain-Wide Delegation feature, posing a risk of unauthorized domain identity access. This design flaw and research paper were responsibly reported to Google in advance as part of the “Bug Hunters” program in August 2023. As of this publication, the flaw remains active. To support organizations in understanding and managing the risks associated with this technique, we’ve provided a proof-of-concept tool and a detailed technical breakdown of our discoveries.
We wrapped up with the "Let’s go hunting" section, detailing threat detection methods and offering recommendations to safeguard against Domain-Wide Delegation attack methods. We are sharing this knowledge and recommendations with the broader community to ensure that any organization is able to safeguard against Domain-Wide Delegation attack methods and we encourage everyone to share with potentially affected parties.
Aug 7, 2023 – Hunters discloses the vulnerability to Google.
Aug 7, 2023 – Google initially responds, identifying the vulnerability as “Abuse Risk”.
Oct 31, 2023 – Google accepts the DeleFriend report.
Important: The DeleFriend POC tool was created as a proof-of-concept tool to increase awareness around OAuth delegation attacks in GCP and Google Workspace and to improve the security posture of organizations that use the Domain-Wide-Delegation feature. DeleFriend POC tool should be used solely for authorized security research purposes. This tool is provided “as is” and Hunters disclaims any and all warranties and liabilities regarding the use/misuse of this tool. Use responsibly.
To stay updated on threat hunting research, activities, and queries, follow Team Axon’s Twitter account (@team__axon).
HUNTERS © 2023 All rights reserved
The Hunters and Axon Team trademarks, service marks and logos used herein are owned by Cyber Hunters Ltd. All other trademarks used herein are the property of their respective owners.