AWS API access key IDs are not Unique and Could Repeat
Executive Summary
Hunters researchers have discovered that temporary Amazon Web Services (AWS) API access key IDs, issued by the Identity and Access Management (IAM) service to users or services requesting temporary credentials (e.g. when assuming roles), are not unique and could repeat.
This means that multiple entities can receive and use separate temporary credentials that share the same access key ID, often at the same time. Additionally, the session token, the only possibly-non-sensitive information that differentiates sets of credentials from each other, is not being logged in CloudTrail.
While credentials generated with the same access key ID operate separately on the permissions level and attackers can't use this behavior to obtain privileges that they don’t already have, this can, in some situations, confuse or even impair the detection, investigation, and forensic analysis capabilities of AWS security tools and methods which rely on CloudTrail logs.
In this blog post we explain the problem in detail and provide mitigation suggestions (from both Hunters and the AWS IAM team) for preventing these situations from impacting your CloudTrail-utilizing capabilities.
What are AWS API Access Keys?
Almost every action in AWS requires an API access key. Are you using AWS Command Line Interface (CLI) to see which buckets are in your account? That requires an access key. Assuming a role? That also requires an access key, and in turn generates a new access key with the requested permissions. Logging into the AWS web console and creating an EC2 instance? Behind the scenes, the web console generates access keys for you on-the-fly to make all those API calls.
With the exception of anonymous access (e.g. to S3 buckets) and API calls invoked by AWS services on your behalf, all API calls require users to provide access key credentials which should have the permissions required for making the desired calls.
Permanent and Temporary Access Keys
AWS generates several types of access keys, but they can be split into two main types: permanent access keys and temporary access keys.
Permanent access keys can only be generated for IAM users. They consist of an access key ID and a secret access key. These allow access to all the resources and API calls the user has permissions for, and they remain valid until they are manually revoked. For this reason, it is against AWS best practices to use permanent API access keys.
Temporary access keys are part of temporary security credentials, and can be requested by using 5 common API calls (AssumeRole, GetFederationToken, GetSessionToken, AssumeRoleWithSAML and AssumeRoleWithWebIdentity) and some other less common ones (see this SalesForce git repository for more information). Like permanent access keys, temporary security credentials consist of an access key ID and a secret access key, but also contain an additional session token, which "lets AWS verify that the temporary security credentials are valid."
Temporary Access Keys Technical Drill-Down
Let's dive deeper into the structure and data of temporary security credentials, specifically those generated when assuming a role -- and see if they have any unique identifier.
When calling AssumeRole, the response contains two objects: An "AssumedRoleUser" object and a "Credentials" object.
- The "AssumedRoleUser" object has two fields: The Amazon Resource Name (ARN) of the temporary security credentials, and the assumed role ID.
- The ARN’s format is arn:aws:sts::<ACCOUNT_ID>:assumed-role/<ROLE_NAME>/<ROLE_SESSION_NAME>.
<ROLE_NAME> is simply the name of the role being assumed. <ROLE_SESSION_NAME> is an arbitrary string that the caller provides in the AssumeRole request. Thus it is clear that the assumed role ARN cannot uniquely refer to a specific assumed role session (or temporary set of credentials), because any AssumeRole on the same role and role session name will result in an identical ARN. - We might hope, then, that the assumed role ID might be a unique identifier of this specific set of temporary security credentials, as mentioned in AWS documentation - "A unique identifier that contains the role ID and the role session name of the role that is being assumed." However, this is not session-unique in any way. As mentioned earlier, the role session name is picked arbitrarily by the caller and the role ID is not session-unique - "The role ID is generated by AWS when the role is created."
- So if neither of these is unique, we are left with the AWS Security Token Service (STS) Credentials object. It has four fields:
Fig 1. The fields of temporary Credentials objects
Given all that we've seen so far, it would stand to reason to assume that access key IDs are unique.
Are Access Key IDs, in Fact, Unique?
During our research of AWS temporary credential mechanisms, we unexpectedly discovered several access key IDs that were each generated more than once, by multiple unique AssumeRole calls happening at different times.
Fig 2. A query that counts how many times each access key ID was seen being generated by AssumeRole calls in a two-day window - we see many key IDs that were generated twice
In effect, we saw the same access key IDs being “recycled” between different sets of temporary credentials.
This left us confused, since this behavior had not been documented. After searching deeper and realizing that this seemed to be happening in all AWS accounts we looked at, we started asking ourselves more questions:
Can access key IDs be recycled across different AWS accounts?
It appears not. Analyzing data from our customers with hundreds of AWS accounts showed that not a single temporary access key ID was recycled across different AWS accounts. All the observed multiple generations of the same access key ID were always occurring within a single AWS account.
The structure of access key IDs -- the format of the ID itself and the components that comprise it -- is not documented in AWS Documentation. However, we suspect that there exists a part of the access key ID that uniquely identifies the account ID to which it belongs - which might explain this result.
Can the same access key ID be generated for different ARNs?
At first, we only saw access key IDs being recycled between different assumed role credentials with the same ARN (same role name and same role session name). However, further analysis of the data revealed that access keys were also being recycled across different assumed roles, i.e. different role names and role session names.
What is the shortest time period observed for a regeneration of an access key ID?
At first, we only saw access key IDs being recycled many days or weeks after they were first generated, thus we suspected that access key IDs can only be recycled after the previous credentials with those key IDs expire.
Once we searched across months of data, we saw access key IDs that were being recycled after short time spans - some of them even being generated at the very same second for multiple sets of credentials!
Fig 3. A single access key ID generated in two different AssumeRole calls at same second
It Gets Weird - they Overlap
The two sets of temporary credentials generated (in Figure 3) had the same access key ID, at the exact same time -- thus they were concurrently valid during the same one-hour period. To make this clear: there were two different identities (or services), each using a different set of credentials, each with a different set of IAM permissions, but both with the same access key ID, simultaneously.
This meant that our previous suspicion was wrong, and access keys can be recycled whether or not there still exist valid credentials at that time that have the same access key ID.
This led us to ask follow-up questions:
When an access key ID is recycled, does its secret access key stay the same?
This is an important distinction because if it did, there might be some sort of “permission leakage,” with each set of credentials temporarily gaining the permissions of the other set of credentials.
As we recall, temporary API credentials not only consist of an access key ID and a secret key, but also of a session token (which does not exist in permanent access keys). In Figure 3, the two sets of credentials had different generated session tokens. This determined that there cannot be any permission leakage between the credentials.
Nonetheless, we wanted to also make sure that the secret access keys were different between sets of recycled access keys IDs. Given that secrets of any kind -- and specifically secret access keys generated for temporary API credentials -- do not appear in CloudTrail logs for security reasons, we could not verify this just by looking at the logs.
In order to get around this, we reproduced the recycling phenomenon ourselves, by writing a Python script that uses the Boto3 AWS Python SDK to make repeated AssumeRole calls, at a rate of 180 calls per second, and checks whether there are any recycled access key IDs within the sets of credentials generated.
Running the script for an hour resulted in three matching pairs of temporary credentials, each pair with an identical access key ID, different secret access key and different session token. We thus concluded that secret access keys do not stay the same for recycled access key IDs.
Fig 4. Recycled access keys, with different secret keys and different session tokens
Finally, we wondered:
Is there a limit to how many times access key IDs can be recycled?
If access key IDs were recycled once, what prevents them from being recycled again?
Indeed, when counting how many times each access key ID was being recycled, we discovered that in the majority of cases, keys were only being recycled once, but in rare cases some keys were being recycled more than once, a few of them even being recycled up to 5 times.
Fig 5. Access key ID recycling statistics in a large AWS environment, over a 1 year time period
Given all we’ve seen so far, we can conclude that an access key ID being recycled is simply a statistical phenomenon. The access key ID “space” is finite, they are generated from within this space at random, and there is a small chance of these random generations over time to match previously generated values.
Looking at how many times each key was being recycled in a large AWS environment with a large number of AssumeRoles (~2 billion) over a course of a year, it appears that the chance of any two random access key generations to result in the same key ID is around 1-3%.
So, the chance of a key being recycled 5 times in a year with this volume of keys might be something like 
Are permanent API access keys also being recycled?
We couldn’t find any permanent API access key IDs being recycled, but this might be because of the significantly lower volume of permanent access keys being generated compared to temporary access keys.
Communication with AWS
After finding this issue, we reported it to AWS Support. Initially, the response from AWS Support was that this shouldn’t be happening and that they had escalated it to the IAM service team. The IAM service team, however, subsequently responded that this is expected behavior:
"This is expected behavior. Due to the sheer amount of AssumeRoll calls made throughout AWS the ASIA**** access key identifier will eventually be generated again (even within the same time frame). Remember that to make an API call with temporary credentials you will have to supply an access key ID, a secret access key and a session token. Those three combinations are permanently unique. I understand that you cannot search CloudTrail via secret access key or session token, and that if an access key is reused it could cause some confusion in some cases.
Access key IDs are unique only in combination with the secret access key and the session token. Thanks to your research, this is now mentioned in our public document."
Fig 6. IAM team’s addition to the documentation
As the session token is the only possibly-non-sensitive information that tells the temporary credential sets apart from each other, we suggested to AWS that this be logged in every event in addition to the access key ID which is already being logged. This would allow searching for events with a given access key ID and session token, thus filtering out API calls done with other credentials that happened to have the same access key ID.
However, they responded that this was not possible:
"There are two reasons for not logging session tokens in CloudTrail:
- CloudTrail event sizes cannot exceed 256Kb. Including the session token would take up from that total size which can be better allocated for the event request/response parameters. This would also be an additional payload that CloudTrail has to process.
- More importantly, the session token is a very long string and including it in every API call would take up some of the size we allocated for events, which may result in events not sent to CloudWatch events/logs if they exceed 256Kb."
Bottom Line: What are the Implications?
As mentioned earlier, there is no permission leakage when access key IDs are recycled. This means that neither attackers nor insiders can use this behavior to their advantage. The likely impact, then, is on workflows, products or services that rely on CloudTrail logs for detection, investigation and forensic capabilities. In such cases, the effect may be temporary confusion or impairment of these capabilities.
For example: imagine a human security analyst or automated investigation service investigating a suspicious GuardDuty alert. The alert was generated on discovery behavior performed by an assumed role, belonging to some business-logic-related service (which we’ll call A), after the service has been successfully compromised by an attacker.
The analyst or investigation service attempts to determine what the attacker did by searching for the access key ID that appeared in the alert (belonging to A’s assumed role) in CloudTrail logs. Unbeknownst to them, that same access key ID was also generated in the same time window for temporary credentials belonging to a different service (which we’ll call B).
In the logs the analyst or investigation service receives, appear both the malicious actions of service A and the legitimate actions of service B. By chance, they go over B’s actions first, see they are benign, and determine the GuardDuty alert to be a false-positive. In this scenario, the attacker happened to be lucky and was not caught because of this issue.
What Can you Do About It?
You can’t actually prevent this behavior from happening - it will statistically and infrequently happen with random access keys IDs.
Having said that, there are two things you can do:
- Be aware of this behavior when working with CloudTrail logs. When filtering by specific access key IDs, also filter by the ARNs you see for those key IDs, in order to prevent seeing other identities with the same recycled keys. If you encounter events with other ARNs than the ones you expect to show up in a search, check whether it is because of this behavior and refine your search accordingly.
- As the AWS IAM team mentions in their response to this finding:
"It is recommended to use unique role session names to identify the user/role for a specific session, and you can even control what session name a user is allowed to use (when making AssumeRole API calls) using sts:RoleSessionName. In example 1 given in the referenced link, you can enforce IAM users to set their aws:username as their role session name when they assume an IAM role in your AWS account. This way you can easily identify a user's sessions in your AWS CloudTrail logs by searching for any Amazon Resource Name (ARN) with that user's aws:username as the role session name."
Following these best-practices will help prevent confusion in cases this behavior happens in your environment.
