Hunters Discovers a Technique that Allows an Attacker to Obfuscate their Use of Stolen Credentials Inside AWS


This blog post was reviewed for technical accuracy by AWS Security

Amazon Virtual Private Cloud (Amazon VPC) is a service that lets customers launch AWS resources in a logically isolated virtual network they define. Customers have complete control over their virtual networking environment, including the selection of their IP address range, creation of subnets, and configuration of route tables and network gateways.

In its effort to identify and detect Attackers’ Techniques across the enterprise: on endpoints, on-premises, and cloud, the Hunters team researched a technique using a VPC feature that allows customers to control their IP address. An attacker could use this to control the IP address written to AWS CloudTrail logs when accessing a compromised account. This can potentially enable an attacker to fool various security protections that rely on the Cloudtrail logs, such as SIEMs and cloud security tools. These include, but are not limited to GuardDuty, Rapid7, and Lacework. In addition, analysts looking for evidence of an attack might miss it.

This blog post reviews the obfuscation technique in technical detail and provides recommendations for actionable measures.

[ You can also watch it as a webinar here ]

 

Attacker control of CloudTrail SourceIP - Why is it Important?

By controlling their source IP address, attackers can hide, potentially bypassing security measures that rely solely and specifically on AWS CloudTrail.

Consequently, it would be possible to make malicious actions look innocent, by making them look as if they were performed by legitimate entities, and it would be very hard to detect attack traces and attribute the intrusion to a specific source.

If your security tools rely solely on the sourceIP field within AWS CloudTrail logs, we recommend you follow the mitigation recommendations below.

Technical Deep Dive

To successfully execute this technique, attackers need to gain access to legitimate AWS credentials from the victim account, then create VPC endpoints (a feature that enables direct communication between EC2 instances in your VPC and AWS services) within their own account, and create a VPC with IP ranges which could obfuscate their traffic when they used the compromised credentials. Below are key features that we need to describe in order to explain how the technique is used:

  1. CloudTrail logs have a "sourceIPAddress" field. This field contains the attacker's IP address, in most cases a public IPv4 address.

  2. However, if an attacker makes an AWS API request within an AWS VPC, through a VPC Endpoint, the IP address logged in the victim’s CloudTrail logs is the "internal" EC2's IP address in the VPC - an IP address the attacker can control.

  3. Customers can set any IPv4 address range, including RFC 1918 or publicly routable IP ranges, within a VPC. Publicly routable IP blocks are only reachable via the Virtual Private Gateway and cannot be accessed over the Internet through the Internet gateway. This allows an attacker to create an IP addressing scheme that mimics a victim’s own network or a reputable external service.

  4. Another important fact is that the userAgent field logged in CloudTrail, which shows the attacker's userAgent string, is entirely controlled by the attacker making the API calls (by design). This is not directly part of the technique, however, when utilized it can help to further make the impersonation process look legitimate.

Example Reproduction

We created a VPC with a private IP CIDR block of our choice (12.34.56.0/24) in our private AWS account (attacker account "A").

Attacker’s VPC

Then, we created an EC2 instance in this VPC, and picked its private IPv4 address to be 12.34.56.78.

Attacker machine with “external” private IPv4 address

We generated API credentials in the victim’s account (“B”) as the credentials that were compromised by the attacker prior to the usage of the technique.

Victim’s AWS account with stolen API credentials

We then configured these “stolen” credentials in the ~/.aws/credentials file on the attacker EC2 instance so that AWS CLI calls made from the instance within the attacker-controlled Account A would use them.

Next, we created VPC endpoints in the attacker account for the services we intended to make API calls to.

We picked a subset of 44 of the most important services out of the 125 AWS services for which VPC endpoints can be created, and for each one we chose a single API call in that service which does not require any specific parameters.

To verify the technique, we executed a test. For comparison, we first performed API calls to the services from the EC2 instance without tunneling it through VPC endpoints. These calls showed up in the CloudTrail logs with the public IP address of the instance as the source IP.

In the next step, we created VPC endpoints for the services mentioned above and re-ran the AWS CLI commands in each service, this time through the VPC endpoints.

Attacker’s AWS account, using the stolen credentials

The test was successful, showing the “internal” IP address selected by the attacker as the source IP in the CloudTrail logs

Obfuscated source IP address in CloudTrail logs

Ways Attackers can Use this Behavior

Attackers can obfuscate their IP address for many different purposes - here are some examples:

 

Attacker chosen IP

Why would an attacker choose it

"Organizational" public IP address

 

 

 

If the attacker gathers information about the victim organization's public internet infrastructure, the attacker can use a public IP that corresponds with some part of its public infrastructure.

Example: the external VPN IP address of the company.

Employee “home” external IP address

 

 

 

If the attacker obtains intelligence on the external IP address an employee uses when away from the office, they can choose that IP, which can falsely direct any investigation at the “innocent” employee.

Example: the home IP address of a cloud IT administrator.

3rd party service provider public IP address

 

 

 

 

 

 

 

An attacker can use a known public IP address belonging to a 3rd party service provider in the AWS customer environment, thus increasing the chance of evading threat Intelligence and reputation services.

Since sometimes these IPs may be whitelisted by AWS customers, the attacker could even perform an even larger volume of sensitive API calls.

Example 1: using an IP address belonging to an enterprise cloud security product (preferably one that the victim organization uses).

Example 2: using 198.20.69.74 from which the popular internet-scanning Shodan scans the entire internet.

A special private, reserved, testing or documentation-only IPv4 subnet block

 

 

There are 4-5 subnets that are defined as “reserved IPs” for ad-hoc purposes. Using one of them can also help evade reputation services. A full list is available here.

Example: using an IP address in the TEST-NET subnets (192.0.2.0/24, 198.51.100.0/24 or 203.0.113.0/24).

 

Which AWS Services Support VPC Endpoints Access?

This technique can only be used to make API calls to AWS services for which VPC endpoints can be created. However, as of May 2021, VPC endpoints can be created for 125 different AWS services, including most of the prominent AWS services (e.g. EC2, S3, Lambda, STS, DynamoDB, Secrets Manager, and more), with the only significant exclusion we've seen being IAM.

Additionally, AWS is continuously expanding the list of VPC-endpoint-supported services. The full list of supported services can be seen in the AWS portal.

Additional Information:

  • The technique does not affect the actual IAM permissions the attacker has when using victims’ compromised AWS API credentials. They can only make the API calls that the compromised credentials have IAM permissions for.
  • When this technique is used, the attacker-controlled source IP is the one logged both when viewing CloudTrail Event history in the CloudTrail service in the AWS Management Console and in the events written by CloudTrail in JSON files to S3 as well.
  • It is impossible to identify the attacker as external to the organization based on the logged IP alone, nor is it possible to determine their true origin. It is, however, possible to determine that the involved VPC does not belong to the organization.

Can this be Used to Bypass IP-based IAM Policies?

We tested whether we can use this technique to bypass IAM source-IP-based policies (usage of the aws:SourceIp key). The test assumes that, in some instances, organizations configure such policies, only allowing certain sensitive AWS actions from specific subnets.

Under this assumption, we tested whether, in such cases, if an attacker creates a VPC with a corresponding "internal" IPv4 CIDR block and uses VPC endpoints, they can succeed in calling the sensitive AWS APIs, thus "bypassing" such policies.

We discovered (as is documented) that when API calls are done through VPC endpoints, the aws:SourceIp key is not available (instead, the aws:VpcSourceIp is available), thus this technique cannot be used to bypass such IAM policies.

How to Detect such activity in your Environment:

In general, when AWS API calls are done through VPC endpoints, the optional vpcEndpointID CloudTrail field is populated (with the ID of the VPC endpoint) - as opposed to all other calls, which do not populate this field. This is the only remnant of usage of this technique in the logs.

Given that, our first recommendation is reviewing the vpcEndpointID field.

Detecting an attacker who uses an IP address that is not in any of the private IP ranges you are using in your VPCs is relatively easy. This can be done simply by looking for any AWS API calls performed through a VPC endpoint (with a populated vpcEndpointId field) and a source IP address that is not within the private IP ranges used in your VPCs.

 

-- we try converting the source IP address to an integer to filter out specific subnets

SELECT parse_ip(sourceIpAddress, 'INET', 1):ipv4::int as source_ip_address_as_int,

           *

 FROM cloudtrail

WHERE vpc_endpoint_id IS NOT NULL

  AND source_ip_address_as_int IS NOT NULL

  AND source_ip_address_as_int NOT BETWEEN 167772160 AND 184549375 -- the lower and upper bounds of 10.0.0.0/8

  AND source_ip_address_as_int NOT BETWEEN 2886729728 AND 2887778303 -- bounds of 172.16.0.0/12

  AND source_ip_address_as_int NOT BETWEEN 3232235520 AND 3232301055 -- bounds of 192.168.0.0/16

  AND user_agent NOT LIKE '%.amazonaws.com' -- remove all AWS-service invoked API calls

Example SQL detection query

However, detecting an attacker who uses an IP address that is also a valid private IP address in any of your VPCs, is significantly harder. This is because if you use VPC endpoints in your environment, the only significant difference between the logs created by legitimate actions and the attacker’s actions is the specific VPC endpoint IDs logged. We recommend addressing this use-case with more anomalous-based detection logic, detecting usage of a new VPC endpoint ID never seen before in the organization

Finally, we recommend AWS CloudTrail users to cross-reference their cloud events with other sensors on endpoints, on-premises, email, identity, and more. Consolidated, automatic detection across surfaces is the most effective way to point out and trace inconsistent logging and missed threats.

 

Watch this technique reproduction in our webinar here.

Listen to "Hands-On Security" podcast episode about the technique here.