Azure Web Apps – Troubleshooting external http/https calls using Kudu

I recently saw some strange issues with a customer’s Azure Web App where as part of the application they relied on an API call to an external source which was failing. This presented almost as a network routing issue to external resources with “connection timeout” errors.

Working with the Azure support team, we confirmed there was an issue (I’m still waiting the full details from Microsoft as to what..) on the underlying VM/machine.

In one case we resolved by scaling up and then scaling back down the Web App instance which made Azure migrate the Web App to a different underlying machine and in another case we had to escalate to a product team in Microsoft for them to manually migrate the Web App to a different machine as we kept being placed back onto the problematic machine.

At first we were unsure if this was an application/site issue, Azure issue or firewall issue at the 3rd parties side.

By using the Kudu tools in Azure Web Apps we can perform some basic diagnostics and testing to ensure that external http/https calls are possible from the underlying machine our PaaS site is running on.

For example, if we have an Azure hosted site, which requires to make lookup calls to an external API hosted by another provider this can be a useful testing/verification step in event of issues.

You can access the Kudu tools from the Azure portal:-

a

b

Or by navigating to https://abc.scm.azurewebsites.net where abc is the name of your Web App.

c

Choose Debug Console > CMD

d

To test a URL enter curl https://yourexternalurlhere.com and press enter

A successful http:// connection will display the raw HTML like this example –

e

You may sometimes see an SSL error when testing some https:// sites successfully this is often perfectly normal as CURL can’t read the cert fully.

f

If you have a DNS issue expect to see this error –

g

For an external URL you are having connectivity issues with, expect to see timeout errors like this –

h

AzureRM automation with service principals, certificates and Invoke-AzureRmResourceAction

I have been working on a project linked to deployment automation with Azure Web Apps.

Some of the challenges have been around the move from Azure Classic Mode (ASM) into Azure Resource Manager (ARM).

For example, in Classic Mode it was possible to do things like use a management certificate to automate remote commands. In ARM I have found this to be misunderstood and fraught with fragmented or out of date documentation (of which I have submitted some suggested edits back to Microsoft!).

In this post we will look at creating a mechanism to issue scripted commands to ARM by authenticating against a local certificate file into a “dummy” Azure Active Directory application and service principal name/account.

Pre-requisites – I believe you will need an Azure portal account with access at Owner subscription level and also Global Admin to your Azure Active Directory tenant. You’ll also need Azure Powershell (https://azure.microsoft.com/en-gb/documentation/articles/powershell-install-configure/) installed.

Getting Started

Launch an Azure Powershell session and run

Add-AzureRmAccount

Once prompted enter your Azure portal login details.

signin

Next we need to confirm you have access to the subscription and also check if you have access to more than one subscription

Get-AzureRmSubscription

getrm

If one subscription is returned use this command to proceed –

$tenant = (Get-AzureRmSubscription).TenantId

If multiple subscriptions are accessible, we need to select the subscription by using

$tenant = (Get-AzureRmSubscription -SubscriptionName “YOURSUBSNAMEHERE").TenantId

To verify this has captured the information use

$tenant

tenant

Creating your certificate

Windows 10/Server 2016 Only

$cert = New-SelfSignedCertificate -CertStoreLocation "cert:\CurrentUser\My" -Subject "CN=PUTYOURUSAGENAMEHERE" -KeySpec KeyExchange

To verify this use

 $cert

cert1

You can only use New-SelfSignedCertificate with Windows 10 and Server 2016.

Prior to Windows 10/Server 2016

If you are using an earlier version of Windows you will need to download the Self Signed certificate generator (https://gallery.technet.microsoft.com/scriptcenter/Self-signed-certificate-5920a7c6) and use the following commands

Import-Module -Name c:\PATHTODOWNLOAD\New-SelfSignedCertificateEx.ps1
New-SelfSignedCertificateEx -Subject "CN=exampleapp" -KeySpec "Exchange" -FriendlyName "exampleapp"
$cert = Get-ChildItem -Path cert:\CurrentUser\My\* -DnsName exampleapp

Retrieving the certificate key

Regardless of certificate creation method, we need to retrieve the key value from the certificate.

$keyValue = [System.Convert]::ToBase64String($cert.GetRawCertData())

Use

$keyValue

on its own afterwards just to check we have the data being pulled through.

getkey

Create your dummy “application” in Azure AD

For Azure Powershell version August 2016 and later –

$azureAdApplication = New-AzureRmADApplication -DisplayName "CHANGEME" -HomePage "https://CHANGEME" -IdentifierUris "https://CHANGEME" -CertValue $keyValue -EndDate $cert.NotAfter -StartDate $cert.NotBefore

If using Azure Powershell before August 2016 version

$azureAdApplication = New-AzureRmADApplication -DisplayName "CHANGEME" -HomePage "https://CHANGEME" -IdentifierUris "https://CHANGEME" -KeyValue $keyValue -KeyType AsymmetricX509Cert -EndDate $cert.NotAfter -StartDate $cert.NotBefore

Verify the new object is setup with

$azureADapplication

verifyapp

Creating the Service Principal

By passing in the ApplicationID of the Active Directory application we can create the Service Principal.

New-AzureRmADServicePrincipal -ApplicationId $azureAdApplication.ApplicationId

spn

Take note (in a safe/secure place) of these variables as we will need them later,

$cert.Thumbprint, $azureAdApplication.ApplicationId, $tenant

variables

At this point you have now created your certificate, dummy application and service principal.

Grant permissions to your resources

As the service principal is now configured, you will be able to grant permissions for it on your Azure subscription either via the Azure portal through the Access Control (IAM) screens like you would for any other Azure AD user or group.

iamiam2

Or using Powershell, for example

To add read access for the full subscription you can use: –

New-AzureRmRoleAssignment -RoleDefinitionName Reader -ServicePrincipalName $azureAdApplication.ApplicationId.Guid

Further details on RBAC options is available at https://azure.microsoft.com/en-gb/documentation/articles/role-based-access-control-configure/

Testing access

Assuming we have our permissions setup on the Azure objects we wish to manipulate, we can start a new/fresh Powershell session to test this out.

Start with this command filling in the values you kept safe earlier where 000000 is stated

Add-AzureRmAccount -ServicePrincipal -CertificateThumbprint 000000 -ApplicationId 000000 -TenantId 0000000

If all goes well your Azure Powershell session will be established and you will be presented with the subscription

test

Automating ARM actions

At this stage you can start issuing ARM commands (assuming you have assigned the relevant permissions to the service principal name against the resources in the subscription.)

A good real world example is automation of Web App slot swapping which we can do by issuing the following:

$ParametersObject = @{targetSlot  = "SLOTNAME"}
Invoke-AzureRmResourceAction -ResourceGroupName NAMEHERE -ResourceType Microsoft.Web/sites -ResourceName WEBAPPHERE -Action slotsswap -Parameters $ParametersObject -ApiVersion 2015-08-01

You can also add -Force at the end of this if placing in a script as this will remove the prompt to confirm action.

Using Invoke-AzureRmResourceAction you can introduce a lot of powerful automation even beyond built in commands as this can perform POST calls against the Azure REST APIs. There is an excellent blog post on this here – http://www.codeisahighway.com/howto-perform-post-call-to-azure-rest-api-using-invoke-azurermresourceaction-powershell-command/

I found it useful to use https://resources.azure.com to get the naming of resources and objects that are expected here. For example, you can navigate down to the Web App in the above example to confirm the naming of the slots.

resources

Exporting the certificate

Once you are comfortable with your automation tasks, you’ll more than likely want to roll this out to your build servers for example. To authorise these, you’ll need to install a copy of the certificate.

Start by exporting the certificate you created earlier on from the Certificate Manager on your machine.

Once exported, you will need to re-import the certificate to the destination machine as required. This is a well-documented standard certificate practice, for example – https://technet.microsoft.com/en-us/library/cc754329(v=ws.11).aspx