Azure “Unmanaged Disks” & TRIM

Unmanaged disks are the disks that most of us will be using with Azure IaaS VMs, e.g non SSD backed on an Azure Storage Account.

If you are using these “unmanaged” standard disks (HDD), you should enable TRIM. TRIM discards unused blocks on the disk so you are only billed for storage that you are actually using. This can save on costs if you create large files and then delete them.

You can run this command to check the TRIM setting. Open a command prompt on your Windows VM and type:

fsutil behavior query DisableDeleteNotify

If the command returns 0, TRIM is enabled correctly. If it returns 1, run the following command to enable TRIM:

fsutil behavior set DisableDeleteNotify 0

Azure Managed Disks GA & Backup In Preview

Azure Managed Disks became GA a few weeks ago and offer much simpler disk management for Azure IaaS VMs by managing the storage accounts associated with the VM disks. You only have to specify the type (Premium or Standard) and the size of disk you need, and Azure creates and manages the disk for you.

As of today Microsoft have also announced preview support for Azure Backup with Azure Managed Disk VMs – so not quite there for those of us using Azure Backup with production VMs but certainly time to get playing (unless you don’t need to use Azure Backup – in which case get going into prod!).

It is also great to see that Microsoft haven’t left the “how do we migrate” conundrum as they often have done in the past. They are providing a wealth of documentation on how to convert disks from a number of scenarios even including an Azure Classic VM conversion into ARM with Managed Disks – https://docs.microsoft.com/en-us/azure/virtual-machines/virtual-machines-windows-migrate-to-managed-disks

Benefits

Simple and scalable VM deployment

Managed Disks handles storage for you behind the scenes. Previously, you had to create storage accounts to hold the disks (VHD files) for your Azure VMs. When scaling up, you had to make sure you created additional storage accounts so you didn’t exceed the IOPS limit for storage with any of your disks. With Managed Disks handling storage, you are no longer limited by the storage account limits (such as 20,000 IOPS / account). You also no longer have to copy your custom images (VHD files) to multiple storage accounts. You can manage them in a central location – one storage account per Azure region – and use them to create hundreds of VMs in a subscription.

Managed Disks will allow you to create up to 10,000 VM disks in a subscription, which will enable you to create thousands of VMs in a single subscription. This feature also further increases the scalability of Virtual Machine Scale Sets (VMSS) by allowing you to create up to a thousand VMs in a VMSS using a Marketplace image.

Better reliability for Availability Sets

Managed Disks provides better reliability for Availability Sets by ensuring that the disks of VMs in an Availability Set are sufficiently isolated from each other to avoid single points of failure. It does this by automatically placing the disks in different storage scale units (stamps). If a stamp fails due to hardware or software failure, only the VM instances with disks on those stamps fail. For example, let’s say you have an application running on five VMs, and the VMs are in an Availability Set. The disks for those VMs won’t all be stored in the same stamp, so if one stamp goes down, the other instances of the application continue to run.

Better security

You can use Azure Role-Based Access Control (RBAC) to assign specific permissions for a managed disk to one or more users. Managed Disks exposes a variety of operations, including read, write (create/update), delete, export, and retrieving a shared access signature (SAS) URI for the disk. You can grant access to only the operations a person needs to perform his job. For example, if you don’t want a person to copy a managed disk to a storage account, you can choose not to grant access to the export action for that managed disk. Similarly, if you don’t want a person to use an SAS URI to copy a managed disk, you can choose not to grant that permission to the managed disk.

Azure Web Apps – Troubleshooting external http/https calls using Kudu

I recently saw some strange issues with a customer’s Azure Web App where as part of the application they relied on an API call to an external source which was failing. This presented almost as a network routing issue to external resources with “connection timeout” errors.

Working with the Azure support team, we confirmed there was an issue (I’m still waiting the full details from Microsoft as to what..) on the underlying VM/machine.

In one case we resolved by scaling up and then scaling back down the Web App instance which made Azure migrate the Web App to a different underlying machine and in another case we had to escalate to a product team in Microsoft for them to manually migrate the Web App to a different machine as we kept being placed back onto the problematic machine.

At first we were unsure if this was an application/site issue, Azure issue or firewall issue at the 3rd parties side.

By using the Kudu tools in Azure Web Apps we can perform some basic diagnostics and testing to ensure that external http/https calls are possible from the underlying machine our PaaS site is running on.

For example, if we have an Azure hosted site, which requires to make lookup calls to an external API hosted by another provider this can be a useful testing/verification step in event of issues.

You can access the Kudu tools from the Azure portal:-

a

b

Or by navigating to https://abc.scm.azurewebsites.net where abc is the name of your Web App.

c

Choose Debug Console > CMD

d

To test a URL enter curl https://yourexternalurlhere.com and press enter

A successful http:// connection will display the raw HTML like this example –

e

You may sometimes see an SSL error when testing some https:// sites successfully this is often perfectly normal as CURL can’t read the cert fully.

f

If you have a DNS issue expect to see this error –

g

For an external URL you are having connectivity issues with, expect to see timeout errors like this –

h

AzureRM automation with service principals, certificates and Invoke-AzureRmResourceAction

I have been working on a project linked to deployment automation with Azure Web Apps.

Some of the challenges have been around the move from Azure Classic Mode (ASM) into Azure Resource Manager (ARM).

For example, in Classic Mode it was possible to do things like use a management certificate to automate remote commands. In ARM I have found this to be misunderstood and fraught with fragmented or out of date documentation (of which I have submitted some suggested edits back to Microsoft!).

In this post we will look at creating a mechanism to issue scripted commands to ARM by authenticating against a local certificate file into a “dummy” Azure Active Directory application and service principal name/account.

Pre-requisites – I believe you will need an Azure portal account with access at Owner subscription level and also Global Admin to your Azure Active Directory tenant. You’ll also need Azure Powershell (https://azure.microsoft.com/en-gb/documentation/articles/powershell-install-configure/) installed.

Getting Started

Launch an Azure Powershell session and run

Add-AzureRmAccount

Once prompted enter your Azure portal login details.

signin

Next we need to confirm you have access to the subscription and also check if you have access to more than one subscription

Get-AzureRmSubscription

getrm

If one subscription is returned use this command to proceed –

$tenant = (Get-AzureRmSubscription).TenantId

If multiple subscriptions are accessible, we need to select the subscription by using

$tenant = (Get-AzureRmSubscription -SubscriptionName “YOURSUBSNAMEHERE").TenantId

To verify this has captured the information use

$tenant

tenant

Creating your certificate

Windows 10/Server 2016 Only

$cert = New-SelfSignedCertificate -CertStoreLocation "cert:\CurrentUser\My" -Subject "CN=PUTYOURUSAGENAMEHERE" -KeySpec KeyExchange

To verify this use

 $cert

cert1

You can only use New-SelfSignedCertificate with Windows 10 and Server 2016.

Prior to Windows 10/Server 2016

If you are using an earlier version of Windows you will need to download the Self Signed certificate generator (https://gallery.technet.microsoft.com/scriptcenter/Self-signed-certificate-5920a7c6) and use the following commands

Import-Module -Name c:\PATHTODOWNLOAD\New-SelfSignedCertificateEx.ps1
New-SelfSignedCertificateEx -Subject "CN=exampleapp" -KeySpec "Exchange" -FriendlyName "exampleapp"
$cert = Get-ChildItem -Path cert:\CurrentUser\My\* -DnsName exampleapp

Retrieving the certificate key

Regardless of certificate creation method, we need to retrieve the key value from the certificate.

$keyValue = [System.Convert]::ToBase64String($cert.GetRawCertData())

Use

$keyValue

on its own afterwards just to check we have the data being pulled through.

getkey

Create your dummy “application” in Azure AD

For Azure Powershell version August 2016 and later –

$azureAdApplication = New-AzureRmADApplication -DisplayName "CHANGEME" -HomePage "https://CHANGEME" -IdentifierUris "https://CHANGEME" -CertValue $keyValue -EndDate $cert.NotAfter -StartDate $cert.NotBefore

If using Azure Powershell before August 2016 version

$azureAdApplication = New-AzureRmADApplication -DisplayName "CHANGEME" -HomePage "https://CHANGEME" -IdentifierUris "https://CHANGEME" -KeyValue $keyValue -KeyType AsymmetricX509Cert -EndDate $cert.NotAfter -StartDate $cert.NotBefore

Verify the new object is setup with

$azureADapplication

verifyapp

Creating the Service Principal

By passing in the ApplicationID of the Active Directory application we can create the Service Principal.

New-AzureRmADServicePrincipal -ApplicationId $azureAdApplication.ApplicationId

spn

Take note (in a safe/secure place) of these variables as we will need them later,

$cert.Thumbprint, $azureAdApplication.ApplicationId, $tenant

variables

At this point you have now created your certificate, dummy application and service principal.

Grant permissions to your resources

As the service principal is now configured, you will be able to grant permissions for it on your Azure subscription either via the Azure portal through the Access Control (IAM) screens like you would for any other Azure AD user or group.

iamiam2

Or using Powershell, for example

To add read access for the full subscription you can use: –

New-AzureRmRoleAssignment -RoleDefinitionName Reader -ServicePrincipalName $azureAdApplication.ApplicationId.Guid

Further details on RBAC options is available at https://azure.microsoft.com/en-gb/documentation/articles/role-based-access-control-configure/

Testing access

Assuming we have our permissions setup on the Azure objects we wish to manipulate, we can start a new/fresh Powershell session to test this out.

Start with this command filling in the values you kept safe earlier where 000000 is stated

Add-AzureRmAccount -ServicePrincipal -CertificateThumbprint 000000 -ApplicationId 000000 -TenantId 0000000

If all goes well your Azure Powershell session will be established and you will be presented with the subscription

test

Automating ARM actions

At this stage you can start issuing ARM commands (assuming you have assigned the relevant permissions to the service principal name against the resources in the subscription.)

A good real world example is automation of Web App slot swapping which we can do by issuing the following:

$ParametersObject = @{targetSlot  = "SLOTNAME"}
Invoke-AzureRmResourceAction -ResourceGroupName NAMEHERE -ResourceType Microsoft.Web/sites -ResourceName WEBAPPHERE -Action slotsswap -Parameters $ParametersObject -ApiVersion 2015-08-01

You can also add -Force at the end of this if placing in a script as this will remove the prompt to confirm action.

Using Invoke-AzureRmResourceAction you can introduce a lot of powerful automation even beyond built in commands as this can perform POST calls against the Azure REST APIs. There is an excellent blog post on this here – http://www.codeisahighway.com/howto-perform-post-call-to-azure-rest-api-using-invoke-azurermresourceaction-powershell-command/

I found it useful to use https://resources.azure.com to get the naming of resources and objects that are expected here. For example, you can navigate down to the Web App in the above example to confirm the naming of the slots.

resources

Exporting the certificate

Once you are comfortable with your automation tasks, you’ll more than likely want to roll this out to your build servers for example. To authorise these, you’ll need to install a copy of the certificate.

Start by exporting the certificate you created earlier on from the Certificate Manager on your machine.

Once exported, you will need to re-import the certificate to the destination machine as required. This is a well-documented standard certificate practice, for example – https://technet.microsoft.com/en-us/library/cc754329(v=ws.11).aspx

Disqus, WordPress and Azure Web Apps – “Unable to connect to the Disqus API servers”

Having used Disqus on other sites, I wanted to add it on my blog as the comments system.

I am using WordPress/Project Nami on Azure Web Apps and had some issues getting Disqus setup.

Upon activating the WordPress plugin and following the Disqus documentation, the configuration wizard would fail with the error “Unable to connect to the Disqus API servers”.

Inside the function wp-content\plugins\disqus-comment-system\lib\api\disqus\url.php on line 55 there is a CURL call to fetch information from Disqus, which was failing on certificate/https validation.

I was able to workaround the issue by undertaking these steps –

  1. Download the cacert.pem file manually from https://curl.haxx.se/docs/caextract.html and place it in the lib\api\disqus\ folder

    (Tip – you can use Kudu with Azure Apps to easily upload the file – http://www.jamessturtevant.com/posts/How-to-add-edit-and-remove-files-in-your-azure-webapp-using-the-kudu-service-dashboard/)

  2. Around Line 55 in the api\disqus\url.php file add the following to the $c_options array
    CURLOPT_SSL_VERIFYPEER => true,
    CURLOPT_CAINFO => __DIR__ .’/cacert.pem’

    (You can edit using Kudu, but I edited this file using the new App Service Editor feature from the Azure portal as I wanted to try it out. The App Service Editor is a web based IDE based on the Monaco code editor which ships with Visual Studio Code)appservice

    Don’t forget the comma at the end of the existing CURLOPT_TIMEOUT => SOCKET_TIMEOUT – I’m not a dev guy so this caught me out initially!

curldisqus

Hopefully if you are suffering the “Unable to connect to the Disqus API servers” error this will help. I’m not sure if this is specific to Azure Web Apps or other web hosts too.