Run cloud native application anywhere with Azure Arc enabled Kubernetes

At the beginning of this year Microsoft released Azure Arc enabled Kubernetes, this enabled you to attach conformant Kubernetes clusters to Azure for management. With a new preview release Microsoft introduced the capabilities of running Azure application services on Kubernetes anywhere you want. This could be on a Kubernetes cluster on the google cloud, AWS, on-premises or even on Azure it self.

The services that you are able to deploy in this preview are:

• Azure App Services
• Azure Logic Apps
• Azure Functions
• Azure Event Grid
• Azure API Management

This functions opens up a lot of new possibilities to developers and also opens up option for using other cloud platforms next to Azure.

How to get started

To get started you are required to have a running Kubernetes cluster, this can be on any cloud platform, on-premises or even on your laptop. For this example I have used a local Kubernetes cluster running on my laptop with the use of Docker for Windows and a Kubernetes cluster within one of my Azure subscriptions.

To be able to deploy app services within Kubernetes we have to perform two main tasks: Connect the Kubernetes cluster to Azure Arc and create a custom location within the Azure platform.

To perform these steps we need the following prerequisites:

• az cli version 2.16.0 or later (you can upgrade the cli via the below command)

az upgrade

• Latest version of Helm

choco install kubernetes-helm

• Kubernetes context connected to the Kubernetes cluster you want to connect to
• az cli extensions: connectedk8s, k8s-extension, customlocation

az extension add --name connectedk8s
az extension add --name k8s-extension
az extension add --name customlocation

Custom locations for Azure services are only supported in the below regions at the time of writing this article:
- East US
- West Europe

Connect Kubernetes cluster to Azure Arc

To connect the Kubernetes cluster to Azure Arc specific resource providers need to be registered on the subscriptions were you want to deploy Azure Arc to. The required resource providers are: Microsoft.Kubernetes, Microsoft.KubernetesConfiguration, Microsoft.ExtendedLocation.

These registration can be done via the Azure Portal but in this article we will use the Azure CLI to perform all the steps.

az provider register --namespace [provider]

Perform the command for all three providers and wait for the registration to finish. In total this could take up to 30 minutes to complete. The status can be monitored by using this command.

az provider show -n [provider] -o table

Azure Arc it self is a Azure resource that needs to be deployed to an Azure Resource group. That is why we will also create a resource group.

az group create -n [group name]-l [location] -o table

Next step is to connect the Kubernetes cluster to Azure Arc. In this step all required resources will be deployed to Kubernetes and Azure were the Azure Arc resource will be created.

az connectedk8s connect -n [Azure Arc Instance name] `
-g [Resource group name]

This command will initiate a new deployment to the Kubernetes cluster via Helm and as you see in the screenshot the deployment done there was to my AKS cluster. Were it is specified that you only have to connect to Azure Arc for enabled services because Azure Monitor and Defender are natively support within AKS.

When this deployment is succeeded the cluster is connected to Azure Arc. In the below screenshots you can see the differences in deployments on the cluster before connecting to Azure Arc and after connecting it to Azure Arc.

Before

After

Adding custom location to Azure

With the Kubernetes cluster connected to Azure Arc a custom location can be added. Adding this location is done in two easy steps by using the extensions on the Azure CLI.

Enable custom locations

To use custom locations the functionality has to be activated on the cluster it self. So make sure u are connected to Azure and still have the active context to the Kubernetes cluster that you would like to connect.

az connectedk8s enable-features -n [cluster name] `
-g [Resource group name] `
--features cluster-connect custom-locations

After enabling the feature the custom location can be activated for three different service instances:

• Azure Arc enabled Data Services
• Azure App Service on Azure Arc
• Event Grid on Kubernetes

Each of these have there own specific command for the activation:

Azure Arc enabled Data Services

az k8s-extension create --name [name for the extension] `
--extension-type microsoft.arcdataservices `
--cluster-type connectedClusters `
-c [name of the Kubernetes cluster] `
-g [resource group name ARC] `
-scope cluster
--release-namespace arc `
--config Microsoft.CustomLocation.ServiceAccount=sa-bootstrapper

Azure App Service on Azure Arc

az k8s-extension create --name [name for the extension] `
--extension-type microsoft.arcdataservices `
--cluster-type connectedClusters `
-c [name of the Kubernetes cluster] `
-g [Resource group name ARC] `
--scope cluster `
--release-namespace arc `
--config Microsoft.CustomLocation.ServiceAccount=sa-bootstrapper

Event Grid on Kubernetes

az k8s-extension create --name [name for the extension] `
--extension-type Microsoft.EventGrid `
--cluster-type connectedClusters `
-c [name of the Kubernetes cluster] `
-g [Resource group name ARC] `
--scope cluster `
--release-namespace eventgrid-ext `
--configuration-protected-settings-file protected-settings-extension.json `
--configuration-settings-file settings-extension.json

The scripts shown above are all standard scripts to get started with the extension. If you for example want to deploy Azure App Services additional configuration is required. To start using App Service try the following command to configure the extension.

az k8s-extension create --resource-group [resource group name] `
--name [name for the extension] `
--cluster-type connectedClusters `
--cluster-name [cluster name] `
--extension-type 'Microsoft.Web.Appservice' `
--release-train stable `
--auto-upgrade-minor-version true `
--scope cluster `
--release-namespace [namespace] `
--configuration-settings "Microsoft.CustomLocation.ServiceAccount=default" `
--configuration-settings "appsNamespace=[namespace]" `
--configuration-settings "clusterName=[cluster name]" `
--configuration-settings "loadBalancerIp=[static ip of cluster]" `
--configuration-settings "keda.enabled=true" `
--configuration-settings "buildService.storageClassName=default" `
--configuration-settings "buildService.storageAccessMode=ReadWriteOnce" `
--configuration-settings "customConfigMap=[namespace]/kube-environment-config" `
--configuration-settings "envoy.annotations.service.beta.kubernetes.io/azure-load-balancer-resource-group=[kubernetes infra resource group name]"

With the extension created some information needs to be retrieved to be able to do the final step. For the final step the resourceId of the Azure Arc enabled Kubernetes cluster is required and the same for the extension that has been activated on the cluster

Resource Id connected cluster

az connectedk8s show -n [Azure Arc enabled cluster] `
-g [Resource group name]  `
--query id -o tsv

Resource Id extension for connected cluster

az k8s-extension show --name [name of the extension] `
--cluster-type connectedClusters `
-c [Azure Arc enabled cluster] `
-g [Resource group name]  `
--query id -o tsv

With this information retrieved the cluster can be defined as a custom location:

az customlocation create -n [location name] `
-g [Resource group name] `
--namespace [namespace in Kubernetes] `
--host-resource-id [cluster id] `
--cluster-extension-ids [extension id]

Things to be aware of when you process the above command:

• Per connection a different namespace needs to be defined.

Next Step

After following the above steps your cluster is connected to Azure Arc and should be available as a custom location. The existing custom location can be checked with the below command.

az customlocation list -o table

These location can also be found by opening the Azure Arc blade within the Azure portal and clicking on custom locations.

As shown in the screenshots four different locations are added to Azure. Two to a local cluster installed on my own laptop and two for the Kubernetes cluster hosted in Azure.

Enable Azure App Services in Kubernetes

Creating services in the specified location requires you to have Azure App Services Kubernetes environment. By creating this environment the cluster is enabled to receive new request for app services.

az appservice kube create --resource-group [resource group] `
--name [name for the environment] `
--custom-location [Id of the custom location] `
--static-ip [static ip used in the extension installation]

The custom location id used in the above command can be retrieved with the following script.

$customLocationId=$(az customlocation show --resource-group [resource group name] `
--name [location name] `
--query id `
--output tsv)

Note: If the "az appservice kube" command does not work make sure you install a defined version with the following script.

az extension remove --name appservice-kube
az extension add --yes `
--source "https://aka.ms/appsvc/appservice_kube-latest-py2.py3-none-any.whl"

When done all kind of new services will have appeared in the resource group used during the connection.

Creating resources on the Kubernetes environment

The Kubernetes environment is now prepared for new Azure App Services. These services can be created through the portal or via the Azure CLI.

Azure CLI

When using the Azure CLI the custom location id needs to be supplied when creating the hosting plan and the web application for example.

az appservice plan create -g [resource group name] `
-n [name hosting plan] `
--custom-location [custom location id] `
--per-site-scaling `
--is-linux `
--sku K1

az webapp create --plan [name hosting plan] `
--resource-group [resource group name] `
--name [web app name] `
--custom-location [custom location id] `
--runtime [runtime]

Azure Portal

On every location were you are able to use custom location the location will appear on the locations drop down.

More to learn

On docs.microsoft.com a lot more articles and information can be found about this subject. The below list points you to some interesting articles:

• Azure Arc enabled Kubernetes
• Create and manage custom locations on Azure Arc enabled Kubernetes
• Create a web app on Azure Arc - Azure App Service
• App Service on Azure Arc - Azure App Service