Installing in Alibaba Cloud Container Service for Kubernetes (ACK)
Alibaba Cloud Container Service for Kubernetes (ACK) integrates virtualization, storage, networking, and security capabilities. ACK allows you to deploy applications in high-performance and scalable containers and provides full lifecycle management of enterprise-class containerized applications. For more information about ACK, see the Alibaba Container Service for Kubernetes documentation.
There are a number of environment-specific steps that you must perform to install
Before you perform the environment-specific steps described below, ensure that you review and fulfill the general requirements listed in Common Kubernetes Prerequisites.
SAP does not support event broker service integration with service meshes. Service meshes include Istio, Cilium, Linkerd, Consul, and others. If deploying to a cluster with a service mesh, you must:
- exclude the
target-namespaceused by advanced event mesh services from the service mesh. - set up connectivity to event broker service in the cluster using LoadBalancer or NodePort. See Exposing Event Broker Services to External Traffic for more information.
The process to deploy event broker services in an ACK private data center is completed in collaboration with a SAP representative. The steps that you (the customer) perform are as follows:
- Create a Kubernetes cluster as described in the ACK Cluster Specifications. For customer-owned deployments, you are responsible for the set up of the Kubernetes cluster, and the maintenance and operation of the cluster.
- Customer-Controlled Regions in China require that you provide a custom domain name registered in China (for example,
*.mycompany.cn) with certificates issued for that domain. Event broker services are accessed using the custom domain name rather than*.messaging.solace.cloud.For more information, see Deployments in China.
- Customer-Controlled Regions in China require that you provide a custom domain name registered in China (for example,
ACK Cluster Specifications
Before you (the customer) install the Mission Control Agent, you must configure the ACK cluster with the technical specifications listed in the sections that follow.
- Alibaba ACK Prerequisites
- Considerations for Deployments in China Regions
- Planning Your Network Requirements
- VPC and vSwitch Requirements
- NAT Gateway Requirements
- Creating the ACK Cluster
- Node Pool Configuration
- Instance Types Requirements
- Storage Class Configuration for Autoscaling
- LoadBalancer Configuration
For more information about using Alibaba ACK, see the User Guide for Kubernetes Clusters on the Alibaba Cloud site.
When created with the following specifications, the ACK cluster has multiple node pools, and is designed to be auto-scaled when new event broker services are created. Each node pool provides the exact resources required by each plan to help optimize the cluster's utilization.
Alibaba ACK Prerequisites
The following are the technical prerequisites for an Alibaba ACK deployment to deploy event broker services:
- Permissions
- An Alibaba account with the following permissions is required to create an ACK cluster:
- Resource Access Management (RAM) and role-based access control (RBAC) must be activated and configured appropriately in the RAM console, see Authorization overview in the Alibaba Cloud documentation.
- Auto Scaling must be activated in the Auto Scaling console.
Considerations for Deployments in China Regions
Additional considerations are required if the private region you deploy to is within China.
-
Instead of the GCP registry, you must use the Azure China registry. The Azure China registry uses a different secret that's required when you later deploy the Mission Control Agent. For example, run the following command where the <username> and <password> is provided by SAP.
kubectl create secret docker-registry cn-reg-secret --namespace kube-system \
--docker-server=solacecloud.azurecr.cn --docker-username=<username> --docker-password=<password>
Planning Your Network Requirements
When creating your ACK cluster, you must specify CIDR blocks for the virtual private cloud (VPC), vSwitches, pods, and Services. It is important to ensure that the CIDR block of the pods and services do not overlap with the CIDR block of the VPC. Therefore SAP recommends that you plan the IP address of each before you create an ACK cluster. The table below outlines the configuration requirements that should be met for each CIDR block.
| Parameter | Configuration |
|---|---|
| VPC CIDR |
You can choose to use an existing VPC, or create a VPC to host your cluster. If creating a new VPC, you will need to define an IPv4 CIDR Block for it and also for the first vSwitch which be created at the same time as the VPC. . The primary IPv4 CIDR Block for the VPC can use one of the following:
The first vSwitch also requires a CIDR Block within the following limits:
|
| vSwitch CIDR |
The IP addresses of ECS (Elastic Compute Service) instances are assigned from the vSwitches. This allows nodes in the cluster to communicate with each other. The CIDR blocks that you specify when you creating vSwitches in the VPC must be subsets of the VPC CIDR block. This means that the vSwitch CIDR blocks must fall within or be the same as the VPC CIDR block. When you set this parameter, take note of the following:
|
|
POD CIDR Block |
The IP addresses of pods are allocated from the pod CIDR block, allowing pods to communicate with each other. When you set configure the this parameter, take note of the following items:
For example, if the VPC CIDR block is 172.16.0.0/12, the CIDR block of pods cannot be 172.16.0.0/16 or 172.17.0.0/16, because these CIDR blocks are subsets of 172.16.0.0/12. |
| Service CIDR |
The CIDR block of Services. Service is an abstraction in Kubernetes. When you set this parameter, take note of the following items:
|
In brief, considering SAP requires only one VPC, and the CIDR block for the VPC is specified when it is created, you simply need to ensure that the CIDR block of pods and the Services do not overlap with the CIDR block of the VPC.
VPC and vSwitch Requirements
One you have planned the CIDR blocks, the VPC and vSwitches can be created.
The cluster requires a single VPC, create it, select its region, name it anything you choose, enter a description, and select a resource group to which it will belong.
During this step, you will must specify an IPv4 CIDR block for the VPC following the information outlined in Planning Your Network Requirements above.
At this stage you also add the vSwitches. SAP recommends that you create at least three vSwitches for every VPC. Each vSwitch must be deployed in different zones to implement cross-zone disaster recovery. For example, the vSwitches could be configured as follows:
-
vSwitch-1 in Zone A
-
vSwitch-2 in Zone B
-
vSwitch-3 in Zone C
When you create the vSwitches, you must specify their private IP address ranges in CIDR notation, see Planning Your Network Requirements for information.
NAT Gateway Requirements
The cluster requires two NAT gateways, each set up in a different zone. This is required so the Mission Control Agent can communicate with the Home Cloud and our monitoring solution can ship metrics and logs.
When configuring the NAT gateways, they must be configured so that vSwitch-1 and vSwitch-2 do not share the same gateway.
vSwitch-3 can be configured to share one of the NAT gateways with either vSwitch-1 or vSwitch-2. This ensures that one zone is always capable of maintaining outgoing internet connections if the other zone goes down.
Creating the ACK Cluster
Once the VPC, vSwitches, and a NAT Gateway are configured, the cluster can be created from the ACK Console. When creating the cluster, configure the parameters with the settings as outlined in the table below. For any parameters where the settings are not specified in the table, you can leave them in their default value.
| Parameter | Configuration |
|---|---|
| Container runtime | Docker 19.03.5 |
| K8S Version | Select 1.16 or above |
| vSwitche zones | Select the three vSwitches you created when creating the VPC. |
| CNI | Flannel |
| Configure SNAT | Enabled |
| Service Account Token Volume Projection | Enabled |
| Keypair | Create a new keypair or use existing |
| Operating System | CentOS based image |
| Remaining Settings | Default values |
| Worker Nodes | 2 x ecs.n2.large (4 core, 16GB) |
Node Pool Configuration
Every node pool you configure must have the following settings:
-
Auto scaling
-
Data disk of 40 GB.
-
CentOS
The following table lists any additional resource and setting requirements for each node pool. SAP recommends that you configure the node pools at the same time as the cluster.
| NodePool | Requirements |
|---|---|
| prod1k-a |
|
| prod1k-b |
|
| prod10k-a |
|
| prod10k-b |
|
| prod100k-a |
|
| prod100k-b |
|
| monitor |
|
Instance Types Requirements
Alibaba cloud recommends that worker nodes have at least 4 cores. SAP also requires all instances to be Intel compatible and meet the requirements outlined in the following table:
| Node Pools | Scaling Tier | Plans Included | Worker Node Cores | Worker Node RAM (GiB) | Example Instance Types |
|---|---|---|---|---|---|
| prod1k-a prod1k-b |
prod1k |
|
4 | 16 | ecs.g6.xlarge ecs.g5.xlarge |
| prod10k-a prod10k-b |
prod10k | Broker 5K and Broker 10K | 4 | 32 | ecs.r6.xlarge ecs.r5.xlarge |
| prod100k-a prod100k-b |
prod100k | Broker 50K and Broker 100K | 8 | 64 | ecs.r6.2xlarge exc.r5.2xlarge |
| monitor | monitor | All Enterprise (HA) plans | 2 | 8 | ecs.n4.xlarge ecs.n1.large ecs.c6.xlarge |
Autoscaling
Your cluster requires autoscaling in order to provide the appropriate level of available resources for your event broker services as their demands change. SAP recommends using the Kubernetes Cluster Autoscaler, which you can find in the Kuberenetes GitHub repository here: https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler.
You should refer to the Auto scaling of nodes documentation on the Container Service for Kubernetes (ACK), User guide for Kubernetes Clusters documentation site to implement the Cluster Autoscaler.
Storage Class Configuration for Autoscaling
Three scaling groups will be required by an auto scaleable cluster. Each of the scaling groups must be configured to use only one of the vSwitches so that each scaling group covers a specific availability zone. This is necessary as the auto-scaling is unable to trigger scale-ups a single group spanning multiple availability zones when using the availability zone nodeSelectors.
The autoscaling for ACK clusters requires a primary and backup StorageClasses for each zone. This allows the autoscaler's predicates to detect which of the zones need to be scaled up. These StorageClasses must be created before deploying the Mission Control Agent.
To support scale-up, the StorageClass must contain the allowVolumeExpansion property, and have it set to "true"
When deploying the Mission Control Agent, the helm parameters for primaryStorageClass and backupStorageClass must be set to the names of the two storage classes you created. For example, in the StorageClass.yaml file below, the names of the storage classes are: alicloud-disk-ssd-primary and alicloud-disk-ssd-backup .
allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: alicloud-disk-ssd-primary
parameters:
type: cloud_ssd
csi.storage.k8s.io/fstype: xfs
provisioner: diskplugin.csi.alibabacloud.com
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowedTopologies:
- matchLabelExpressions:
- key: topology.diskplugin.csi.alibabacloud.com/zone
values:
- <Zone of vSwitch1>
---
allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: alicloud-disk-ssd-backup
parameters:
type: cloud_ssd
provisioner: diskplugin.csi.alibabacloud.com
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowedTopologies:
- matchLabelExpressions:
- key: topology.diskplugin.csi.alibabacloud.com/zone
values:
- <Zone of vSwitch2>
---
allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: alicloud-disk-ssd-topology
parameters:
type: cloud_ssd
provisioner: diskplugin.csi.alibabacloud.com
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
To apply the storage classes run the following command:
kubectl apply -f StorageClass.yaml
LoadBalancer Configuration
Each SAP event broker service created will use one Server Load Balancer (SLB). By default, a shared SLB with a public IP address (EIP) will be created for each service.
The Mission Control Agent accepts a list of arbitrary annotations that will be passed to the LoadBalancer services that it creates for event broker services. This allows it to define various configurations settings on the SLB it creates.
For example to make an SLB private (No EIP public IP) you would need to specify the "service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type: intranet" annotation, along with another annotation to configure where the SLB will be connecting to (vSwitch or VPC).