Kubernetes Cluster API Provider IBM Cloud

Kubernetes-native declarative infrastructure for IBM Cloud.

What is the Cluster API Provider IBM Cloud

The Cluster API brings declarative, Kubernetes-style APIs to cluster creation, configuration and management.

The API itself is shared across multiple cloud providers allowing for true IBM Cloud hybrid deployments of Kubernetes. It is built atop the lessons learned from previous cluster managers such as kops and kubicorn.

CAPIBM Supported Infrastructure-as-a-Service (IaaS)

Currently, the CAPIBM project exclusively facilitates the deployment of Kubernetes (K8s) clusters solely on two IBM infrastructure offerings, namely IBM VPC (Virtual Server Instances) and IBM PowerVS.

Quick Start

Check out the getting started section to create your first Kubernetes cluster on IBM Cloud using Cluster API.

Tilt-based development environment

See developer guide section for details.

Documentation

Please see our Book for in-depth user documentation.

Additional docs can be found in the /docs directory, and the index is here.

Getting involved and contributing

Are you interested in contributing to cluster-api-provider-ibmcloud? We, the maintainers and community, would love your suggestions, contributions, and help! Also, the maintainers can be contacted at any time to learn more about how to get involved.

In the interest of getting more new people involved, we tag issues with good first issue. These are typically issues that have smaller scope but are good ways to start to get acquainted with the codebase.

We also encourage all active community participants to act as if they are maintainers, even if you don’t have “official” write permissions. This is a community effort, we are here to serve the Kubernetes community. If you have an active interest and you want to get involved, you have real power! Don’t assume that the only people who can get things done around here are the “maintainers”.

We also would love to add more “official” maintainers, so show us what you can do!

This repository uses the Kubernetes bots. See a full list of the commands here.

Join us

The community holds a weekly meeting every Friday at 09:00 (IST) / 03:30 (UTC) on Zoom. Subscribe to the SIG Cluster Lifecycle Google Group for access to documents and calendars

Other ways to communicate with the contributors

Please check in with us in the #cluster-api-ibmcloud channel on Slack.

Github issues

Bugs

If you think you have found a bug please follow the instructions below.

• Please spend a small amount of time giving due diligence to the issue tracker. Your issue might be a duplicate.
• Get the logs from the cluster controllers. Please paste this into your issue.
• Open a bug report.
• Remember users might be searching for your issue in the future, so please give it a meaningful title to helps others.

Tracking new features

We also use the issue tracker to track features. If you have an idea for a feature, or think you can help Cluster API Provider IBMCloud become even more awesome, then follow the steps below.

• Open a feature request.
• Remember users might be searching for your issue in the future, so please give it a meaningful title to helps others.
• Clearly define the use case, using concrete examples. EG: I type this and cluster-api-provider-ibmcloud does that.
• Some of our larger features will require some design. If you would like to include a technical design for your feature please include it in the issue.
• After the new feature is well understood, and the design agreed upon we can start coding the feature. We would love for you to code it. So please open up a WIP (work in progress) pull request, and happy coding.

Getting Started

For prerequisites, check the respective sections for VPC and PowerVS

Now that we’ve got all the prerequisites in place, let’s create a Kubernetes cluster and transform it into a management cluster using clusterctl.

Provision local boostrap management cluster:

1. Create simple, local bootstrap cluster with a control-plane and worker node

   Using kind:

   ~ kind create cluster --name my-bootstrap --config bootstrap.yaml
   

   Example bootstrap.yaml:

   kind: Cluster
   apiVersion: kind.x-k8s.io/v1alpha4
   nodes:
      - role: control-plane
      - role: worker
   

   Make sure the nodes are in Ready state before moving on.

   ~ kubectl get nodes
   NAME                         STATUS   ROLES                  AGE   VERSION
   my-bootstrap-control-plane   Ready    control-plane,master   46h   v1.20.2
   my-bootstrap-worker          Ready    <none>                 46h   v1.20.2
   

2. Set workload cluster environment variables

   Make sure these value reflects your API Key for your target VPC environment or PowerVS environment in IBM Cloud.

   export IBMCLOUD_API_KEY=<YOUR_API_KEY>
   

3. To deploy workload cluster with Custom Service Endpoint, Set SERVICE_ENDPOINT environmental variable in semi-colon separated format:

   ${ServiceRegion1}:${ServiceID1}=${URL1},${ServiceID2}=${URL2};${ServiceRegion2}:${ServiceID1}=${URL1...}.

   Supported ServiceIDs include - vpc, powervs, rc

    export SERVICE_ENDPOINT=us-south:vpc=https://us-south-stage01.iaasdev.cloud.ibm.com,powervs=https://dal.power-iaas.test.cloud.ibm.com,rc=https://resource-controller.test.cloud.ibm.com
   

   Note: Refer Regions-Zones Mapping for more information.

4. For enabling debug level logs for the controller, set the LOGLEVEL environment variable(defaults to 0).

   export LOGLEVEL=5
   

5. Initialize local bootstrap cluster as a management cluster

   When executed for the first time, the following command accepts the infrastructure provider as an input to install. clusterctl init automatically adds to the list the cluster-api core provider, and if unspecified, it also adds the kubeadm bootstrap and kubeadm control-plane providers, thereby converting it into a management cluster which will be used to provision a workload cluster in IBM Cloud.

   ~ clusterctl init --infrastructure ibmcloud:<TAG>
   

   Note: If the latest release version of the provider is available, specifying TAG can be avoided. In other cases, you can specify any prerelease version compatible with the supported API contract as the TAG.
   Example: clusterctl init --infrastructure ibmcloud:v0.2.0-alpha.5

   Output:

   Fetching providers
   Installing cert-manager Version="v1.5.3"
   Waiting for cert-manager to be available...
   Installing Provider="cluster-api" Version="v0.4.4" TargetNamespace="capi-system"
   Installing Provider="bootstrap-kubeadm" Version="v0.4.4" TargetNamespace="capi-kubeadm-bootstrap-system"
   Installing Provider="control-plane-kubeadm" Version="v0.4.4" TargetNamespace="capi-kubeadm-control-plane-system"
   Installing Provider="infrastructure-ibmcloud" Version="v0.1.0-alpha.2" TargetNamespace="capi-ibmcloud-system"
   
   Your management cluster has been initialized successfully!
   
   You can now create your first workload cluster by running the following:
   
   clusterctl generate cluster [name] --kubernetes-version [version] | kubectl apply -f -
   

6. Once the management cluster is ready with the required providers up and running, proceed to provisioning the workload cluster. Check the respective sections for VPC and PowerVS to deploy the cluster.

7. For deploying with your workload cluster with Cloud Controller manager or Cluster Class template, refer to deploy with cloud controller manager and deploy PowerVS cluster with cluster class template sections respectively.

Deploy with Cloud Contoller manager

1. To deploy VPC workload cluster with IBM cloud controller manager, set the PROVIDER_ID_FORMAT environmental variable.

   export PROVIDER_ID_FORMAT=v2
   export EXP_CLUSTER_RESOURCE_SET=true
   

2. To deploy workload cluster with PowerVS cloud controller manager, set the POWERVS_PROVIDER_ID_FORMAT environmental variable.

   export POWERVS_PROVIDER_ID_FORMAT=v2
   export EXP_CLUSTER_RESOURCE_SET=true
   

Note: EXP_CLUSTER_RESOURCE_SET should be set for deploying workload cluster with Cloud Controller manager.

Deploy PowerVS cluster with ClusterClass template

To deploy workload cluster with PowerVS clusterclass-template. Set the following environmental variables.

export POWERVS_PROVIDER_ID_FORMAT=v2
export EXP_CLUSTER_RESOURCE_SET=true
export CLUSTER_TOPOLOGY=true

Note: Currently, both ClusterClass and ClusterResourceset are experimental feature so we need to enable the feature gate by setting EXP_CLUSTER_RESOURCE_SET, CLUSTER_TOPOLOGY environmental variables.

IBM Cloud Machine Images for CAPIBM Clusters

CAPIBM requires a “machine image” containing pre-installed, matching versions of kubeadm and kubelet. Machine image is required during the cluster creation in the AWSMachineTemplate spec.

Pre-built public Images are published by the maintainers regularly for each new Kubernetes version.

Note: These images are only for the test purpose

VPC Images

Note: These images are built using the image-builder tool and more information can be found here

Power VS Images

Note: These images are built using the image-builder tool and more information can be found here

Topics

This section contains information about using IBM Cloud features with Cluster API Provider IBM Cloud.

VPC Cluster

Contents

• Prerequisites
• Uploading an image
• Creating a cluster

Prerequisites

1. Install kubectl (see here). Because kustomize was included into kubectl and it’s used by cluster-api-provider-ibmcloud in generating yaml files, so version 1.14.0+ of kubectl is required, see integrate kustomize into kubectl for more info.
2. You can use either VM, container or existing Kubernetes cluster act as the bootstrap cluster.
   • If you want to use container, install kind. This is preferred.
   • If you want to use VM, install minikube, version 0.30.0 or greater.
   • If you want to use existing Kubernetes cluster, prepare your kubeconfig.
3. Install a driver if you are using minikube. For Linux, we recommend kvm2. For MacOS, we recommend VirtualBox.
4. An appropriately configured Go development environment
5. Install clusterctl tool (see here)

Build workload cluster image:

1. Build a qcow2 image suitable for use as a Kubernetes cluster machine as detailed in the image builder book.

   Note: Rename the output image to add the .qcow2 extension. This is required by the next step.

2. Upload the VPC Gen2 custom image to IBM Cloud following this section or the detailed explainations in the VPC documentation.

Uploading an image to the IBM Cloud

Build the Ubuntu image as described in the previous VPC section. Make sure to build the qcow2 version by following the instructions for raw image build.

Since the IBM Cloud does not support dots before the qcow2 extension, rename the file as follows:

ubuntu-2004-ibmcloud-kube-v1-23-4.qcow2

Upload VM image:

1. Create an IBM COS instance
2. Create a bucket in the COS instance.
3. Upload the image
   1. Upload via aspera
      • Install the browser extension for Aspera
      • Downloading the Aspera tool
      • Selecting the image via Aspera dialog
      • Upload the image via aspera
   2. Using minio cli
      • Install minio cli
      • Creating a service credential with hmac=true for the bucket
      • Example upload for eu-de:

        mc alias set uploadcos https://s3.eu-de.cloud-object-storage.appdomain.cloud <hmac access id> <hmac secret key>
        

        mc cp <image-name>.qcow2 uploadcos/<my-bucket-name>
        

Add VM image to VPC

1. Make sure you have editor rights for all/most VPC services
2. Add additional read rights for:

src: service VPC Infrastructure Services resourceType equals image
target: serviceInstance string equals <your-Cloud-Object Storage-VM-plain-name>

Add write rights for:

Service VPC Infrastructure Services in Resource_group <your_resource_group_or_account> resourceType equals image
target: service Cloud object storage in resource_group <your_resource_group_or_account>

3. Go to https://cloud.ibm.com/vpc-ext/provision/customImage

• Fill in imagename, resource group or account
• Choice box: Cloud Object Storage
• Set Filter: <your_cos_plain_name> <eu-de_or_other> <your_vm_bucket>
• Choice box: Select your image
• Select base os (ubuntu-20-04-amd64 for example)
• Click Create Image

Now you can provision a VM with your own VM image. Then please continue with creating a cluster.

Make sure you take the ImageID from your VM image. The ImageID can be determined using ibmcloud cli. In addition, the Kubernetes version must be set to match the image. In this example:

v1.23.4

Provision workload Cluster in IBM Cloud VPC

Now that we have a management cluster ready, you can create your workload cluster by following the steps below.

1. Using clusterctl, render the yaml through templates and deploy the cluster

   Note: the IBMVPC_IMAGE_NAME value below should reflect the Name of the custom qcow2 image

   IBMVPC_REGION=us-south \
   IBMVPC_ZONE=us-south-1 \
   IBMVPC_RESOURCEGROUP=4f15679623607b855b1a27a67f20e1c7 \
   IBMVPC_NAME=ibm-vpc-0 \
   IBMVPC_IMAGE_NAME=capibm-vpc-ubuntu-2004-kube-v1-25-2 \
   IBMVPC_PROFILE=bx2-4x16 \
   IBMVPC_SSHKEY_NAME=capi-vpc-key \
   clusterctl generate cluster ibm-vpc-0 --kubernetes-version v1.19.9 \
   --target-namespace default \
   --control-plane-machine-count=1 \
   --worker-machine-count=2 | kubectl apply -f -
   

   Output:

   cluster.cluster.x-k8s.io/ibm-vpc-5 created
   ibmvpccluster.infrastructure.cluster.x-k8s.io/ibm-vpc-5 created
   kubeadmcontrolplane.controlplane.cluster.x-k8s.io/ibm-vpc-5-control-plane created
   ibmvpcmachinetemplate.infrastructure.cluster.x-k8s.io/ibm-vpc-5-control-plane created
   machinedeployment.cluster.x-k8s.io/ibm-vpc-5-md-0 created
   ibmvpcmachinetemplate.infrastructure.cluster.x-k8s.io/ibm-vpc-5-md-0 created
   kubeadmconfigtemplate.bootstrap.cluster.x-k8s.io/ibm-vpc-5-md-0 created
   

   Note: Refer below for more detailed information on VPC variables.

   • IBMVPC_REGION
   • IBMVPC_ZONE
   • IBMVPC_RESOURCEGROUP
   • IBMVPC_IMAGE_NAME
   • IBMVPC_PROFILE
   • IBMVPC_SSHKEY_NAME

2. Check the state of the provisioned cluster and machine objects within the local management cluster

   Clusters

   ~ kubectl get clusters
   NAME         PHASE
   ibm-vpc-0    Provisioned
   

   Kubeadm Control Plane

   ~ kubectl get kubeadmcontrolplane
   NAME                       INITIALIZED   API SERVER AVAILABLE   VERSION   REPLICAS   READY   UPDATED   UNAVAILABLE
   ibm-vpc-0-control-plane    true          true                   v1.19.9   1          1       1
   

   Machines

   ~ kubectl get machines
   ibm-vpc-0-control-plane-vzz47     ibmvpc://ibm-vpc-0/ibm-vpc-0-control-plane-rg6xv   Running        v1.19.9
   ibm-vpc-0-md-0-5444cfcbcd-6gg5z   ibmvpc://ibm-vpc-0/ibm-vpc-0-md-0-dbxb7            Running        v1.19.9
   ibm-vpc-0-md-0-5444cfcbcd-7kr9x   ibmvpc://ibm-vpc-0/ibm-vpc-0-md-0-k7blr            Running        v1.19.9
   

3. Deploy Container Network Interface (CNI)

   Example: calico

   ~ clusterctl get kubeconfig ibm-vpc-0 > ~/.kube/ibm-vpc-0
   ~ export KUBECONFIG=~/.kube/ibm-vpc-0
   ~ kubectl apply -f https://docs.projectcalico.org/v3.15/manifests/calico.yaml
   

4. Check the state of the newly provisioned cluster within IBM Cloud

   ~ kubectl get nodes
   NAME                             STATUS   ROLES    AGE   VERSION
   ibm-vpc-0-control-plane-rg6xv    Ready    master   41h   v1.18.15
   ibm-vpc-0-md-0-4dc5c             Ready    <none>   41h   v1.18.15
   ibm-vpc-0-md-0-dbxb7             Ready    <none>   20h   v1.18.15
   

Provision workload Cluster with Load Balancer and external cloud provider in IBM Cloud VPC

⚠️ WARNING: This feature is currently in experimental stage

Steps

• To deploy a VPC workload cluster with Load Balancer IBM external cloud provider, create a cluster configuration with the external cloud provider template
• The external cloud provider template will use clusterresourceset and will create the necessary config map, secret and roles to run the cloud controller manager
• As a prerequisite set the provider-id-fmt flag with value v2

Deploy VPC cluster with Load Balancer and IBM cloud provider

  IBMCLOUD_API_KEY="XXXXXXXXXXXXXXXXXX" \
  IBMVPC_REGION=us-south \
  IBMVPC_ZONE=us-south-1 \
  IBMVPC_RESOURCEGROUP_NAME="ibm-hypershift-dev" \
  IBMVPC_RESOURCEGROUP=4f15679623607b855b1a27a67f20e1c7 \
  IBMVPC_NAME=ibm-vpc-0 \
  IBMVPC_IMAGE_ID=r134-ea84bbec-7986-4ff5-8489-d9ec34611dd4 \
  IBMVPC_PROFILE=bx2-4x16 \
  IBMVPC_SSHKEY_ID=r134-2a82b725-e570-43d3-8b23-9539e8641944 \
  IBMACCOUNT_ID="ibm-accountid" \
  BASE64_API_KEY=$(echo -n $IBMCLOUD_API_KEY | base64) \
  clusterctl generate cluster ibm-vpc-0 --kubernetes-version v1.25.2 \
  --target-namespace default \
  --control-plane-machine-count=1 \
  --worker-machine-count=2 \
  --flavor=load-balancer | kubectl apply -f -

Change disk size for the boot volume

There are two following variables for controlling the volume size for the boot disk.

• IBMVPC_CONTROLPLANE_BOOT_VOLUME_SIZEGIB: Size of the boot volume for the control plane nodes, default set to 20GiB
• IBMVPC_WORKER_BOOT_VOLUME_SIZEGIB: Size of the boot volume for the worker nodes, default set to 20GiB

Note: Default value is set to 20GiB because the images published for testing are of size 20GiB(default size in the image-builder scripts as well).

Power VS Cluster

Contents

• Prerequisites
• Creating a cluster
• Creating a cluster with external cloud provider
• Creating a cluster from ClusterClass
• Using autoscaler with scaling from 0 machine

Prerequisites

1. Install kubectl (see here). Because kustomize was included into kubectl and it’s used by cluster-api-provider-ibmcloud in generating yaml files, so version 1.14.0+ of kubectl is required, see integrate kustomize into kubectl for more info.
2. You can use either VM, container or existing Kubernetes cluster act as the bootstrap cluster.
   • If you want to use container, install kind. This is preferred.
   • If you want to use VM, install minikube, version 0.30.0 or greater.
   • If you want to use existing Kubernetes cluster, prepare your kubeconfig.
3. Install a driver if you are using minikube. For Linux, we recommend kvm2. For MacOS, we recommend VirtualBox.
4. An appropriately configured Go development environment
5. Install clusterctl tool (see here)
6. Install pvsadm tool (see here)
7. Install ibmcloud tool (see here)

PowerVS Prerequisites

Create an IBM Cloud account.

If you don’t already have one, you need a paid IBM Cloud account to create your Power Systems Virtual Server instance. To create an account, go to: cloud.ibm.com.

Create an IBM Cloud account API key

Please refer to the following documentation to create an API key.

Create Power Systems Virtual Server Service Instance

After you have an active IBM Cloud account, you can create a Power Systems Virtual Server service. To do so, perform the following steps:

1. TO-DO

Create Network

A public network is required for your kubernetes cluster. Perform the following steps to create a public network for the Power Systems Virtual Server service instance created in the previous step.

1. Create Public Network

   ~ ibmcloud pi network-create-public capi-test --dns-servers "8.8.8.8 9.9.9.9"
   

   Output:

   Network capi-test created.
                       
   ID                fea9ac26-693d-402b-b22f-aa3d90ed0a31   
   Name              capi-test  
   Type              pub-vlan   
   VLAN              2008   
   CIDR Block        192.168.150.96/29   
   IP Range          [192.168.150.98 192.168.150.102]   
   Public IP Range   [158.175.161.98 158.175.161.102]   
   Gateway           192.168.150.97   
   DNS               8.8.8.8, 9.9.9.9
   

Import the machine boot image:

$ export IBMCLOUD_API_KEY=<API_KEY>
$ pvsadm image import --pvs-instance-id <SERVICE_INSTANCE_ID> -b <BUCKETNAME> --object <OBJECT> --pvs-image-name <POWERVS_IMAGE_NAME> --bucket-region <REGION> --public-bucket

e.g:

$ pvsadm image import --pvs-instance-id 6d892c30-5387-4685-85d0-4999d9c22a8c -b power-oss-bucket --object capibm-powervs-centos-streams8-1-24-2.ova.gz --pvs-image-name capibm-powervs-centos-streams8-1-24-2 --bucket-region us-south --public-bucket

For more information about the images can be found at machine-images section

Provision workload cluster in IBM Cloud PowerVS

Now that we have a management cluster ready, you can create your workload cluster by following the steps below.

1. Create PowerVS network port

   ~ pvsadm create port --description "capi-port" --network <NETWORK_NAME> --instance-id <SERVICE_INSTANCE_ID>
   

   Output:

   I1125 15:24:20.581757 1548881 port.go:89] Successfully created a port, id: ac18ef17-8517-40e3-889d-4f246e9bd17e
   +---------------------+------------+------+-----------------+-------------------+--------------------------------------+-------------+--------+
   |     DESCRIPTION     | EXTERNALIP | HREF |    IPADDRESS    |    MACADDRESS     |                PORTID                | PVMINSTANCE | STATUS |
   +---------------------+------------+------+-----------------+-------------------+--------------------------------------+-------------+--------+
   | capi-port |            |      | 192.168.151.125 | fa:16:3e:34:2c:ef | 7eff02b5-040c-4934-957a-18209e65eca4 |             | DOWN   |
   +---------------------+------------+------+-----------------+-------------------+--------------------------------------+-------------+--------+
   

   ~ pvsadm get ports --instance-id <SERVICE_INSTANCE_ID> --network <NETWORK_NAME>
   

   Output:

   +-------------------+-----------------+-----------------+-------------------+--------------------------------------+--------+
   |    DESCRIPTION    |   EXTERNALIP    |    IPADDRESS    |    MACADDRESS     |                PORTID                | STATUS |
   +-------------------+-----------------+-----------------+-------------------+--------------------------------------+--------+
   | capi-port         | 158.175.162.125 | 192.168.151.125 | fa:16:3e:34:2c:ef | 7eff02b5-040c-4934-957a-18209e65eca4 | DOWN   |
   +-------------------+-----------------+-----------------+-------------------+--------------------------------------+--------+
   

2. Use clusterctl to render the yaml through templates and deploy the cluster

   Note: To deploy workload cluster with Power VS cloud controller manager which is currently in experimental stage follow these steps.

   Note: the IBMPOWERVS_IMAGE_ID value below should reflect the ID of the custom qcow2 image, the kubernetes-version value below should reflect the kubernetes version of the custom qcow2 image.

   IBMPOWERVS_SSHKEY_NAME="my-pub-key" \
   IBMPOWERVS_VIP="192.168.151.22" \
   IBMPOWERVS_VIP_EXTERNAL="158.175.162.22" \
   IBMPOWERVS_VIP_CIDR="29" \
   IBMPOWERVS_IMAGE_NAME="capibm-powervs-centos-8-1-22-4" \
   IBMPOWERVS_SERVICE_INSTANCE_ID="7845d372-d4e1-46b8-91fc-41051c984601" \
   IBMPOWERVS_NETWORK_NAME="capi-test-3" \
   clusterctl generate cluster ibm-powervs-1 --kubernetes-version v1.22.4 \
   --target-namespace default \
   --control-plane-machine-count=3 \
   --worker-machine-count=1 \
   --flavor=powervs | kubectl apply -f -
   

   Output:

   cluster.cluster.x-k8s.io/ibm-powervs-1 created
   ibmpowervscluster.infrastructure.cluster.x-k8s.io/ibm-powervs-1 created
   kubeadmcontrolplane.controlplane.cluster.x-k8s.io/ibm-powervs-1-control-plane created
   ibmpowervsmachinetemplate.infrastructure.cluster.x-k8s.io/ibm-powervs-1-control-plane created
   machinedeployment.cluster.x-k8s.io/ibm-powervs-1-md-0 created
   ibmpowervsmachinetemplate.infrastructure.cluster.x-k8s.io/ibm-powervs-1-md-0 created
   kubeadmconfigtemplate.bootstrap.cluster.x-k8s.io/ibm-powervs-1-md-0 created
   

   Additional parameters for modifying PowerVS Control-Plane spec

   IBMPOWERVS_CONTROL_PLANE_MEMORY
   IBMPOWERVS_CONTROL_PLANE_PROCESSORS
   IBMPOWERVS_CONTROL_PLANE_SYSTYPE
   IBMPOWERVS_CONTROL_PLANE_PROCTYPE
   

   Additional parameters for modifying PowerVS Compute node spec

   IBMPOWERVS_COMPUTE_MEMORY
   IBMPOWERVS_COMPUTE_PROCESSORS
   IBMPOWERVS_COMPUTE_SYSTYPE
   IBMPOWERVS_COMPUTE_PROCTYPE
   

3. Check the state of the provisioned cluster and machine objects within the local management cluster

   Clusters

   ~ kubectl get clusters
   NAME         PHASE
   ibm-powervs-1    Provisioned
   

   Kubeadm Control Plane

   ~ kubectl get kubeadmcontrolplane
   NAME                       INITIALIZED   API SERVER AVAILABLE   VERSION   REPLICAS   READY   UPDATED   UNAVAILABLE
   ibm-powervs-1-control-plane    true          true                   v1.22.4   1          1       1
   

   Machines

   ~ kubectl get machines
   ibm-powervs-1-control-plane-vzz47     ibmpowervs://ibm-powervs-1/ibm-powervs-1-control-plane-rg6xv   Running        v1.22.4
   ibm-powervs-1-md-0-5444cfcbcd-6gg5z   ibmpowervs://ibm-powervs-1/ibm-powervs-1-md-0-dbxb7            Running        v1.22.4
   ibm-powervs-1-md-0-5444cfcbcd-7kr9x   ibmpowervs://ibm-powervs-1/ibm-powervs-1-md-0-k7blr            Running        v1.22.4
   

4. Deploy Container Network Interface (CNI)

   Example: calico

   ~ clusterctl get kubeconfig ibm-powervs-1 > ~/.kube/ibm-powervs-1
   ~ export KUBECONFIG=~/.kube/ibm-powervs-1
   ~ kubectl apply -f https://docs.projectcalico.org/v3.15/manifests/calico.yaml
   

5. Check the state of the newly provisioned cluster within IBM Cloud

   ~ kubectl get nodes
   NAME                             STATUS   ROLES    AGE   VERSION
   ibm-powervs-1-control-plane-rg6xv    Ready    master   41h   v1.22.4
   ibm-powervs-1-md-0-4dc5c             Ready    <none>   41h   v1.22.4
   ibm-powervs-1-md-0-dbxb7             Ready    <none>   20h   v1.22.4
   

IBM Power VS External Cloud Provider

⚠️ WARNING: This feature is currently in experimental stage

Steps

• To deploy a Power VS workload cluster with IBM Power VS external cloud provider, create a cluster configuration with the external cloud provider template
• The external cloud provider template will use clusterresourceset and will create the necessary config map, secret and roles to run the cloud controller manager
• As a prerequisite set the powervs-provider-id-fmt flag with value v2

Deploy Power VS cluster with IBM Power VS cloud provider

IBMPOWERVS_SSHKEY_NAME="my-pub-key" \
IBMPOWERVS_VIP="192.168.151.22" \
IBMPOWERVS_VIP_EXTERNAL="158.175.162.22" \
IBMPOWERVS_VIP_CIDR="29" \
IBMPOWERVS_IMAGE_NAME="capibm-powervs-centos-8-1-22-4" \
IBMPOWERVS_SERVICE_INSTANCE_ID="7845d372-d4e1-46b8-91fc-41051c984601" \
IBMPOWERVS_NETWORK_NAME="capi-test-3" \
IBMACCOUNT_ID="ibm-accountid" \
IBMPOWERVS_REGION="powervs-region" \
IBMPOWERVS_ZONE="powervs-zone" \
BASE64_API_KEY=$(echo -n $IBMCLOUD_API_KEY | base64) \
clusterctl generate cluster ibm-powervs-1 --kubernetes-version v1.22.4 \
--target-namespace default \
--control-plane-machine-count=3 \
--worker-machine-count=1 \
--flavor=powervs-cloud-provider | kubectl apply -f -

When the cluster is created with above parameters, The IBM Power VS cloud provider will

1. Initialize the node by fetching appropriate VM information such as IP, zone, region from Power Cloud.

Create IBM Power VS Cluster Using ClusterClass

Steps

• To deploy Power VS workload cluster using ClusterClass, create a cluster configuration from the clusterclass-template
• The Power VS cluster will use external cloud provider. As a prerequisite set the powervs-provider-id-fmt flag with value v2
• The clusterclass-template will use clusterresourceset and will create the necessary config map, secret and roles to run the cloud controller manager

Deploy Power VS cluster with IBM Power VS cloud provider

IBMPOWERVS_CLUSTER_CLASS_NAME="powervs-cc" \
IBMPOWERVS_SSHKEY_NAME="my-pub-key" \
IBMPOWERVS_VIP="192.168.151.22" \
IBMPOWERVS_VIP_EXTERNAL="158.175.162.22" \
IBMPOWERVS_VIP_CIDR="29" \
IBMPOWERVS_IMAGE_NAME="capibm-powervs-centos-8-1-22-4" \
IBMPOWERVS_SERVICE_INSTANCE_ID="7845d372-d4e1-46b8-91fc-41051c984601" \
IBMPOWERVS_NETWORK_NAME="capi-test-3" \
IBMACCOUNT_ID="ibm-accountid" \
IBMPOWERVS_REGION="powervs-region" \
IBMPOWERVS_ZONE="powervs-zone" \
BASE64_API_KEY=$(echo -n $IBMCLOUD_API_KEY | base64) \
clusterctl generate cluster ibm-powervs-1 --kubernetes-version v1.22.4 \
--target-namespace default \
--control-plane-machine-count=3 \
--worker-machine-count=1 \
--flavor=simple-powervs-clusterclass | kubectl apply -f -

Using Autoscaler to scale machines from 0 machine

The autoscaler project supports cluster-api. With this enhancement now the user can use cluster-api feature to scaling from 0 machine.

Settinng up the workload cluster

While creating a workload cluster, We need to set the below annotations to machinedeployment inorder to enable the autoscaling, This is one of the prerequisites for autoscaler.

apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment
metadata:
  name: "${CLUSTER_NAME}-md-0"
  annotations:
    cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: "5"
    cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size: "0"

Setting up the cluster-autoscaler

1. Clone the autoscaler repository

git clone https://github.com/kubernetes/autoscaler.git

2. Build the autoscaler binary

cd cluster-autoscaler 
go build .

3. Start the autoscaler

./cluster-autoscaler \
--cloud-provider=clusterapi \
--v=2 \
--namespace=default \
--max-nodes-total=30 \
--scale-down-delay-after-add=10s \
--scale-down-delay-after-delete=10s \
--scale-down-delay-after-failure=10s \
--scale-down-unneeded-time=5m \
--max-node-provision-time=30m \
--balance-similar-node-groups \
--expander=random \
--kubeconfig=<workload_cluster_kubeconfig> \
--cloud-config=<management_cluster_kubeconfig>

Note:

1. autoscaler can be run in different ways the possible ways are described here.
2. autoscaler supports various command line flags and more details about it can be found here.

Use case of cluster-autoscaler

1. Create a workload cluster with 0 worker machines
2. Create a sample workload

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: busybox
  name: busybox-deployment
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: busybox
  template:
    metadata:
      labels:
        app: busybox
    spec:
      containers:
        - command:
            - sh
            - -c
            - echo Container 1 is Running ; sleep 3600
          image: busybox
          imagePullPolicy: IfNotPresent
          name: busybox
          resources:
            requests:
              cpu: "0.2"
              memory: 3G

3. Scale the deployment to create addition pods

kubectl scale --replicas=2 deployment/busybox-deployment 

4. Obeserve the status of new pods

kubectl get pods                                        
NAME                                  READY   STATUS    RESTARTS   AGE
busybox-deployment-7c87788568-qhqdb   1/1     Running   0          48s
busybox-deployment-7c87788568-t26bb   0/1     Pending   0          5s

5. On the management cluster verify that the new machine creation is being triggered by autoscaler

NAME                                        CLUSTER               NODENAME                                  PROVIDERID                                                                                      PHASE          AGE     VERSION
karthik-ibm-powervs-control-plane-smvf7     karthik-ibm-powervs   karthik-ibm-powervs-control-plane-pgwmz   ibmpowervs://osa/osa21/3229a-af54-4212-bf60-6202b6fd0a07/809cd0f2-7502-4112-bf44-84d178020d8a   Running        82m     v1.24.2
karthik-ibm-powervs-md-0-6b4d67ccf4-npdbm   karthik-ibm-powervs   karthik-ibm-powervs-md-0-qch8f            ibmpowervs://osa/osa21/3229a-af54-4212-bf60-6202b6fd0a07/50f841e5-f58c-4569-894d-b40ba0d2696e   Running        76m     v1.24.2
karthik-ibm-powervs-md-0-6b4d67ccf4-v7xv9   karthik-ibm-powervs                                                                                                                                             Provisioning   3m19s   v1.24.2

6. After sometime verify that the new node being added to the cluster and pod is in running state

kubectl get nodes
NAME                                      STATUS   ROLES           AGE   VERSION
karthik-ibm-powervs-control-plane-pgwmz   Ready    control-plane   92m   v1.24.2
karthik-ibm-powervs-md-0-n8c6d            Ready    <none>          42s   v1.24.2
karthik-ibm-powervs-md-0-qch8f            Ready    <none>          85m   v1.24.2

kubectl get pods
NAME                                  READY   STATUS    RESTARTS   AGE
busybox-deployment-7c87788568-qhqdb   1/1     Running   0          19m
busybox-deployment-7c87788568-t26bb   1/1     Running   0          18m

7. Delete the deployment to observe the scale down of nodes by autoscaler

kubectl delete deployment/busybox-deployment

kubectl get nodes
NAME                                      STATUS   ROLES           AGE    VERSION
karthik-ibm-powervs-control-plane-pgwmz   Ready    control-plane   105m   v1.24.2
karthik-ibm-powervs-md-0-qch8f            Ready    <none>          98m    v1.24.2

capibmadm CLI

Kubernetes Cluster API Provider IBM Cloud Management Utility

1. Power VS commands

capibmadm powervs <commands>

1. Power VS commands

• network
  • create
  • list

Power VS Network Commands

1. capibmadm powervs network create

Usage:

Create Power VS network.

Environmental Variable:

IBMCLOUD_API_KEY: IBM Cloud api key.

Arguments:

--service-instance-id: Power VS service instance id.

--dns-servers: Comma separated list of DNS Servers to use for this network, Defaults to 8.8.8.8, 9.9.9.9.

Example:

export IBMCLOUD_API_KEY=<api-key>
capibmadm powervs network create <network-name> --service-instance-id <service-instance-id>

2. capibmadm powervs network list

Usage:

List PowerVS networks.

Environmental Variable:

IBMCLOUD_API_KEY: IBM Cloud api key.

Arguments:

--service-instance-id: PowerVS service instance id.

--zone: PowerVS service instance zone.

Example:

export IBMCLOUD_API_KEY=<api-key>
capibmadm powervs network list --service-instance-id <service-instance-id> --zone <zone>

Developer Guide

• Rapid iterative development with Tilt
• Guide for API conversions
• Release process
• How to build machine boot images

Rapid iterative development with Tilt

Overview

This document describes how to use kind and Tilt for a simplified workflow that offers easy deployments and rapid iterative builds.

Prerequisites

1. Docker v19.03 or newer
2. kind v0.9 or newer (other clusters can be used if preload_images_for_kind is set to false)
3. kustomize
4. Tilt v0.30.8 or newer
5. envsubst or similar to handle clusterctl var replacement
6. Clone the Cluster API repository locally
7. Clone the cluster-api-provider-ibmcloud repository you want to deploy locally as well

Create a kind cluster

First, make sure you have a kind cluster and that your KUBECONFIG is set up correctly:

kind create cluster

This local cluster will be running all the cluster api controllers and become the management cluster which then can be used to spin up workload clusters on IBM Cloud.

Create a tilt-settings.yaml file

Next, create a tilt-settings.yaml file and place it in your local copy of cluster-api. Here is an example:

Example tilt-settings.yaml for CAPI-IBM clusters:

Make sure to replace the parameter IBMCLOUD_API_KEY with a valid API key.

default_registry: "gcr.io/you-project-name-here"
provider_repos:
- ../cluster-api-provider-ibmcloud
enable_providers:
- ibmcloud
- kubeadm-bootstrap
- kubeadm-control-plane
kustomize_substitutions:
  IBMCLOUD_API_KEY: "XXXXXXXXXXXXXXXXXX"

Add following extra_args to log Power VS REST API Requests/Responses

extra_args:
  ibmcloud:
    - '-v=5'

1. Configuration to deploy Power VS workload cluster with external cloud controller manager

To deploy workload cluster with Power VS cloud controller manager which is currently in experimental stage, Set POWERVS_PROVIDER_ID_FORMAT to v2 and enable cluster resourceset feature gate under kustomize_substitutions. Currently, ClusterResourceset is experimental feature so we need to enable the feature gate by setting EXP_CLUSTER_RESOURCE_SET variable under kustomize_substitutions.

default_registry: "gcr.io/you-project-name-here"
provider_repos:
- ../cluster-api-provider-ibmcloud
enable_providers:
- ibmcloud
- kubeadm-bootstrap
- kubeadm-control-plane
kustomize_substitutions:
  IBMCLOUD_API_KEY: "XXXXXXXXXXXXXXXXXX"
  POWERVS_PROVIDER_ID_FORMAT: "v2"
  EXP_CLUSTER_RESOURCE_SET: "true"

2. Configuration to deploy VPC workload cluster with external cloud controller manager

To deploy workload cluster with cloud controller manager which is currently in experimental stage, Set PROVIDER_ID_FORMAT to v2 and enable cluster resourceset feature gate under kustomize_substitutions. Currently, ClusterResourceset is experimental feature so we need to enable the feature gate by setting EXP_CLUSTER_RESOURCE_SET variable under kustomize_substitutions.

default_registry: "gcr.io/you-project-name-here"
provider_repos:
- ../cluster-api-provider-ibmcloud
enable_providers:
- ibmcloud
- kubeadm-bootstrap
- kubeadm-control-plane
kustomize_substitutions:
  IBMCLOUD_API_KEY: "XXXXXXXXXXXXXXXXXX"
  PROVIDER_ID_FORMAT: "v2"
  EXP_CLUSTER_RESOURCE_SET: "true"

3. Configuration to deploy workload cluster from ClusterClass template

To deploy workload cluster with clusterclass-template under kustomize_substitutions set POWERVS_PROVIDER_ID_FORMAT to v2. Currently, both ClusterClass and ClusterResourceset are experimental feature so we need to enable the feature gate by setting EXP_CLUSTER_RESOURCE_SET, CLUSTER_TOPOLOGY variable under kustomize_substitutions.

default_registry: "gcr.io/you-project-name-here"
provider_repos:
- ../cluster-api-provider-ibmcloud
enable_providers:
- ibmcloud
- kubeadm-bootstrap
- kubeadm-control-plane
kustomize_substitutions:
  IBMCLOUD_API_KEY: "XXXXXXXXXXXXXXXXXX"
  POWERVS_PROVIDER_ID_FORMAT: "v2"
  EXP_CLUSTER_RESOURCE_SET: "true"
  CLUSTER_TOPOLOGY: "true"

4. Configuration to deploy workload cluster with Custom Service Endpoint

To deploy workload cluster with Custom Service Endpoint, Set SERVICE_ENDPOINT environmental variable in semi-colon separated format: ${ServiceRegion}:${ServiceID1}=${URL1},${ServiceID2}=${URL2...}

default_registry: "gcr.io/you-project-name-here"
provider_repos:
- ../cluster-api-provider-ibmcloud
enable_providers:
- ibmcloud
- kubeadm-bootstrap
- kubeadm-control-plane
kustomize_substitutions:
  IBMCLOUD_API_KEY: "XXXXXXXXXXXXXXXXXX"
  SERVICE_ENDPOINT: "us-south:vpc=https://us-south-stage01.iaasdev.cloud.ibm.com,powervs=https://dal.power-iaas.test.cloud.ibm.com,rc=https://resource-controller.test.cloud.ibm.com"
  IBMCLOUD_AUTH_URL: "https://iam.test.cloud.ibm.com"

NOTE: For information about all the fields that can be used in the tilt-settings.yaml file, check them here.

Run Tilt

To launch your development environment, run:

tilt up

Kind cluster becomes a management cluster after this point, check the pods running on the kind cluster by running kubectl get pods -A.

Create workload clusters

To provision your workload cluster, check the Creating a cluster section for VPC and PowerVS.

After deploying it, check the tilt logs and wait for the clusters to be created.

Clean up

Before deleting the kind cluster, make sure you delete all the workload clusters.

kubectl delete cluster <clustername>
tilt up (ctrl-c)
kind delete cluster

Guide for API conversions

Introduction

The purpose of this document is to help/assist contributors with future API conversions using conversion-gen tool.

Prerequisites

1. Create a new API version.

kubebuilder create api --group <group> --version <version> --kind <kind>

2. Copy over existing types, and make the required changes.
3. Mark a storage version, add marker +kubebuilder:storageversion to concerned version package.

NOTE: Refer for more detailed information about prerequisites.

Conversion flow

1. In each “spoke” version package, add marker +k8s:conversion-gen directive pointing to the “hub” version package. It must be in doc.go. Refer
2. In “hub” version package, create doc.go file without any marker. Refer
3. In “spoke” version package, add a var localSchemeBuilder = &SchemeBuilder.SchemeBuilder in groupversion_info.go so the auto-generated code would compile. Refer
4. In “hub” version package, create a conversion.go to implement the “hub” methods. Refer
5. Run target make generate-go-conversions-core, this will generate zz_generated.conversion.go in the spoke version package.
6. In “spoke” version package, update {kind}_conversion.go to implement Convertible for each type. When conversion-gen stops generating methods because of incompatibilities or we need to override the behavior, we stick them in this source file. Our “spoke” versions need to implement the Convertible interface. Namely, they’ll need ConvertTo and ConvertFrom methods to convert to/from the hub version. Refer

References

• What are hubs and spokes?
• Sample Tutorial

Release Process

Alpha/Beta releases

• Create a tag and push

  git clone git@github.com:kubernetes-sigs/cluster-api-provider-ibmcloud.git
  git tag -s -m "v0.2.0-alpha.3" v0.2.0-alpha.3
  git push origin v0.2.0-alpha.3
  

• Wait for the google cloud build to be finished
• Create a draft release with release notes for the tag
• Tick the prerelease checkbox
• Download the artifacts once cloud build is finished

  gsutil -m cp \
   "gs://artifacts.k8s-staging-capi-ibmcloud.appspot.com/components/v0.2.0-alpha.3/cluster-template-powervs.yaml" \
   "gs://artifacts.k8s-staging-capi-ibmcloud.appspot.com/components/v0.2.0-alpha.3/cluster-template.yaml" \
   "gs://artifacts.k8s-staging-capi-ibmcloud.appspot.com/components/v0.2.0-alpha.3/infrastructure-components.yaml" \
   "gs://artifacts.k8s-staging-capi-ibmcloud.appspot.com/components/v0.2.0-alpha.3/metadata.yaml" \
   .
  

• Upload the downloaded artifacts into the release asset
• Publish the drafted release

Note: In the above instructions, v0.2.0-alpha.3 is the version/tag is being released

GA Releases

• Create a tag and push

  git clone git@github.com:kubernetes-sigs/cluster-api-provider-ibmcloud.git
  git tag -s -m "v0.1.0" v0.1.0
  git push origin v0.1.0
  

• Wait for the google cloud build to be finished
• Create a draft release with release notes for the tag
• Perform the image promotion process:
  • Clone and pull down the latest from kubernetes/k8s.io
  • Create a new branch in your fork of kubernetes/k8s.io.
  • The staging repository is here.
  • Once image is present in the above staging repository, find the sha256 tag for the image by following instructions

  $ manifest-tool inspect --raw gcr.io/k8s-staging-capi-ibmcloud/cluster-api-ibmcloud-controller:v0.1.0 | jq '.[0].Digest'
  "sha256:6c92a6a337ca5152eda855ac27c9e4ca1f30bba0aa4de5c3a0b937270ead4363"
  

  • In your kubernetes/k8s.io branch edit k8s.gcr.io/images/k8s-staging-capi-ibmcloud/images.yaml and add an entry for the version using the sha256 value got from the above command. For example: "sha256:6c92a6a337ca5152eda855ac27c9e4ca1f30bba0aa4de5c3a0b937270ead4363": ["v0.1.0"]
  • You can use this PR as example
  • Wait for the PR to be approved and merged
  • Run make release command
  • Copy the content from out directory to release asset
  • Publish the drafted release

Note: In the above instructions, v0.1.0 is the version/tag is being released

How to build the machine boot images

VPC

TO-DO

Power VS

Compose the user-varibales.json file contains the information for the Power VS

{
  "account_id": "",
  "apikey": "",
  "capture_cos_access_key": "",
  "capture_cos_bucket": "",
  "capture_cos_region": "",
  "capture_cos_secret_key": "",
  "key_pair_name": "",
  "kubernetes_deb_version": "",
  "kubernetes_rpm_version": "",
  "kubernetes_semver": "",
  "kubernetes_series": "",
  "region": "",
  "service_instance_id": "",
  "ssh_private_key_file": "",
  "zone": ""
}

• account_id: IBM Cloud account ID
• apikey: IBM Cloud API Key
• capture_cos_access_key: Access key for the IBM Cloud Object Storage(COS) to export the image to
• capture_cos_bucket: IBM Cloud Object Storage(COS) bucket name
• capture_cos_region: IBM Cloud Object Storage(COS) bucket region
• capture_cos_secret_key: IBM Cloud Object Storage(COS) secret key
• key_pair_name: SSH key name present in the Power VS
• kubernetes_deb_version: Kubernetes deb version, e.g: 1.24.2-00
• kubernetes_rpm_version: Kubernetes RPM package version, e.g: 1.24.2-0
• kubernetes_semver: e.g: v1.24.2
• kubernetes_series: e.g: v1.24
• region: Power VS region, e.g: osa
• service_instance_id: Power VS service instance ID
• ssh_private_key_file: Path to the SSH private key file used to connect to the vm while image preparation, e.g: /Users/manjunath/.ssh/id_rsa
• zone: Power VS zone, e.g: osa21

# Clone the image-builder repository
$ git clone https://github.com/kubernetes-sigs/image-builder.git
$ cd image-builder/images/capi
$ PACKER_VAR_FILES=user-variables.json make build-powervs-centos-8

Modules and tools dependencies

CAPI Packages

• K8s Packages

• Test Packages

Note: The K8s and Test packages are subject to updates with each new CAPI package release.

IBM Packages

Tools used by E2E tests.

Other Tools

Troubleshooting

Reference

API References

Cluster API Provider IBMCloud currently exposes the following APIs:

• The Cluster API Provider IBMCloud Custom Resource Definitions (CRDs): documentation

• Golang APIs: godoc

Regions-Zones Mapping

References:

1. IBM Cloud doc for the PowerVS locations
2. IBM Cloud doc for the VPC Regions