Kubernetes comes with resources like Pods, Deployments, Configmaps, PersistentVolumes & many more. Kubernetes is extensible & allows users to create Custom Resources (CR). Before creating a CR, it's required to create a Custom Resource Definition (CRD) that will define the structure & meta-attributes of the future resource.
For example, the Prometheus Operator adds a "PrometheusRule" resource. Alerts can be defined and edited with kubectl edit prometheusrule some-rule.
- Natan Yellin, CEO Robusta.dev
There are many ways to create CRDs for Kubernetes. You can write CRDs from scratch as well, but some amazing tools out there will scaffold the skeleton structure for you to make things easier. A few of them are kubebuilder, operator-sdk, etc.
Kubernetes operators require you to define & create CRDs. Often, when you develop operators, the basic requirement would be validating different fields in CRDs. There are many ways in kubebuilder to perform basic validations like setting maximum/minimum length of a field, required/optional validation checks for a field, etc.
Before Kubernetes 1.25, the only way to create complex validations in CRDs was to write & deploy a validating webhook. Each CRD would have its own validating webhook deployment running on the system. This is an operational & development overhead when you have to develop & deploy numerous CRDs. This issue is addressed in the Kubernetes 1.25 release with the introduction of CEL(Common Expression Language) validation rules. In this post, we will see the process of creating immutable CRDs before & after introduction of CEL in Kubernetes.
Warning!!!
Kindly note this feature is still in the beta phase & is subject to changes. The following sections assume you know basics of Golang, Kubernetes, & operator development.
Task
In this demo, we will try to create an immutable CRD, i.e., no one can edit the object once the CR is made. If someone tries to edit the object, the API server must reject the change & throw an error.
For this demo, we will use kubebuilder. Code available here - rewanthtammana/crd-immutable-validation-webhook
CRD validation in Kubernetes 1.23
As mentioned above, we need to create a webhook for validation. But before that, let's scaffold the skeleton.
Create a Kubernetes 1.23 cluster
Create a repository & initialize it with go mod.
mkdir /tmp/one cd /tmp/one go mod init one
Initialize kubebuilder
On M1,
kubebuilder init --domain rewanthtammana.com --license none --owner "rewanthtammana" --plugins=go/v4-alpha
Others,
kubebuilder init --domain rewanthtammana.com --license none --owner "rewanthtammana"
Create an API with ImmutableKind. Say yes to creating a controller & resource.
kubebuilder create api --version v1 --group validate --kind ImmutableKind
Create a webhook for validation
kubebuilder create webhook --group validate --version v1 --kind ImmutableKind --programmatic-validation
Add the validating webhook logic to the codebase, api/v1/immutablekind_webhook.go. In this case, we want our object to be immutable, so all update operations must be blocked.
func (r *ImmutableKind) ValidateUpdate(old runtime.Object) error {
immutablekindlog.Info("validate update", "name", r.Name)
return apierrors.NewForbidden(
schema.GroupResource{
Group: "validate.rewanthtammana.com",
Resource: "ImmutableKind",
}, r.Name, &field.Error{
Type: field.ErrorTypeForbidden,
Field: "*",
BadValue: r.Name,
Detail: "Invalid value: \"object\": Value is immutable",
},
)
}
The default webhook will not work because of certificate issues. To fix this, you must install cert-manager for operational ease.
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.9.1/cert-manager.yaml
Edit configurations from the skeleton to enable webhook, perform cainjection, etc. deployments
Uncomment patches/webhook_in_immutablekinds.yaml and patches/cainjection_in_immutablekinds.yaml in config/crd/kustomization.yaml
Uncomment ../certmanager and ../webhook directories, manager_webhook_patch.yaml & entire CERTMANAGER replacements block in config/default/kustomization.yaml
Create custom CRDs
make manifests
Build & push the webhook code logic to dockerhub. In this case, I'm pushing the image to my personal dockerhub account for deployment. You can change the image name accordingly.
make docker-build docker-push IMG=rewanthtammana/immutablekindwebhook:v1
Install & deploy the CRD & webhook
make install deploy IMG=rewanthtammana/immutablekindwebhook:v1
Deploy a sample CR.
kubectl apply -f ./config/samples/validate_v1_immutablekind.yaml
apiVersion: validate.rewanthtammana.com/v1
kind: ImmutableKind
metadata:
labels:
app.kubernetes.io/name: immutablekind
app.kubernetes.io/instance: immutablekind-sample
app.kubernetes.io/part-of: immutable-validation-webhook
app.kuberentes.io/managed-by: kustomize
app.kubernetes.io/created-by: immutable-validation-webhook
mutate: maybe
name: immutablekind-sample
spec:
To validate the immutability feature let's edit the deployed CR.
To keep things simple, let's remove all labels from the above snippet & just deploy the immutablekind-sample CR again.
echo "apiVersion: validate.rewanthtammana.com/v1
kind: ImmutableKind
metadata:
name: immutablekind-sample" | kubectl apply -f-
The Kubernetes API server should throw an error blocking the update.
CRD validation in Kubernetes 1.25
With the introduction of fantastic CEL, we can see a clear difference in the complexity of implementing CRD validation. The initial scaffolding steps are gonna remain the same.
Create a Kubernetes 1.25 cluster
Create a repository & initialize it with go mod.
mkdir /tmp/two
cd /tmp/two
go mod init two
Initialize kubebuilder
On M1,
kubebuilder init --domain rewanthtammana.com --license none --owner "rewanthtammana" --plugins=go/v4-alpha
Others,
kubebuilder init --domain rewanthtammana.com --license none --owner "rewanthtammana"
Create an API with ImmutableKind. Say yes to creating a controller & resource.
kubebuilder create api --version v1 --group validate --kind ImmutableKind
No need to create a webhook for CRD validation with CEL
We aren't validating a specific field in this task. We want to protect the entire object & all its subsequent fields
The best way to achieve our goal is to embed the kubebuilder marker comments for the entire kind struct object
The CEL immutable validation check looks as below
// +kubebuilder:validation:XValidation:rule="self == oldSelf", message="Value is immutable"
The above marker comment in CEL format is parsed by controller-gen to generate CRDs. The XValidation field in CEL rule translates to x-Kubernetes-validation in the CRD
The validation rule specified above, ensures that the new request object (self) is always equal to the old deployed object (oldSelf). If it's any different, the CEL validation throws an error message
A lot more granular validation on each field is possible with CEL. But it's not required for our demo use case
In this case, we created ImmutableKind struct & want to make sure it's CRs are immutable. Add the above validation marker comments to the struct
The ImmutableKind struct exists in api/v1/immutablekind_types.go
type ImmutableKind struct {
metav1.TypeMeta `json:",inline"`
metav1.ObjectMeta `json:"metadata,omitempty"`
Spec ImmutableKindSpec `json:"spec,omitempty"`
Status ImmutableKindStatus `json:"status,omitempty"`
}
Create custom CRDs & install them
make manifests
make install
Deploy a sample CR
kubectl apply -f ./config/samples/validate_v1_immutablekind.yaml
apiVersion: validate.rewanthtammana.com/v1
kind: ImmutableKind
metadata:
labels:
app.kubernetes.io/name: immutablekind
app.kubernetes.io/instance: immutablekind-sample
app.kubernetes.io/part-of: immutable-validation-webhook
app.kuberentes.io/managed-by: kustomize
app.kubernetes.io/created-by: immutable-validation-webhook
mutate: maybe
name: immutablekind-sample
spec:
To validate the immutability feature, let's edit the deployed CR.
To keep things simple, let's remove all labels from the above snippet & just deploy the immutablekind-sample CR again.
echo "apiVersion: validate.rewanthtammana.com/v1
kind: ImmutableKind
metadata:
name: immutablekind-sample" | kubectl apply -f-
The object update request will be failed.
Just a one-line marker comment, removed all the complexity of creating a webhook deployment, certificate management/requirement of cert-manager deployment, etc.
The CRD configuration is located in ./config/crd/bases/validate.rewanthtammana.com_immutablekinds.yaml
The above marker comment embeds x-Kubernetes-validations field to openAPIV3Schema when you generate the manifest files.
Conclusion
This is just the tip of the iceberg. We can achieve numerous other things with the combination of CEL & Kubernetes. You can check the references for further usage.
References
