Gatekeeper Rules Helm Library
An armor to the traditional gatekeeper rules library with helm templatization to ease the operational & maintenance overhead
With the deprecation of Pod Security Policies in Kubernetes 1.24 and the re-introduction of it as an admission controller which is still in the beta phase, it's complicated to use PSP/equivalent in larger organizations. We can use them but it's still in the beta phase, so we aren't sure of the surprises Kubernetes is gonna bring us. So, what can we do about it?
Introduction to Admission Webhooks
Admission webhooks in Kubernetes allows us to enforce policies to resource creations & deployments. Some tools help us to write extensible policies to achieve better control over the environment like OPA Gatekeeper, Kyverno, etc.
Challenges
OPA Gatekeeper rules are written in Rego language which is a pain to write & maintain. Kyverno offers a much simpler solution but it's not very flexible/extensible like OPA Gatekeeper. Though Gatekeeper is having complexity, it's effective to write custom rules according to the infrastructure.
For infrastructures compromising different teams/large numbers, it's easier to go with Kyverno/similar for the ease of getting started. But if we want to achieve granular level customizations, we have to go with OPA Gatekeeper. It's often complex to write rego rules for each customization. How do we achieve this?
Solution
We need to develop a solution that's extensible, easy to use & maintain. The gatekeeper team has done an amazing job with creating a library of rules. This helps to achieve a better understanding of the different rules we can write in OPA Gatekeeper.
If that's amazing, where's the catch? Well, it's easy to use but difficult to configure when we want to use it across multiple teams/clusters. The developers/DevOps/other teams have to dive into the templates & CRDs to further customize it. Unfortunately, not everyone is good with customizing rego rules/editing templates. Also, it's not an ideal approach to create multiple files for each customization.
Taking inspiration from the OPA team's work on rules library, we have customized their work to integrate with Helm.
Now, developers/DevOps/other teams can install rules with just the helm install
command. Since we are using helm, the values of all templates/CRDs are available in values.yaml
. This single file works as an interface for the end-users to understand/gain information on the list of rules/policies that are applied.
# Source: https://github.com/rewanthtammana/gatekeeper-rules-helm-library/blob/main/values.yaml
globalImport:
includeNamespaces:
- "default"
excludeNamespaces:
- "kube-system"
includeNamespacesDefaultFlag: true
excludeNamespacesDefaultFlag: false
K8sPSPAllowPrivilegeEscalationContainer:
includeNamespacesDefaultFlag: false #Overrides global flag. Remove field, if not required
excludeNamespacesDefaultFlag: false #Overrides global flag. Remove field, if not required
includeNamespaces:
- "default"
excludeNamespaces:
- "kube-system"
scope: "Cluster"
K8sPSPCapabilities:
includeNamespacesDefaultFlag: false #Overrides global flag. Remove field, if not required
excludeNamespacesDefaultFlag: true #Overrides global flag. Remove field, if not required
includeNamespaces:
- "default"
excludeNamespaces:
- "kube-system"
scope: "Cluster"
parameters:
allowedCapabilities:
- "hello"
requiredDropCapabilities:
- "KILL"
- "MKNOD"
- "SETUID"
- "SETGID"
exemptImages:
- "nginx:latest"
...
In daily use, there will be occurrences where we have to add exclusions like excluding a namespace from the specific rule(s). So, we have created something called globalExcludeNamespace
in values.yaml
. You can add the list of namespaces you want to exclude at the global level. The helm template will parse the input & add the list of namespaces at the global level in the exclusion list for all rules. You can override this exclusion by toggling the excludeNamespacesDefaultFlag
variable. This makes it easy to organize & understand things. Similarly, we have given a feature to specify includeNamespacesDefaultFlag
but it's recommended not to use it because by default the rules are applied for all namespaces.
Helm installation gives the luxury of creating multiple releases with a different set of rules like one chart for PSPs, one chart for general rules, etc. You can edit/delete the set of rules easily with helm.
helm list -A
will return the list of installed release information.
Code outline & structure
Each rule/entry in values.yaml
looks something like this. You can tweak the values accordingly & customize the templating according to your use case.
K8sPSPForbiddenSysctls:
includeNamespacesDefaultFlag: true #Overrides global flag. Remove field, if not required
excludeNamespacesDefaultFlag: false #Overrides global flag. Remove field, if not required
includeNamespaces:
- "default"
excludeNamespaces:
- "kube-system"
scope: "Cluster"
parameters:
forbiddenSysctls:
- "*"
The associated template for the above rule looks like as below.
# Source: https://github.com/rewanthtammana/gatekeeper-rules-helm-library/blob/main/templates/pod-security-policy/forbidden-sysctls.yaml
{{- if .Values.K8sPSPForbiddenSysctls -}}
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sPSPForbiddenSysctls
metadata:
name: psp-forbidden-sysctls-{{.Release.Name}}
spec:
match:
kinds:
- apiGroups: [""]
kinds: ["Pod"]
{{- /* Check if any custom values are defined */ -}}
{{- if .Values.K8sPSPForbiddenSysctls}}
{{- if or (.Values.K8sPSPForbiddenSysctls.includeNamespaces) (hasKey .Values.K8sPSPForbiddenSysctls "includeNamespacesDefaultFlag")}}
{{/* Check if any specific namespaces are defined */ -}}
namespaces:
{{- /* Check for globalimport include */ -}}
{{- if hasKey .Values.K8sPSPForbiddenSysctls "includeNamespacesDefaultFlag"}}
{{- if .Values.K8sPSPForbiddenSysctls.includeNamespacesDefaultFlag}}
{{- range .Values.globalImport.includeNamespaces}}
- {{. | quote -}}
{{end -}}
{{end -}}
{{- else if .Values.globalImport.includeNamespacesDefaultFlag}}
{{- range .Values.globalImport.includeNamespaces}}
- {{. | quote -}}
{{end -}}
{{end -}}
{{- range .Values.K8sPSPForbiddenSysctls.includeNamespaces}}
- {{. | quote -}}
{{end -}}
{{end -}}
{{/* Check if excludeNamespaces are defined */ -}}
{{- if or (.Values.K8sPSPForbiddenSysctls.excludeNamespaces) (hasKey .Values.K8sPSPForbiddenSysctls "excludeNamespacesDefaultFlag")}}
excludedNamespaces:
{{- /* Check for globalimport include */ -}}
{{- if hasKey .Values.K8sPSPForbiddenSysctls "excludeNamespacesDefaultFlag"}}
{{- if .Values.K8sPSPForbiddenSysctls.excludeNamespacesDefaultFlag}}
{{- range .Values.globalImport.excludeNamespaces}}
- {{. | quote -}}
{{end -}}
{{end -}}
{{- else if .Values.globalImport.excludeNamespacesDefaultFlag}}
{{- range .Values.globalImport.excludeNamespaces}}
- {{. | quote -}}
{{end -}}
{{end -}}
{{- range .Values.K8sPSPForbiddenSysctls.excludeNamespaces}}
- {{. | quote -}}
{{end -}}
{{end -}}
{{/* Check if any scope is defined (Cluster/Namespace) */ -}}
{{- if .Values.K8sPSPForbiddenSysctls.scope}}
scope: {{.Values.K8sPSPForbiddenSysctls.scope | quote -}}
{{end -}}
{{end -}}
{{- /* Check if any custom values are defined */ -}}
{{- if .Values.K8sPSPForbiddenSysctls}}
{{- if .Values.K8sPSPForbiddenSysctls.parameters}}
{{- if .Values.K8sPSPForbiddenSysctls.parameters.forbiddenSysctls}}
parameters:
forbiddenSysctls:
{{- range .Values.K8sPSPForbiddenSysctls.parameters.forbiddenSysctls}}
- {{. | quote -}}
{{end -}}
{{end}}
{{end}}
{{end}}
{{- end -}}
Installation
Create all CRDs. The CRDs are available in
./crds
folder.bash kubectl create -f ./crds/general/ kubectl create -f ./crds/pod-security-policy/
Install the templates. The
values.yaml
can be tweaked to adjust template values.bash helm install rules-helm .
Conclusion
This kind of structuring allows simple management with Gatekeeper rules & allows the rules library for easy extension. Separate values.yaml
can be created. For example, psp-values.yaml
with PSP template values, general-values.yaml
with the general template values, etc. removing the complexity from the operations team.
References
https://github.com/rewanthtammana/gatekeeper-helm-library
https://github.com/open-policy-agent/gatekeeper-library