With the deprecation of Pod Security Policies in Kubernetes 1.24 and the re-introduction of it as an admission controller which is still in the beta phase, it's complicated to use PSP/equivalent in larger organizations. We can use them but it's still in the beta phase, so we aren't sure of the surprises Kubernetes is gonna bring us. So, what can we do about it?
Introduction to Admission Webhooks
Admission webhooks in Kubernetes allows us to enforce policies to resource creations & deployments. Some tools help us to write extensible policies to achieve better control over the environment like OPA Gatekeeper, Kyverno, etc.
Challenges
OPA Gatekeeper rules are written in Rego language which is a pain to write & maintain. Kyverno offers a much simpler solution but it's not very flexible/extensible like OPA Gatekeeper. Though Gatekeeper is having complexity, it's effective to write custom rules according to the infrastructure.
For infrastructures compromising different teams/large numbers, it's easier to go with Kyverno/similar for the ease of getting started. But if we want to achieve granular level customizations, we have to go with OPA Gatekeeper. It's often complex to write rego rules for each customization. How do we achieve this?
Solution
We need to develop a solution that's extensible, easy to use & maintain. The gatekeeper team has done an amazing job with creating a library of rules. This helps to achieve a better understanding of the different rules we can write in OPA Gatekeeper.
If that's amazing, where's the catch? Well, it's easy to use but difficult to configure when we want to use it across multiple teams/clusters. The developers/DevOps/other teams have to dive into the templates & CRDs to further customize it. Unfortunately, not everyone is good with customizing rego rules/editing templates. Also, it's not an ideal approach to create multiple files for each customization.
Taking inspiration from the OPA team's work on rules library, we have customized their work to integrate with Helm.
Now, developers/DevOps/other teams can install rules with just the helm install command. Since we are using helm, the values of all templates/CRDs are available in values.yaml. This single file works as an interface for the end-users to understand/gain information on the list of rules/policies that are applied.
globalImport:
includeNamespaces:
- "default"
excludeNamespaces:
- "kube-system"
includeNamespacesDefaultFlag: true
excludeNamespacesDefaultFlag: false
K8sPSPAllowPrivilegeEscalationContainer:
includeNamespacesDefaultFlag: false
excludeNamespacesDefaultFlag: false
includeNamespaces:
- "default"
excludeNamespaces:
- "kube-system"
scope: "Cluster"
K8sPSPCapabilities:
includeNamespacesDefaultFlag: false
excludeNamespacesDefaultFlag: true
includeNamespaces:
- "default"
excludeNamespaces:
- "kube-system"
scope: "Cluster"
parameters:
allowedCapabilities:
- "hello"
requiredDropCapabilities:
- "KILL"
- "MKNOD"
- "SETUID"
- "SETGID"
exemptImages:
- "nginx:latest"
...
In daily use, there will be occurrences where we have to add exclusions like excluding a namespace from the specific rule(s). So, we have created something called globalExcludeNamespace in values.yaml. You can add the list of namespaces you want to exclude at the global level. The helm template will parse the input & add the list of namespaces at the global level in the exclusion list for all rules. You can override this exclusion by toggling the excludeNamespacesDefaultFlag variable. This makes it easy to organize & understand things. Similarly, we have given a feature to specify includeNamespacesDefaultFlag but it's recommended not to use it because by default the rules are applied for all namespaces.
Helm installation gives the luxury of creating multiple releases with a different set of rules like one chart for PSPs, one chart for general rules, etc. You can edit/delete the set of rules easily with helm.
helm list -A will return the list of installed release information.
Code outline & structure
Each rule/entry in values.yaml looks something like this. You can tweak the values accordingly & customize the templating according to your use case.
K8sPSPForbiddenSysctls:
includeNamespacesDefaultFlag: true
excludeNamespacesDefaultFlag: false
includeNamespaces:
- "default"
excludeNamespaces:
- "kube-system"
scope: "Cluster"
parameters:
forbiddenSysctls:
- "*"
The associated template for the above rule looks like as below.
{{- if .Values.K8sPSPForbiddenSysctls -}}
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sPSPForbiddenSysctls
metadata:
name: psp-forbidden-sysctls-{{.Release.Name}}
spec:
match:
kinds:
- apiGroups: [""]
kinds: ["Pod"]
{{- /* Check if any custom values are defined */ -}}
{{- if .Values.K8sPSPForbiddenSysctls}}
{{- if or (.Values.K8sPSPForbiddenSysctls.includeNamespaces) (hasKey .Values.K8sPSPForbiddenSysctls "includeNamespacesDefaultFlag")}}
{{/* Check if any specific namespaces are defined */ -}}
namespaces:
{{- /* Check for globalimport include */ -}}
{{- if hasKey .Values.K8sPSPForbiddenSysctls "includeNamespacesDefaultFlag"}}
{{- if .Values.K8sPSPForbiddenSysctls.includeNamespacesDefaultFlag}}
{{- range .Values.globalImport.includeNamespaces}}
- {{. | quote -}}
{{end -}}
{{end -}}
{{- else if .Values.globalImport.includeNamespacesDefaultFlag}}
{{- range .Values.globalImport.includeNamespaces}}
- {{. | quote -}}
{{end -}}
{{end -}}
{{- range .Values.K8sPSPForbiddenSysctls.includeNamespaces}}
- {{. | quote -}}
{{end -}}
{{end -}}
{{/* Check if excludeNamespaces are defined */ -}}
{{- if or (.Values.K8sPSPForbiddenSysctls.excludeNamespaces) (hasKey .Values.K8sPSPForbiddenSysctls "excludeNamespacesDefaultFlag")}}
excludedNamespaces:
{{- /* Check for globalimport include */ -}}
{{- if hasKey .Values.K8sPSPForbiddenSysctls "excludeNamespacesDefaultFlag"}}
{{- if .Values.K8sPSPForbiddenSysctls.excludeNamespacesDefaultFlag}}
{{- range .Values.globalImport.excludeNamespaces}}
- {{. | quote -}}
{{end -}}
{{end -}}
{{- else if .Values.globalImport.excludeNamespacesDefaultFlag}}
{{- range .Values.globalImport.excludeNamespaces}}
- {{. | quote -}}
{{end -}}
{{end -}}
{{- range .Values.K8sPSPForbiddenSysctls.excludeNamespaces}}
- {{. | quote -}}
{{end -}}
{{end -}}
{{/* Check if any scope is defined (Cluster/Namespace) */ -}}
{{- if .Values.K8sPSPForbiddenSysctls.scope}}
scope: {{.Values.K8sPSPForbiddenSysctls.scope | quote -}}
{{end -}}
{{end -}}
{{- /* Check if any custom values are defined */ -}}
{{- if .Values.K8sPSPForbiddenSysctls}}
{{- if .Values.K8sPSPForbiddenSysctls.parameters}}
{{- if .Values.K8sPSPForbiddenSysctls.parameters.forbiddenSysctls}}
parameters:
forbiddenSysctls:
{{- range .Values.K8sPSPForbiddenSysctls.parameters.forbiddenSysctls}}
- {{. | quote -}}
{{end -}}
{{end}}
{{end}}
{{end}}
{{- end -}}
Installation
Clone https://github.com/rewanthtammana/gatekeeper-rules-helm-library
Create all CRDs. The CRDs are available in ./crds folder. bash kubectl create -f ./crds/general/ kubectl create -f ./crds/pod-security-policy/
Install the templates. The values.yaml can be tweaked to adjust template values. bash helm install rules-helm .
Conclusion
This kind of structuring allows simple management with Gatekeeper rules & allows the rules library for easy extension. Separate values.yaml can be created. For example, psp-values.yaml with PSP template values, general-values.yaml with the general template values, etc. removing the complexity from the operations team.
References
https://github.com/rewanthtammana/gatekeeper-helm-library
https://github.com/open-policy-agent/gatekeeper-library
Authors
Siddharth Tanna
Rewanth Tammana
