Kyverno Resource Cleanup Rules

While deep diving on multiple admission controllers I came across a feature I had not expected but highly welcomed in Kyverno called clean up rules. One scenario of providing multi-tenant sandboxes would be that you need to eventually have a clean-up process for resources. Most solutions would involve a process outside of a kubernetes’ native functionality. This could be done with a cron job or a function app, that tracked the lifecycle of the resources. All things that would take a bit of engineering effort to build and monitor.

The gist of what will be accomplished is with a small minor tweak to Kyverno’s basic installation, Kyverno will monitor labels on resources and you can flag them for deletion.

Warning: this feature is listed as beta at the moment.

Prerequisites

Walk through the Kyverno documentation to install it into your kubernetes cluster.

You can modify the cluster role that Kyverno is using. In my example I have given it access to manage namespaces and some other objects that differ from the basic install.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    app.kubernetes.io/component: cleanup-controller
    app.kubernetes.io/instance: kyverno
    app.kubernetes.io/part-of: kyverno
    namespace: kyverno
  name: kyverno:cleanup-pods
rules:
- apiGroups:
  - ""
  resources:
  - pods 
  - namespaces # added namespaces
  verbs:
  - get
  - watch
  - list
  - delete

after applying the configuration to the namespace kyverno is implemented in, you should now be able to use the beta feature for cleaning up resources.

If you deploy a namespace and a resource within the namespace and provide the label: cleanup.kyverno.io/ttl , Kyverno will trigger the namespace to be deleted (along with any resources inside the namespace) after the value you place in the label has been reached.

apiVersion: v1
kind: Namespace
metadata:
  labels:
    cleanup.kyverno.io/ttl: 2m # in 2 mins, kyverno will delete the namespace
  name: test-delete
---
apiVersion: v1
kind: Pod
metadata:
  labels:
    team: test-delete
  namespace: test-delete
  name: foo
spec:
  containers:
    image: nginx:latest
    name: nginx

Conclusion

This is a great feature to keep your clusters clean of resources that you consider temporary. I can see this being used for ephemeral namespaces in sandbox clusters.