Safe K8s Deployment with Open Policy Agent

Introduction

This demo shows how to enforce security policies for K8s deployments using Open Policy Agent, and the corresponding Gatekeeper project.

For details on the demo application see hello spring boot application.

Open Policy Agent

Before diving into the Kubernetes part with Gatekeeper let's look into Open Policy Agent how open policies work in general.

OPA decouples policy decision-making from policy enforcement. When your software needs to make policy decisions it queries OPA and supplies structured data (e.g., JSON) as input. OPA works similar to a BPM engine, or a state machine by decoupling processes from input and output data.

opa

(Source: https://www.openpolicyagent.org)

Policies in OPA are written using OPA's own language called Rego.

package kubernetes.admission                                                

deny[msg] {                                                                 
  input.request.kind.kind == "Pod"                                          
  image := input.request.object.spec.containers[_].image                    
  not startswith(image, "myreg.com/")                                       
  msg := sprintf("image '%v' comes from untrusted registry", [image])       
}

This example checks if the container image to be deployed on a K8s cluster origins from a trusted container registry called (myreg.com).

To evaluate and play with Rego you can use The Rego Playground

Gatekeeper

Gatekeeper installs an admission controller on K8s that contains the Open Policy Agent to enforce policies for deployments on the Kubernetes cluster.

opa

(Source: https://www.openpolicyagent.org)

Lab

To install Gatekeeper just follow the installation instructions or use the script deploy-gatekeeper.sh in this folder.

Rego policies cannot be deployed directly into a K8s cluster, instead Gatekeeper uses the OPA Constraint Framework.

Here you first deploy a constraint template and then a corresponding constraint using the template. In this step we will enforce that all Pod specifications require to include a security context disallowing privilege escalation by setting allowPrivilegeEscalation to false.

To enforce this, please execute the script deploy-constraint.sh in this folder.

Deploy the application (will be denied)

First we want to see the Gatekeeper in action so that our deployment is denied because of allowing privilege escalation. The corresponding container image is pulled from andifalk/hello-rootless-jib docker hub repository.

The application is deployed using the following deployment yaml file k8s/deploy_denied.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: hello-opa-gatekeeper-denied
  name: hello-opa-gatekeeper-denied
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hello-opa-gatekeeper-denied
  template:
    metadata:
      labels:
        app: hello-opa-gatekeeper-denied
    spec:
      automountServiceAccountToken: false
      securityContext:
        runAsNonRoot: true
      containers:
      - image: andifalk/hello-rootless-jib:latest
        name: hello-opa-gatekeeper-denied
        resources:
          limits:
            cpu: "1"
            memory: "512Mi"
          requests:
            cpu: "0.5"
            memory: "256Mi"
        securityContext:
          readOnlyRootFilesystem: true
          allowPrivilegeEscalation: true
          privileged: false
          runAsNonRoot: true
          capabilities:
            drop:
              - ALL
        readinessProbe:
          httpGet:
            path: /
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 5
        livenessProbe:
          httpGet:
            path: /actuator/health
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 5
        volumeMounts:
          - name: tmp-volume
            mountPath: /tmp
      restartPolicy: Always
      volumes:
        - name: tmp-volume
          emptyDir: {}

You will get an error message from the Gatekeeper denying the deployment.

admission webhook "validation.gatekeeper.sh" denied the request: [denied
      by psp-allow-privilege-escalation-container] Privilege escalation container
      is not allowed: hello-opa-gatekeeper-denied'

Deploy the application (will be deployed successfully)

Now we want to see that Gatekeeper accepts our deployment because of disallowing privilege escalation now. The corresponding container image is pulled from andifalk/hello-rootless-jib docker hub repository.

The application is deployed using the following deployment yaml file k8s/deploy.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: hello-opa-gatekeeper
  name: hello-opa-gatekeeper
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hello-opa-gatekeeper
  template:
    metadata:
      labels:
        app: hello-opa-gatekeeper
    spec:
      automountServiceAccountToken: false
      securityContext:
        runAsNonRoot: true
      containers:
      - image: andifalk/hello-rootless-jib:latest
        name: hello-opa-gatekeeper
        resources:
          limits:
            cpu: "1"
            memory: "512Mi"
          requests:
            cpu: "0.5"
            memory: "256Mi"
        securityContext:
          readOnlyRootFilesystem: true
          allowPrivilegeEscalation: false
          privileged: false
          runAsNonRoot: true
          capabilities:
            drop:
              - ALL
        readinessProbe:
          httpGet:
            path: /
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 5
        livenessProbe:
          httpGet:
            path: /actuator/health
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 5
        volumeMounts:
          - name: tmp-volume
            mountPath: /tmp
      restartPolicy: Always
      volumes:
        - name: tmp-volume
          emptyDir: {}

This should now be deployed without any error.