Deception in container environments

Valuable assets are now more than ever defined in containers, pipelines and clusters. Their ability to quickly spin up / down, and yet remain statically defined makes them both a moving target, while still being predictable.

You may think of embedding deception into this infrastructure differently to traditional environments, however we'll cover some easy deployment ideas making it just as easy.

To start off, we'll look at Canaries and how to add detection to your container environments.

Canary Containers:

Canaries can run as a container alongside your existing workloads within a cluster.

Simply spin them up as their own containers with launch commands, or declare them in YAML files.

A reachable and discoverable Canary workload can detect attackers who enumerate cluster services or attempt lateral movement from a compromised workload. Ensure the Canary is intentionally reachable from the workloads and namespaces whose compromise you want to detect, while preventing it from reaching sensitive production systems.

Consider the following personalities for your deployment:

Linux Database
Confluence 9
Drupal 10
GitLab Server
JBoss Login
Joomla Server
Kibana Server

Or customise them further with some extra services:

SSH Server
HTTP/S Webservers
File Transfer (FTP)
Custom TCP Service
MySQL
GIT Repository
Redis
MongoDB

We cover this topic further in our deployment guide↗.

Canary K8s Cluster:

A Canary can be positioned as a Kubernetes cluster alongside your real cluster hosts.

This is an attractive target since compromising the control plane of a cluster, provides the attacker access over all subordinate workloads.

To do this, simply deploy your preference in platform Canary to your cluster subnet, and start off with the "standard linux" personality.

From here we'll expand on this personality further with a couple custom TCP services to match the real deal.

You'll want to enable the following ports on your Canary's custom TCP services↗:

2379
2380
6443
10250
10256

Canarytokens:

Canarytokens are digital tripwires which will tip you off to an attacker's presence within existing resources.

For example, a credential deployed to a container, which raises an alert when exposed to and used by an attacker.

We'll cover a couple deployment ideas for these below:

Secrets in environment variables

Environment variables are a common way to inject information into a container and are one of the first places attackers will look after landing in a container.

Accessible through the env or printenv commands, or by reading the /proc/1/environ file.

Mint an AWS API Key Token↗ and expose it to your pod either through docker launch commands:

docker run -d --name billing-worker \
  -e AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE \
  -e AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY \
  -e AWS_DEFAULT_REGION=us-east-1 \
  your-image:latest

If you're a Kubernetes cluster administrator, declare the fake secrets in the workloads YAML:

apiVersion: v1
kind: Pod
metadata:
  name: billing-worker
spec:
  containers:
    - name: worker
      image: your-image:latest
      env:
        - name: AWS_ACCESS_KEY_ID
          value: "AKIAIOSFODNN7EXAMPLE"
        - name: AWS_SECRET_ACCESS_KEY
          value: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
        - name: AWS_DEFAULT_REGION
          value: "us-east-1"

The Token will be accessible from within the container:

/ # env | grep AWS
AWS_DEFAULT_REGION=us-east-1
AWS_ACCESS_KEY_ID=wJalrXUtnFEMI
AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

Mounted secrets:

Canarytokens can be embedded into a container as mounted secrets. A concept better explained in the Kubernetes documentation↗. Here we will once again mint a unique AWS API Key Token, and inject it into our container, right where we'd expect AWS Keys to exist.

In our example we’ll mount to /root/.aws file path since we have a root user directory present in the container.

This may differ in your environment, in which case you may want to mount the Token at different filepaths like /app or /app/secrets/.

This technique can similarly be used for Slack API Keys↗, WireGuard VPN Profiles↗ and Guardrail Triggers↗.

apiVersion: v1
kind: Secret
metadata:
  name: aws-dotfile-fake-secret
  namespace: production
stringData:
  credentials: |
    [default]
    aws_access_key_id = AKIAIOSFODNN7EXAMPLE
    aws_secret_access_key = wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
    region = us-east-1
---
apiVersion: v1
kind: Pod
metadata:
  name: billing-worker
  namespace: production
spec:
  volumes:
    - name: aws-creds
      secret:
        secretName: aws-dotfile-fake-secret
        defaultMode: 0400
  containers:
    - name: worker
      image: your-image:latest
      volumeMounts:
        - name: aws-creds
          readOnly: true
          mountPath: "/root/.aws"   # the hidden file appears as /root/.aws/credentials

The Token file and credentials will now be accessible from within the container:

root@canarypod:~# ls -a
.  ..  .aws  .bashrc  .cache  .profile
root@canarypod:~# cd .aws
root@canarypod:~/.aws# ls
credentials
root@canarypod:~/.aws# cat credentials 
[default]
aws_access_key_id = AKIAIOSFODNN7EXAMPLE
aws_secret_access_key = wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
region = us-east-1

The Kubeconfig Token

Kubeconfig files↗ are self-contained text files with appropriate credentials and configuration to help a tool connect to a cluster.

This also means that they're particularly useful for attackers to find since they point directly to a cluster host for remote management.

Our Kubeconfig Token↗ is a small file which can be deployed onto bastion hosts, developer endpoints and administrator machines as bait for an attacker looking to pivot to your clusters.

The file is right at home in its natural ~/.kube/config file path (visible in the KUBECONFIG environment variable).

Attackers that use the file to try to connect to the fake cluster defined within it will set off an alert.