beesting

Research project to better understand how to inject and monitor honey tokens into a kubernetes workload, with minimal configurations.

You can find the individual research directions by looking at the poc-vx branches.

PoCv1

This PoC focused on injecting the honey token via a OCI hook that runs on container creation.

For the hooks to be registered with the container runtime, the NRI (node resource interface) is used. NRI appears to be a standardized way supported by containerD, as well as CRI-O (and probably others, this still needs to be evaluated) of hooking into the creation of containers and modifying them before creation. Kinda like a modifying admission web hook, but instead of Kubernetes API level, it is on container runtime level.

The agent embedded a second hook binary, that gets unpacked on startup. This hook binary then writes the honey token at a specified location inside the container.

More details can be found here.

Pros

• Restarts of the nodes should not matter here, as the container filesystem gets restored
• Very flexible, as the hook can setup anything in the container it wants

Cons

• Requries a hostPath volume, which does allow executions

PoCv2

This PoC mounts a token file into the container via a bind mount. To achieve this, once again NRI was used to modify the container before it is send to the low level runtime.

More details can be found here.

Pros

• hostPath does not need to allow executions

Cons

• Restarts of the nodes will cause the token file to be lost, as it is a bind mount into the container

PoCv3

Based on PoCv2, when the token file is mounted into the container, the inode is retrieved and added to eBPF map for monitoring. There is an eBPF program that attaches a kprobe to the security_file_open function, which tests if the inode of the opened file is in the map of watched inodes. For the PoC this is good enough, but additional security function would need to be hooked to also catch creating of symlinks.

If a file open for a monitored inode is detected, a message to an ringbuffer is send. The sent event contains the timestamp when the event was recorded, the process identity (which is PID, as well as start time), the cgroup_id and the accessed inode. In userspace this event is then parsed and transformed to an alert to the user.

To also be able to monitor existing containers, the Synchronize NRI hook is used, which will pass all already existing containers. Beesting then watches the inodes of the corresponding container token files.

More details can be found here.

Pro

• Easy to get inodes to watch since all of them are also accessible from the beesting pod

Cons

• Restarts of the nodes will cause token files to be lost