A simple introduction to mirrord

Mirrord is a tool that can enrich your CI/CD-pipeline when developing software running on Kubernetes. It does one thing and it does it well: Mirrord can intercept and steal/duplicate specific traffic on your remote Kubernetes cluster and send it to a local running process, imitating a remote pod of your cluster. That being said we can identify two key use cases.

Key Use Cases

As mentioned above, we can identify two key use cases:

• The first one is debugging and troubleshooting your Kubernetes application by sending traffic of a faulty pod e.g. right into the debugger of your local IDE.
• Mirrord also provides the necessary capabilities to test newly developed code super quick against staging environments and therefore can speed up development. It aims at reducing the costs associated with a staging environment by taking „Deploy to staging“ out of the loop below, where you can test new code on staging without going through your CI-pipeline.

Mirrord is being very non-invasive, letting multiple developers work simultaneously on the same environment without stepping on each other’s toes. This will become clear in the architecture section.

How does mirrord work?

Mirrord runs in two places. First, there is your local process (mirrord-layer), imitating the remote pod under scrutiny. Second, a short lived container (mirrod-agent) gets created in your Kubernetes cluster, sniffing and forwarding traffic that reaches the remote pod to mirrord-layer.

When mirrord gets initiated by mirrord exec --target <pod/remote-pod-under-scrutiny> <local process> the mirrord-layer hooks syscalls for your local process, overriding its network behaviour to listen to incoming traffic from the agent. It specifically does so by overriding libc functions through the LD_PRELOAD environment variable which can be used to load a shared library into a process before loading anything else. At this point mirrord provides its own implementations of file and socket functions having it read and write files and traffic remotely without changing a single line of code. The cumbersome task of overriding these libc functions on different systems is facilitated by Frida-gum’s handy inline hooking interceptor.

For example, when debugging a faulty node-js service in a cluster via:  mirrord exec --target pod/node-user-service yarn -- dev the  node development server gets booted with mirrord-layer loaded onto it. But the default port 3000 does not get bound, instead, mirrord under the hood creates its own port (eg. 3432) and binds it to the mirrord-agent. This agent, a Kubernetes Job, is in parallel deployed via kubectl on the cluster. Using setns The agent enters the container’s network namespace of the pod node-user-service. Once it has entered the namespace, it sniffs the network packets and mirrors the traffic to the local fake port 3432, giving us the illusion that our process is running on the remote pod.

Some interesting options

Network Traffic

By default mirrord does not steal incoming traffic from the remote pod but rather duplicates it and sends it to the local mirrord-layer where the traffic can be processed in the context of the cluster environment. Any outgoing traffic from mirrord-layer is dropped and hence the cluster can operate undisrupted. However, if mocking a pod is desired, all traffic to the target container can be rerouted using the --steal flag. Stealing can be enhanced with a filter, which checks request headers against regular expressions to decide if the request gets stolen or not.

File access

File access is relayed to the target pod by default. Once a request to open a file is issued by the local process, it is forwarded to the mirrord-agent which in turn hands it over to the container of the remote pod. This behaviour can also be disabled using the --fs-mode local flag to serve local files.

Environment variables

Environment variables present in the remote pod will be loaded into the local process. This way you can effortlessly use remote database connections for example. Environment variables can be overriden using the --override-env-vars-include/exclude.

Agent deployment mode

Using the --ephemeral-container mirrord agent will be launched in an ephemeral container, a relatively new Kubernetes feature. The ephemeral container shares by default common namespaces like the network namespace and resource allocations with the debugged pod.

Requirements

MacOS and Linux (64bit) are supported. Windows users can opt for WSL. Mirrord is available as a CLI-tool, a VS Code extension and an Intellij plugin, making debugging inside the IDE quite simple. It supports all languages/frameworks that use libc, which is the majority (PHP, Rust, Node, Python, Java, Kotlin, Ruby etc.) and specifically Go aswell.

A good alternative: Telepresence

Telepresence is probably the closest alternative to mirrord, a mature tool and definitely a good candidate if you do not like mirrord. However, here are the key differences between them:

• By overriding dynamically linked libc methods, mirrord can run on the local process level. Telepresence in comparison requires you to change local settings or needs to be run in a container.
• Mirrord lets you duplicate traffic and does not intercept/steal by default.
• Mirrord does not require you to install anything on the remote cluster. The remote mirrord-agent is automatically created and cleaned up by the local mirrord script.

Some final thoughts

By hooking into clib functions mirrord naturally integrates very smooth with the local development environment. The same smoothness can also be accomplished at cluster level through the use of ephemeral containers. However, clib detours on different systems are fragile by nature and will require steady maintenance for a stable development experience.

Mirrord offers a really helpful tool to tunnel into the context of a cloud microservice application with almost no setup required. It shines where development teams are concurrently debugging and testing staging environments without interfering with each other. It is by no means a substitute to full-flegded Kubernetes CI/CD tools like Skaffold, but rather a complement, smoothing out the difference between local development and staging environments.

Mirrord is convincincly fast to set up and easy to use. It is a young and promising project with a lively community on Discord. New features are likely to be added soon, giving the tool more flexibility. On your next Kubernetes debugging session you could definitely give it a shot! Best start here!