Introduction to Original Destination in Envoy

Envoy is a dynamically configurable, open-source service mesh proxy with a ton of built-in capabilities. In this post, I’ll be discussing the why, what, and how of the Original Destination feature in Envoy.

Introduction

The Envoy proxy provides a set of xDS APIs via which a control plane can program and control its behavior. Following are some of the key configurable elements in Envoy.

• Listener — A listener typically opens a port in Envoy for a downstream client to send traffic to.
• Filter — A filter reads metadata about incoming connections/requests and enhances it further to affect routing decisions.
• Route — A route is used for mapping incoming HTTP requests using its attributes to clusters.
• Cluster — A cluster is generally a collection of endpoints, configured with a load balancing policy.
• Endpoint — An endpoint is the address of an upstream server or another Envoy if it's a service mesh.

Generally, most of these components have to be pre-configured either statically or dynamically in order for Envoy to start routing traffic. However, there can be cases where traffic needs to be routed more dynamically i.e. without prior knowledge of the destination.

A couple of examples where you cannot/you’d prefer not to pre-configure endpoints are if:

• You want to literally route to previously unknown destinations, think of a general egress proxy.
• You want to route to user-specified, arbitrary, upstream addresses i.e. without load balancing.

The Original Destination implementation in Envoy can be used to solve this.

Original Destination

There are 2 main Original Destination related components available in Envoy.

1. Original Destination Listener Filter
2. Original Destination Cluster

Original Destination Listener Filter

The Original Destination listener filter reads the SO_ORIGINAL_DST socket option at the listener for incoming connections that were redirected to Envoy using iptables rules.

This filter can parse connection metadata to identify the original destination address that the client intended to send traffic to.

Original Destination Cluster

The Original Destination cluster dynamically modifies the cluster state to handle new upstream addresses at runtime.

This cluster acts like a load balancer that always routes to a single host i.e. just to the original destination address, but this host can differ for each connection.

Original Destination in Action

Setup

To see Original Destination in action, we first need to set up an Envoy container with the Original Destination listener and cluster configured. And, to further see iptables in action, we can use docker-compose to set up another “sleep” container with a shared network.

For this example, I’ve set up the following directory structure.

$ tree .
.
├── Dockerfile
├── docker-compose.yaml
└── envoy.yaml

The Dockerfile looks like so:

Here’s the envoy.yaml:

And finally, here’s the docker-compose.yaml:

Demo

The first step is to start the docker containers using the docker-compose up command. We’re then ready to run the rest of the commands from within the “sleep” container.

# Route traffic from the "sleeper" user on port 80 to Envoy.
$ docker exec -u root -it sleep \
    iptables -t nat \
             -A OUTPUT \
             -p tcp \
             --dport 80 \
             -j REDIRECT \
             --to-port 9000 \
             -m owner \
             --uid-owner sleeper
# Traffic on port 80 is routed to Envoy on port 9000 via iptables.
# The DNS within the sleep container resolves httpbin.org to an IP
# like 54.166.163.67, which Envoy then identifies and routes to.
$ docker exec -it sleep curl httpbin.org:80/status/418

    -=[ teapot ]=-
       _...._
     .'  _ _ `.
    | ."` ^ `". _,
    \_;`"---"`|//
      |       ;/
      \_     _/
        `"""`
# Traffic is routed to Envoy on port 9001 with a special header.
# The special x-envoy-original-dst-host header tells the Original
# Destination cluster to route to the said address.
$ docker exec -it sleep \
    curl -H "x-envoy-original-dst-host: 54.166.163.67:80" \
         127.0.0.1:9001/status/418

    -=[ teapot ]=-
       _...._
     .'  _ _ `.
    | ."` ^ `". _,
    \_;`"---"`|//
      |       ;/
      \_     _/
        `"""`

Note that enabling Original Destination based routing essentially allows any user to literally route to any other host, so it can be potentially misused without proper controls (like mTLS, RBAC, etc.)

Conclusion

The Original Destination feature in Envoy can be a powerful tool to route to previously unknown destinations in cases where it’s either undesirable to preconfigure hosts, or if it’s not possible to do so. Using this feature also calls for proper security measures to be in place to avoid potential misuse.

Thanks for reading through! Please leave me a comment if you have any feedback.

If you liked this article, do check out my other articles on venilnoronha.io.

Disclaimer: My postings are my own and don’t necessarily represent Stripe’s positions, strategies or opinions.