Introduction to Original Destination in Envoy

Envoy is a dynamically configurable, open-source service mesh proxy with a ton of built-in capabilities. In this post, I’ll be discussing the why, what, and how of the Original Destination feature in Envoy.

Introduction

The Envoy proxy provides a set of xDS APIs via which a control plane can program and control its behavior. Following are some of the key configurable elements in Envoy.

  • Listener — A listener typically opens a port in Envoy for a downstream client to send traffic to.
  • Filter — A filter reads metadata about incoming connections/requests and enhances it further to affect routing decisions.
  • Route — A route is used for mapping incoming HTTP requests using its attributes to clusters.
  • Cluster — A cluster is generally a collection of endpoints, configured with a load balancing policy.
  • Endpoint — An endpoint is the address of an upstream server or another Envoy if it's a service mesh.

Generally, most of these components have to be pre-configured either statically or dynamically in order for Envoy to start routing traffic. However, there can be cases where traffic needs to be routed more dynamically i.e. without prior knowledge of the destination.

A couple of examples where you cannot/you’d prefer not to pre-configure endpoints are if:

  • You want to literally route to previously unknown destinations, think of a general egress proxy.
  • You want to route to user-specified, arbitrary, upstream addresses i.e. without load balancing.

The Original Destination implementation in Envoy can be used to solve this.

Original Destination

There are 2 main Original Destination related components available in Envoy.

  1. Original Destination Listener Filter
  2. Original Destination Cluster

Original Destination Listener Filter

The Original Destination listener filter reads the SO_ORIGINAL_DST socket option at the listener for incoming connections that were redirected to Envoy using iptables rules.

This filter can parse connection metadata to identify the original destination address that the client intended to send traffic to.

Original Destination Cluster

The Original Destination cluster dynamically modifies the cluster state to handle new upstream addresses at runtime.

This cluster acts like a load balancer that always routes to a single host i.e. just to the original destination address, but this host can differ for each connection.

Original Destination in Action

Setup

To see Original Destination in action, we first need to set up an Envoy container with the Original Destination listener and cluster configured. And, to further see iptables in action, we can use docker-compose to set up another “sleep” container with a shared network.

For this example, I’ve set up the following directory structure.

$ tree .
.
├── Dockerfile
├── docker-compose.yaml
└── envoy.yaml

The Dockerfile looks like so:

Here’s the envoy.yaml:

And finally, here’s the docker-compose.yaml:

Demo

The first step is to start the docker containers using the docker-compose up command. We’re then ready to run the rest of the commands from within the “sleep” container.

# Route traffic from the "sleeper" user on port 80 to Envoy.
$ docker exec -u root -it sleep \
iptables -t nat \
-A OUTPUT \
-p tcp \
--dport 80 \
-j REDIRECT \
--to-port 9000 \
-m owner \
--uid-owner sleeper
# Traffic on port 80 is routed to Envoy on port 9000 via iptables.
# The DNS within the sleep container resolves httpbin.org to an IP
# like 54.166.163.67, which Envoy then identifies and routes to.
$ docker exec -it sleep curl httpbin.org:80/status/418

-=[ teapot ]=-
_...._
.' _ _ `.
| ."` ^ `". _,
\_;`"---"`|//
| ;/
\_ _/
`"""`
# Traffic is routed to Envoy on port 9001 with a special header.
# The special x-envoy-original-dst-host header tells the Original
# Destination cluster to route to the said address.
$ docker exec -it sleep \
curl -H "x-envoy-original-dst-host: 54.166.163.67:80" \
127.0.0.1:9001/status/418

-=[ teapot ]=-
_...._
.' _ _ `.
| ."` ^ `". _,
\_;`"---"`|//
| ;/
\_ _/
`"""`

Note that enabling Original Destination based routing essentially allows any user to literally route to any other host, so it can be potentially misused without proper controls (like mTLS, RBAC, etc.)

Conclusion

The Original Destination feature in Envoy can be a powerful tool to route to previously unknown destinations in cases where it’s either undesirable to preconfigure hosts, or if it’s not possible to do so. Using this feature also calls for proper security measures to be in place to avoid potential misuse.

Thanks for reading through! Please leave me a comment if you have any feedback.

If you liked this article, do check out my other articles on venilnoronha.io.

Disclaimer: My postings are my own and don’t necessarily represent Stripe’s positions, strategies or opinions.

--

--

--

I do Service Meshy stuff @ Stripe

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Journey of Managing Sales Orders in SAP Fiori: #4 Key User Adaptation

Creating a Physics-Based Character Controller in Unity

Spring — J2EE Container | Code Factory

CI/CD for Cloud Run with Terraform

Passport Authentication

Why Traditional Productivity Metrics Fail to Show True Development Progress

A month of Flutter: a look back

iOS Development: Leave the Tutorials Behind

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Venil Noronha

Venil Noronha

I do Service Meshy stuff @ Stripe

More from Medium

Export JMX metrics from Confluent KSQL to Datadog

Integration testing of SSO workflows using Cypress

Three easy steps to mitigate the risk of Injection

Modern Authorization requires defense in depth