This document summarizes Squarespace's transition from a monolithic architecture to a microservices architecture and their implementation of a service mesh using Envoy proxy. It describes how Squarespace grew from less than 50 engineers in 2013 to over 200 engineers in 2017, necessitating the move to microservices for scalability. It outlines their initial use of Consul for service discovery and Netflix OSS libraries. It then introduces the concept of a service mesh and how Envoy proxy deployed as a sidecar can provide advanced control, observability, and support for multiple languages. It details how Envoy uses Consul and its xDS APIs for dynamic service discovery and configuration. Finally, it discusses future work including integrating orchestration and abstracting common service
1. Building a Service Mesh with Envoy
Doug Jones
djones@squarespace.com
@dougfjones
2. Microservices: A Story of Growth
Monolith
Background Jobs
DBQueue
2013: <50 engineers
● “Whatever works”
● Build product
● Grow fast
3. Microservices: A Story of Growth
2014: ~75 engineers
● “Whatever works”
● Too much firefighting
● Not enough new features
● Inflexible monolith architecture
Monolith
Background Jobs
DBQueue
4. Microservices: A Story of Growth
Monolith
Background Jobs
DBQueue
2016: 100+ engineers
● Microservices
● Scalable + Reliable
● Developers can move faster
● Squarespace can move faster
5. Microservices: A Story of Growth
Monolith
Background Jobs
DBQueue
2017: 200+ engineers
● Even More Microservices
● Independent, full stack teams
● Self Service Infra with
Kubernetes
● Desire flexible, reliable infra
● Desire better tooling
6. Microservices at Squarespace
● Building started late 2014
● Java API Servers
● Virtual Machines
● Consul Service Discovery
● Service client based on Netflix OSS (Hystrix, Ribbon, RxNetty)
● Now in progress: migrating from VMs to Kubernetes
Microservices Platform
8. DC 2DC 1
Consul Cross DC
Consul
Service A
Service B
Service B
Consul
Service C
Service D
Service D
Cross DC Gossip
9. Service Mesh
● Service client functionality moves to its own process (sidecar)
○ No longer trapped in a library
● This process can be configured and updated independently of the
application it serves
● Advanced operational control through APIs
● Improved observability
● Opens the door to better support for service development in multiple
programming languages
Why Service Mesh?
10. Service Mesh
● Envoy proxy
○ Co-located with each service instance (sidecar)
○ Proxies ingress and egress traffic
● Dynamic configuration API
○ Provide service discovery information
○ Change routing table and circuit breaker configuration
○ Big upgrade in our capability as operators
Service Mesh with Envoy
16. Envoy xDS Proto
● EDS -> Consul health endpoint list Nodes for Service
○ Uses same Consul index to version mapping
○ Background polling for updates
● RDS
○ Simple HTTP routing rule matching header value to cluster name
Other xDS Requests
18. Future
● Build orchestration features into our new discovery system
○ Harness xDS to push updates that make traffic routing changes
● APIs to abstract common service mesh operations
● Dashboard for operators
Future Work