Service Mesh Fundamentals with Linkerd
Decoupling at Layer 5

Lee Calcote

calcotestudios.com/talks

Lee Calcote

linkedin.com/in/leecalcote

@lcalcote

layer5.io

cloud native and its management

calcotestudios.com/talks

leecalcote

Service Mesh Patterns

calcotestudios.com/talks

Our virtual lab assistant

Abishek Kumar

@abd_abishek

Connect
 

Collaborate
 

Contribute

Join Slack http://slack.layer5.io

Prereq

What is a Service Mesh?

a services-first network

@lcalcote

Traffic Control

control over chaos

Resilency

content-based traffic steering

Observability

what gets people hooked on service metrics

Security

identity and policy

Service Mesh Functionality

Expect more from your infrastructure

Business Logic

in-network application logic

@lcalcote

What is Linkerd?

an open platform to connect, manage, and secure microservices

  • Observability

  • Resiliency

  • Traffic Control

  • Security

  • Policy Enforcement

linkerd.io

github.com/linkerd

@linkerd

@lcalcote

Reviews v1

Reviews Pod

Reviews v2

Reviews v3

Product Pod

Details Container

Details Pod

Ratings Container

Ratings Pod

Product Container

Reviews Service

Ratings Service

Details Service

Product Service

BookInfo Sample App

@lcalcote

BookInfo Sample App on Service Mesh

Reviews v1

Reviews Pod

Reviews v2

Reviews v3

Product Pod

Details Container

Details Pod

Ratings Container

Ratings Pod

Product Container

Proxy sidecar

Proxy sidecar

Proxy sidecar

Proxy sidecar

Proxy sidecar

Reviews Service

Proxy sidecar

Proxy  ingress

Product Service

Ratings Service

Details Service

@lcalcote

Observability

what gets people hooked on service metrics

Goals

  • Metrics without instrumenting apps

  • Consistent metrics across fleet

  • Trace flow of requests across services

  • Portable across metric back-end providers

You get a metric!  You get a metric!  Everyone gets a metric!

@lcalcote

Traffic Control

control over chaos

  • Traffic splitting
    • L7 tag based routing?
  • Traffic steering
    • Look at the contents of a request and route it to a specific set of instances.
  • Ingress and egress routing

Resilency

 

  • Systematic fault injection

  • Timeouts and Retries with timeout budget

  • Control connection pool size and request load

  • Circuit breakers and Health checks

 

content-based traffic steering

Missing: application lifecycle management, but not by much

Missing: distributed debugging; provide nascent visibility (topology)

@lcalcote

Confirm Prerequisites 

 

  1. Start Docker Desktop, Minikube or other.
    (either single-node or multi-node clusters will work)

  2. Verify that you have a functional Docker environment by running :

$ docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
1b930d010525: Pull complete
Digest: sha256:0e11c388b664df8a27a901dce21eb89f11d8292f7fca1b3e3c4321bf7897bffe
Status: Downloaded newer image for hello-world:latest

Hello from Docker!

Docker and Kubernetes

Prereq

@lcalcote

Prepare Docker Desktop

Ensure your Docker Desktop VM has 4GB of memory assigned.

Ensure Kubernetes is enabled.

Prereq

@lcalcote

Deploy Kubernetes

  1. Confirm access to your Kubernetes cluster.
$ kubectl version --short
Client Version: v1.19.2
Server Version: v1.18.8
$ kubectl get nodes
NAME             STATUS   ROLES    AGE   VERSION
docker-desktop   Ready    master   10m   v1.18.8

Prereq

v1.9 or higher

@lcalcote

meshery.io

Deploy Management Plane

Management
Plane

Provides expanded governance, backend system integration, multi-mesh, federation, expanded policy, and dynamic application and mesh configuration.

Control Plane

Data Plane

brew tap layer5io/tap
brew install mesheryctl
mesheryctl system start

Install Meshery

Prereq

@lcalcote

Using brew

Using bash

curl -L https://git.io/meshery | sudo bash -  
kubectl config view --minify --flatten > config_minikube.yaml

Using minikube:

Relating to

Service Meshes

Which is why...

 I have a container orchestrator.

@lcalcote

Core

Capabilities

  • Cluster Management

    • Host Discovery

    • Host Health Monitoring

  • Scheduling

  • Orchestrator Updates and Host Maintenance

  • Service Discovery

  • Networking and Load Balancing

  • Stateful Services

  • Multi-Tenant, Multi-Region

Additional

Key Capabilities

  • Application Health and Performance Monitoring

  • Application Deployments

  • Application Secrets

minimal capabilities required to qualify as a container orchestrator

Service meshes generally rely on these underlying layers.

@lcalcote

Which is why...

 I have an API gateway.

@lcalcote

Microservices API Gateways

north-south vs. east-west

Which is why...


 I have client-side libraries.

Enforcing consistency is challenging.

Foo Container

Flow Control

Foo Pod

Go Library
A v1

Network Stack

Service Discovery

Circuit Breaking

Application / Business Logic

Bar Container

Flow Control

Bar Pod

Go Library
A v2

Network Stack

Service Discovery

Circuit Breaking

Application / Business Logic

Baz Container

Flow Control

Baz Pod

Java Library
B v1

Network Stack

Service Discovery

Circuit Breaking

Application / Business Logic

Retry Budgets

Rate Limiting

@lcalcote

Help with Modernization

 

  • Can modernize your IT inventory without:

    • Rewriting your applications

    • Adopting microservices, regular services are fine

    • Adopting new frameworks

    • Moving to the cloud

address the long-tail of IT services

Get there for free

@lcalcote

Why use a Service Mesh?

  • Bloated service (application) code

  • Duplicating work to make services production-ready

    • Load balancing, auto scaling, rate limiting, traffic routing...

  • Inconsistency across services

    • Retry, tls, failover, deadlines, cancellation, etc., for each language, framework

    • Siloed implementations lead to fragmented, non-uniform policy application and difficult debugging

  • Diffusing responsibility of service management

to avoid...

@lcalcote

DEVELOPER

OPERATOR

Decoupling at Layer 5

where Dev, Ops, Product meet

Empowered and independent teams can iterate faster

PRODUCT OWNER

@lcalcote

DEVELOPER

OPERATOR

Decoupling at Layer 5

where Dev, Ops, Product meet

Empowered and independent teams can iterate faster

PRODUCT OWNER

@lcalcote

Q&A

Service Mesh Architectures

Data Plane

  • Touches every packet/request in the system.

  • Responsible for service discovery, health checking, routing, load balancing, authentication, authorization, and observability.

Ingress Gateway

Egress Gateway

Service Mesh Architecture

@lcalcote

No control plane? Not a service mesh.

Control Plane

  • Provides policy, configuration, and platform integration.

  • Takes a set of isolated stateless sidecar proxies and turns them into a service mesh.

  • Does not touch any packets/requests in the data path.

Data Plane

  • Touches every packet/request in the system.

  • Responsible for service discovery, health checking, routing, load balancing, authentication, authorization, and observability.

Ingress Gateway

Egress Gateway

Service Mesh Architecture

@lcalcote

Control Plane

Data Plane

  • Touches every packet/request in the system.

  • Responsible for service discovery, health checking, routing, load balancing, authentication, authorization, and observability.

  • Provides policy, configuration, and platform integration.

  • Takes a set of isolated stateless sidecar proxies and turns them into a service mesh.

  • Does not touch any packets/requests in the data path.

You need a management plane.

Ingress Gateway

Management
Plane

  • Provides backend system integration, expanded policy and governance, continuous delivery integration, workflow, chaos engineering,  configuration and performance management and multi-mesh federation.

Egress Gateway

Service Mesh Architecture

@lcalcote

Pilot

Citadel

Mixer

Control Plane

Data Plane

istio-system namespace

policy check

Foo Pod

Proxy Sidecar

Service Foo

tls certs

discovery & config

Foo Container

Bar Pod

Proxy Sidecar

Service Bar

Bar Container

Out-of-band telemetry propagation

telemetry

 

reports

Control flow

application traffic

Application traffic

application namespace

telemetry reports

Istio Architecture

Galley

Ingress Gateway

Egress Gateway

@lcalcote

Control Plane

Data Plane

octa-system namespace

policy check

Foo Pod

Proxy
Sidecar

Service Foo

discovery & config

Foo Container

Bar Pod

Service Bar

Bar Container

Out-of-band telemetry propagation

telemetry

 

reports

Control flow

application traffic

Application traffic

application namespace

telemetry reports

Policy
Engine

Security Engine

Visibility
Engine

+

Proxy
Sidecar

+

@lcalcote

Octarine Architecture

Leader

Agent

Control Plane

Data Plane

intentions

Foo Pod

Proxy Sidecar

Service Foo

discovery, config,

 

tls certs

Foo Container

Bar Pod

Proxy Sidecar

Service Bar

Bar Container

Control flow

application traffic

Application traffic

application namespace

Follower

Consul Client

Consul Servers

Follower

policy

 

check

Consul Architecture

layer5.io/service-mesh-architectures

WASM Filter

node

@lcalcote

Our service mesh of study: Linkerd

Deploy Linkerd

@lcalcote

  1. Install Linkerd using Meshery

     
  2. Find control plane namespace

     
  3. Inspect control plane services

     
  4. Inspect control plane components
open http://localhost:9081
kubectl get namespaces
kubectl get svc -n linkerd
kubectl get pod -n linkerd

Lab 1

github.com/layer5io/linkerd-service-mesh-workshop

Control Plane

Data Plane

linkerd namespace

Foo Pod

Proxy Sidecar

Service Foo

Foo Container

Bar Pod

Proxy Sidecar

Service Bar

Bar Container

Out-of-band telemetry propagation

telemetry

 

scarping

Control flow during request processing

application traffic

Application traffic

application namespace

telemetry scraping

Architecture

Prometheus

Grafana

web

CLI

public-api

Linkerd

linkerd-controller

sp-validator

Control Plane

destination

tap

web

CLI

proxy-injector

identity

@lcalcote

Service Proxy Sidecar

- A Rust-based L4/L7 proxy

- Low memory footprint

- Designed for service mesh use cases

@lcalcote

Capabilities:

  • Transparent, zero-config proxying for HTTP, HTTP/2, and arbitrary TCP protocols.

  • Automatic Prometheus metrics export for HTTP and TCP traffic.

  • Transparent, zero-config WebSocket proxying.

  • Automatic, latency-aware, layer-7 load balancing.

  • Automatic layer-4 load balancing for non-HTTP traffic.

  • Automatic TLS.

  • An on-demand diagnostic tap API.

The proxy supports service discovery via DNS and the destination gRPC API.

the included, non-swappable battery

Data Plane

Pod

Proxy sidecar

App Container

the workhorse

What's Identify for?

@lcalcote

  • Verifiable identity
    • Issues certs
    • Certs distributed to service proxies
    • Mounted as a Kubernetes secret
  • Secure naming / addressing
  • Traffic encryption

security at scale

security by default

Orchestrate Key & Certificate:

  • Generation
  • Deployment
  • Rotation
  • Revocation

 

linkerd namespace

Prometheus

Grafana

web

CLI

public-api

linkerd-controller

sp-validator

destination

tap

web

CLI

proxy-injector

identity

Q&A

Break

Deploy Sample App

@lcalcote

  1. Multi-language, multi-service application

  2. Automatic vs manual  sidecar injection

  3. Verify install

Use Meshery or install manually

Lab 2

emoji Pod

web Pod

web
Container

emoji-svc Service

web Service

Emojivoto Sample App

@lcalcote

emoji
Container

voting Pod

voting-svc Service

voting
Container

emoji Pod

web Pod

web
Container

emoji-svc Service

web Service

Emojivoto Sample App on the Service Mesh

@lcalcote

emoji
Container

voting Pod

voting-svc Service

voting
Container

Linkerd sidecar

Linkerd sidecar

Linkerd sidecar

Sidecar Injection

@lcalcote

Automatic sidecar injection leverages Kubernetes' Mutating Webhook Admission Controller.

 

  1. Verify whether your Kubernetes deployment supports these APIs by executing:


     
  2. Inspect the linkerd-proxy-injector-webhook-config webhook:
 

 

Lab 2

github.com/layer5io/linkerd-service-mesh-workshop

kubectl get mutatingwebhookconfigurations
kubectl get mutatingwebhookconfigurations linkerd-proxy-injector-webhook-config -o yaml
kubectl api-versions | grep admissionregistration

Sidecars proxy can be either manually or automatically injected into your pods.

Deploy Sample App with Automatic Sidecar Injection

@lcalcote

  1. Verify presence of the sidecar injector
kubectl get deployment linkerd-proxy-injector -n linkerd

Lab 2

github.com/layer5io/linkerd-service-mesh-workshop

2.   Confirm namespace annotation

kubectl describe namespace linkerd

sidecar proxy

 Deploy Sample App with Manual Sidecar Injection

@lcalcote

In namespaces without the linkerd.io/inject annotation, you can use linkerd inject to manually inject Linkerd proxies into your application pods before deploying them:

kubectl kustomize kustomize/deployment | \
    linkerd inject - | \
    kubectl apply -f -

Lab 2

github.com/layer5io/linkerd-service-mesh-workshop

sidecar proxy

Expose Emojivoto via NGINX Ingress Controller

@lcalcote

 

Inspect the Linkerd proxy of the web pod

NGINX  Ingress

application traffic

Lab 3

github.com/layer5io/linkerd-service-mesh-workshop

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: web-ingress
  namespace: emojivoto
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_set_header l5d-dst-override $service_name.$namespace.svc.cluster.local:$service_port;
      grpc_set_header l5d-dst-override $service_name.$namespace.svc.cluster.local:$service_port;

spec:
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              serviceName: web-svc
              servicePort: 80
  1. Deploy the NGINX Ingress Controller

2. Configure NGINX Ingress for Emojivoto

Q&A

Break

layer5.io/landscape

It's meshy out there.

Strengths of Service Mesh Implementations

Different tools for different use cases

a sample

@lcalcote

Service mesh abstractions 

Meshery is interoperable with these abstractions.

Service Mesh Interface
(SMI)

Multi-Vendor Service Mesh Interoperation (Hamlet)

Service Mesh Performance (SMP)

A standard interface for service meshes on Kubernetes.

A set of API standards for enabling service mesh federation.

A format for describing and capturing service mesh performance.

to the rescue

@lcalcote

  • benchmarking of service mesh performance
     

  • exchange of performance information from system-to-system / mesh-to-mesh
     

  • apples-to-apples performance comparisons of service mesh deployments.
     

  • MeshMark - a universal performance index to gauge a service mesh’s efficiency against deployments in other organizations’ environments

 

Service Mesh Performance

https://smp-spec.io

Directly provides:

Indirectly facilitates:

  • a vendor neutral specification for capturing details of infrastructure capacity, service mesh configuration, and workload metadata.

@lcalcote

Service Mesh Interface

Four API specifications

Service Mesh Interface aims to provide:

• Standard interface for service mesh on Kubernetes

• Basic feature set for most common mesh use cases

• Extensible to support new features

• Space for the ecosystem to innovate

Traffic Split

Traffic Specs

Traffic Metrics

Traffic Access Control

APIs:

@lcalcote

Service Mesh Interface (SMI) Conformance

Operate and upgrade with confirmation of SMI compatibility

✔︎ Learn Layer5 sample application used for validating test assertions.
 

 âœ”︎ Defines compliant behavior.

 âœ”︎ Produces compatibility matrix.

 âœ”︎ Ensures provenance of results.

 âœ”︎ Runs a set of conformance tests.

 âœ”︎ Built into participating service mesh’s release pipeline.

@lcalcote

Telemetry

Service Mesh Interface (SMI)

Service Mesh Performance (SMP)

Configuration

Security

Telemetry

Control Plane

Data
Plane

service mesh ns

Foo Pod

Proxy Sidecar

Service Foo

Foo Container

Bar Pod

Proxy Sidecar

Service Bar

Bar Container

Out-of-band telemetry propagation

Control flow

application

traffic

Application traffic

application namespace

Meshery Architecture

Ingress Gateway

Egress Gateway

Management
Plane

meshery

adapters

gRPC

kube-api

kube-system

generated load

http / gRPC traffic

fortio

wrk2

nighthawk

UI

API

workloads

Meshery WASM Filter

CLI

perf analysis

patterns

@mesheryio

Metrics Dashboard

@lcalcote

Lab 4

  1. Expose Grafana with `linkerd`


     

    2.   
linkerd dashboard &

Grafana

github.com/layer5io/linkerd-service-mesh-workshop

@lcalcote

Lab 6

Distributed Tracing

Jaeger

github.com/layer5io/linkerd-service-mesh-workshop

Distributed Tracing

Jaeger

@lcalcote

The istio-proxy collects and propagates the following headers from the incoming request to any outgoing requests:

  • x-request-id
  • x-b3-traceid
  • x-b3-spanid
  • x-b3-parentspanid
  • x-b3-sampled
  • x-b3-flags
  • x-ot-span-context

Lab 6

cat >> config.yaml << EOF
tracing:
  enabled: true
EOF

github.com/layer5io/linkerd-service-mesh-workshop

linkerd upgrade --config config.yaml | kubectl apply -f -

Q&A

Lee Calcote

cloud native and its management

@lcalcote

calcotestudios.com/talks

linkedin.com/in/leecalcote

layer5.io/subscribe