Linkerd External Traffic Split With Nginx Ingress Controller

Previously, I wrote an article about observing or usually we call it “Tapping” the network traffic in Linkerd, which is the most lightweight service mesh available so far. This is something I wanted to implement in one of my projects with external traffic and Nginx ingress controller. I wanted to use the external traffic split with Linkerd Service Mesh in order to implement “Blue/Green” also called “Red/Black” deployments to provide a deployment strategy for new codes. I faced some challenges combining this with Nginx ingress controller, when I’m reaching to my application externally. I was able to get it fixed with the support of an amazing group of people at Linkerd community. So, I thought to put this up in an article for my followers.

I’d love to mention that the Linkerd traffic split functionality can be leveraged to achieve the Blue/Green which switch the 100% of the traffic to the new code, as well as gradually increase the traffic to the new code, we call it canary deployments. Ultimately, the primary goal of these two approaches is the same and Linkerd can be used to achieve both the implementation on Kubernetes. Linkerd’s traffic splitting feature can be configured with the dynamically allocated weights to the services which serve the traffic from the underline services. This feature is controlled by a CRD (Custom Resource Definition). Let’s see how we can play around with this.

How To Use Traffic Splitting Internally

I have deployed two services which act as the blue and green services and I want to use  Linkerd’s traffic split CRD to split traffic with the given weights. Another service was created to act as the main service which is the endpoint to other services and pods.

The speciality of this main service (also called as the apex service) is, it doesn’t have an endpoint which usually selects with the selectors. Selectors configured to select the same label itself. Here is the sample main service manifest.

#Internal Service without the pod selectors
apiVersion: v1
kind: Service
metadata:
  name: web-apex-satellite-prod
  namespace: satellite-prod
spec:
  ports:
  - name: http
    port: 80
  selector:
    app: web-apex-satellite-prod
  type: ClusterIP

After the deployment of this service it doesn’t point to backend endpoints

Read More:

• Run Gitlab Pipelines On K3s Powered Civo In Minutes
• What Is VMware Test Drive And What Are The Benefits?
• Deploy Applications Easily With Kubeapps On Kubernetes
• Darkbit MKIT – Managed Kubernetes Inspection Tool
• AWS Systems Manager (SSM) Hybrid Activations With On Premises Virtual Machines

Once we are good with the service implementation, we can configure the traffic split CRD with the allocated  weights. For blue/green deployments weights should be configured as 1000m and 0m for the service. Basically these values are the traffic ratio for each and every backend services. The service configured with 1000m will get all the traffic while the other service is not getting the traffic. So Blue/Green or the Red/Black is achieved. Below traffic-split was created to provide the blue/green with Linkerd

#Internal Traffic split for blue/green
apiVersion: split.smi-spec.io/v1alpha1
kind: TrafficSplit
metadata:
  name: web-svc-ts-internal
  namespace: satellite-prod
spec:
  service: web-apex-satellite-prod
  backends:
  - service: frontend-blue
    weight: 0m
  - service: frontend-green
    weight: 1000m

The service, in this case web-svc-ts-internal will get and IP address and internal requests will route the traffic according to the weights specified. Here is the output of the above traffic-split implementation.

With the effect of adjusted traffic this can be leveraged as the Canary implementation.

Traffic -split for the Canery deployment

Here is the output of the internal traffic routing

External traffic split with Nginx ingress controller was bit different and let’s see how I achieved the external traffic split.

Traffic Split Externally With Nginx Ingress Controller

I created three services similar to the above internal step along with the Nginx ingress controller in my cluster. I’d like to highlight the below points when configuring the traffic split.

Since traffic split is a part of the Linkerd configuration, Nginx ingress should be aware of the Linkerd configuration, which also means that Nginx controllers should be annotated to inject the linkerd2-proxy as a sidecar. I had to restart the Nginx DaemonSet in order to apply the changes.

Special annotations in the ingress resource should be updated as stated in the documentation.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: main-ingress
  namespace: satellite-prod
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_set_header l5d-dst-override $service_name.$namespace.svc.cluster.local:$service_port;
      grpc_set_header l5d-dst-override $service_name.$namespace.svc.cluster.local:$service_port;
spec:
  rules:
  - host: main.techcrumble.cloud
    http:
      paths:
      - pathType: Prefix
        path: "/"
        backend:
          service:
            name: frontend-blue
            port:
              number: 80

Configurations seem all “OK” but I faced a very bizarre situation where my external traffic is not reaching the pods. Nginx doesn’t seem like the services without the actual endpoints. It stopped delivering the external traffic to a service.

I was searching for a solution and finally I have reached the Linkerd community slack with my question and I knew that this is the best place where I can get the support for my query.

I was informed that, there is no absolute necessity of a service without the endpoints for the main traffic routing and, one of the services taking part to the Blue/Green services can be leveraged for this.

I have updated the “traffi-split” CRD as below and I was able to achieve what I was expecting with this.

Here is the traffic-split CRD which worked.

apiVersion: split.smi-spec.io/v1alpha1
kind: TrafficSplit
metadata:
  name: frontend-svc-ts
  namespace: satellite-prod
spec:
  service: frontend-blue
  backends:
  - service: frontend-blue
    # Identical to resources, 1 = 1000m
    weight: 0m
  - service: frontend-green
    weight: 1000m

Basically, my ingress resource is forwarding the traffic to the blue service and it is also the primary service of my traffic-split object. The difference here with the previous step is removing the third service which doesn’t have endpoints. After implementing blue/green services with the ingress resource with the added annotations. I was able to implement Blue/Green with Linkerd. When I’m reaching to the main external url in this case “main.techcrumble.cloud” is delivering the backend service as I specified in the traffic-split CRD object.

Related Discourse thread is here.