Previously, I wrote an article about observing or usually we call it “Tapping” the network traffic in Linkerd, which is the most lightweight service mesh available so far. This is something I wanted to implement in one of my projects with external traffic and Nginx ingress controller. I wanted to use the external traffic split with Linkerd Service Mesh in order to implement “Blue/Green” also called “Red/Black” deployments to provide a deployment strategy for new codes. I faced some challenges combining this with Nginx ingress controller, when I’m reaching to my application externally. I was able to get it fixed with the support of an amazing group of people at Linkerd community. So, I thought to put this up in an article for my followers.
I’d love to mention that the Linkerd traffic split functionality can be leveraged to achieve the Blue/Green which switch the 100% of the traffic to the new code, as well as gradually increase the traffic to the new code, we call it canary deployments. Ultimately, the primary goal of these two approaches is the same and Linkerd can be used to achieve both the implementation on Kubernetes. Linkerd’s traffic splitting feature can be configured with the dynamically allocated weights to the services which serve the traffic from the underline services. This feature is controlled by a CRD (Custom Resource Definition). Let’s see how we can play around with this.
How To Use Traffic Splitting Internally
I have deployed two services which act as the blue and green services and I want to use Linkerd’s traffic split CRD to split traffic with the given weights. Another service was created to act as the main service which is the endpoint to other services and pods.
The speciality of this main service (also called as the apex service) is, it doesn’t have an endpoint which usually selects with the selectors. Selectors configured to select the same label itself. Here is the sample main service manifest.
#Internal Service without the pod selectors apiVersion: v1 kind: Service metadata: name: web-apex-satellite-prod namespace: satellite-prod spec: ports: - name: http port: 80 selector: app: web-apex-satellite-prod type: ClusterIP
After the deployment of this service it doesn’t point to backend endpoints
- Run Gitlab Pipelines On K3s Powered Civo In Minutes
- What Is VMware Test Drive And What Are The Benefits?
- Deploy Applications Easily With Kubeapps On Kubernetes
- Darkbit MKIT – Managed Kubernetes Inspection Tool
- AWS Systems Manager (SSM) Hybrid Activations With On Premises Virtual Machines
Once we are good with the service implementation, we can configure the traffic split CRD with the allocated weights. For blue/green deployments weights should be configured as 1000m and 0m for the service. Basically these values are the traffic ratio for each and every backend services. The service configured with 1000m will get all the traffic while the other service is not getting the traffic. So Blue/Green or the Red/Black is achieved. Below traffic-split was created to provide the blue/green with Linkerd
#Internal Traffic split for blue/green apiVersion: split.smi-spec.io/v1alpha1 kind: TrafficSplit metadata: name: web-svc-ts-internal namespace: satellite-prod spec: service: web-apex-satellite-prod backends: - service: frontend-blue weight: 0m - service: frontend-green weight: 1000m
The service, in this case web-svc-ts-internal will get and IP address and internal requests will route the traffic according to the weights specified. Here is the output of the above traffic-split implementation.
With the effect of adjusted traffic this can be leveraged as the Canary implementation.
Traffic -split for the Canery deployment
Here is the output of the internal traffic routing
External traffic split with Nginx ingress controller was bit different and let’s see how I achieved the external traffic split.
Traffic Split Externally With Nginx Ingress Controller
I created three services similar to the above internal step along with the Nginx ingress controller in my cluster. I’d like to highlight the below points when configuring the traffic split.
Since traffic split is a part of the Linkerd configuration, Nginx ingress should be aware of the Linkerd configuration, which also means that Nginx controllers should be annotated to inject the linkerd2-proxy as a sidecar. I had to restart the Nginx DaemonSet in order to apply the changes.
Special annotations in the ingress resource should be updated as stated in the documentation.
apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: main-ingress namespace: satellite-prod annotations: kubernetes.io/ingress.class: "nginx" nginx.ingress.kubernetes.io/configuration-snippet: | proxy_set_header l5d-dst-override $service_name.$namespace.svc.cluster.local:$service_port; grpc_set_header l5d-dst-override $service_name.$namespace.svc.cluster.local:$service_port; spec: rules: - host: main.techcrumble.cloud http: paths: - pathType: Prefix path: "/" backend: service: name: frontend-blue port: number: 80
Configurations seem all “OK” but I faced a very bizarre situation where my external traffic is not reaching the pods. Nginx doesn’t seem like the services without the actual endpoints. It stopped delivering the external traffic to a service.
I was searching for a solution and finally I have reached the Linkerd community slack with my question and I knew that this is the best place where I can get the support for my query.
I was informed that, there is no absolute necessity of a service without the endpoints for the main traffic routing and, one of the services taking part to the Blue/Green services can be leveraged for this.
I have updated the “traffi-split” CRD as below and I was able to achieve what I was expecting with this.
Here is the traffic-split CRD which worked.
apiVersion: split.smi-spec.io/v1alpha1 kind: TrafficSplit metadata: name: frontend-svc-ts namespace: satellite-prod spec: service: frontend-blue backends: - service: frontend-blue # Identical to resources, 1 = 1000m weight: 0m - service: frontend-green weight: 1000m
Basically, my ingress resource is forwarding the traffic to the blue service and it is also the primary service of my traffic-split object. The difference here with the previous step is removing the third service which doesn’t have endpoints. After implementing blue/green services with the ingress resource with the added annotations. I was able to implement Blue/Green with Linkerd. When I’m reaching to the main external url in this case “main.techcrumble.cloud” is delivering the backend service as I specified in the traffic-split CRD object.
Related Discourse thread is here.