Loading Posts...

Tap Kubernetes Cluster Communication With Linkerd Service Mesh

Service Mesh is an added Infrastructure layer in order to maintain security, reliability and observability in the cloud-native applications independently with the running applications. Linkerd is one of the oldest and well known solutions to provide the service mesh functionalities to the cloud-native applications. I have been using Linkerd a while now and I thought to put up an article with the functionalities, specifically the “Tap” functionality which allows you to see through the cluster communication.

If we are talking about the history of Linkerd, Linkerd 1.x version was introduced in 2016 on Twitter’s Finagle Library and it was donated to the Cloud Native Computing Foundation (CNCF) in 2017 becoming the fifth open source project hosted by the CNCF. In the year of 2018 it was rewritten in Rust and Go which is much faster than the earlier version and released as Linkerd 2.0. In this post I deployed a Kubernetes cluster with version 1.19.4, installed Linkerd 2.9.1 and the “Emojivoto” demo app provided as the sample app for the Linkerd. It was really easy for me to understand the concept behind the service mesh with Linkerd and Emojivoto app.

Read More:

Installing Linkerd On Kubernetes

Installing Linkerd was quite an easy task. First I installed the Linkerd CLI and then the tool itself provided the steps until it fully up and running in my cluster. Just executed the below command to install the Linkerd CLI tool. You can also find the complete steps guide in the  Linkerd website.

curl -sL https://run.linkerd.io/install | sh

Then the Linkerd path needed to be added and all the commands were listed in the terminal.

export PATH=$PATH:/home/ec2-user/.linkerd2/bin

Pre cluster validation should be performed

linkerd check --pre

Once the pre validation is succeeded, install the linkerd using below command.

linkerd install | kubectl apply -f - 

Once it succeeds, Linkerd will be installed on the Kubernetes cluster in the Linkerd namespace. If you need to install Linkerd in to a different namespace you can do it by specifying the namespace with “-l” flag.

linkerd install -l linkerdtest | kubectl apply -f -

Post cluster validation is also needed in order to verify the Linkerd installation.

linkerd check

Once it is completed you are done with the installation. Here is the output of the created components in the Linkerd namespace.

Here is the Linkerd dashboard after installing it on my Kubernetes Cluster.

Now we need to have a test application in order to test the basic functionality of the Linkerd Service Mesh. Linkerd itself provided us with the sample application called emojivoto with a couple of microservices. Let’s give it a go.

Installing the Emojivoto Application

To install the emojivoto application use the following kubectl command

kubectl apply -f https://run.linkerd.io/emojivoto.yml

By default service mesh is not installed to the emojivoto application. We need to inject the linkerd2 proxy in the components manually or to the namespace, which is the recommended and most suitable method as it injects the proxy to all the pods in the namespace.

kubectl annotate ns emojivoto linkerd.io/inject=enabled

Restart the deployments in the emojivoto namespace.

kubectl rollout restart deploy -n emojivoto

In order to verify the proxy injection, check the pod status and we should see it as 2/2 under the “Ready” header in the output.

Now our application is ready and let’s see how to utilize the Linkerd Tap command to observe the traffic.

“Tapping” The Kubernetes Resource Communication With Linkerd Service Mesh

Linkerd offers a great feature to tap the cluster communication between the kubernetes components with its “Linkerd-tap” component of the control plane. I personally like to use this command as it provides valuable information about the traffic flow of the components. 

For example let’s see the prometheus component which is installed as a part of the Linkerd installation and its communication in between the other components. Commands are easy to understand and interact with if you are familiar with the kubectl tool, which is obvious at this point.

linkerd tap -n linkerd deploy/linkerd-prometheus

This output provides you a valuable information relative to the linkerd proxy in the Prometheus deployment pods such as:

  • Request, response and the end of the response
  • Direction of the traffic flow
  • Source and the destination IP addresses
  • Traffic encryption status with the Linkerd Mutual TLS certificates
  • Other information such as request duration, latency and response latency

In order to get the same output Linkerd dashboard can be used. Selecting the “tap” option at the left menu and selecting the namespace and the component to observe the traffic will give the similar output in the dashboard.

Tap option should be started after selecting the component as below and real time traffic will be displayed. The respective command line query will also be displayed in the dashboard.

Realtime traffic will be listed as below and can be stopped manually.

More filters are available for granular and deeper information observation

To view the traffic from web deployment to voting deployment in the emojivoto namespace below command can be leveraged

linkerd tap deployment/web --to deployment/voting -n emojivoto

Similar output can be obtained through the dashboard adding the filters.

If you want to see the exact path which communicates with the destination service –path option can be leveraged

linkerd tap deployment/web --namespace emojivoto --to deployment/voting --path /emojivoto.v1.VotingService/VoteDoughnut

Path can be specified in the advanced filters for this as well

Also, you can obtain the json output with the “-o json” flag with the output. See the below output.

linkerd tap deployment/web --namespace emojivoto --to deployment/voting --path /emojivoto.v1.VotingService/VoteDoughnut -o json

Real Time Traffic Status And Statistics

Similar to the kubectl top command you can see the real time sorted traffic with Linkerd top command. See the statistics it provides.

linkerd top deploy/web -n emojivoto 

Also, in the dashboard top option is available and tap option will be available directly tap the request. It will lead to the customised tap query for the request.

One of the main benefits of running a service mesh on a kubernetes cluster is that, obtaining the golden metrics in the traffic flow. Linkerd does provide the golden matrices, the latencies success and error rates and the requests per second. Let’s see how you can see these golden matrices in the terminal. 

If we need to see the traffic statistics for deployments in the emojivoto namespace, we can observe it through the below command.

linkerd stat deploy -n emojivoto

Similar to the kubectl tool, we can see more information about the traffic with the -o wide flag.

linkerd stat deploy -n emojivoto -o wide

To see more granular and specific information, we can leverage the “–to” and “–from” flags in the query. To see the matrices in the web deployment to emoji deployment below query can be used.

linkerd stat deploy/web --to deploy/emoji -n emojivoto 

To see the traffic from a deployment below query can be used

linkerd stat deploy/emoji --from deploy/web -n emojivoto 

All the traffic comes to the deployments in the emojivoto namespace also observed as below

linkerd stat deploy --from deploy/web -n emojivoto

I suppose this post is valuable for anyone who needs to observe the traffic in a Kubernetes cluster with Linkerd.

More Links:

Click to rate this post!
[Total: 3 Average: 5]

Aruna Lakmal

Associate Technical Specialist at Pearson, Sri Lanka. Technology junky, enthusiast, a VMware vExpert and a blogger with more than 7 years of Experience in Information Technology more focusing on VMware Virtualization, Microsoft and Datacenter Technologies.

Get Updates Directly To Your Inbox!


Leave a Comment

Loading Posts...