Introduction

In my previous blog in this series, I spoke with Sunil Amin about his work on the Cisco Telemetry Broker, the hot new product that allows customers to finally have the telemetry across their business be programmable and available to any analytics platform.

Today I’m here with Ajit Thyagarajan who is responsible for the architecture of the Cisco Telemetry Broker. Ajit will talk about early prototypes, lessons learned in designing and building the product, and give us a sneak peek into what’s coming to Cisco Telemetry Broker in the future.

Interview

TK: Ajit, first tell us a little about your background, because you and I go way back to the early days of the Internet and have adapted to a lot of changes over the years.

Ajit: TK, thank you for doing this – super excited to be participating in this interview. Yes, I am quite fortunate to have been involved in the evolution of the Internet and to have spent time learning from some of the visionaries who formed a lot of the core ideas in place today. I spent a lot of my graduate school days working on Internet Protocols. I developed the idea for the current version of the Internet Group Management Protocol (IGMP version 3), worked extensively on various multicast protocols, and some of changes to the current version of the Network Time Protocol (NTP) around autonomous configuration. It was during this phase that I really developed a liking for packet processing.

My very first real job after school was with an Internet router startup building next-generation hardware-based routers to keep up with the demand for faster Internet speeds. It was an exhilarating experience, and I haven’t looked back since, having been involved with various companies building networking products.

I started focusing on network security nearly a decade ago. It’s been a fascinating area, where I’ve been able to leverage my background to understand how threats are delivered over the network. Even at Cisco, I have spearheaded the effort around Encrypted Traffic Analytics in Secure Network Analytics, a very exciting way to analyze network traffic intelligently and perceive threats.

TK: You began at Cisco towards the end of 2018 and were first assigned to improve on a few of the Secure Network Analytics (known as Stealthwatch at that time) products that process packets. How did you go from those achievements to the prototype of CTB? What problems were you inspired to fix?

Ajit: When I first joined Cisco, I was assigned to the team that was looking into how to combine the cloud and on-prem products into a single offering. It was during my first off-site meeting that it was brought to my attention that the FlowSensor was experiencing performance issues and that given my background with packet processing, I may want to help. I was really intrigued and jumped right into the code. Within a couple of weeks, I had analyzed the architecture and found a couple of optimizations which immediately fixed the core performance issues, so that was a huge win! Since then, we’ve added a number of enhancements like moving to the Data Plane Development Kit (DPDK) to get much higher performance and it also resulted in us getting a new FlowSensor 4240 to the market a year or so ago.

The UDP Director (UDPD) was one of the other products the same team owned, and I started to dive deep into its architecture and realized that it was ripe for a major refresh. I also started to understand the type of problems that were being solved by the UDPD and realized that there was a much bigger opportunity in the marketplace. It was around that time that I was introduced to Vector Packet Processing (VPP) technology and realized the game-changing opportunity in front of us.

TK: Our audience might not be familiar with VPP so it would be great if you would tell us what it is and why it became a fundamental part of the Cisco Telemetry Broker Nodes.

Ajit: I was intrigued by VPP when I was first introduced to it, especially as a complete data plane for high-speed packet processing. At its core, VPP leverages modern day CPU architectures to process packets not individually, but as groups or vectors. This takes advantage of both the instruction and data caches to optimize the processing of packets in a pipelined manner. The resulting performance boost can be one to two orders of magnitude. Additionally, the VPP architecture came pre-packaged with a number of plug-ins for common packet processing such as support for IPv6, various tunneling formats, various NICs, etc. and given that it was actively being supported by a great team within Cisco, it made a lot of sense for us to leverage this as a data plane, not just for CTB but other products that required high speed packet processing.

TK: The rest of the development team and I interact with highly technical folks in Sales and Customer Success for feedback. What are some of the features in Cisco Telemetry Broker that were not known when we began but were picked up along the way?

Ajit: It has been incredible collaborating with the Sales and Customer Success folks to hone the features of the Cisco Telemetry Broker. There are several features that weren’t originally planned. For example, High Availability (HA) as implemented in the UDP Director allowed only for two nodes with one node as the active and the other as the standby. With CTB, we realized we could do better and implemented it in active/active mode with more than two nodes being part of a cluster. Another was being able to classify all telemetry traffic as syslog, NetFlow, sflow, etc. in a port-agnostic manner. This is incredibly useful to know which sources are sending what telemetry.

TK: I have a few favorites so if you don’t mind, I would like you to speak to the feature code named Dead Destination Detection (DDD) and why it is important to customers.

Ajit: So DDD essentially monitors every destination for potential issues with configuration or traffic delivery. When you have a destination that receives traffic only over UDP, it’s very hard to figure out whether the destination is alive or not. So, we implemented a heartbeat mechanism that allowed us to detect potential issues with the receiving system and thus potentially prevent gigabytes of data from flooding the network and overwhelming switches and routers. In many cases, switches and routers send back ICMP unreachable packets if the destination is down, compounding the traffic issue further, so eliminating this traffic storm in the case of a down destination is super useful.

TK: One incredible feature of Cisco Telemetry Broker is the ability to transform legacy protocols to modern consumers on the fly and to transform modern protocols to legacy consumers. Let’s talk about the VPC Flow Logs to IPFIX transformation available to customers. Why did you do it and what were the challenges?

Ajit: Customers today are increasingly adopting hybrid cloud infrastructures. One of the common issues is that the tools used for analytics or visibility have not kept up with the ability to monitor both cloud and on-prem assets and this brought about the need to transform cloud flow data to IPFIX for consumption by some of these tools. Cloud providers also aggressively roll out new features and not all tools have the ability to take advantage of them. Having a broker be able to transform some of this data for tools to consume in the format they understand allows for uninterrupted operation.

TK: I have seen that architects really love Cisco Telemetry Broker and the concept infrastructure that solves all of their telemetry routing. I heard one customer say, “I can finally treat all my telemetry for the enterprise as code!” Without spilling too much, what is next for Cisco Telemetry Broker?

Ajit: We really want customers to think about CTB not as a single box solution that takes telemetry in and pushes telemetry out. Instead, we want them to think about CTB as the next generation Intelligent Telemetry Plane (ITP), where all of the CTB nodes operate in tandem to manage all of the telemetry in the network, giving the customer unprecedented visibility into who is generating what telemetry and how that telemetry is being consumed by which consumers. Simplifying this by intelligently managing the telemetry in the customer environment and providing a single pane of glass will be how CTB evolves going forward. Expect to see more details around this as we roll out this ground-breaking functionality.

TK: It has been a real pleasure interviewing you and even more of a pleasure working with you on Cisco Telemetry Broker! Any parting thoughts you’d like to share before we wrap things up? How should our audience find you on the Interwebz?

Ajit: Thank you TK! It’s been a real pleasure participating in this interview. I am super excited about the opportunity in front of us. We are really looking to expanding the scope of this to other areas as well for full stack visibility, so stay tuned. If you are interested to learn more about CTB or any of my interests, do connect with me on LinkedIn.

Learn more about the Cisco Telemetry Broker. If you’d like to see some of Ajit’s more recent presentations from Cisco Live, be sure to check them out..