Security professionals are currently facing a big data conundrum

If you work in IT networking or security, you’re all too familiar with how difficult it can be to effectively manage and analyze large volumes of network data. If so, you aren’t alone – many organizations face significant challenges when it comes to effectively managing the collection and storage of their network and security event telemetry in an efficient and scalable manner and then applying security-focused analytics at scale to detect more threats. Unfortunately, as networks continue to grow in both size and complexity this problem will only continue to get worse over time. At its core, this is a big data problem. Simply put, the growth of today’s networks has led us into a new paradigm which the network analytics tools of the past were never built to handle the explosion of network data that exists today. This problem especially pronounced for large enterprises and service providers with massive network footprints and exceptionally high flow per second volumes, as they are now faced with challenges related to ingestion bandwidth, query performance, long-term data retention, and data resiliency.

Listed below are some of the most common network telemetry collection, storage, and analysis challenges that organizations are facing today:

• Ingestion Challenges: Organizations with large or expanding network footprints face scalability challenges and increased expenses as they must continuously purchase additional sensors to handle continuously growing ingest volumes.
• Query Performance Challenges: For large enterprises, the task of running queries on large data sets is incredibly computationally expensive and can take upwards of 24 hours – this leads to operational inefficiencies by hindering the ability to detect threats within massive data sets in a timely manner, effectively slowing down remediation efforts and draining finite computational bandwidth.
• Data Retention Challenges: Many organizations are unable to retain the amount of network telemetry that they need to fulfill compliance requirements, forcing them to either purchase expensive third-party storage solutions or free up room in their proprietary databases to avoid legal risks should they be audited.
• Data Resiliency Challenges: Organizations that lack sufficient backup storage capacity are at risk of losing valuable data if one of their critical backup data storage systems fails.

For many large organizations, the challenges listed above have impeded their ability to perform routine network data management and threat detection efforts. These organizations need a solution that can provide scalable network telemetry collection and storage, highly responsive query times, and reliable data resiliency as core capabilities.

Introducing the Data Store

The Secure Network Analytics Data Store has been created with large enterprises and service providers in mind and specifically designed to solve the unique network telemetry management challenges outlined above through an improved database architecture design to enable new ways of managing data more efficiently.

How it works

The Data Store, introduced in version 7.3.0, sits between the Secure Network Analytics Manager and Flow Collectors. Flow Collectors ingest, de-duplicate and stitch together network flow data, then send it to the Data Store cluster. Flow data is then distributed across a resilient database cluster, which is comprised of a minimum of three Data Node appliances. This new database architecture offers scalable storage, increased flow rate ingestion capacity, improved resiliency versus the traditional model and higher-performance queries.

The illustration above depicts the components and architecture of a Secure Network Analytic deployment with a Data Store. Similarly, to the current Secure Network Analytics deployment model, Flow Collectors still ingest and process enterprise telemetry such as NetFlow. However, unlike the traditional model, the processed telemetry is not stored locally on each Flow Collector. Instead telemetry is written and stored across each Data Node within the Data Store. This new design allows for ingest and data storage functions to be performed independently from one another enabling the following benefits:

1. Increased ingest capacity: Data Stores can be combined to create a single cluster that is capable of monitoring over 3 million flows per second to alleviate ingestion bandwidth limitations for organizations with high flow volumes.
2. Storage scalability: The Data Store offers organizations with growing networks enhanced flexibility around data storage scalability through the ability to add additional database clusters.
3. Long-term data retention: Scalable and long-term telemetry storage capabilities enable long-term flow retention of up to 1-2 years’ worth of data with no need to add additional Flow Collectors.
4. Enterprise-class data resiliency: Telemetry data is stored redundantly across nodes to allow for seamless data availability during single node failures helping to ensure against loss of telemetry data.
5. Query and reporting response times improved by a significant magnitude: The Data Store provides drastically improved query performance and reporting response times of at least 10x faster than those offered by other deployment models.

And the above list still is not exhaustive – the Data Store also provides additional follow-on benefits. Take the Data Store’s long-term storage capabilities for instance – with 1-2 years’ worth of data at your fingertips this capability not only enables you to perform investigations on larger data sets, but also make it easier to fulfill regulatory and compliance requirements should you get audited. Moreover, it also reduces both costs and complexity by eliminating the need to purchase expensive and non-integrated third-party storage solutions for data retention purposes.

The Data Store transforms weaknesses to strengths by flipping the
‘big data problems’ that burden many organizations on their heads

In summary, modern networks remain on track to continue expanding – both in terms of size and complexity – without end. Because of this, it has never been more critical that security practitioners adopt tools capable of scaling to the challenges of growing network environments. The Data Store stands in a class of its own as the only solution on the market that was specifically built, not only to enable practitioners to effectively manage, analyze, and retain today’s endlessly increasing volumes of network data, but that is also capable of allowing them to leverage this data to their advantage.

Next steps

Check out the Secure Network Analytics Data Store Solution Overview, or contact your local Cisco Account Representative to learn more.