As this year draws to a close, most of us will take time to reflect on what has come before, and what we might expect to see in the coming year. This is especially true this year, as we make the leap into the 2020s.

It’s incredible to think that just 20 years ago, organizations everywhere were in a state of panic. Not so much about any long-term changes the next decade would bring, but rather about whether their machines would be operational on January 1.

With Y2K an ancient memory and yet another transition into a new decade right around the corner, we’re no longer worried about whether our computers can handle a simple date change. But are we actually prepared for 2020 and beyond, particularly when it comes to the volume, variety and velocity of data we are generating today?

The Data Decade in Asia

From my perspective, the transition from this decade to the next will be all about data, and how it can drive and reinvent organizations, sectors and even entire industries.

Asia is already leading the world in terms of producing data, thanks in part to the large amount of industrial robotics in the region[1]. Presently, Asia has approximately 33 zettabytes (ZB) of data now in storage (a ZB is equivalent to a trillion gigabytes—the same GB you know from your mobile phone plan).  That number is expected to reach around 44 ZB in the region next year, and a whopping 170 ZB by 2025, with enterprises comprising around 60% of that 170 ZB total.

Today, data is the true differentiator for businesses, allowing them to make massive gains and innovations if utilized, or to rue missed opportunities if ignored. Over this past year, I have personally witnessed many examples of just how rapidly industries in Asia are reinventing themselves by leveraging data, AI and the cloud to level up almost everything about how they work and create phenomenal results.

Because of this, it has never been more evident to me that this is truly the data decade – and that a company-wide data strategy and data-driven culture can create tremendous change in a very short timeframe. But first, organizations need to break down their data silos to understand it fully.

Understanding Your Data Estate

Today we see organizational data being stored in multiple locations, both virtually and geographically, including operational databases, data warehouses and data lakes, creating a digital whirlwind of information from an ever-increasing number of sources.

However, in today’s world, driving business transformation requires a modernized data platform that breaks down those silos in order to unleash data’s full potential.

At Microsoft, we call this the Data Estate, and designing and managing it well can be the difference between mediocre and excellent business performance. It allows organizations to unlock the full potential of data from the first moments of receiving it through analytics and to final outputs like actionable business intelligence. A data estate ingests, stores, prepares, models, serves and visualizes data to produce insights that can support, boost and even transform operations.

A good example of this in use is the Australian branch of Japanese construction equipment maker Komatsu. This company processes up to one million records per day, and collects 20-30 GB of data from each of its almost 30,000 connected tools, machines and pieces of equipment every year. That translates to more than 900,000 GB of data per year.

On one’s own, finding any specific file – much less discerning meaningful patterns – would be like finding a needle in a haystack. However, through AI and the cloud, Komatsu has been able to identify customer patterns and unlock valuable information which they then use to improve efficiencies and outputs. By better understanding their demand and supply cycles, and finding and addressing gaps in stability and operational issues, Komatsu has enjoyed a 49% reduction in costs and performance gains of between 25% and 30%. A huge advantage in today’s ultra-competitive business environment.

Microsoft’s Data Journey

Microsoft fully realized the importance of data and the challenges organization face by undergoing our own transformation journey, modernizing our own data estate and establishing a data-driven culture.  Much of our data historically operated in silos, and keeping this data in disparate systems had encouraged our teams to make isolated decisions, making it difficult to connect the dots.

In general, businesses are largely forced to maintain two types of analytical systems: Data warehouses and data lakes. Data warehouses provide critical overall insights on business health. Data lakes can uncover important signals on customers, products, employees and processes. Both are critical, yet usually operate independently of one another, which can lead to uninformed decisions, or action without purpose. Applying advanced analytics and machine learning to this disparate array of information can be challenging, putting deeper insights out of reach.

To address the issue, we united data from across our entire enterprise, making it discoverable, trustworthy, understandable and trackable. Creating and implementing the data estate methodology for our own company has allowed us to optimize our own data for advanced analytics, machine learning and AI insights that have dramatically transformed our business, and sped up innovation.

Connecting Synapses for Business Growth

Based on our journey, our engineering teams began to examine why many other companies had also struggled to get the full benefit of their analytics products, and found that not being able to align and combine data storage sets in order to apply AI to a company’s entire set of information was a key barrier to innovation in many organizations.

From this insight, we recently launched Azure Synapse Analytics, a crucial component of our entire data platform that enables anybody working with data in those disparate places to manage and analyze it from within a single service. It can be used to analyze structured and unstructured data, using standard SQL. Azure Synapse Analytic integrates seamlessly with easy-to-use business intelligence and reporting tool Power BI, as well as Azure Machine Learning for building models. Organizations can put their data to work much more quickly, productively and securely, pulling together insights from all data sources, data warehouses, and big data analytics systems.

Azure Synapse Analytics allows customers to build apps faster and more efficiently, using fewer data scientists and developers. It can scale across proprietary SQL and open source databases and manage analytics workloads to provide fast, cost-effective power over any data – whether it is structured, semi-structured, run in real-time or globally distributed. And whether it’s inside your organization or on the cloud, or both.

In the past, most in-depth business intelligence reports – let’s say for example a detailed sales report on specific products in specific markets in Asia – would require a data engineer to come in and prepare the data, then build a model and test it first and perhaps even a data architect to ensure all the data could be brought across the various platforms. Now it can be done much more easily with existing data or IT staff, meaning business decisions and initiatives can be put into place that much faster, and organizations can flex and adapt to changes and opportunities almost immediately.

Changes Ahead

As the year – and decade – come to a close, it’s amazing to realize what an incredible transformation all industries across Asia are undergoing and will continue to go through in the coming year.

Now as we embark on this new data decade, I greet the future with optimism and excitement, and look forward to seeing many more organizations transform how they work, operate and ultimately succeed using data. Here’s to a fantastic 2020 – and beyond!

[1] https://www.researchandmarkets.com/reports/4769146/asia-pacific-industrial-robotics-market-by

The post Data Estate: A Winning Formula for the New Decade appeared first on Microsoft Malaysia News Center.