Source – https://www.forbes.com/
Self-driving cars, lifelike robots, and autonomous delivery drones are the sexy, headline-grabbing face of the digital transformation that we see all around us today.
None of these would be possible, though, without data – the oil of the fourth industrial revolution – and the analytic technology we’ve built to allow us to interpret and understand it.
Big Data is a term that’s come to be used to describe the technology and practice of working with data that’s not only large in volume but also fast and comes in many different forms. For every Elon Musk with a self-driving car to sell, or Jeff Bezos with a cashier-less convenience store, there is a sophisticated Big Data operation and an army of clever data scientists who’ve turned a vision into reality.
The term Big Data itself may not be as ubiquitous as it was a few years ago, and that’s purely because many of the concepts it embodies have been thoroughly embedded into the world around us. But just because we’ve heard about it for a while, though, doesn’t mean it’s old news. The fact is that even today, most organizations struggle to get value from a lot of the data they have access to. As a business practice, it’s still very much in its infancy.
So here’s my look at some of the key trends that will influence how data and analytics are used for work, play, and everything in between, this year and in the near future.
AI drives deeper insights and increasingly sophisticated automation
Using VR To Step Inside Your Data: VR Or AR-Enabled AnalyticsThe Amazing Ways You Can Combine AI, 5G, And Machine Vision To Transform Fish FarmingApple Vs. Facebook – Who Will Win The Data Privacy War?
Artificial intelligence (AI) has been a gamechanger for analytics. With the huge amount of structured and unstructured data generated by companies and their customers, even automated manual forms of analytics can only scratch at the surface of what’s to be found.
The simplest way to think of AI, as it is used today, is machines – computers and software – that are capable of learning for themselves. For a simple example, let’s look at a problem we might use a computer to solve today. Which one of our customers is the most valuable to us?
If we only have traditional, non-learning computing available to us, we might be able to take a stab by creating a database showing us which customers spend the most money. But what if a new customer appears who spends $100 in their first transaction with us? Are they more valuable that a customer who has spent $10 a month for the past year? To understand that we need a lot more data, such as the average customer’s lifetime value, and perhaps personal data about the customer themselves such as their age, spending habits or income level would also be useful!
Interpreting, understanding, and drawing insights from all of those datasets is a far more complicated task. AI is useful here because it can attempt to interpret all of the data together and come up with predictions about what the potential lifetime value of a customer may be based on everything we know – whether or not we understand the connections ourselves. An important element of this is that it doesn’t necessarily come up with “right” or “wrong” answers – it provides a range of probabilities and then refines its results depending on how accurate those predictions turn out to be.
Rich new ways to explore and interpret data
Data visualization is the “final mile” of the analytics process before we take action based on our findings. Traditionally, communication between machines and humans is carried out by visualization, taking the form of graphs, charts, and dashboards that highlight key findings and help us to determine what the data is suggesting needs to be done.
The problem here has been that not all people are great at spotting a potentially valuable insight hidden in a pile of statistics. As it becomes increasingly important that everyone within an organization is empowered to act on data-driven insight, new ways of communicating these findings are constantly evolving.
One area where important breakthroughs have been made is the use of human language. Analytics tools that allow us to ask questions of data and to receive answers in clear, human language will greatly increase access to data and improve overall data capabilities in the organization. This field of technology is known as natural language processing (NLP).
Another is new technologies that allow us to get a better visual overview and understanding of data by fully immersing ourselves within it. Extended reality (XR) – a term that includes virtual reality (VR) and augmented reality (AR) will clearly be seen to drive innovation here. VR can be used to create new kinds of visualizations that allow us to impart richer meaning from data, while AR can show us directly how the results of data analytics impact the world in real-time. For example, a mechanic trying to diagnose a problem with a car may be able to look at the engine wearing AR glasses and be given predictions on what components are likely to be problematic and may need replacing. In the near future, we should expect to see new ways of visualizing or communicating data, widening accessibility to analytics and insights.
Hybrid cloud and the edge
Cloud computing is another technology trend that has had a massive impact on the way Big Data analytics are carried out. The ability to access vast data stores and act on real-time information without needing expensive on-premises infrastructure has fuelled the boom in apps and startups offering data-driven services on-demand. But relying entirely on public cloud providers is not the best model for every business, and when you trust your entire data operations to third parties, there are inevitably concerns around security and governance.
Many companies now find themselves looking towards hybrid cloud systems, where some information is held on Amazon Web Service, Microsoft Azure, or Google Cloud servers, while other, perhaps more personal or sensitive data, remains within the proprietary walled garden. Cloud providers are increasingly on-board with this trend, offering “cloud-on-premises” solutions that potentially provide all of the rich features and robustness of public cloud but allowing data owners full custody of their data.
Edge computing is another strong trend that will affect the impact that Big Data and analytics have on our lives over the next year. Essentially this means devices that are built to process data where it is collected, rather than sending it to the cloud for storage and analysis. Some data simply needs to be acted on too quickly to risk sending it backwards and forwards – a good example here is the data gathered from sensors on autonomous vehicles. In other situations, consumers can be reassured that they have an additional level of privacy when insights can be gleaned directly from their devices without them having to send data to any third party. For example, the Now Playing feature on Google’s new Android phones continuously scans the environment for music so it can tell us the names of songs playing in the supermarket or movies we’re watching. This wouldn’t be possible with a purely cloud-based solution as users would reject the idea of sending a constant 24/7 stream of their audio environment to Google.
The rise of DataOps
DataOps is a methodology and practice that borrows from the DevOps framework often deployed in software development. While those in DevOps roles manage ongoing technology processes around service delivery, DataOps is concerned with the end-to-end flow of data through an organization. In particular, this means removing obstacles that limit the usefulness or accessibility of data and deployment of third-party “as-a-service” data tools.
There’s no formal training needed to work in DataOps. The evolution of the role makes it a great opportunity for anyone with experience or interest in an IT career that wants to work on the most exciting and innovative projects, which are often data projects. We will also see the growth in popularity of “DataOps-as-a-service” vendors, offering end-to-end management of data processes and pipelines on-tap and pay-as-you-go. This will continue to lower the barriers of entry to small and startup organizations with great ideas for new data-driven services but without access to the infrastructure needed to make them a reality.