Source:- insidebigdata.com
In this special guest feature, Yuval Greenfield from MissingLink.ai, discusses the rise of “DeepOps” aka AIOps and how it is where DevOps was in the 1990s—a nascent field that is becoming increasingly necessary. It starts with simple, but key facets like addressing some of the biggest pain points in the deep learning development lifecycle, such as automating version control. Yuval has been an engineer and data enthusiast for the past 13 years in the fields of military cybersecurity, computer vision medical diagnostics, gaming, 360 cameras, and deep-learning tools. He holds a B.Sc. in Physics and Mathematics from the Hebrew University of Jerusalem as part of the IDF Talpiot program. At MissingLink, Yuval is in charge of developer relations, using the MissingLink platform for deep learning research, building tutorials, marketing content, and technical presentations.
Before DevOps was a common practice, the worlds of development and operations existed in a constant state of friction. Developers were instructed to innovate and think in new ways, whereas operations staff were charged with keeping all systems stable and predictable, creating two siloed work streams that frustrated both parties equally.
From those frustrations, a culture of DevOps was born, bringing development and operations together to enable organizations to deliver applications and services faster and more efficiently than ever before. Engineers now work across the entire development lifecycle and the increased collaboration among engineers and members of other teams (quality assurance, security, etc.) leads to a more streamlined development procedure. The basics of DevOps, which includes writing and running tests, meticulously tracking the changes through automated processes, has completely changed how software development works in a very short time.
Today, we need to look beyond revolutionizing software development. DevOps has come, it has seen and it has conquered. We must now look to see how we can take the lessons learned from creating a successful DevOps culture and apply them to other critical technology challenges such as those facing deep learning.
Why Deep Learning Needs “DeepOps”
Deep learning is currently one of the most technically advanced fields in the technology industry, and has even been referred to as “Software 2.0.” It powers everything from social network algorithms, your favorite voice assistant, medical devices and so much more, but the development of that technology is being hindered by outdated and inefficient tools and practices that continue to hold it back from reaching its potential. The deep learning development process is similar to software development in that code is being written and tested over and over, but layer on top of that a few added twists that introduce a lot more complications to the matter: data, models and experiments.
Data scientists and engineers working on deep learning projects today spend inordinate time spinning and setting up machines, copying data onto machines, modeling and training data, painstakingly reviewing code to ensure version control and many other tasks that just take too much time away from the actual science. Data scientists need to be able to streamline some of this work and make their workflows more efficient.
Enter DeepOps.
Where to Start
DeepOps today is where DevOps was in the 1990s—a nascent field that is becoming increasingly necessary. It starts with simple, but key facets like addressing some of the biggest pain points in the deep learning development lifecycle, such as automating version control. Data scientists have to run thousands of experiments on different data, different code and different computing environments to make sure they’re working toward the right solution; making sure you’re working with the right data is paramount to achieving the desired results. But with the sheer amount of files and their exorbitant sizes, this is extremely hard to manage without a DeepOps structure in place. If implemented, DeepOps helps to simplify some of the biggest pain points in machine learning development including saving and versioning data, being able to manipulate, slice and process enormous data sets, helping new data scientists pick up where others left off and much more.
What’s Next and What’s Possible
While DeepOps is still a fairly new concept, some organizations are actually already implementing parts of DeepOps culture and calling it AIOps, MLOps, AI Infrastructure or SystemML. While they’re all related, DeepOps maps closest to the trajectory that DevOps went through, and we expect it to reach the mainstream among deep learning experts in the next five years.