Tuesday, September 27, 2022
HomeBig DataThe Rise of DataOps - Atlan

The Rise of DataOps – Atlan

Information is getting even greater, and conventional knowledge administration simply doesn’t work. DataOps is on the rise, promising to tame right now’s chaos and context challenges.

Let’s face it — conventional knowledge administration doesn’t work. Right now, 75% of executives don’t belief their very own knowledge, and solely 27% of information tasks are profitable. These are dismal numbers in what has been known as the “golden age of information”.

As knowledge simply retains rising in dimension and complexity, we’re struggling to maintain it underneath management. To make issues worse, knowledge groups and their members, instruments, infrastructure, and use instances have gotten extra various on the similar time. The result’s knowledge chaos like we’ve by no means seen earlier than.

DataOps has been round for a number of years, however proper now it’s on hearth as a result of it guarantees to resolve this drawback. Only a week aside, Forrester and Gartner not too long ago made main shifts towards recognizing the significance of DataOps.

On June 23 of this 12 months, Forrester launched the newest model of its Wave report about knowledge catalogs — however as an alternative of being about “Machine Studying Information Catalogs” like regular, they renamed the class to “Enterprise Information Catalogs for DataOps”. Every week later, on the thirtieth, Gartner launched its 2022 Hype Cycle, predicting that DataOps will totally penetrate the market in 2-5 years and transferring it from the far left facet of the curve to its “Peak of Inflated Expectations”.

Google Trends data for "DataOps" searches since 2015
Google Developments knowledge on world searches for “DataOps” since 2015. The y-axis reveals “curiosity over time”, or a normalized model of search curiosity. 100 represents peak reputation for the time period within the given time and area.

However the rise of DataOps isn’t simply coming from analysts. At Atlan, we work with fashionable knowledge groups around the globe. I’ve personally seen DataOps go from an unknown to vital, and a few firms have even constructed complete methods, capabilities, and even roles round DataOps. Whereas the outcomes range, I’ve seen unbelievable enhancements in knowledge groups’ agility, velocity, and outcomes.

On this weblog, I’ll break down every little thing you need to learn about DataOps — what it’s, why you need to care about it, the place it got here from, and how one can implement it.

What’s DataOps?

The primary, and maybe most vital, factor to learn about DataOps is that it’s not a product. It’s not a instrument. Actually, it’s not something you should buy, and anybody attempting to let you know in any other case is attempting to trick you.

As an alternative, DataOps is a mindset or a tradition — a means to assist knowledge groups and folks work collectively higher.

DataOps generally is a bit arduous to understand, so let’s begin with just a few well-known definitions.

DataOps is a collaborative knowledge administration observe targeted on enhancing the communication, integration and automation of information flows between knowledge managers and knowledge shoppers throughout a corporation.


DataOps is the flexibility to allow options, develop knowledge merchandise, and activate knowledge for enterprise worth throughout all expertise tiers from infrastructure to expertise.


DataOps is an information administration methodology that emphasizes communication, collaboration, integration, automation and measurement of cooperation between knowledge engineers, knowledge scientists and different knowledge professionals.

Andy Palmer

As you’ll be able to inform, there’s no commonplace definition for DataOps. Nonetheless, you’ll see that everybody talks about DataOps by way of being past tech or instruments. As an alternative, they give attention to phrases like communication, collaboration, integration, expertise, and cooperation.

In our thoughts, DataOps is absolutely about bringing right now’s more and more various knowledge groups collectively and serving to them work throughout equally various instruments and processes. Its rules and processes assist groups drive higher knowledge administration, save time, and cut back wasted effort.

Why must you care about DataOps?

The brief reply: It helps you tame the info chaos that each knowledge individual is aware of all too properly.

Now for the longer, extra private reply…

At Atlan, we began as an information staff ourselves, fixing social good issues with large-scale knowledge tasks. The tasks have been actually cool — we started working with organizations just like the UN and Gates Basis on large-scale tasks affecting tens of millions of individuals.

However internally, life was chaos. We handled each hearth drill that might presumably exist, resulting in lengthy chains of irritating telephone calls and hours spent attempting to determine what went unsuitable. As an information chief myself, this was a personally weak time, and I knew it couldn’t proceed.

We put our minds to fixing this drawback, did a bunch of analysis, and discovered the thought of “knowledge governance”. We have been an agile, fast-paced staff, and conventional knowledge governance didn’t seem to be it match us. So we got here collectively, reframed our issues as “How May We” questions, and began an inner challenge to resolve these questions with new tooling and practices. By bringing inspiration from various industries again to the info world, we stumbled upon what we now know as DataOps.

It was throughout this time that we noticed what the suitable tooling and tradition can do for an information staff. The chaos decreased, the identical large knowledge tasks grew to become exponentially quicker and simpler, and the late-night calls grew to become splendidly uncommon. And in consequence, we have been capable of accomplish way more with far much less. Our favourite instance: we constructed India’s nationwide knowledge platform, executed by an eight-member staff in simply 12 months, a lot of whom had by no means pushed a line of code to manufacturing earlier than.

We later wrote down our learnings in our DataOps Tradition Code, a set of rules to assist an information staff work collectively, construct belief, and collaborate higher.

That’s in the end what DataOps does, and why it’s all the fad right now — it helps knowledge groups cease losing time on the infinite interpersonal and technical velocity bumps that stand between them and the work they like to do. And in right now’s economic system, something that saves time is priceless.

Atlan's DataOps Culture Code
A part of our DataOps Tradition Code

The 4 elementary concepts behind DataOps

Some individuals prefer to say that knowledge groups are identical to software program groups, and so they attempt to apply software program rules on to knowledge work. However the actuality is that they couldn’t be extra completely different.

In software program, you’ve some degree of management over the code you’re employed with. In any case, a human someplace is writing it. However in an information staff, you usually can’t management your knowledge, as a result of it comes from various supply techniques in a wide range of continuously altering codecs. If something, an information staff is extra like a producing staff, remodeling a heap of unruly uncooked materials right into a completed product. Or maybe an information staff is extra like a product staff, taking that product to all kinds of inner and exterior finish shoppers.

The best way we like to consider DataOps is, how can we take one of the best learnings from different groups and apply them to assist knowledge groups work collectively higher? DataOps combines one of the best components of Lean, Product Pondering, Agile, and DevOps, and making use of them to the sector of information administration.

The 4 elementary concepts behind DataOps


Key concept: Scale back waste with Worth Stream Mappings.

Although its roots return to Benjamin Franklin’s writings from the 1730s, Lean comes from Toyota’s work within the Nineteen Fifties. Within the shadow of World Warfare II, the auto business — and the world as a complete — was getting again on its toes. For automotive producers all over the place, workers have been overworked, orders delayed, prices excessive, and clients sad.

To resolve this, Toyota created the Toyota Manufacturing System, a framework for conserving assets by eliminating waste. It tried to reply the query, how are you going to ship the best high quality good with the bottom value within the shortest time? Certainly one of its key concepts is to eradicate the eight kinds of waste in manufacturing wherever doable — from overproduction, ready time, transportation, underutilized employees, and so forth — with out sacrificing high quality.

The TPS was the precursor to Lean, coined in 1988 by businessman John Krafcik and popularized in 1996 by researchers James Womack and Daniel Jones. Lean targeted on the thought of Worth Stream Mapping. Similar to you’ll map a producing line with the TPS, you map out a enterprise exercise in excruciating element, establish waste, and optimize the method to take care of high quality whereas eliminating waste. If part of the method doesn’t add worth to the shopper, it’s waste — and all waste must be eradicated.

What does a Worth Stream Mapping really appear to be? Let’s begin with an instance in the actual world.

Lean: Example Value Stream Mapping
The Worth Stream Mapping for ordering a espresso in a restaurant

Say that you simply personal a restaurant, and also you need to enhance how your clients order a cup of espresso. Step one is to map out every little thing that occurs when a buyer takes once they order a espresso: taking the order, accepting fee, making the espresso, handing it to the shopper, and so forth. For every of those steps, you then clarify what can go unsuitable and the way lengthy the step can take — for instance, a buyer having bother finding the place they need to order, then spending as much as 7 minutes ready in line as soon as they get there.

How does this concept apply to knowledge groups? Information groups are much like manufacturing groups. They each work with uncooked materials (i.e. supply knowledge) till it turns into a product (i.e. the “knowledge product”) and reaches clients (i.e. knowledge shoppers or finish customers).

So if a provide chain has its personal worth streams, what would knowledge worth streams appear to be? How can we apply these similar rules to a Information Worth Stream Mapping? And the way can we optimize them to eradicate waste and make knowledge staff extra efficients?

Product pondering

Key concept: Ask what job your product is absolutely carrying out with the Jobs To Be Completed framework.

The core idea in product pondering is the Jobs To Be Completed (JTBD) framework, popularized by Anthony Ulwick in 2005.

The best approach to perceive this concept is thru the Milkshake Concept, a narrative from Clayton Christensen. A quick meals restaurant wished to extend the gross sales of their milkshakes, so that they tried quite a lot of completely different modifications, equivalent to making them extra chocolatey, chewier, and cheaper than opponents. Nonetheless, nothing labored and gross sales stayed the identical.

Subsequent, they despatched individuals to face within the restaurant for hours, amassing knowledge on clients who purchased milkshakes. This led them to comprehend that just about half of their milkshakes have been bought to single clients earlier than 8 am. However why? After they got here again the subsequent morning and talked to those individuals, they realized that these individuals had an extended, boring drive to work and wanted a breakfast that they may eat within the automotive whereas driving. Bagels have been too dry, doughnuts too messy, bananas too fast to eat… however a milkshake was excellent, since they take some time to drink and hold individuals full all morning.

As soon as they realized that, for these clients, a milkshake’s function or “job” was to supply a satisfying, handy breakfast throughout their commute, they knew they wanted to make their milkshakes extra handy and filling — and gross sales elevated.

The JTBD framework helps you construct merchandise that folks love, whether or not it’s a milkshake or dashboard. For instance, a product supervisor’s JTBD could be to prioritize completely different product options to realize enterprise outcomes.

How does this concept apply to knowledge groups? Within the knowledge world, there are two most important kinds of clients: “inner” knowledge staff members who have to work extra successfully with knowledge, and “exterior” knowledge shoppers from the bigger group who use merchandise created by the info staff.

We will use the JTBD framework to know these clients’ jobs. For instance, an analyst’s JTBD could be to supply the analytics and insights for these product prioritization selections. Then, when you create a JTBD, you’ll be able to create a listing of the duties it takes to realize it — every of which is a Information Worth Stream, and will be mapped out and optimized utilizing the Worth Stream Mapping course of above.


Key concept: Improve velocity with Scrum and prioritize MVPs over completed merchandise.

In the event you’ve labored in tech or any “fashionable” firm, you’ve in all probability used Agile. Created in 2001 with the Agile Software program Growth Manifesto, Agile is a framework for software program groups to plan and monitor their work.

The core concept in Agile is Scrum, an iterative product administration framework based mostly on the thought of making an MVP, or minimal viable product.

Right here’s an instance: if you happen to wished to construct a automotive, the place must you begin? You could possibly begin with conducting interviews, discovering suppliers, constructing and testing prototypes, and so forth… however that may take a very long time, throughout which the market and world can have modified, and you could find yourself creating one thing that folks don’t really like.

Agile: Benefits of a Minimum Viable Product (MVP)
Six ways in which an MVP helps enhance improvement

An MVP is about shortening the event course of. To create an MVP, you ask what the JTBD is — is it actually about making a automotive, or is it about offering transportation? The primary, quickest product to resolve this job might be a motorcycle quite than a automotive.

The objective of Scrum is to create one thing as fast as doable that may be taken to market and be used to collect suggestions from customers. In the event you give attention to discovering the minimal answer, quite than creating the best or dream answer, you’ll be able to be taught what customers really need once they check your MVP — as a result of they normally can’t categorical what they really need in interviews.

How does this concept apply to knowledge groups? Many knowledge groups work in a silo from the remainder of the group. When they’re assigned a challenge, they’ll usually work for months on an answer and roll it out to the corporate solely to be taught that their answer was unsuitable. Perhaps the issue assertion they got was incorrect, or they didn’t have the context they wanted to design the suitable answer, or perhaps the group’s wants modified whereas they have been constructing their answer.

How can knowledge groups use the MVP strategy to cut back this time and are available to a solution faster? How can they construct a transport mindset and get early, frequent suggestions from stakeholders?

Agile can be utilized to open up siloed knowledge groups and enhance how they work with finish knowledge shoppers. It will possibly assist knowledge groups discover the suitable knowledge, carry knowledge fashions into manufacturing and launch knowledge merchandise quicker, permitting them to get suggestions from enterprise customers and iteratively enhance and adapt their work as enterprise wants change.


Key concept: Enhance collaboration with launch administration, CI/CD, and monitoring.

DevOps was born in 2009 on the Velocity Convention Motion, the place engineers John Allspaw and Paul Hammond offered about enhancing “dev & ops cooperation”.

The normal pondering on the time was that software program moved in a linear circulation — the event staff’s job is so as to add new options, then the operations staff’s job is to maintain the options and software program secure. Nonetheless, this discuss launched a brand new concept: each dev and ops’ job is to allow the enterprise.

DevOps turned the linear improvement circulation right into a round, interconnected one which breaks down silos between these two groups. It helps groups work collectively throughout two various capabilities by way of a set course of. Concepts like launch administration (implementing set “transport requirements” to make sure high quality), and operations and monitoring (creating monitoring techniques to alert when issues break), and CI/CD (steady integration and steady supply) make this doable.

DevOps toolchain
The DevOps toolchain (created by Kharnagy on Wikipedia)

How does this concept apply to knowledge groups? Within the knowledge world, it’s straightforward for knowledge engineers and analysts to operate independently — e.g. engineers handle knowledge pipelines, whereas analysts construct fashions — and blame one another when issues inevitably break. As an alternative of options, this simply results in bickering and resentment. As an alternative, it’s vital to carry them collectively underneath a standard objective — making the enterprise extra data-driven.

For instance, your knowledge scientists could depend upon both engineering or IT now to deploy their fashions—from exploratory knowledge evaluation to deploying machine studying algorithms. With DataOps, they’ll deploy their fashions themselves and carry out evaluation rapidly — no extra dependencies.

Going from DevOps to DataOps
Going from DevOps to DataOps

Word: I can not emphasize this sufficient — DataOps isn’t simply DevOps with knowledge pipelines. The issue that DevOps solves is between two extremely technical groups, software program improvement and IT. DataOps solves complicated issues to assist an more and more various set of technical and enterprise groups create complicated knowledge merchandise, every little thing from a pipeline to a dashboard or documentation. Study extra.

How do you really implement DataOps?

Each different area right now has a targeted enablement operate. For instance, SalesOps and Gross sales Enablement give attention to enhancing productiveness, ramp time, and success for a gross sales staff. DevOps and Developer Productiveness Engineering groups are targeted on enhancing collaboration between software program groups and productiveness for builders.

Why don’t we have now the same operate for knowledge groups? DataOps is the reply.

Determine the top shoppers

Reasonably than executing knowledge tasks, the DataOps staff or operate helps the remainder of the group obtain worth from knowledge. It focuses on creating the suitable instruments, processes, and tradition to assist different individuals achieve success at their work.

Key consumers of DataOps
The important thing shoppers of DataOps

Create a devoted DataOps operate

A DataOps technique is handiest when it has a devoted staff or operate behind it. There are two key personas on this operate:

  • DataOps Enablement Lead: They perceive knowledge and customers, and are nice at cross-team collaboration and bringing individuals collectively. DataOps Enablement Leads usually come from backgrounds like Data Architects, Information Governance Managers, Library Sciences, Information Strategists, Information Evangelists, and even extroverted Information Analysts and Engineers.
  • DataOps Enablement Engineer: They’re the automation mind within the DataOps staff. Their key power is sound information of information and the way it flows between techniques/groups, appearing as each advisors and executors on automation. They’re usually former Builders, Information Architects, Information Engineers, and Analytics Engineers.
How WeWork structured their DataOps function around 2 personas
How WeWork structured their DataOps operate round two key personas

Map out worth streams, cut back waste, and enhance collaboration

Firstly of an organization’s DataOps journey, DataOps leaders can use the JBTD framework to establish frequent knowledge “jobs” or duties, also referred to as Information Worth Streams. Then, with Lean, they’ll do a Worth Stream Mapping train to establish and eradicate wasted effort and time in these processes.

In the meantime, the Scrum ideology from Agile helps knowledge groups perceive how construct knowledge merchandise extra effectively and successfully, whereas concepts from DevOps present how they’ll collaborate higher with the remainder of the group on these knowledge merchandise.

Learnings from the 4 movements behind DataOps
What knowledge groups can be taught from the 4 actions behind DataOps

Making a devoted DataOps technique and performance is much from straightforward. However if you happen to do it proper, DataOps has the potential to resolve a few of right now’s greatest knowledge challenges, save time and assets throughout the group, and improve the worth you get from knowledge.

In our subsequent blogs, we’ll dive deeper into the “how” of implementing a DataOps technique, based mostly on greatest practices we’ve seen from the groups we’ve labored with — how one can establish knowledge worth streams, how one can construct a transport mindset, how one can create a greater knowledge tradition, and extra. Keep tuned, and let me know in case you have any burning questions I ought to cowl!

To get future DataOps blogs in your inbox, join my publication: Metadata Weekly

Header picture by Chris Liverani on Unsplash



Please enter your comment!
Please enter your name here

Most Popular