Report: Data engineers spend 2 days per week firefighting bad data quality
Image Credit: Jay Yuno // Getty Images
Were you unable to attend Transform 2022? Check out all of the summit sessions in our on-demand library now! Watch here.
Past industry surveys have shown that most data professionals struggle with poor data quality, despite the amount of time and resources invested in fixing it. To expand on these insights and help data leaders understand where and how to invest resources, data reliability company, Monte Carlo, partnered with Wakefield Research to survey more than 300 data professionals to reveal the state of data quality in the broader market.
Results from the study show that teams are still struggling with the quality of data and wasting precious time, money and resources to fix issues that arise. Respondents reported that on average, 40% of their time is spent evaluating or checking data quality. Additionally, more than half of respondents ranked building or fixing pipelines as taking the most amount of their time throughout the day.
When data incidents occurred, the majority of data pros — 62% — said their time to detection was five to eight hours. The average time for resolving data incidents was nine hours.
Data quality and the bottom line
Poor data quality was also shown as affecting the bottom line — with 47% of respondents estimating that bad data affected 25% or more of their company revenue.
Of the 300 data professionals surveyed:
- 75% take four or more hours to detect a data quality incident.
- About half said it takes an average of nine hours to resolve the issue once identified.
- 58% said the total number of incidents has increased somewhat or greatly over the past year, often as a result of more complex pipelines, bigger data teams, greater volumes of data, and other factors.
However, there is some positive news. Roughly 90% of respondents reported they were already investing or planning to invest in data quality solutions, like data observability, within six months.
The survey of more than 300 data professionals was conducted between April 28 and May 11, 2022, using an email invitation and an online survey.
Read the full report from Monte Carlo.
VentureBeat’s mission is to be a digital town square for technical decision-makers to gain knowledge about transformative enterprise technology and transact. Learn more about membership.