SAN FRANCISCO, CA--(Marketwired - Nov 3, 2016) - The results from the first Application Testing survey conducted by ClusterHQ ™ revealed that 43 percent of app developers spend between 10 and 25 percent of their time debugging application errors discovered in production, rather than developing new features. In an effort to uncover why bugs in production are so commonplace, survey takers were asked to identify the most common causes. The top three challenges reported were:
- An inability to fully recreate production environments in testing (33%)
- Interdependence on external systems that makes integration testing difficult (27%)
- Testing against unrealistic data before moving into production (26%)
With more organizations transitioning to microservices-based applications, it is apparent that legacy testing processes are no longer sufficient to meet the speed and agility requirements of modern businesses.
"Our Application Testing survey highlights that legacy software development practices, like relying on narrow subsets of synthetic data for testing, no longer cut it for teams focused on maximizing the amount of time they spend building features that users value," said Mark Davis, CEO, ClusterHQ. "Forward looking software developer leaders understand that to deliver innovation to customers they must effectively manage the entire application lifecycle across a diverse range of infrastructures, a process that begins with identifying and eliminating bugs as early as possible so that teams can focus on adding end-user value."
In this survey, ClusterHQ polled 386 people, 41 percent of whom identified as DevOps team members, followed by developers (37%), other (8%), operations (7%), QA (5%) and security (2%). Respondents work for organizations that range in size from 1 to 100 employees (49%), 101-500 employees (20%), 501-2500 employees (10%), 2501-5000 employees (5%), 5001-10,000 employees (3%) and 10,000+ employees (13%).
Errors found in production are costly Respondents were asked to identity the environment in which they encounter bugs that are most costly to fix. Unsurprisingly, the majority (62%) selected production as the most expensive stage of app development to fix errors, followed by development (18%), staging (7%), QA (7%) and testing (6%).
Next, the survey took a deeper look at how often bugs are being encountered in production as a result of incomplete testing. Findings revealed:
- Every day: 11%
- Two to three times per week: 13%
- Once per week: 12%
- Every other week: 15%
- Once a month: 25%
- Don't know: 21%
- Never: 3%
A quarter of respondents report encountering bugs discovered in production one or more times per week. Developing new features should be priority number one in agile software development, however when asked what percentage of development time is spent debugging application errors discovered in production instead of working on new features, results showed:
- Less than 10%: 34%
- Between 10-25%: 43%
- Between 25-50%: 18%
- Between 50-75%: 4%
- More than 75%: 1%
Causes of bugs in production Testing is a crucial part of an application's lifecycle, but it's inherently challenging to ensure that tests done in development will mirror what happens in production. Survey takers were asked to select the top challenge associated with testing that causes bugs to appear in production. They responded:
- Inability to fully recreate production environments in testing: 33%
- Interdependence on external systems, making integration testing difficult: 27%
- Testing against unrealistic data before moving into production: 26%
- Difficulty sharing test data across different teams: 10%
- Difficulty creating staging environments for testing: 4%
Recreating production environments was cited as the leading cause of bugs appearing in production, supporting the notion that testing on a laptop is like "flying" in a flight simulator -- the experience can be very different when you actually get up in the air. This challenge was followed closely by interdependence on external systems that makes integration testing cumbersome, which leads into the third most cited challenge: testing against unrealistic data. At present, data is difficult to move between all the places that it is needed, including in test infrastructure. As a result, unrealistic, mock data sets are often used to test applications. However, these unrealistic data sets cannot prepare applications for all real world variables, and thus cause serious, expensive, and time consuming issues down the line.
Testing apps against realistic data A resounding number of respondents (88%) would like the ability to test with realistic data during application development. However, this is not as easy as it sounds, and there are real challenges that prevent this from being the norm. When asked to identify the biggest challenge to testing against realistic data sets, respondents reported:
- Keeping test data up-to-date: 23%
- Moving data to all the places it is needed for testing: 19.5%
- Keeping track of different versions of data to be used for different purposes: 19.5%
- Managing access controls to data: 18%
- Making copies of production data: 14%
- Storage costs: 6%
In addition to this news, ClusterHQ today announced the beta availability of two new products -- FlockerHub and Fli. With FlockerHub and Fli, DevOps teams can seamlessly move data between laptops, test environments, data centers and clouds, all with version and access controls. This flexibility to have data where you need it, when you need it, has an immediate and positive impact on DevOps teams doing continuous integration, continuous delivery and automated testing. To learn more, visit: https://clusterhq.com/2016/11/03/flockerhub-fli-release/
The full survey polled 386 respondents. Not all respondents answered every question. Results shown are based on the percent of people who answered each individual question. You can download the full results here .
During KubeCon 2016, November 8-9 in Seattle, stop by booth 24 for demos of FlockerHub and Fli and to learn more about ClusterHQ.
Can't make it to KubeCon? Learn more about FlockerHub during an online meetup on November 15 at 9am PT. Register today: https://clusterhq.com/register/flockerhub-webinar/
Interact with ClusterHQ
- Follow us on Twitter: @ClusterHQ
- Find us on GitHub: ClusterHQ
- Like us on Facebook: ClusterHQ
About ClusterHQ ClusterHQ makes data as portable, flexible and agile as the containers powering modern computing. The company's suite of products- FlockerHub, Fli and Flocker-give developers and operations teams the control needed to manage the data powering microservices. At the center of the ClusterHQ platform is FlockerHub, which is like GitHub® for data. With it, teams can store and share any Docker data volume with access-controlled users or machines. Fli, a Git-like command line interface, snapshots, clones, pushes and pulls data volumes to FlockerHub. Used together, DevOps teams can seamlessly move data between laptops, test environments, data centers and clouds all with version and access controls. In addition to FlockerHub and Fli, customers worldwide use Flocker, the leading open-source volume orchestrator designed to make stateful containers portable in production. Flocker works with over 15 cloud and enterprise storage systems including four of the five global leaders: Dell, EMC, HPE and NetApp. ClusterHQ provides unparalleled data management across entire containerized application lifecycles. We are the Container Data People™.
Media Contact Bhava Communications for ClusterHQ Brianna Galloway Email Contact 925-922-0708
