Product utilization analytics are an untapped gold mine for software program high quality assurance groups. This text exhibits how linking person habits and connecting to high quality technique via information graphs, functions can enhance each technical and enterprise high quality.
Creator: Vignesh Govindarajan Ravichandran
The Wake-Up Name
At virtually three within the morning, my pager blared with the dreaded message: “Utility Well being Verify-Failure.” This was the third main system outage in two months. I needed to log into my PC with a purpose to acquire some perspective on the state of affairs and grapple with the uncomfortable reality: our classical high quality assurance method had, but once more, did not catch the faults that have been inflicting in depth downtime and irretrievably misplaced enterprise alternatives.
Within the subsequent root-cause evaluation, the respective module High quality lead acknowledged “However we had 92% take a look at protection; All vital paths have been examined.”
This second modified all the things about my pondering regarding software program high quality. With practically 100% take a look at protection, we nonetheless had did not seize what actually mattered: how our customers navigated and interacted with the system in real-world use.
Past Check Protection: The High quality Analytics Revelation
In monetary providers, a single calculation mistake can result in compliance infractions or big monetary losses. In opposition to this backdrop, conventional or typical high quality metrics similar to take a look at protection or defect charges, in some circumstances, can present solely a partial perception. Fairly the true high quality of the software program is a measure of its functionality to fulfill person wants and expectations for any utility with respect to how these requests or expectations could reveal themselves by way of usability and shopper satisfaction/expertise.
Thus, our journey began with a quite simple query: what if we may flip the product utilization information into actionable high quality insights? In collaboration with our product analytics crew, we’ve developed a extra detailed framework to seize, analyze, and visualize using our threat administration platform in actual eventualities. The findings have been deeply distressing:
- 72% of customers utterly skipped our painstakingly well-defined working journey for threat profiling
- Efficiency reporting – essentially the most examined function – was utilized by solely 11% of our customers
- 83% of buyer help tickets stemmed from 4 minor person journeys that got very low testing focus
Essentially the most compelling was the identification of a powerful linkage between outlined utilization patterns and shopper churn. Prospects who deserted particular workflows or had particularly elongated response instances in key transaction paths have been 4.6 instances extra more likely to both scale back their utilization or stroll away from the app onto different functions/information sources quite than increase a ticket inside a interval not exceeding six months.

Constructing the Information Graph
We constructed a information graph that established connections between utilization habits, system efficiency, error states, and enterprise outcomes. This data graph serves as the premise for our predictive churn mannequin in addition to high quality transformation.
The information graph method contributed to a number of benefits over typical analytics:
1. Relationship Visibility: Fairly than treating remoted metrics as a foundation for high quality choices, we may visualize how explicit actions carried out by customers infused with sure metrics can hyperlink to enterprise outcomes beneath clearer high quality priorities.
2. Sample Recognition: If not for the method of mapping motion sequences, the tendency of the graph to disclose lacking workflows would have gone unnoticed by particular person occasions monitoring.
3. Predictive Functionality: Authoring predictive habits patterns that helped in figuring out at-risk shoppers previous to them truly coming into the churn zone was doable with the assistance of the graph.
The broader platform that underlies our information graph entails the next 5 steps:
1. Occasion Sequence: This record contains person workflows throughout the system to incorporate each practical (present flows) and additive (new workflows on account of newest modifications scheduled within the launch).
2. Efficiency of every Node: Subsequent, it’s the identification of efficiency metrics for particular steps in person journeys. These would possibly then deliver to gentle some steps whose longer response instances would possibly have an effect on vital paths.
3. Consumer Motion and Error Correlation: Positioning relationships between person actions and error states- when customers don’t report problems- is Step 3.
4. Consumer Abandonment Sample: After the above processes, examine product analytics to watch screens/fields the place customers persistently abandon processes: indicators for usability or worth notion points.
5. Measure Enterprise Affect: Lastly, construct hyperlinks from the technical patterns again to enterprise outcomes: help prices, renewal charges, and have adoption.
There have been a number of different hurdles in the way in which of implementation; privateness facets, dealing with information quantity, and integrating with already present monitor programs grew to become a problem. However, a lot worth had been envisaged for this mission that govt sponsorship was acquired for a devoted cross-functional crew to deliver it to life.
Remodeling Testing Technique
With the plethora of expertise gained from the information graph, we utterly revised our method to high quality assurance. The breaking modifications within the QA technique will be summarized succinctly in 5 elements:
1. From Protection to Affect
As an alternative of blindly pursuing arbitrary code protection necessities, we employed a mannequin the place checks could be allotted based on how usually totally different facets of our utility have been used and the way necessary these facets have been to the enterprise. Enterprise-critical path protection grew to become the brand new commonplace for fulfillment: giving heavy testing to these workflows that have been essentially the most central to shopper retention and satisfaction. For the shoppers, this translated to about 60% of testing completed towards paths associated to threat calculations with extra permutations of account sorts: the paths that the customers spent essentially the most time on and the place the enterprise impression of defective testing was essentially the most severe.
2. From Hypothetical to Precise Eventualities
We not employed generic take a look at circumstances; as an alternative, we created checks immediately based mostly on actual person habits. By replaying precise utilization sequences of our most precious shoppers, we ensured that our testing could be aligned extra carefully with actual interplay patterns quite than idealized workflows.
In doing so, we unearthed that loads of our shoppers in asset administration utilized our portfolio re-balancing capabilities in a way that we by no means anticipated – constructing a number of situation analyses earlier than truly executing trades. Conventional take a look at circumstances totally missed recognizing this habits. From Equal Remedy to Danger-Based mostly Prioritization. The information graph allowed us to derive threat scores for numerous elements and paths based mostly on their correlation with shopper churn and help incidents. Now, we might take a look at not all code equally, however quite take a look at a mixture of code paths to totally different extents based mostly on threat profiles.
3. From Equal Remedy to Danger-Based mostly Prioritization
The information graph allowed us to supply threat scores for numerous elements and pathways based mostly on their interactions with shopper churn and help incidents. Testing depth was now dramatically differentiated based mostly on threat profiles as an alternative of equal weight being given to all code paths.
4. From Separate to Built-in Monitoring
This section helped us to interrupt down the wall between manufacturing monitoring and pre-release testing. Check circumstances have been influenced by manufacturing efficiency and error information as they have been generated and prioritized in real-time, representing a suggestions loop.
5. From Technical to Consumer-Centered Metrics
Probably, an important of all was that person expertise, as an alternative of technical correctness, was into the bottom of how we outlined our high quality metrics. Consumer completion charges, time-on-task, and a lower within the variety of help tickets have been success indicators as an alternative of the presence of defects.
The Outcomes: Past Lowered Churn
Past Lowered Churn After having our high quality technique based mostly on our information graph working for 18 months, the outcomes got here in above what we had hoped to realize:
- Consumer churn decreased by 32%.
- Software program-related help tickets have been decreased by 47%.
- Launch velocity, greater than anticipated, went up by 28% with a corresponding more durable take a look at effort.
- Internet Promoter Rating elevated from 24 to 41.
One very fascinating end result was the invention of a hike within the unknown main person path. Our analytics confirmed that asset administration shoppers wanted to have the ability to produce compliance experiences instantly after portfolio rebalance fairly often one thing that linked two workflows that our product crew had by no means linked. After optimizing for this path and supply, we noticed a 74% enhance within the utilization of our compliance reporting function and vastly improved shopper satisfaction scores.
Conclusion: The Untapped Gold Mine
Product utilization analytics represents an untapped gold mine for high quality groups. By linking person habits and connecting to high quality technique via information graphs, functions can dramatically enhance each technical high quality in addition to enterprise high quality. Re-imagining the classical high quality method will want funding in analytical capabilities and breaking down silos between high quality and product and shopper success groups. Engineering leaders want to rework silos into collaborative work norms to reap the rewards of much less churn, higher effectivity, and stronger shopper relationships. Investing In High quality Product Utilization Analytics Teaches Us To Hear First Earlier than We Construct Higher Any Factor. Listening to customers’ habits permits us to construct higher software program and never simply checks.
In regards to the Creator
Vignesh Govindarajan Ravichandran leads High quality groups for Danger, Analytics, and Funding Science at a number one asset administration agency, making certain resilient monetary programs. An ASQ Monetary Providers Group Chief, Check Automation Summit 2025 speaker, and Gartner Peer Ambassador, he explores testing, DevOps, and finance.
References
Agarwal, R., & Dhar, V. (2023). Past the clicking: Information-driven high quality engineering. Harvard Enterprise Overview, 101(3), 112-120.
Ebert, C., & Cain, J. (2022). Consumer analytics in monetary software program high quality. IEEE Software program, 39(5), 36-42.
Monetary Providers High quality Consortium. (2024). Annual benchmark report: High quality in asset administration software program. FSQC Publications.
Huang, J., & Zhang, T. (2023). Information graphs for predictive high quality. Communications of the ACM, 66(8), 78-85.
Worldwide Group for Standardization. (2023). ISO/IEC 25022:2023: Programs and software program High quality Necessities and Analysis (SQuaRE) – Measurement of high quality in use. ISO.
Kim, G., Humble, J., Debois, P., & Willis, J. (2021). The DevOps handbook: The way to create world-class agility, reliability, & safety in know-how organizations (2nd ed.). IT Revolution Press.
Malhotra, R., & Shah, P. (2024). Consumer retention via high quality engineering. Journal of Monetary Know-how, 12(2), 145-162.
Singh, A., & Johnson, M. (2023). Graph-based analytics for monetary software program testing. Proceedings of the Worldwide Convention on Software program Testing, 234-248.
Viswanathan, Ok. (2024). The brand new high quality paradigm: From defect detection to enterprise worth. Monetary Software program Press.