The Problem
In the realm of Quality Engineering, you can save time by increasing the speed of automation execution, so developers get faster feedback cycles. But that’s not all. There is an enormous amount of time spent analyzing automation failures, which includes test failures due to flakiness, environmental issues and reasons that are not real issues. As you practice your fail-fast, test-fast and fail-often concepts, you might spend a lot of time analzing automation failures. Saving a small % of this over a period of time, would result in a huge ROI at the end, in terms of increased productivity.
How can you save time on test analysis? If it takes 10 mins to analyze a test failure, is there any way to reduce it by 50-60% or more?
That would save engineers 60% of the time they’re spending analyzing automation failures.
Often, we use static HTML or reporting plugins for test reporting in build systems themselves, but this is not a centralized one and they don’t help you see the trend of each test and allow you to debug deeper. Test Flakiness is a common issue in testing. To analyze if the test is flaky or a real failure, to see test runs against different versions of your software, you often must see the trend of how the test has behaved in past.
But the bigger picture is as everything is moving to the cloud. From test execution to browser farm, how would you have a centralized automation reporting not just for you and your team but for your entire organization, that anyone can access, that would help you spend the least amount of time necessary to analyze test failures if you didn’t move to the cloud?
Cloud Based Dashboard
Why Cloud?
Storage/Scaling: Cloud helps with alternative data storage/scaling using databases as opposed to storing in excel or build systems
Data Reliability – Cloud multi zone features or automatic backup features can be used to make the data reliable.
Audit – Cloud helps us keep the test automation reports for longer periods of time for auditing purposes.
Large Blobs – Large blobs like video, network logs, screenshot and html reports require alternative cheaper storage like S3.
Multi dimension – Database gives multidimensional aspect to the data where you can handle multi linked data from various sources.
Data Lake for AI/ML– Capturing the test automation data continuously will finally turn your data into Data Lake, which can be utilized by AI/ML for future use.
The cloud-based dashboard would be a perfect solution, which would allow you to collate data for multiple teams in your organization and provide you with trend analysis and most of all have your data available at your fingertip. Save time and analyze more results. Make your decisions based on data in the fastest time. Cloud is the best option for this as other cloud components like S3 could be used to store the video, screenshot and link it to the database where other meta data are stored.
What should the cloud-based automation dashboard have?
A centralized place to store all automation report artefacts of the entire organization.
The data stored should be able to span over weeks and months which will allow you to build trends.
Identify individual tests over a period of time with all its past executions.
Challenge
But most importantly, your framework should have the ability to collect these meta data of each test such as test name, execution status, screen recording of the execution, html report, error message, browser version, name, capabilities, logs, traces, etc and move them to the cloud as your test suite execution is in progress.
Solution
If you want to store the data on the cloud and display the data in the cloud dashboard you need to build an in-house solution. You would need a connector in your automation framework that can move your data to the cloud. You would need to spend thousands of dollars and would need a highly skilled team of engineers to build it.
You have a better option now!
This is what The Cloud Runner aka DigyRunner is all about.
Digy4 has created the Cloud Runner, called DigyRunner. This runner can function as your regular runner e.g., TestNg, but also can collect individual test execution meta data and store them in the cloud. Some of the meta data include test execution status, test case name, browser name, error messages, team name, project name to name a few.
Cloud Runner can also handle high concurrency when you run 100s of tests in parallel and still store them in the backend with high accuracy.
DigyRunner is part of our DigyKube BYOC (Bring Your Own Cloud) solution which provides highly scalable browser farms across your organization. DigyRunner helps to store every one of your tests results in the cloud, group them based on suite, team, project and visualize via our DigyKube Dashboard. You can review the test results and their historical trend and act on them faster than ever before.
If a cloud is a direction your testing team is taking, DigyRunner is your connection between your automation framework and cloud infrastructure.
Instead of developing, you can focus on adopting the solution and accelerate your migration to the cloud.
Adopt DigyKube BYOC in your testing pipeline and experience all that DigyRunner has to offer. Make test analysis faster.
For this to happen, you don’t have to do anything except install the DigyRunner library in your automation framework. No need to change your automation code.
Our next blog will talk about the DigyKube BYOC Dashboard, which will show you how the data collected by DigyRunner is visualized and can provide faster test analysis.