Over the last couple of weeks after my presentation in Applitool’s conference “Future of Testing – Canada”, received a lot of interest in the 360 Automation Platform that I talked about. Inspired me to expand on my previous posts about architecting the 360 Automation Platform at the enterprise level.
The building blocks for such an Automation Platform can be broken up into 3 dimensions.
X-Axis: Types of Testing.
The platform that supports all types of testing happening in the organization: Web, Mobile, API, Database, Accessibility, Visual testing, Non-functional testing, etc.
Y-Axis – DevOps Model
The platform should allow the testing teams to adopt the DevOps model of testing by creating a continuous testing pipeline that supports the different phases of testing: Test Planning, Script Development, Code Repository, CI/CD, Execution, Test Data Platform, Reporting. It should allow running the entire test pipeline in an automated way without manual intervention. With the help of the platform, organizations will be able to achieve their goal of a full-scale DevOps pipeline – starting from the development to release.
Z-Axis – Scalability
Test Infrastructure and Operational optimization – Scaling of automated testing capacity needs test infrastructure to expand. The platform needs to scale up or scale down on demand to meet the test infrastructure requirements by the integration of cloud solutions.
To handle the increase in testing activities over time and increase operational efficiency, AI will play a critical part in the automation tooling strategy. The platform needs to integrate AI-based features and also handle all the digital data from the testing operations to feed the AI Models to increase the accuracy of predictions.
X-Axis: Types of Testing
The platform needs to be built by integrating all the tools that are required to support all the digital channels (e.g., API, Web, Mobile, IoT, etc.) of testing and also support all the different types of testing that the organization needs. The key point to keep in mind, you are not building the tools from the ground up but building integration among the tools and breaking down the walls of silos, thus providing the testing teams in your organization to align to a standard, accelerated, and integrated way of developing automation. Integration should be built in a way that testing teams can develop tests that run across multiple digital channels within the same test case or test suites- API, Mobile, Web, etc. to cover an end-to-end user journey. I like to call it hybrid tests.
The successful implementation will depend on the tool selection process. While selecting tools for each type of testing, key integration factors need to be considered.
Key integration factors:
Benefits
These integration factors will allow you to avoid building tests in silos of each tool’s ecosystem and help you to build an integrated ecosystem for automation.
These will facilitate building automation coverage for an end-to-end user journey across multiple digital channels. No-one tool will support all the digital channels and all types of testing. You have to build it through the integration of multiple tools.
This will allow leveraging script developed for one type of testing to be used in another type of testing.
Example:
The platform can be the one-stop-shop for running and tracking all the tests in the organization. The same platform can run different levels of tests in sequence and displays stage-wise testing results – Unit, System, Integration, Performance, Accessibility, etc.
Pick your tools thoughtfully to build a unified platform of test automation.
Y-Axis: Continuous Testing Pipeline
Every organization that wants to adopt quality engineering is looking to have a well-orchestrated continuous testing pipeline. The continuous testing pipeline requires close integrating of all the phases of testing and automation support for those phases. It starts with test planning and goes all the way through test reporting. The pipeline should have the ability to hook up to the development pipeline on one end and the release pipeline on the other end.
Phases to be considered for Automated Continuous testing pipeline:
Test Planning, Script Development, Code Repository, Continuous Integration, Execution, Test Data Platform, Reporting
There are numerous tools in each of these areas to pick from. There are some important considerations while selecting the best tool for your organization. This is a dimension where you want to analyze the tool stack that exists in your organization so that you can align your tool choices to closely work with them or adopt the existing tools in your organization wherever possible.
Test Planning:
Let’s consider test planning. Most of the organization these days have tools for project management and development lifecycle management. Example – Jira, TFS, Azure DevOps, etc. If your organization already uses such a tool, plan to adopt test management tools that are closely coupled with these project management tools.
Benefits:
Script Development:
Adopt the programming IDE or language whichever your dev teams are using if possible. You can reuse a lot of dev code for your test development. Integration between Dev and QA code becomes more seamless.
Code Repository:
Leverage the code repository that your dev team is using if possible. Less repository maintenance overhead. Building continuous integration with the dev pipeline will be easier and more closely coupled.
Continuous Integration:
Same here – it would be easier to leverage the Continuous integration platform your dev and release management teams are using.
Execution:
The number of digital channels we have to cover these days is increasing. It often becomes hard to maintain the whole infrastructure of test execution – all browsers and mobile devices on on-premise hardware. Many organizations rely on cloud-based solutions to support the execution of cross-browser and cross-device testing. 360 Platform needs to build an integration to these on-demand cloud-based tools to provide on-demand support for cross-browser and cross-device testing. When working at the organizational level you will find one cloud platform doesn’t provide all the functionalities that the organization needs, and you have to use multiple platforms depending on the type of test you are running. You may also want to leverage in house PaaS based browser farm or device farm for running noncritical tests or to reduce cost.
360 Automation Platform should allow you to mix and match the tooling platforms from different vendors and in-house tools. The testing teams can select which test to run on which platform by configuration switching. It will allow the effective use of cloud platforms for increased coverage and cost optimizations.
Test Data Platform:
Test data is the heart of the Continuous testing pipeline. We often focus on building tests and forget to build the automated test data pipeline which can feed the test execution. As a result, you end up having two major issues
While orchestrating a continuous testing pipeline consider an automated test data generation platform as a key part of the initiative. I like to call it (#TDA). There would two types of test data that we have to deal with in testing.
Reusable data: Test Data that can be reused again and again during execution and only need one time set up. This type of data can be easily handled by setting it up once.
Disposable data: Test data that get consumed after one or a certain number of test execution and needs to be refilled or reset. Continuous testing needs to have ways to refill or reset these data continuously to feed to the execution pipeline.
We often end up doing these refills or reset process as part of the test automation script. It can work but will extend your test execution time. Test data may end up significantly increasing your total execution time.
You are always running against time to publish the test result. One way to achieve faster execution can be to create the test data ahead of execution and have them stored in the test data pool. Test execution can pick the test data during execution and run the test with a smaller execution time.
One other option may be to use service virtualization or stubs to generate mock data, this may also reduce the test execution time and create more stability for your test.
When it comes to the test data world, there can be several other options based on business scenarios, 360 Automation Platform should integrate those capabilities into the pipeline.
Continuous testing pipeline without continuous data feed is a car without fuel.
Reporting:
In a continuous testing pipeline, reporting can be done at multiple levels.
High-level reporting: High-level test results can be sent back to the test management tool which sits on the project management tool. This will allow the project team to get a real-time status update on the test execution without having to contact the testers every time they need a status update. Test management tools will also allow them to build custom dashboards to fit the need of the project teams. The project management team can remain on their project management tool and get all the testing updates without having to understand all the complexities of the test tool ecosystem. The project team will love to find all the information at their fingertips.
Detail level reporting: You may not want to dump all the test result details on the project management tool as it may sometimes impact the performance of the project management tool. You can maintain the detail level test results like step-by-step screenshots, videos, debug log in custom test results and link them to the test report on the project management tool. Store them on the cloud or in the code repository.
Build the continuous testing pipeline that runs through all the testing phases in an automated way. Manual intervention anywhere in the pipeline will impact your ability to run an automated test pipeline.
Plan, integrate, automate, and execute.
Z-Axis: Scalability
The platform needs to include two key features to address scalability – Test Infrastructure and Operational optimization.
Test infrastructure
Test infrastructure is becoming a more and more important topic these days as the volume of execution and variety of digital channels to support are increasing.
The world of data is moving toward Big Data and real-time analytics. Testing of these is fast becoming a priority for testing teams. Infrastructure again plays a big role when it comes to supporting volume testing within a highly limited time window.
The cost of the test infrastructure is increasing. Dedicated hardware can no longer be the backbone of test infrastructure. IT infrastructure is moving toward cloud-based infrastructure to facilitate on-demand scaling and pay as you go model. Testing teams need to adopt the cloud as their new home for test infrastructure.
In the cloud world, you have many options – IaaS, PaaS, and SaaS. SaaS solutions are easy to adopt as you can avoid the overhead of infrastructure maintenance but the cost of running things in SaaS is extremely high. PaaS is a good alternate to implemented in addition to SaaS solutions. By adopting a hybrid model of using SaaS and PaaS infrastructure team can scale at an optimized cost in the cloud world.
Similar to how the larger IT industry is adopting cloud-based models to simplify the complexity of infrastructure maintenance and let the development team focus on business logic, the 360 automation platforms need to take care of the infrastructure complexity and let the testing teams focus on test development without worrying about infrastructure scaling.
One other aspect to consider for 360 Automation Platform is the adoption of AI. At one point there would be too many results to analyze in the test operations. Manual analysis will become a bottleneck in the testing operation. The AI-infused test operation will help to remove the roadblocks and help the test pipeline to keep up with the pace of the Digital Transformation pace.
One of the keys factors of effective AI adoption would be to store all the data from the testing operations and analyzing the data by AI model. AI will never be able to give effective predictions if we can’t provide enough and accurate data to the AI model. To effectively use AI 360 Automation Platform should be able to store all the data in the data lake and use it to make AI predictions more accurate over time.
Overall Cloud and AI are essential parts of the 360 Automation Platform that should allow testing teams to Scale on-demand and at an optimized cost.
A holistic approach for Automation Platform needs all the 3 dimensions to be looked into. Anything less than that will not be able to scale your team enough to catch up with the pace of digital transformation in today’s IT world.
Watch out for more details on “360 Automation” in my upcoming posts…