We are delighted to offer another special guest blog in collaboration with the Australian and New Zealand Testing Board (ANZTB), one of the ISTQB® Member Boards.
Automation is a hot topic, and within the iSQI professional development portfolio, we see significant demand for ISTQB® Certified Tester – Test Automation Engineer (CT-TAE), A4Q Certified Selenium Tester Foundation, and A4Q Foundation Level Tester For Appium.
In this blog, our guest authors explore using automation utilities for continuous testing.
Introduction
Continuous testing is the process of executing automated tests as part of the software delivery pipeline to obtain immediate feedback on the business risks associated with a software release candidate. When it comes to test automation, most organizations would focus on test automation for Continuous testing by automating the regression test cases for web, mobile, or desktop.
Problem Statement
The real challenge or hidden effort for every organization is to start the automated regression tests successfully. There are so many manual tasks, communication and coordination - that need to perform before starting the actual automation for each release. These manual tasks are given less priority/focus to automate and sit idle in the testing backlog.
Sample manual tasks include,
- Test Environment Setup - ( e.g., Browser updates, OS updates )
- Test Data Generation
- Test Data Reset/Clean up
- Restart the Test Servers
- Ensure the required services are running
- Software updates for testing tools, frameworks, libraries, utilities
- Modifying pipelines variables ( e.g., Test plan, build ) for every new release
- Monthly patching and vulnerability scan for Test Servers, virtual desktops
- To monitor/analyze the logs for health, performance, automation failures, and specific errors
- Alerts on any test environment-specific outages (going to occur soon, occurred)
- Testing Insights and Metrics
Solution
Identify the technologies and programming languages to automate the manual tasks (as per your own organization’s need), which helps to start the automated regression successfully, increases the automation efficiency, and minimizes the automation failures. Try to build effective automation utilities that can be used across your organization.
Stable open-source programming languages and scripting with a continuous integration tool is the best choice. We are going to discuss a couple of options here that help to achieve the goal.
- Open source tools (e.g., Ansible for infrastructure-related automation, Kibana for log analysis and creating dashboards)
- Scripting languages (e.g., Powershell, python, yaml – become the top most needed DevOps skills)
- Any Continuous integration tool (e.g., Jenkins, GitLab, Jira, Azure DevOps) to run your script
Ansible
Ansible is a suite of software tools that enables infrastructure as code. It is an open-source, command-line IT automation software application written in Python. The suite includes software provisioning, configuration management, and application deployment functionality.
Ansible playbook is written in YAML syntax, contains one or more plays and each play within a playbook can run one or more tasks, and each task invokes an Ansible module, which is used to accomplish cross platform automation tasks. Additionally, the win_modules for Ansible typically aim to interact with PowerShell through the Ansible Python interface in the backend when communicating with the Windows Server.
YAML
YAML is a data serialization language that is often used for writing configuration files. It is a popular programming language because it is human-readable and easy to understand. JSON files are valid in YAML.
YAML uses Python-style indentation to indicate nesting. The structure of a YAML file is a map or a list. Maps allow you to associate key-value pairs. Each key must be unique, and the order doesn't matter. Think of a Python dictionary or a variable assignment in a Bash script. A list includes values listed in a specific order and may contain any number of items needed. A list sequence starts with a dash (-) and a space; you can think of a sequence as a Python list or an array in Bash or Perl. A list can be embedded into a map.
YAML also contains scalars, which are arbitrary data (encoded in Unicode) that can be used as values such as strings, integers, dates, numbers, or booleans. The yamllint command can help to ensure you’ve created a valid YAML file before you hand it over to an application.
Python
A rich library of useful packages and ready-to-use ingredients for automation greatly facilitates testing in Python. Python is object-oriented and functional. It allows choosing what suits your tasks better – functions or classes. To create custom utilities, python is recommended by many DevOps developers. For automating your tasks you can easily develop python custom ansible modules.
PowerShell
PowerShell is the most powerful cross-platform task automation solution made up of a command-line shell, a scripting language, and a configuration management framework in Windows. It programmatically lets system administrators to automate IT administration and DevOps-related tasks they would normally perform from the GUI. Ansible also relies heavily on PowerShell to interact with Windows Servers.
If artificial intelligence, machine learning, voice recognition, and voice activation are enabled, then PowerShell automation utilities can be converted as a Testing BOT. Testing BOTs will be in massive demand soon.
Kibana
Kibana is a free and open frontend application that helps in searching, viewing, and visualizing data indexed in Elasticsearch and analyzing the data through the creation of bar charts, pie charts, tables, histograms, and maps. A dashboard view combines these visual elements to then be shared via browser to provide real-time analytical views into large data volumes in support of use cases like logging and log analytics, Infrastructure metrics, Application performance monitoring,
Observability enables you to add and monitor your logs, system metrics, uptime data, and application traces as a single stack. Configuring and monitoring API health endpoints for each feature and test environment via Kibana is the first one need to be set up before running large regression tests.
Continuous Integration tools
Analyze the Continuous integrations tools support for artificial intelligence and machine learning like test slicing, parallel testing with multiple agents, automatic retries, running tests based on tags or filters, identifying flaky tests, enabling advance logging, and classifying test failures. Also, monitoring while running the automated tests, support for configuring the schedule, and triggers to start and stop test execution based on conditions.
Framework
Connecting these tools and creating a framework of useful automation utilities or libraries is the final goal. These utilities can be scaled to automate different manual tasks across different teams and organizations. A few example use cases are discussed below,
Use case 1: Selenium Grid configured in 8 nodes; when monthly patching starts each node has to be updated with the new browser version for 3 browsers to run the regression test. So a tester has to copy the latest browser exe files into each node and, install the browsers, restart the selenium service manually. This manual task can be easily automated using ansible script configured in a continuous integration tool
Use case 2: Your event source is watching API health for a microservice against a test environment and discovers that the microservice is not responding or down—recognizing this as an event. Event-Driven Ansible receives this event and finds the corresponding Ansible Rulebook that indicates: when a “500 Internal Server Error” is encountered from this API, restart the microservice in the test environment. Event-Driven Ansible triggers the instructions in the rulebook and the microservice is restarted, restoring it to normal function so that the automated regression suite which is planned to run for this API microservice is successfully run.
Benefits
- Helps manual testers by reducing environment-related issues a lot
- High Return on Investment can be achieved by developing the automation utilities
- Faster and stable testing cycles, automated regression
- Encourages Testers to learn DevOps skills and automate manual tasks.
About our Guest Authors
Pushparajan Balasubramanian (PB) is working as a Test Lead at Dynamo Recruitment Limited and he is one of the board members of Australia and New Zealand Testing Board (ANZTB). PB is focused on delivering the highest quality to his stakeholders and he has decades of experience in the field of Software Quality Assurance, Test Automation, Business Process Automation, Performance and Security Testing, Product Modelling, Product Management, Fulfilment, Assurance, and Billing. PB specializes in testing-based content writing and mentors aspiring IT professionals who want to move their careers into software testing.
. 
Vignesh Kumar Balasubramanian is working as a Core Automation Manager at CCL and he is one of the associate board members of Australia and New Zealand Testing Board (ANZTB). He has testing experience in various technology domains Cloud Technologies, Mobile Computing Devices, REST API/ Webservices, Mobile Operating Systems, and Wireless technologies. Practicing different flavors of Agile Methodologies (Scrum, SAFe, DevOps) and implementing Agile Test Strategies, building self-organized testing teams. Focused on using automation tools and frameworks for continuous testing (both functional and non-functional) to provide early feedback in all test phases.
