Many web developers get excited and thrilled at the same time when it comes to releasing a new update to one of our plugins. They are excited since there are more features and improvements to be delivered to customers, yet, this also means starting of another round of manual and rigorous release testing. Therefore, you should know the best way to improve this testing process and one option is automation. In this article, you can reveal how to implement it, so make sure that you read all of the sections in this article.

Release Testing Relay
Since there is a 3-person testing relay, usually we approach release testing. You have to ensure that your current set up involves a Google spreadsheet for each product with worksheets for each of the plugins and addons. Each sheet has lots of test scripts to perform, covering the functionality and edge cases for the plugins.

The first person will run the scripts and look for any issues. If an issue is found, then the test is considered as a fail and raised on GitHub to be fixed. Once the first person has finished and the issues have been fixed, the next person in the relay can start their round of testing. This process will be continued and repeated until there are no bugs found. Hence, this test will surely take lots of time and cost.

The testing process above includes unit tests that use PHPUnit and run by Travis Cl every time code is pushed to our GitHub repositories. Even though it is highly automated and able to catch issues with code before the release testing stage, we still need real people to test it, as unit testing is only a small part of the big testing puzzle.

Codeception
Codeception is better known as a behavior-driven testing framework that automates acceptance, functional, and unit testing. Similar to Behat, codeception is also written in PHP, the test is clear and very descriptive.

Although there are only some elements that can be clicked and it doesn’t handle JavaScript interactions, by using the WebDriver module, your acceptance tests can be run.

Installation
At the first time, you may find that setting this up can take much time, but after it ends, you will discover that writing tests is so much faster.  Like PHPUnit, we have Codeception installed in our repository using composer:

Aside from that, we also install WPBrowser, a set of WordPress extensions for Codeception. This extension provides helpful methods allowing the test ‘actor’ to interact easily with a WordPress site.

Environment
It is better for you to roll your own bash script to set up everything needed to run your acceptance tests, instead of using the WPBrowser’s WPLoader module to set up the test WordPress site. The script applies WP-CLI and includes:

1. Installing WordPress in a tmp directory inside your repo (.gitignored of course)
2. Build your plugins from source and install them on the site
3. Download either PhantomJS or Selenium and fire it up to be used by the WebDriver module.
4. Run Codeception tests
5. Shut down PhantomJS or Selenium
6. Kill any Firefox instances

Installation
You can find more requirements for the projects with detailed installation steps in the README.md. After everything is installed, you can fire up the test by running sh run-tests.sh.

The Scenario
The main test file, known as a ‘cept’ is a file that contains all the steps and assertions for a given scenario. The scenario is to test the core functionality of WP offload S3 Lite – upload a file to the WordPress Media Library and make sure that it gets offloaded to S3:

The steps are very understandable and show clearly the process that show what an actual tester would do, such as login, add a file to the media library, ensure that the URL is rewritten to an S3 URL, then check with AWS to find out if the file has been uploaded.

In conclusion
As it is stated above, you will need a hard work to get this all set up. Remember that the scenario above only for a single site install of WordPress. Therefore, a custom database module for Codeception is needed to make switching between databases and installs during testing as convenient as possible.

You also require script maintenance as code and product usability changes over time. Hence, the best solution is to replace two out of the three of your human testers with automated tests using Codeception. This will surely give more advantages to your release testing process.