WordPress testing suite vs Windows 10 and WSL

I’m quite a fan of the Windows operating system in general and Windows 10 Pro in particular. Pretty much all of my projects are written in PHP and JavaScript and developed locally using a great tool called OSPanel. The only exception is a project where I investigate Go programming language and its goroutines to scrap one public API and store data in Elastic Search database (and where I used Docker with 2 separate containers for Go and ES). I’ve enabled Hyper-V on Windows and have a native Docker running locally, and thus – can’t use Vagrant (and tools like Local by Flywheel, see this issue) because of BSODs.

Recently, I was given a commit access to the CMB2 library, which is a developer’s toolkit for building metaboxes, custom fields, options pages and forms for WordPress that will blow your mind. Its paramount priority is backward compatibility, and I totally understand and support that for this particular project. To effectively contribute I need to be able to run tests locally and not push and wait for Travis to do its job. And here is where my adventures begin.

Develop on Windows 10, run tests in WSL

CMB2 uses unit tests, and inherits a testing suite from WordPress, using its tools to install everything needed. At first, I decided to use a WSL, a Linux subsystem on Windows. So I had to install a bunch of packages in WSL using regular sudo apt-get install *. After that, I had to do a bunch of steps of different complexity and confusion:

1. turn off the server (apache and mysql) on my host machine inside OSPanel to release ports
2. run sudo service apache2 start and sudo service mysql start on WSL
3. install a tests library using bash tests/bin/install-wp-tests.sh wordpress_test root '', should be run on WSL – this is a one time task
4. run composer install and npm install – this is a one time task
5. run tests inside the library folder using ./vendor/bin/phpunit.
6. to continue development on Windows (where all of my projects located) – run sudo service apache2 stop and sudo service mysql stop on WSL…
7. …and turn on the server on a host machine in OSPanel.

Seems to be just 7 steps, but 5 of them should be repeated over and over again when you develop on Windows and run tests in WSL. It’s a (somewhat manageable) headache.

Develop and run tests in WSL

This one looks to be more logical, but the issue is that you have different tools on 2 different servers and environments, and the cost is that you have to constantly switch between them AND also start/stop services either on Windows or in WSL. That’s really annoying.

And there is no way that I can migrate all projects completely to WSL. Reason – a killer OSPanel feature, see the screenshot below.

As you can see I can select PHP on a server starting from 5.2 and up to 7.2, as well as Apache2 (from 2.2) and MySQL (from 5.1, or MariaDB 10). This is super helpful while performing a manual testing of various plugins on various PHP/WordPress versions. Yeah, I know about writing tests, but that requires a ton of initial time investment and I don’t have such luxury at this point.

And as far as I remember, having several PHP versions in Ubuntu is a problematic situation, especially with things like extensions compatibility. Frankly speaking, I’ve not dived too deep in an investigation, but I’m not an Ubuntu fun anyways, so will postpone this as long as I can.

Develop and run tests using Docker

I’ve tried that too. Imported an original image with the latest WordPress, Apache and PHP 7.2, tried to set up a volume, but for some reason, it was failing for me all the time with various issues: either container doesn’t start with no errors, or directory (that is linked using volumes) is visible but empty. Most likely I haven’t spent enough time reading manuals.

I still want to dive into this, but I suspect that for my needs I will need to have a ton of containers, to be able to switch PHP versions. And with a lack of deep knowledge of this technology that will take me a ton of time. I need different PHP containers linked to particular Apache versions and all of them working with the same MySQL container, so DB is shared no matter which php-apache pair is currently active.

Develop and run tests on Windows 10

So the next thing, that I tried recently, is having all the tests running on Windows itself. That was several hours of WTF and scratching head.

First of all, I had to use Git Bash, and not cmd/powershell, as installing the test suite requires running a bash script. So:

1. make sure Apache/MySQL are both running on Windows (using OSPanel in my case)
2. using Git Bash (comes with Git for Windows) run bash tests/bin/install-wp-tests.sh wordpress_test root ''
3. it may fail on its very last steps – when removing existing and creating a new wordpress_test database, but this can be easily done manually. For me it failed because Git Bash doesn’t know about mysql command outside of the OSPanel tools, see below
4. now we need to find where the script has installed WordPress and its tests library, for me it was in Appdata/Local/Temp
5. as this temp directory is not reliable, I’ve moved /wordpress/ and /wordpress-tests-lib/ directories to another convenient place under the OSPanel server
6. as Git Bash doesn’t know about PHP (it’s installed separately and available in OSPanel own command line tool – ConEmu) I need to go to ConEmu and install composer and npm packages for a library
7. now in ConEmu when I try to run ./vendor/bin/phpunit.bat – I get errors of paths (can’t find files etc)
8. investigations in /cmb2/tests/bootstrap.php, /wordpress-tests-lib/wp-tests-config.php taught me that I need to do several things described below
9. define a path to the WordPress codebase I’d like to test, the default hardcoded value is /tmp/wordpress/, change in my case to:
   define( 'ABSPATH', 'C:\OpenServer\domains\test\wordpress\' );
10. set up an environment variable to a /wordpress-tests-lib directory in Apache configuration was NOT helpful:
    SetEnv WP_TESTS_DIR "C:\OpenServer\domains\test\wordpress-tests-lib"
    Although it’s still helpful to have it for future, when working in a browser, I guess.
11. so the only way to make it work – use a console command to set the Windows environment variable and run tests in that same session right after that (no, editing system environment variables tool in Control Panel doesn’t help – PHP doesn’t see it):
    set WP_TESTS_DIR=C:\OpenServer\domains\test\wordpress-tests-lib
./vendor/bin/phpunit.bat
12. Only after that, I got FINALLY tests working. I had 1 failed test because of the Windows environment and that it generates paths using backslashes \ instead of forward slashes /, but I will deal with it later, perhaps a simple paths normalization will help.

Summary

It’s hard to run WordPress core-based tests library in Windows. It requires a ton of source code reading and actually modifying tests environment by tweaking hard-coded paths. Why some parts can be configured using environment variables, and others cannot? Why tests are so focused to be run on Unix-based systems?

I can’t move completely to WSL, and I need to actually find time to dive into Docker extensively, as this seems to be the only cross-platform solution that will allow me to use native tools.

With CMB2 I will stick with running/writing tests inside WSL when the actual development will be handled by Windows 10 and OSPanel.