Running parallel test on local and CI

05 Apr 2014

 

- One problem with long running tests is that team starts to ignore them. It's also a blocker for deployment and a productivity debt for developers to switch context back and forth between builds.

- Parallelizing tests is the obvious way to make them run significantly faster. Either you are running your test suite locally or remotely on CI servers, there's good chance that you have multi-core machines. So, why's not?

- Usually your test suites have integration-, API- and UI-level tests. Unit-level tests already run quite fast, so they’re unlikely to be the bottleneck in your builds. Integration and UI level tests are usually the culprits.

- Running test suite in parallel means spawning multiple instance of the server app. One of the biggest hassles is to bootstrap server app with shared dependencies such as test databases, caching layer (memcache, redis), etc... and those varies with each individual app.

- In the scope of this article, I'm running a stack including rails sitting on top of MySQL and redis. The test suite are rspec and cucumber

Tools

parallel_test gem (https://github.com/grosser/parallel_tests)

From their README:

Speedup Test::Unit + RSpec + Cucumber by running parallel on multiple CPUs (or cores).
ParallelTests splits tests into even groups(by number of tests or runtime) and runs each group in a single process with its own database.

Config

- In your database.yml

[code]
test:
database: yourproject_test<%= ENV['TEST_ENV_NUMBER'] %>
[/code]

- For redis, either we turn it off or add a prefix namespace to the key

[code]
# In config/environments/test.rb
config.cache_store = :null_store
[/code]

or

[code]
config.cache_store = :redis_store, {
:host => AppConfig.redis.host,
:port => AppConfig.redis.port,
:password => AppConfig.redis.password,
:db => AppConfig.redis.cache_db,
:namespace => AppConfig.redis.namespace || ENV['TEST_ENV_NUMBER'],
}
[/code]

- Running tests is very straight forward

[code]
rake parallel:create # Create additional database
rake parallel:prepare # Copy development schema
rake parallel:spec # RSpec
rake parallel:features # Cucumber
[/code]

- On CI, we can also pass the profile name into each test, and serialize the output for better readability.

[code]
bin/parallel_cucumber -o "--profile ci" --serialize-stdout features
bin/parallel_rspec -o "--profile ci" --serialize-stdout spec
[/code]

Coming up next: Test perf benchmark