parallel_tests

Gem Version
Build status

Speedup Minitest + RSpec + Turnip + Cucumber + Spinach by running parallel on multiple CPU cores.

ParallelTests splits tests into balanced groups (by number of lines or runtime) and runs each group in a process with its own database.

Setup for Rails

RailsCasts episode #413 Fast Tests

Install

Gemfile:

gem 'parallel_tests', group: [:development, :test]

Add to config/database.yml

ParallelTests uses 1 database per test-process.

Process number123
ENV[‘TEST_ENV_NUMBER’]“‘2’‘3’

test:
  database: yourproject_test<%= ENV['TEST_ENV_NUMBER'] %>

Create additional database(s)

rake parallel:create

(Multi-DB) Create individual database

rake parallel:create:
rake parallel:create:secondary

Copy development schema (repeat after migrations)

rake parallel:prepare

Run migrations in additional database(s) (repeat after migrations)

rake parallel:migrate

(Multi-DB) Run migrations in individual database

rake parallel:migrate:

Setup environment from scratch (create db and loads schema, useful for CI)

rake parallel:setup

Drop all test databases

rake parallel:drop

(Multi-DB) Drop individual test database

rake parallel:drop:

Run!

rake parallel:test # Minitest
rake parallel:spec # RSpec
rake parallel:features # Cucumber
rake parallel:features-spinach # Spinach

rake parallel:test[1] –> force 1 CPU –> 86 seconds
rake parallel:test –> got 2 CPUs? –> 47 seconds
rake parallel:test –> got 4 CPUs? –> 26 seconds
…

Test by pattern with Regex (e.g. use one integration server per subfolder / see if you broke any ‘user’-related tests)

rake parallel:test[test/unit] # every test file in test/unit folder
rake parallel:test[user] # run users_controller + user_helper + user tests
rake parallel:test[‘user|product’] # run user and product related tests
rake parallel:spec[‘spec\/(?!features)’] # run RSpec tests except the tests in spec/features

Example output

2 processes for 210 specs, ~ 105 specs per process
… test output …

843 examples, 0 failures, 1 pending

Took 29.925333 seconds

Run an arbitrary task in parallel

RAILS_ENV=test parallel_test -e "rake my:custom:task"
# or
rake parallel:rake[my:custom:task]
# limited parallelism
rake parallel:rake[my:custom:task,2]

Running things once

require "parallel_tests"

# preparation:
# affected by race-condition: first process may boot slower than the second
# either sleep a bit or use a lock for example File.lock
ParallelTests.first_process? ? do_something : sleep(1)

# cleanup:
# last_process? does NOT mean last finished process, just last started
ParallelTests.last_process? ? do_something : sleep(1)

at_exit do
  if ParallelTests.first_process?
    ParallelTests.wait_for_other_processes_to_finish
    undo_something
  end
end

Even test group runtimes

Test groups will often run for different times, making the full test run as slow as the slowest group.

Step 1: Use these loggers (see below) to record test runtime
Step 2: Your next run will use the recorded test runtimes (use --runtime-log if you picked a location different from below)

RSpec

Rspec: Add to your .rspec_parallel (or .rspec) :

–format progress
–format ParallelTests::RSpec::RuntimeLogger –out tmp/parallel_runtime_rspec.log

To use a custom logfile location (default: tmp/parallel_runtime_rspec.log), use the CLI: parallel_test spec -t rspec --runtime-log my.log

Minitest

Add to your test_helper.rb:

require 'parallel_tests/test/runtime_logger' if ENV['RECORD_RUNTIME']

results will be logged to tmp/parallel_runtime_test.log when RECORD_RUNTIME is set,
so it is not always required or overwritten.

TODO: add instructions for other frameworks

Loggers

RSpec: SummaryLogger

Log the test output without the different processes overwriting each other.

Add the following to your .rspec_parallel (or .rspec) :

–format progress
–format ParallelTests::RSpec::SummaryLogger –out tmp/spec_summary.log

RSpec: FailuresLogger

Produce pasteable command-line snippets for each failed example. For example:

rspec /path/to/my_spec.rb:123 # should do something

Add to .rspec_parallel or use as CLI flag:

–format progress
–format ParallelTests::RSpec::FailuresLogger –out tmp/failing_specs.log

(Not needed to retry failures, for that pass –only-failures to rspec)

RSpec: VerboseLogger

Prints a single line for starting and finishing each example, to see what is currently running in each process.

# PID, parallel process number, spec status, example description
[14403] [2] [STARTED] Foo foo
[14402] [1] [STARTED] Bar bar
[14402] [1] [PASSED] Bar bar

Add to .rspec_parallel or use as CLI flag:

–format ParallelTests::RSpec::VerboseLogger

Cucumber: FailuresLogger

Log failed cucumber scenarios to the specified file. The filename can be passed to cucumber, prefixed with ‘@’ to rerun failures.

Usage:

cucumber –format ParallelTests::Cucumber::FailuresLogger –out tmp/cucumber_failures.log

Or add the formatter to the parallel: profile of your cucumber.yml:

parallel: –format progress –format ParallelTests::Cucumber::FailuresLogger –out tmp/cucumber_failures.log

Note if your cucumber.yml default profile uses <%= std_opts %> you may need to insert this as follows parallel: <%= std_opts %> --format progress...

To rerun failures:

cucumber @tmp/cucumber_failures.log

Setup for non-rails

gem install parallel_tests
# go to your project dir
parallel_test
parallel_rspec
parallel_cucumber
parallel_spinach

  • use ENV['TEST_ENV_NUMBER'] inside your tests to select separate db/memcache/etc. (docker compose: expose it)

  • Only run a subset of files / folders:

    parallel_test test/bar test/baz/foo_text.rb

  • Pass test-options and files via --:

    parallel_rspec -- -t acceptance -f progress -- spec/foo_spec.rb spec/acceptance

  • Pass in test options, by using the -o flag (wrap everything in quotes):

    parallel_cucumber -n 2 -o '-p foo_profile --tags @only_this_tag or @only_that_tag --format summary'

Options are:

-n [PROCESSES] How many processes to use, default: available CPUs
-p, –pattern [PATTERN] run tests matching this regex pattern
–exclude-pattern [PATTERN] exclude tests matching this regex pattern
–group-by [TYPE] group tests by:
found - order of finding files
steps - number of cucumber/spinach steps
scenarios - individual cucumber scenarios
filesize - by size of the file
runtime - info from runtime log
default - runtime when runtime log is filled otherwise filesize
-m, –multiply-processes [FLOAT] use given number as a multiplier of processes to run
-s, –single [PATTERN] Run all matching files in the same process
-i, –isolate Do not run any other tests in the group used by –single(-s)
–isolate-n [PROCESSES] Use ‘isolate’ singles with number of processes, default: 1.
–highest-exit-status Exit with the highest exit status provided by test run(s)
–specify-groups [SPECS] Use ‘specify-groups’ if you want to specify multiple specs running in multiple
processes in a specific formation. Commas indicate specs in the same process,
pipes indicate specs in a new process. Cannot use with –single, –isolate, or
–isolate-n. Ex.
$ parallel_tests -n 3 . –specify-groups ‘1_spec.rb,2_spec.rb|3_spec.rb’
Process 1 will contain 1_spec.rb and 2_spec.rb
Process 2 will contain 3_spec.rb
Process 3 will contain all other specs
–only-group INT[,INT] Only run the given group numbers. Note that this will force the ‘filesize’
grouping strategy (even when the runtime log is present) unless you explicitly
set it otherwise via the ‘-group-by’ flag.
-e, –exec [COMMAND] execute this code parallel and with ENV[‘TEST_ENV_NUMBER’]
-o, –test-options ‘[OPTIONS]’ execute test commands with those options
-t, –type [TYPE] test(default) / rspec / cucumber / spinach
–suffix [PATTERN] override built in test file pattern (should match suffix):
‘spec.rb$’ - matches rspec files
‘(test|spec).rb$’ - matches test or spec files
–serialize-stdout Serialize stdout output, nothing will be written until everything is done
–prefix-output-with-test-env-number
Prefixes test env number to the output when not using –serialize-stdout
–combine-stderr Combine stderr into stdout, useful in conjunction with –serialize-stdout
–non-parallel execute same commands but do not in parallel, needs –exec
–no-symlinks Do not traverse symbolic links to find test files
–ignore-tags [PATTERN] When counting steps ignore scenarios with tags that match this pattern
–nice execute test commands with low priority.
–runtime-log [PATH] Location of previously recorded test runtimes
–allowed-missing [INT] Allowed percentage of missing runtimes (default = 50)
–unknown-runtime [FLOAT] Use given number as unknown runtime (otherwise use average time)
–first-is-1 Use "1” as TEST_ENV_NUMBER to not reuse the default test environment
–fail-fast Stop all groups when one group fails (best used with –test-options ‘–fail-fast’ if supported
–verbose Print debug output
–verbose-command Displays the command that will be executed by each process and when there are failures displays the command executed by each process that failed
–quiet Print only tests output
-v, –version Show Version
-h, –help Show this.

You can run any kind of code in parallel with -e / –exec

parallel_test -n 5 -e ‘ruby -e “puts %[hello from process #{ENV[:TEST_ENV_NUMBER.to_s].inspect}]”’
hello from process “2”
hello from process “”
hello from process “3”
hello from process “5”
hello from process “4”

1 Process2 Processes4 Processes
RSpec spec-suite18s14s10s
Rails-ActionPack88s53s44s

TIPS

RSpec

Cucumber

  • Add a parallel: foo profile to your config/cucumber.yml and it will be used to run parallel tests
  • ReportBuilder can help with combining parallel test results
    • Supports Cucumber 2.0+ and is actively maintained
    • Combines many JSON files into a single file
    • Builds a HTML report from JSON with support for debug msgs & embedded Base64 images.

General

  • [ZSH] use quotes to use rake arguments rake "parallel:prepare[3]"
  • [Memcached] use different namespaces
    e.g. config.cache_store = ..., namespace: "test_#{ENV['TEST_ENV_NUMBER']}"
  • Debug errors that only happen with multiple files using --verbose and cleanser
  • export PARALLEL_TEST_PROCESSORS=13 to override default processor count
  • Shell alias: alias prspec='parallel_rspec -m 2 --'
  • [Spring] Add the spring-commands-parallel-tests gem to your Gemfile to get parallel_tests working with Spring.
  • --first-is-1 will make the first environment be 1, so you can test while running your full suite.
    export PARALLEL_TEST_FIRST_IS_1=true will provide the same result
  • email_spec and/or action_mailer_cache_delivery
  • zeus-parallel_tests
  • Distributed Parallel Tests on CI systems) learn how parallel_tests can run on distributed servers such as Travis and GitLab-CI. Also shows you how to use parallel_tests without adding TEST_ENV_NUMBER-backends
  • Capybara setup
  • Sphinx setup
  • Capistrano setup let your tests run on a big box instead of your laptop

Contribute your own gotchas to the Wiki or even better open a PR :)

Authors

inspired by pivotal labs

Contributors

Michael Grosser

michael@grosser.it

License: MIT