Speedup Test::Unit + RSpec + Cucumber by running parallel on multiple CPUs (or cores).

ParallelTests splits tests into even groups(by number of tests or runtime) and runs each group in a single process with its own database.

upgrading from 0.6 ?

Setup for Rails

still using Rails 2?

Install

If you use RSpec: ensure you got >= 2.4

As gem

# add to Gemfile
gem "parallel_tests", :group => :development

OR as plugin

rails plugin install git://github.com/grosser/parallel_tests.git

# add to Gemfile
gem "parallel", :group => :development

Add to config/database.yml

ParallelTests uses 1 database per test-process.

Process number123
ENV['TEST_ENV_NUMBER']“‘2’‘3’

test:
  database: yourproject_test<%= ENV['TEST_ENV_NUMBER'] %>

Create additional database(s)

rake parallel:create

Copy development schema (repeat after migrations)

rake parallel:prepare

Run!

rake parallel:test # Test::Unit
rake parallel:spec # RSpec
rake parallel:features # Cucumber

rake parallel:test[1] –> force 1 CPU –> 86 seconds
rake parallel:test –> got 2 CPUs? –> 47 seconds
rake parallel:test –> got 4 CPUs? –> 26 seconds
…

Test by pattern (e.g. use one integration server per subfolder / see if you broke any ‘user’-related tests)

rake parallel:test[test/unit] # every test file in test/unit folder
rake parallel:test[user] # run users_controller + user_helper + user tests
rake parallel:test[‘user|product’] # run user and product related tests

Example output

2 processes for 210 specs, ~ 105 specs per process
… test output …

843 examples, 0 failures, 1 pending

Took 29.925333 seconds

Run an arbitrary task in parallel

RAILS_ENV=test parallel_test -e "rake my:custom:task"
# or
rake parallel:rake[my:custom:task]

Running things once

# effected by race-condition: first process may boot slower the second
# either sleep a bit or use a lock for example File.lock
ParallelTests.first_process? ? do_something : sleep(1)

at_exit do
  if ParallelTests.first_process?
    ParallelTests.wait_for_other_processes_to_finish
    undo_something
  end
end

Loggers

Even process runtimes

Log test runtime to give each process the same runtime.

Rspec: Add to your .rspec_parallel (or .rspec) :

If installed as plugin: -I vendor/plugins/parallel_tests/lib
–format progress
–format ParallelTests::RSpec::RuntimeLogger –out tmp/parallel_runtime_rspec.log

Test::Unit: Add to your test_helper.rb:

require 'parallel_tests/test/runtime_logger'

RSpec: SummaryLogger

This logger logs the test output without the different processes overwriting each other.

Add the following to your .rspec_parallel (or .rspec) :

If installed as plugin: -I vendor/plugins/parallel_tests/lib
–format progress
–format ParallelTests::RSpec::SummaryLogger –out tmp/spec_summary.log

RSpec: FailuresLogger

This logger produces pasteable command-line snippets for each failed example.

E.g.

rspec /path/to/my_spec.rb:123 # should do something

Add the following to your .rspec_parallel (or .rspec) :

If installed as plugin: -I vendor/plugins/parallel_tests/lib
–format progress
–format ParallelTests::RSpec::FailuresLogger –out tmp/failing_specs.log

Setup for non-rails

gem install parallel_tests
# go to your project dir
parallel_test test/
parallel_rspec spec/
parallel_cucumber features/

  • use ENV[‘TEST_ENV_NUMBER’] inside your tests to select separate db/memcache/etc.

  • Only run selected files & folders:

    parallel_test test/bar test/baz/foo_text.rb

Options are:

-n [PROCESSES] How many processes to use, default: available CPUs
-p, –pattern [PATTERN] run tests matching this pattern
–group-by [TYPE] group tests by:
found - order of finding files
steps - number of cucumber steps
default - runtime or filesize
-m, –multiply-processes [FLOAT] use given number as a multiplier of processes to run
-s, –single [PATTERN] Run all matching files in the same process
-i, –isolate Do not run any other tests in the group used by –single(-s)
-e, –exec [COMMAND] execute this code parallel and with ENV[‘TEST_ENV_NUM’]
-o, –test-options ‘[OPTIONS]’ execute test commands with those options
-t, –type [TYPE] test(default) / rspec / cucumber
–serialize-stdout Serialize stdout output, nothing will be written until everything is done
–non-parallel execute same commands but do not in parallel, needs –exec
–no-symlinks Do not traverse symbolic links to find test files
–ignore-tags [PATTERN] When counting steps ignore scenarios with tags that match this pattern
-v, –version Show Version
-h, –help Show this.

You can run any kind of code in parallel with -e / –execute

parallel_test -n 5 -e ‘ruby -e "puts %[hello from process #{ENV[:TEST_ENV_NUMBER.to_s].inspect}]”’
hello from process “2”
hello from process “”
hello from process “3”
hello from process “5”
hello from process “4”

1 Process2 Processes4 Processes
RSpec spec-suite18s14s10s
Rails-ActionPack88s53s44s

TIPS

  • [RSpec] add a .rspec_parallel to use different options, e.g. no –drb
  • [RSpec] delete script/spec
  • [Spork] does not work with parallel_tests
  • [RSpec] remove –loadby from you spec/*.opts
  • [RSpec] Instantly see failures (instead of just a red F) with rspec-instafail
  • [Bundler] if you have a Gemfile then bundle exec will be used to run tests
  • [Cucumber] add a parallel: foo profile to your config/cucumber.yml and it will be used to run parallel tests
  • Capybara setup
  • Sphinx setup
  • Capistrano setup let your tests run on a big box instead of your laptop
  • [SQL schema format] use :ruby schema format to get faster parallel:prepare`
  • export PARALLEL_TEST_PROCESSORS=X in your environment and parallel_tests will use this number of processors by default
  • [ZSH] use quotes to use rake arguments rake "parallel:prepare[3]"
  • email_spec and/or action_mailer_cache_delivery
  • [Memcached] use different namespaces e.g. config.cache_store = ..., :namespace => "test_#{ENV['TEST_ENV_NUMBER']}"

TODO

  • make tests consistently pass with --order random in .rspec
  • fix tests vs cucumber >= 1.2 unknown option --format
  • add integration tests for the rake tasks, maybe generate a rails project …
  • add unit tests for cucumber runtime formatter
  • make jRuby compatible basics
  • make windows compatible

Authors

inspired by pivotal labs

Contributors

Michael Grosser

michael@grosser.it

License: MIT