Parallelize RSpec

Testing RSpec
1

I am looking for some advice on the best way to parallelize a large
set of browser-based regression tests written in rspec. Just as a
note; we are running these specs with RSpec 2.5 on JRuby 1.6.1.

Our current set of specs takes about 4 hours to run when it is not
parallelized. We have implemented a simple “bucket” parallelization
scheme that basically takes each spec file and divides them evenly
across a specified number of forked buckets. This simple solution
has problems:

  • Some forked buckets finish early and exit. They don’t get the
    chance to contribute to finishing the remaining work.
  • It is somewhat difficult to aggregate all the results into one
    spot (Not really, but annoying).

So… What do I want?

  1. Is RSpec the right tool? If no, what would you suggest?
  2. It seems like having a queue of specs and a thread pool would
    address my two points above. But… I don’t think RSpec is thread
    safe (Specifically RSpec::Core.world and RSpec::Core.configuration).
    Thoughts?
  3. Other ideas?

Thanks for any help you can give.

AE

2

Have you already tried Parallel Tests?

It helped a lot for me…

Hope it helps,

Rodrigo.

Em 28-04-2011 16:54, Adam E. escreveu:

3

You can also take a look at Test Load Balancer · GitHub

Best,
Sidu.
http://c42.in

4

Wouldn’t using threads require that the code being tested also be
threadsafe?

Best,
Sidu.
http://c42.in

5

Both Test Load Balancer · GitHub and
GitHub - grosser/parallel_tests: Ruby: 2 CPUs = 2x Testing Speed for RSpec, Test::Unit and Cucumber seem more complicated than
needed. They both require other moving parts (database; other
server). It seems now with ruby 1.9 and jruby a simpler solution
(maybe harder to code) would be to use a queue and native threads.

Thoughts?

AE

6

Em 29-04-2011 00:19, Adam E. escreveu:

Both Test Load Balancer · GitHub and
GitHub - grosser/parallel_tests: Ruby: 2 CPUs = 2x Testing Speed for RSpec, Test::Unit and Cucumber seem more complicated than
needed. They both require other moving parts (database; other
server). It seems now with ruby 1.9 and jruby a simpler solution
(maybe harder to code) would be to use a queue and native threads.

Thoughts?

The multiple databases will be necessary even if you do have multiple
threads if you are accessing your database in your tests, unless either:
- You run all your examples that hit the database in a single
thread, or;
- You will have to careful choose how to write your tests in a
thread-safe way, which will is very likely error prone and you should
make your tests as simple as possible.

On the other hand, having as many databases as many threads/processes
you have is a much simpler solution and it is not that complicated since
Rails can generate them for you: “rake parallel:create_databases” or
something like that.

Cheers, Rodrigo.

7

On Fri, Apr 29, 2011 at 1:19 PM, Adam E. [email protected]
wrote:

Both Test Load Balancer · GitHub and
GitHub - grosser/parallel_tests: Ruby: 2 CPUs = 2x Testing Speed for RSpec, Test::Unit and Cucumber seem more complicated than
needed. They both require other moving parts (database; other
server). It seems now with ruby 1.9 and jruby a simpler solution
(maybe harder to code) would be to use a queue and native threads.

Maybe dripdrop can make it easy to write some simple performant queues?
The nice thing is that if you do need to grow beyond on machine zeromq
makes that mauch less painful than it was before.
There was a thread.new in DripDrop the last time I looked, and I
haven’t had the time to see if making it optionnal (vs some
observable) is possible/easy.

HTH?

Sidu.

scheme that basically takes each spec file and divides them evenly

  1. Is RSpec the right tool? If no, what would you suggest?
    rspec-users mailing list
    http://rubyforge.org/mailman/listinfo/rspec-users

–
πόλλ’ οἶδ ἀλώπηξ, ἀλλ’ ἐχῖνος ἓν μέγα
[The fox knows many things, but the hedgehog knows one big thing.]
Archilochus, Greek poet (c. 680 BC – c. 645 BC)
http://wiki.hedgehogshiatus.com

8

On Fri, Apr 29, 2011 at 3:52 AM, Sidu P. [email protected]
wrote:

Wouldn’t using threads require that the code being tested also be threadsafe?

Yes. In my case, we are testing a web application through the
browser (regression tests). If the web application is not thread
safe, we want to know about it :slight_smile: