RSpec test failures related to ordering may sometimes appear, particularly on larger and older projects.

These intermittent errors can be difficult to trace, and as they may only occur during a long build process after other tasks have succeeded, can impact productivity.

Fortunately RSpec includes an option to help reproduce errors like these, when used in conjunction with random test ordering.

Randomized tests

Random test ordering may be specified at the command line when running RSpec, for example:

$ rspec --order rand

There is also a more permanent way to enable this option, by using a preferences file named .rspec:

--order rand

Seed values and reproducing a test run

Randomized tests are managed by a test seed, which is displayed in test output such as Randomized with seed 49229. And this holds the key to reproducing test runs.

For example, here is a test with an ordering problem:

# sentence_builder_spec.rb

RSpec.describe "Randomness" do
  before(:context) do
    @sentence = ""
  end

  it "says hello" do
    @sentence << "hello "
  end

  it "says world" do
    @sentence << "world"

    expect(@sentence).to eq("hello world")
  end
end

This test is using state (poorly) to demonstrate an ordering problem–it will pass when the first test is run followed by the second, but when the order is reversed, the spec will fail.

$ rspec --order random sentence_builder_spec.rb

Randomized with seed 10450
..

Finished in 0.00652 seconds (files took 0.15641 seconds to load)
2 examples, 0 failures

This run passed, but the next fails:

$ rspec --order random sentence_builder_spec.rb

Randomized with seed 49229
F.

Failures:

  1) Randomness says world
     Failure/Error: expect(@sentence).to eq("hello world")

       expected: "hello world"
            got: "world"

       (compared using ==)
     # ./sentence_builder_spec.rb:13:in `block (2 levels) in <top (required)>'

Finished in 0.0227 seconds (files took 0.15411 seconds to load)
2 examples, 1 failure

This test will fail about half the time, but what if there were a failure that only occurs very rarely? In that case, one can keep running a (potentially lengthy) test suite repeatedly, or this seed value can be used to re-run the specs in the same order.

$ rspec --order random:49229 sentence_builder_spec.rb

Randomized with seed 49229
F.

Failures:

  1) Randomness says world
     Failure/Error: expect(@sentence).to eq("hello world")

       expected: "hello world"
            got: "world"

       (compared using ==)
     # ./sentence_builder_spec.rb:13:in `block (2 levels) in <top (required)>'

Finished in 0.01941 seconds (files took 0.16408 seconds to load)
2 examples, 1 failure

This time the same seed value was specified with the option --order random:49229 to duplicate the previous test run, the error is reproduced, and will be easier to troubleshoot since it can be generated on-demand.

Aliases

These options all execute the same command:

  1. rspec --order random:1234 example_spec.rb
  2. rspec --order rand:1234 example_spec.rb
  3. rspec --seed 1234 example_spec.rb

Closing

When dealing with intermittent errors with randomized specs due to an ordering problem–or even a suspected one–specifying a seed value is the best solution.

RSpec command documentation