RSpec test failures related to ordering may sometimes appear, particularly on larger and older projects.
These intermittent errors can be difficult to trace, and as they may only occur during a long build process after other tasks have succeeded, can impact productivity.
Fortunately RSpec includes an option to help reproduce errors like these, when used in conjunction with random test ordering.
Random test ordering may be specified at the command line when running RSpec, for example:
$ rspec --order rand
There is also a more permanent way to enable this option, by using a preferences file named
Seed values and reproducing a test run
Randomized tests are managed by a test seed, which is displayed in test output such as
Randomized with seed 49229. And this holds the key to reproducing test runs.
For example, here is a test with an ordering problem:
# sentence_builder_spec.rb RSpec.describe "Randomness" do before(:context) do @sentence = "" end it "says hello" do @sentence << "hello " end it "says world" do @sentence << "world" expect(@sentence).to eq("hello world") end end
This test is using state (poorly) to demonstrate an ordering problem–it will pass when the first test is run followed by the second, but when the order is reversed, the spec will fail.
$ rspec --order random sentence_builder_spec.rb Randomized with seed 10450 .. Finished in 0.00652 seconds (files took 0.15641 seconds to load) 2 examples, 0 failures
This run passed, but the next fails:
$ rspec --order random sentence_builder_spec.rb Randomized with seed 49229 F. Failures: 1) Randomness says world Failure/Error: expect(@sentence).to eq("hello world") expected: "hello world" got: "world" (compared using ==) # ./sentence_builder_spec.rb:13:in `block (2 levels) in <top (required)>' Finished in 0.0227 seconds (files took 0.15411 seconds to load) 2 examples, 1 failure
This test will fail about half the time, but what if there were a failure that only occurs very rarely? In that case, one can keep running a (potentially lengthy) test suite repeatedly, or this seed value can be used to re-run the specs in the same order.
$ rspec --order random:49229 sentence_builder_spec.rb Randomized with seed 49229 F. Failures: 1) Randomness says world Failure/Error: expect(@sentence).to eq("hello world") expected: "hello world" got: "world" (compared using ==) # ./sentence_builder_spec.rb:13:in `block (2 levels) in <top (required)>' Finished in 0.01941 seconds (files took 0.16408 seconds to load) 2 examples, 1 failure
This time the same seed value was specified with the option
--order random:49229 to duplicate the previous test run, the error is reproduced, and will be easier to troubleshoot since it can be generated on-demand.
These options all execute the same command:
rspec --order random:1234 example_spec.rb
rspec --order rand:1234 example_spec.rb
rspec --seed 1234 example_spec.rb
When dealing with intermittent errors with randomized specs due to an ordering problem–or even a suspected one–specifying a seed value is the best solution.