why you should regenerate your spec_helper - ivo's awfully random tech blog

If your project is like most ruby projects, your spec_helper.rb looks something like this:

# encoding: utf-8

if RUBY_ENGINE == 'rbx'
  require 'codeclimate-test-reporter'
  CodeClimate::TestReporter.start
end

require 'dry/container'
require 'dry/container/stub'

Dir[Pathname(__FILE__).dirname.join('support/**/*.rb').to_s].each do |file|
  require file
end

Pretty straightforward, and also missing on a lot of cool rspec features.

Modern rspec versions (as I’m writing this, the latest version is 3.5.4) have a lot of cool features that are not enabled by default, so as not to break backwards compatibility. What this means is that in practice, unless you turn them on, you’re missing out on a lot of goodies and cool features that could help you spot potential issues in your code and specs.

The quickest way to enable them is to ask rspec to generate a pair of new spec_helper.rb and .rspec files for your project.

generating a new `.rspec` and `.spec_helper`

Make sure you’re using the latest rspec:

$ bundle update rspec
# ... bundler output ...
Bundle updated!
$ rspec -v
3.5.4

Save your existing files (if any) by renaming them

$ mv .rspec old_dot_rspec
$ mv spec/spec_helper.rb spec/old_spec_helper.rb

…then ask rspec to generate you a new set of files.

$ rspec --init
  create   .rspec
  create   spec/spec_helper.rb

Let’s take a look at these files to see what they get you.

The newly-generated `.rspec`

--color
--require spec_helper

The .rspec file specifies default flags that get passed to the rspec command when you run your tests. So if you want one of the options you see listed on rspec --help to apply by default, you can add them here.

The two flags that are added by default are --color, that enables coloring in the rspec output, and --require spec_helper, that automatically requires the spec_helper file whenever you run rspec, rather than you have to manually add require 'spec_helper' at the top of your specs.

The latter option is more important than it seems. The spec_helper.rb we just generated (and that I’ll go into detail below) sets a number of options, but you only get them if, obviously, the spec_helper.rb is loaded by your specs. Forgetting the require may mean that when a spec is executed in isolation it may give a different result than when you execute your whole spec suite, which is troubling.

So rather than worrying about the next time you’ll forget to include the magic require, just let the .rspec always do it for you, and never worry about this again!

The newly-generated `spec/spec_helper.rb`

So we arrive at last to the meaty part. So what are these newfangled features that we get by regenerating the spec_helper, then?

The `include_chain_clauses_in_custom_matcher_descriptions` option

config.expect_with :rspec do |expectations|
  # This option will default to `true` in RSpec 4. It
  # makes the `description` and `failure_message` of
  # custom matchers include text for helper methods
  # defined using `chain`, e.g.:
  #   be_bigger_than(2).and_smaller_than(4).description
  #     # => "be bigger than 2 and smaller than 4"
  #   ...rather than:
  #     # => "be bigger than 2"
  expectations
    .include_chain_clauses_in_custom_matcher_descriptions =
      true
end

The include_chain_clauses_in_custom_matcher_descriptions option gets you improved descriptions and failure messages when you’re using multiple custom matchers, and you can find a handy example in rspec’s documentation.

Of course, this one only applies if you’re using your own matchers, so let’s move on to bigger fish.

The `verify_partial_doubles` option

config.mock_with :rspec do |mocks|
  # Prevents you from mocking or stubbing a method that does
  # not exist on a real object. This is generally
  # recommended, and will default to `true` in RSpec 4.
  mocks.verify_partial_doubles = true
end

This option is truly awesome. When you enable it, rspec also checks during the test suite execution that any mocks you setup are consistent with the original class being mocked.

What does this mean in practice? Well, consider that you’re mocking the :delete operation on the File class because you don’t want your test suite to be deleting real files during its execution, but you end up mistyping :delete as :delte.

allow(File).to receive(:delte)

If you then exercise your code, your specs may still pass, but instead of mocking the delete operation, you may be doing real deletes!

When you enable this option you’ll instead get as an output of your test run:

 Failure/Error: allow(File).to receive(:delte)
   File does not implement: delte
 # ./spec/foo_spec.rb:4:in `block (2 levels) in <top (required)>'

Another big benefit of this option is when mocking external libraries. Whenever you upgrade to a new major version a library, you may be led to believe that your code still works if you ran your specs and you’re still all green. But, however, if you’re mocking methods that no longer exist in the new library version, you can be in for a big surprise when you deploy your changes.

Using the verified doubles, your specs will fail right away if you’re using mocked objects wrongly. So upgrading major versions just got easier and your test suite leaves you more confident!

Note that these verified doubles check not only that a method exists, but also the number of arguments, and whenever keyword arguments are in use, if the proper required and optional keyword arguments are being used. So a whole lot of awesome 🎉!

The `shared_context_metadata_behavior` option

# This option will default to `:apply_to_host_groups`
# in RSpec 4 (and will have no way to turn it off --
# the option exists only for backwards compatibility
# in RSpec 3). It causes shared context metadata to
# be inherited by the metadata hash of host groups
# and examples, rather than triggering implicit
# auto-inclusion in groups with matching metadata.
config
  .shared_context_metadata_behavior = :apply_to_host_groups

This is also a very minor change to behavior of shared contexts, but the new behavior will be the default and only option in rspec 4, so enabling this today helps makes your specs forward-compatible with the next major rspec release.

For more details on this option see the rspec 3.5 release announcement and the api documentation.

Suggested settings in `spec/spec_helper.rb`

If you’re following along an automatically-generated spec_helper.rb as I suggested above, you’ll now get to a section of suggested options that are by default commented out.

# The settings below are suggested to provide a good
# initial experience with RSpec, but feel free to customize
# to your heart's content.

I believe most of them are rather useful, so let’s start turning them on!

The `filter_run_when_matching` option

# This allows you to limit a spec run to individual
# examples or groups you care about by tagging them with
# `:focus` metadata. When nothing is tagged with
# `:focus`, all examples get run. RSpec also provides
# aliases for `it`, `describe`, and `context` that include
# `:focus` metadata: `fit`, `fdescribe` and `fcontext`,
# respectively.
config.filter_run_when_matching :focus

The filter_run_when_matching option allows you to easily restrict your test suite execution to a specific subset of specs. This is helpful when you’re working on a number of classes and don’t want any other specs to run for now.

To trigger this behavior, just add the :focus option after describe, context or it to any specs or groups of specs you want to select:

RSpec.describe 'Foo', :focus do
  # ...
end

RSpec.describe 'Bar' do
  context 'when made out of iron', :focus do
    # ...
  end
end

RSpec.describe 'Baz' do
  it 'does stuff', :focus do
    # ...
  end
end

In the above example, all specs tagged with :focus will be executed, and any other specs in the test suite will be left out.

As a shorthand, you can also prepend an f to describe (fdescribe), context (fcontext), or it (fit) to get the same effect:

RSpec.fdescribe 'Foo' do
  # ...
end

RSpec.describe 'Bar' do
  fcontext 'when made out of iron' do
    # ...
  end
end

RSpec.describe 'Baz' do
  fit 'does stuff' do
    # ...
  end
end

Personally, I favor adding :focus as it’s harder to accidentally forget one in a commit ;)

The `example_status_persistence_file_path` option

# Allows RSpec to persist some state between runs in order
# to support the `--only-failures` and `--next-failure` CLI
# options. We recommend you configure your source control
# system to ignore this file.
config.example_status_persistence_file_path =
  'spec/examples.txt'

The example_status_persistence_file_path makes rspec remember which specs failed from your test suite.

This allows you to focus on fixing each failing spec one-by-one by running rspec --next-failure, or to just limit the test suite execution to the failing specs with rspec --only-failures.

As an example, consider the following execution:

$ rspec

Foo
  should not fail (FAILED - 1)
  when you really want to Foo
    really should not fail (FAILED - 2)

Bar
  works nicely

Failures:

  1) Foo should not fail
     Failure/Error: fail
     RuntimeError:
     # ./spec/foo_spec.rb:3:in `block (2 levels) in <top (required)>'

  2) Foo when you really want to Foo really should not fail
     Failure/Error: fail
     RuntimeError:
     # ./spec/foo_spec.rb:8:in `block (3 levels) in <top (required)>'

Finished in 0.00085 seconds (files took 0.07439 seconds to load)
3 examples, 2 failures

Failed examples:

rspec ./spec/foo_spec.rb:2 # Foo should not fail
rspec ./spec/foo_spec.rb:7 # Foo when you really want to Foo really should not fail

In this case, I ran the test suite, and it resulted in 2 failed specs.

Now, if I run again with rspec --only-failures, rspec will only retry the failed specs, in this case, those for Foo, and will not run the spec for Bar, which was working correctly.

$ rspec --only-failures
Run options: include {:last_run_status=>"failed"}

Foo
  should not fail (FAILED - 1)
  when you really want to Foo
    really should not fail (FAILED - 2)

Failures:

  1) Foo should not fail
     Failure/Error: fail
     RuntimeError:
     # ./spec/foo_spec.rb:3:in `block (2 levels) in <top (required)>'

  2) Foo when you really want to Foo really should not fail
     Failure/Error: fail
     RuntimeError:
     # ./spec/foo_spec.rb:8:in `block (3 levels) in <top (required)>'

Finished in 0.00074 seconds (files took 0.07481 seconds to load)
2 examples, 2 failures

Failed examples:

rspec ./spec/foo_spec.rb:2 # Foo should not fail
rspec ./spec/foo_spec.rb:7 # Foo when you really want to Foo really should not fail

Finally, if you really want to focus and tackle failing specs one-by-one, you can use rspec --next-failure and only the first of the failing specs will be executed.

$ rspec --next-failure
Run options: include {:last_run_status=>"failed"}

Foo
  should not fail (FAILED - 1)

Failures:

  1) Foo should not fail
     Failure/Error: fail
     RuntimeError:
     # ./spec/foo_spec.rb:3:in `block (2 levels) in <top (required)>'

Finished in 0.00055 seconds (files took 0.07577 seconds to load)
1 example, 1 failure

Failed examples:

rspec ./spec/foo_spec.rb:2 # Foo should not fail

This option is somewhat like :focus above, but rspec automatically focuses you on failing specs.

The `disable_monkey_patching!` option

# Limits the available syntax to the non-monkey patched
# syntax that is recommended. For more details, see:
#   - http://rspec.info/blog/2012/06/rspecs-new-expectation-syntax/
#   - http://www.teaisaweso.me/blog/2013/05/27/rspecs-new-message-expectation-syntax/
#   - http://rspec.info/blog/2014/05/notable-changes-in-rspec-3/#zero-monkey-patching-mode
config.disable_monkey_patching!

Monkey-patching, e.g. the practice of extending or changing system or external library code during runtime, is both a blessing and a curse in Ruby.

Earlier versions of rspec employed a number of monkey patches to provide the DSL syntax used in tests. Unfortunately, this collided with ruby’s delegation mechanisms and thus a few releases ago the rspec developers came up with a new matching syntax (the expect-based one) that avoids the need for monkey patching.

For compatibility reasons, some of the monkey patches are still enabled by default, but their use is no longer recommended, and the disable_monkey_patching! option disables them for good.

In most projects I’ve found that the only monkey patches still in use are top-level describe and context blocks:

describe 'Foo' do
  context 'when fooing' do
    # ...
  end
end

context 'when the current planet is the earth' do
  describe 'clouds' do
    # ...
  end
end

Running these specs with disable_monkey_patching! will yield something like:

$ rspec
spec/foo_spec.rb:1:in `<top (required)>': undefined method `describe' for main:Object (NoMethodError)

To fix this, just prefix any top-level describe and context with the RSpec class:

RSpec.describe 'Foo' do
  context 'when fooing' do
    # ...
  end
end

RSpec.context 'when the current planet is the earth' do
  describe 'clouds' do
    # ...
  end
end

In most cases adding RSpec. to top-level blocks should be enough. Notice that contexts and describes nested inside the top-level blocks do not need to be modified.

If, however, some of your specs still make use of the old should matching syntax, consider using the transpec gem to automatically update your code to the newer expect syntax—no manual work needed!

The `warnings` option

# This setting enables warnings. It's recommended, but in
# some cases may be too noisy due to issues in dependencies.
config.warnings = true

This is one of the options I recommend the most. By default, even when running rspec, ruby does not emit warnings for valid, but possibly buggy code, such as:

Require loops (file foo.rb requires bar.rb, which requires foo.rb)
Unused variables
Using the value of an uninitialized variable
Redefining a method in the same scope as the original definition
Obsolete library usage
…

This option enables these warnings while running your test suite. You may find interesting things, not only on your suite, but on your own dependences—many of which also have warnings disabled on their own test suites and thus never noticed that these potential issues were there (see also below for a discussion on this).

Enable detailed rspec output when running only for a single file

# Many RSpec users commonly either run the entire suite or
# an individual file, and it's useful to allow more verbose
# output when running an individual spec file.
if config.files_to_run.one?
  # Use the documentation formatter for detailed output,
  # unless a formatter has already been configured
  # (e.g. via a command-line flag).
  config.default_formatter = 'doc'
end

For larger test suites, it’s usual to use the default rspec output where you print a dot for a passed spec, and an F for a failed one.

$ rspec
................

Finished in 0.00192 seconds (files took 0.07474 seconds to load)
16 examples, 0 failures

This tweak changes rspec’s behavior to output a detailed description of the tests when you run it with a single file (or only a single file is in :focus):

$ rspec spec/foo_spec.rb

Foo
  should not fail
  when you really want to Foo
    really should not fail

Finished in 0.0006 seconds (files took 0.07574 seconds to load)
2 examples, 0 failures

Simple, but useful :)

The `profile_examples` option

# Print the 10 slowest examples and example groups at the
# end of the spec run, to help surface which specs are
# running particularly slow.
config.profile_examples = 10

Remember the golden days of your project, when the entire test suite ran in under 10 seconds? The profile_examples option lists the slowest-running specs in your suite, which really highlights where time is being spent—perhaps the sleep you just added isn’t that innocent after all.

Personally, I find that 10 examples is a bit too noisy, and set it to either 3 or 5, but I find this option really useful to spot slow specs that are candidates for simplification, allowing your test suite to be blazingly-fast again.

$ rspec

Foo
  should not fail
  when you really want to Foo
    really should not fail

Bar
  works nicely

Top 3 slowest examples (0.00019 seconds, 20.5% of total time):
  Foo should not fail
    0.00009 seconds ./spec/foo_spec.rb:2
  Bar works nicely
    0.00005 seconds ./spec/foo_spec.rb:14
  Foo when you really want to Foo really should not fail
    0.00005 seconds ./spec/foo_spec.rb:7

Top 2 slowest example groups:
  Foo
    0.00023 seconds average (0.00045 seconds / 2 examples) ./spec/foo_spec.rb:1
  Bar
    0.00015 seconds average (0.00015 seconds / 1 example) ./spec/foo_spec.rb:13

Finished in 0.00092 seconds (files took 0.07551 seconds to load)
3 examples, 0 failures

Random spec ordering

# Run specs in random order to surface order dependencies.
# If you find an order dependency and want to debug it, you
# can fix the order by providing the seed, which is printed
# after each run.
#     --seed 1234
config.order = :random

# Seed global randomization in this process using the
# `--seed` CLI option.
# Setting this allows you to use `--seed` to
# deterministically reproduce test failures related to
# randomization by passing the same `--seed` value as the
# one that triggered the failure.
Kernel.srand config.seed

By default, rspec runs your specs in the order that they are defined (and your specs in alphabetic order). This helps during development—and I find myself temporarily enabling in-order execution when writing specs with rspec --order defined often—but may be hiding bugs.

How so? Consider a very simple example from my did_you_know.ruby? presentation:

class Bar
  def self.hello
    message =
      ENV['MESSAGE'] || 'hello'
    puts message
  end
end

RSpec.describe Bar do
  describe '#hello' do
    it 'prints hello' do
      expect(Bar).to receive(:puts).with('hello')
      Bar.hello
    end

    it 'is set prints the message in ENV['MESSAGE']' do
      ENV['MESSAGE'] = 'hello world'
      expect(Bar).to receive(:puts).with('hello world')
      Bar.hello
    end
  end
end

These specs pass when executed in order, but fail whenever the spec that sets the ENV['MESSAGE'] runs before the one that tests that it prints hello.

Now this example is pretty trivial, but in a big test suite it’s possible that quite accidentally one spec is leaving behind state that makes other tests either pass when they shouldn’t, or fail mysteriously.

Thus, the rspec order option enables random spec ordering, making rspec pick a randomized order for executing your specs, making sure that sooner or later code or specs that fail sporadically is found.

Whenever rspec runs with a randomized test ordering it prints a random seed—a value which you can use to repeat that exact ordering, when something files.

To run your specs with that seed, just run rspec --seed seed_value. And if you find yourself scratching your head finding which specs are interfering with each other, you can see an example of using rspec’s --bisect option on my presentation.

My test suite was green and now it fails! :(

Some of the options I suggest above may uncover issues in your code (in the case of verified partial doubles or random spec ordering) or on your test suite when you still used outdated rspec syntax (in the case of the disable monkey patching option).

If you find yourself in this situation, I recommend disabling the options again and enabling them one-by-one until you see the culprit. And when you do, I recommend looking into it, as you may have just uncovered a bug somewhere in your code.

Easiest open-source contribution ever!

As I’ve alluded to above, many existing gems also have outdated rspec configurations and cause lots of warnings when executed.

This means that contributing updated rspec configurations and fixes for warnings is one of the easiest ways you can start contributing to a project.

So that’s exactly what I’ve been doing:

So if you’ve been eyeing a project, here’s your chance! And if you just enabled warnings in your own code, and see that everyone else’s is causing warnings, help fix them! :)

Thanks for reading thus far!

generating a new .rspec and .spec_helper

The newly-generated .rspec

The newly-generated spec/spec_helper.rb

The include_chain_clauses_in_custom_matcher_descriptions option

The verify_partial_doubles option

The shared_context_metadata_behavior option

Suggested settings in spec/spec_helper.rb

The filter_run_when_matching option

The example_status_persistence_file_path option

The disable_monkey_patching! option

The warnings option