After upgrade to Rails 4: Specs fail in combination while passing in isolation

Question

I'm struggling with this for quite a while now: I'm trying to upgrade an app from Rails 3.2 to Rails 4. While on Rails 3.2 all specs are passing, they fail under certain conditions in Rails 4.

Some specs are passing in isolation while failing when run together with other specs.

Example Video

https://www.wingolf.org/ak-internet-files/Spec_Behaviour.mp4 (4 mins)

This example video shows:

1. Running 3 specs using :focus–––green.
2. Running them together with another spec–––two specs passing before now fail.
3. Running the 3 specs, but inserting two empty lines–––one spec fails.
   1. Undo does not help when using guard.
   2. focus/unfocus does not help.
   3. Restarting guard does not help.
   4. Running all specs and then running the 3 specs again does help and make them green again. But adding the other task makes two specs fail, again.

As one can see, some specs are red when run together with other specs. Even entering blank lines can make a difference.

More Observations

• For some specs, passing or failing occurs randomly when run several times.
• The behavior is not specific to one development machine but can be reproduced on travis.
• To delete the database completely between the specs using database_cleaner does not help.
• To Rails.cache.clear between the specs does not help.
• Wrapping each spec in an ActiveRecord::Base.transaction does not help.
• This does occur in Rails 4.0.0 as well as in Rails 4.1.1.
• Using this minimal spec_helper.rb without spring or anything does not help.
• Using guard vs. using bundle exec rspec some_spec.rb:123 directly doesn't make a difference.
• This behavior goes for model specs, thus doesn't have to do anything with parallel database connections for features specs.
• I've already tried to keep as many gems at the same version as in the (green) Rails-3.2 branch, including guard, rspec, factory_girl, etc.–––does not help.

Update: Observations Based on Comments & Answers

• Thanks to engineerDave, I've inserted require 'pry'; binding.pry; into one of the concerning specs. Using the cd and show-source of pry, it was ingeniously easy and fun to narrow down the problem: Apparently, this has_many :through relation does not return objects when run together with other specs, even when called with (true).

  has_many(:groups,
    -> { where('dag_links.descendant_type' => 'User').uniq },
    through: :memberships,
    source: :ancestor, source_type: 'Group'
    )
  

  If I call groups directly, I get an empty result. But if I go through the memberships, the correct groups are returned:

  @user.groups                                #  => []
  @user.groups(true)                          #  => []
  @user.memberships.collect { |m| m.group }   # returns the correct groups
  

  Has Rails changed the has many through behavior in Rails 4 in a way that could be responsible? (Remember: The spec works in isolation.)

Any help, insights and experiences are appreciated. Thanks very much in advance!

Code

• Current master branch on Rails 3.2––all green.
• Rails-4 branch––strange behavior.
• The file/commit seen in the video––strange behavior.
• All specs passing on travis for Rails 3.2.
• Diff of the Gemfile.lock (or use git diff master..sf/rails4-minimal-update Gemfile.lock |grep rspec)

How to Reproduce

This is how one can check if the issue still exists:

Preparation

git clone git@github.com:fiedl/wingolfsplattform.git
cd wingolfsplattform
git checkout sf/rails4-minimal-update
bundle install
# please create `config/database.yml` to your needs.
bundle exec rake db:create db:migrate db:test:prepare

Run the specs

bundle exec rspec ./vendor/engines/your_platform/spec/models/user_group_membership_spec.rb
bundle exec rspec ./vendor/engines/your_platform/spec/models/user_group_membership_spec.rb:213

The problem still exists, if the spec :213 is green in the second call but is red when run together with the other specs in the first call.

Answer 1

Based on that you're using:

• the should syntax
• that you indicate you've upgraded recently (perhaps a bundle update?)
• that your failure messages indicate a NilObject error.

Is something like this perhaps what is causing it?

https://stackoverflow.com/a/16427072/793330

Are you are calling an object in your test which hasn't been instantiated yet?

I think this might be an rspec 3 upgrade issue where should is deprecated.

Have you ruled out an rspec gem upgrade to the new rspec 3 syntax (2.99 or 3.0.0+) as the culprit?

"Remove support for deprecated expect(...).should. (Myron Marston)"

IMO this behavior would not be caused by a Rails 4 update as its centered around your test suite.

Update (with pry debug):

Also you could use the pry gem to get a window into what is going on in your specs.

Essentially you can put a big "stop" sign (similar to a debug break) right before the spec executes to get a handle on the environment at that point.

it {
  require 'pry'; binding.pry
  should == something
}

Although beaware sometimes these pry calls wreck havoc on guard's threading and you have to kill it with CTRL+Z and then kill -9 PID that shows.

Update #2: Looking at updated answer.

You might be running up against FactoryGirl issues based on your has_many issue

You may need to trigger a before action in your Factory to pre-populate the associated record. Although this could get messy, i.e. here be monsters, you can trigger after and before callbacks in your factory that will bring these objects into being.

after(:create) do |instance|
    do stuff here
end