Gitlab CI: bundle could not find in locally installed gems

Question

I have quite a simple GitLab CI-setup:

1. Once per branch, a job runs which creates the backoffice image for my tests. The idea is to do the repetitive stuff once by creating a custom docker-image to save resources. That contains installing libraries with apt and gem and bundle.
2. With that backoffice image I execute all the tests. That runs very quickly because all the libraries are already installed in my custom image. No downloads. Nice.

It was working perfectly when I built it.

Now, a few weeks later, I come back to it and it is broken. The tests complain about missing gems:

/usr/local/bundle/gems/bundler-2.3.23/lib/bundler/definition.rb:508:in `materialize': 
Could not find
 passenger-6.0.14,
 sinatra-2.2.1,
 sinatra-contrib-2.2.1,
 haml-5.2.2,
 mustermann-1.1.2,
 rack-protection-2.2.1,
 tilt-2.0.10 
in locally installed gems (Bundler::GemNotFound)

While my custom backoffice image shows that everything was installed correctly:

+ bundle install
Fetching gem metadata from https://rubygems.org/............
Resolving dependencies...
[... stuff removed]
Installing passenger 6.0.15 with native extensions
Fetching rack-protection 3.0.2
Installing rack-protection 3.0.2
Fetching sinatra 3.0.2
Installing sinatra 3.0.2
Fetching sinatra-contrib 3.0.2
Installing sinatra-contrib 3.0.2
Bundle complete! 6 Gemfile dependencies, 16 gems now installed.

What broke?

Answer 1

tl;dr:

Installing and running must use the same Gemfile.lock - or none at all.

The Reason

The eagle-eyed among you might have found the problem right away. The versions are different:

passenger-6.0.15 installed vs. passenger-6.0.14 required.

The source of the problem is the Gemfile.lock as suggested here in a different context.

When creating the backoffice image I copied the Gemfile, but not the Gemfile.lock. Without the lock-file, bundle will install the latest of whatever matches the rather loose instructions in the Gemfile.

But the execution-environment for testing had a Gemfile.lock in place, which requests specific versions of all installed gems.

The Background

When the setup was new, the latest-and-greatest was installed. And from that bundle would create the lock-file. Thus even without the stringent instructions of the lock-file, the simple Gemfile would come to the same setup - for now.

Weeks later, there were newer versions of the gems and latest-and-greatest versions diverged from the versions locked-in with the included Gemfile.lock. That caused the CI-pipeline to fail with this rather generic error-message. Cause and effect being so far apart caused me a lot of headache, which I want to save you by writing this specific article.