2

I'm using Gitlab-CI to build my application and have a weird error on some runners (Docker executors) when my service is starting. Indeed, on some runners, the service crashes at startup with a ClassNotFoundException, and works perfectly on others runners.

This is the part of my .gitlab-ci.yml which is relevant :

mvn-build:
  services:
    - name: my-repository.com:5002/networklocation/configuration-api:1.0.23
      alias: configuration
      variables:
        CNB_PLATFORM_API: ""
        SPRING_PROFILES_ACTIVE: docker

This configuration-api application is a Java Spring Boot application built with Paketo Cloud Native Buildpacks.

Here are the logs when the job is scheduled on a non-working runner :

[service:configuration-api] 2023-02-23T08:30:33.319822839Z Error: Could not find or load main class org.springframework.boot.loader.JarLauncher
[service:configuration-api] 2023-02-23T08:30:33.320160182Z Caused by: java.lang.ClassNotFoundException: org.springframework.boot.loader.JarLauncher

The only workaround I have found is to connect to the runner and do a docker volume prune. After that operation, the service is working correctly for some time, until a new build breaks it on some runners.

All runners are configured as docker executors and are in the same version (currently 15.8.1)

One hypothesis I have is that Cloud Native Buildpacks creates image without timestamp, may this cause some issues ? Is there anything else that may cause this issue ?

Xavier FRANCOIS
  • 652
  • 4
  • 15
  • If a prune fix the problem, could it be your runner host runs out of disk space after a while and thus not pull the complete image/all layers for this service ? – Tensibai Feb 27 '23 at 10:37
  • No, I checked and it seems that the runners are not full at all – Xavier FRANCOIS Feb 27 '23 at 14:49
  • One way to ensure the image is ok could be to use the image tag (on top of the version) as explained [here](https://docs.gitlab.com/ee/ci/docker/using_docker_images.html#use-checksum-to-keep-your-image-secure). Something like `my-repository.com:5002/networklocation/configuration-api:1.0.23@sha256:......` – Tensibai Feb 27 '23 at 16:22
  • Yes, I tried that too to ensure it's the correct image, and it is. – Xavier FRANCOIS Feb 28 '23 at 07:55
  • Have you tried disabling cache volumes in your runner configuration? – sytech Mar 02 '23 at 07:17
  • No, because these runners are shared runners and belong to my company. We made a debuging session with the shared runners owners, that's why I know disk is not full and that pruning volume "solve" the problem, but I can't ask them to change the runner configuration just for my project – Xavier FRANCOIS Mar 02 '23 at 08:42

2 Answers2

1

I finally found the root cause of the issue, unfortunately I didn't provide enough details on the questions for someone to resolve the issue.

The problem was that my configuration-api image, which is used as a Gitlab-CI service was build with CNB buildpacks by calling the lifecycle endpoint. The final Docker image was created with a directory structure of /builds/ because CNB_APP_DIR was set to this value. In fact, the underlying issue is that it's not possible to decorelate source and target folders with CNB lifecycle.

The unintended consequence was that by using this image as a Gitlab-CI service, it conflicted with the volume created by Gitlab-CI.

I fix this issue by recreating my configuration-api with a CNB_APP_DIR of /workspace. I did that by copying my $CI_PROJECT_DIR to the workspace directory.

Xavier FRANCOIS
  • 652
  • 4
  • 15
0

Build it with maven debug enabled and simply see the difference. There is some difference in java version / maven version / dependency resolution (for project or for plugins) that you haven't found.

As a debug step dump dependency tree before the real build (mvn -X dependency:tree) and store it job artifact and compare between working and nonworking builds.

Piotr
  • 317
  • 1
  • 13