1

I haven't changed a bit in the cmake configuration of the project. The only thing that has changed is the "in-house" built release of a new version of the opensource library (OpenCV). What is strange is that some of the libraries from the locally exracted folder are linked and found and others are not.

  • When cmake is called OPENCV_LIB variable is created with all of the libraries needed - this variable is used for linking
  • All such libraries are present in the folder
  • Whole library was built in one step (all for x64 configuration)
  • The project is correctly linked with the libraries from the library

Yet, when the ldd is called on the final executable, the result is quite confusing.

libopencv_core.so.406 => /LOCAL_FOLDER/OpenCV/lib/x64/libopencv_core.so.406 (0x00007f974e2df000)    
libopencv_calib3d.so.406 => /LOCAL_FOLDER/OpenCV/lib/x64/libopencv_calib3d.so.406 (0x00007f974d852000)    
libopencv_imgcodecs.so.406 => /LOCAL_FOLDER/OpenCV/lib/x64/libopencv_imgcodecs.so.406 (0x00007f974d795000)    
libopencv_imgproc.so.406 => /LOCAL_FOLDER/OpenCV/lib/x64/libopencv_imgproc.so.406 (0x00007f974b79c000)    
libopencv_features2d.so.406 => not found                                                                                                                                           
libopencv_flann.so.406 => not found

All of the libraries are present in the OPENCV_LIB variable which is used for linking and all of them are present in the LOCAL_FOLDER. When ldd is called on the libraries which results in not found in the LOCAL_FOLDER some of their dependencies are not found as well. How come, some libraries are found and some are not, when they are from in the same folder?

The process of OpenCV library inclusion to the project:

Script creates the dependencies.cmake file which is included with include(dependencies.cmake). The file contains amongst others, the following lines:

set(OPENCV_INCLUDE_DIR "/LOCAL_FOLDER/OpenCV/include") 
include_directories(${OPENCV_INCLUDE_DIR})
link_directories("/LOCAL_FOLDER/OpenCV/lib/x64/")
set(OPENCV_LIB ${OPENCV_LIB} *.so) # * as substitution for all the libs that are "set" this way.

Then in a CMakeLists of the sub project it is used like so

target_link_libraries(${PROJECT_NAME} ${OPENCV_LIB} "-Wl,--disable-new-dtags")

ldd -v result for one of the libraries

/LOCAL_FOLDER/OpenCV/lib/x64/libopencv_calib3d.so.406:
                libgcc_s.so.1 (GCC_3.0) => /lib/x86_64-linux-gnu/libgcc_s.so.1
                libgcc_s.so.1 (GCC_4.0.0) => /lib/x86_64-linux-gnu/libgcc_s.so.1
                libm.so.6 (GLIBC_2.29) => /lib/x86_64-linux-gnu/libm.so.6
                libm.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libm.so.6
                libstdc++.so.6 (GLIBCXX_3.4.20) => /lib/x86_64-linux-gnu/libstdc++.so.6
                libstdc++.so.6 (CXXABI_1.3.8) => /lib/x86_64-linux-gnu/libstdc++.so.6
                libstdc++.so.6 (GLIBCXX_3.4.19) => /lib/x86_64-linux-gnu/libstdc++.so.6
                libstdc++.so.6 (GLIBCXX_3.4.9) => /lib/x86_64-linux-gnu/libstdc++.so.6
                libstdc++.so.6 (GLIBCXX_3.4.29) => /lib/x86_64-linux-gnu/libstdc++.so.6
                libstdc++.so.6 (GLIBCXX_3.4.26) => /lib/x86_64-linux-gnu/libstdc++.so.6
                libstdc++.so.6 (GLIBCXX_3.4.11) => /lib/x86_64-linux-gnu/libstdc++.so.6
                libstdc++.so.6 (CXXABI_1.3) => /lib/x86_64-linux-gnu/libstdc++.so.6
                libstdc++.so.6 (GLIBCXX_3.4.14) => /lib/x86_64-linux-gnu/libstdc++.so.6
                libstdc++.so.6 (GLIBCXX_3.4.21) => /lib/x86_64-linux-gnu/libstdc++.so.6
                libstdc++.so.6 (GLIBCXX_3.4.15) => /lib/x86_64-linux-gnu/libstdc++.so.6
                libstdc++.so.6 (GLIBCXX_3.4) => /lib/x86_64-linux-gnu/libstdc++.so.6
                libc.so.6 (GLIBC_2.4) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.14) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.3.4) => /lib/x86_64-linux-gnu/libc.so.6

The RPATH is set to origin like so

set_target_properties(${PROJECT_NAME} PROPERTIES INSTALL_RPATH "$ORIGIN")

I have also tried to set it as suggested in the post here Nothing has solved the problem.

Setting it explicitly to LOCAL_FOLDER does nothing.

When I download the older "in-house" built version of the same library and call ldd on some libraries that shows not found in the new one, it ends up in the same situation. Linking is always ok. Basically with older version of the library, the ldd shows the same for the libraries, but the resulting binary is ok (all libs are located properly) and can be run without issues. The only difference between the old library and the new one is that the old was built with GCC 8 and under Ubuntu 18. The new one with GCC 11 and under Ubuntu 20.

Croolman
  • 1,103
  • 13
  • 40
  • You can add your local path to `/etc/ld.so.conf` and then call `sudo ldconfig -v` and see if all libraries have been found. – pptaszni Jul 18 '22 at 08:27
  • That works. That is good, but I consider this to be a "hacky" way around this situation. Not sufficient. – Croolman Jul 18 '22 at 08:38
  • Could be an rpath issue: it could be set for the first four libraries, but not for the last two, during linkage. – 9769953 Jul 18 '22 at 08:39
  • How do I fix that? – Croolman Jul 18 '22 at 09:38
  • 4
    @pptaszni don't **ever** do that on a production system. It opens a big security hole into the system. Consider using the binaries `DT_RPATH` or the `LD_LIBRARY_PATH` environment variable to set the loader search paths temporarly. On overview can be found on the [`ld.so` manpage](https://man7.org/linux/man-pages/man8/ld.so.8.html) – Jakob Stark Jul 18 '22 at 11:44
  • @Croolman can you show how you include and use the opencv package in your cmake file? Usually cmake will do all the correct things for you if you include the package correctly... – Jakob Stark Jul 18 '22 at 11:52
  • I have added the process description. I hope that is sufficient amount of info. – Croolman Jul 18 '22 at 12:20
  • @Croolman Well there's two things that come to my mind: Do you need the `-Wl,--disable-new-dtags` option, and if yes for what exactly. I ask because it messes with the runtime path settings of your linker, and thats where you got problems... Second, where is `OPENCV_LIB` set for the first time? Did you use `find_package()`? If yes why do you need to modify it? – Jakob Stark Jul 18 '22 at 14:15
  • Also could you include the output of `ldd -v` (the verbose output of `ldd`). It may be, that the runtime path of your binary is set correctly, but that the runtime path of the libraries is not set correctly for recursive dependencies. – Jakob Stark Jul 18 '22 at 14:21
  • `link_directories("/LOCAL_FOLDER/OpenCV/lib/x64/")` tells the compiler where to look for libraries specified with a lib name only. Afaik it does not result in any modification of the runtime search path. You could set the [`BUILD_RPATH`](https://cmake.org/cmake/help/latest/prop_tgt/BUILD_RPATH.html) and/or [`INSTALL_RPATH`](https://cmake.org/cmake/help/latest/prop_tgt/INSTALL_RPATH.html) target properties to add info about where to find the libs at runtime. – fabian Jul 18 '22 at 17:14
  • 1
    Does this answer your question? [How to set RPATH in CMAKE?](https://stackoverflow.com/questions/43551483/how-to-set-rpath-in-cmake) – raxonpc Jul 18 '22 at 21:16
  • @raxonpc nope, I do not want to install the library. The goal is to build a project which dependencies are in locally stored apt repository. So the deb package is downloaded and extracted locally, then used for build and tests. – Croolman Jul 19 '22 at 05:03
  • @JakobStark The `-Wl,--disable-new-dtags` were added at some point due to some issues with the resulting library not finding the right libraries, if I remmeber correctly. I wasnt the one setting them, so cant really reason about it. The `OPENCV_LIB` is not set anywhere. Since the `deb` package is extracted locally (in the `LOCAL_FOLDER`), `find_package` is not used. I have added the result of `ldd -v` for one of the libs since it is huge. – Croolman Jul 19 '22 at 05:19
  • @fabian There is the `INSTALL_RPATH` property set like so `set_target_properties(${PROJECT_NAME} PROPERTIES INSTALL_RPATH "$ORIGIN")`. – Croolman Jul 19 '22 at 05:23
  • @JakobStark The structure of our `cmake` files is a mess. We use properties and set properties which are against best practises. Whne I removed the `-Wl,--disable-new-dtags` flags, even more references were lost (only to the libs in `LOCAL_FOLDER`) – Croolman Jul 19 '22 at 05:40
  • I might have found the issue. The old lib uses `rpath`, new version uses `runpath` and all of the libs using the new lib are build using `rpath`. The theory is, that if I force the new lib to use `rpath` instead of `runpath` it might work. – Croolman Jul 28 '22 at 11:28

2 Answers2

1

You have specified the RPATH for your target. So its direct dependencies are searched in the program's directory. I think that your program directly depends only on libopencv_core.so.406, libopencv_calib3d.so.406, libopencv_imgcodecs.so.406 and libopencv_imgproc.so.406. The other two libraries are indirect dependecies (i.e. they are direct dependencies of one of the former libraries). So your programs RPATH is not applied when these indirect depenendcies are searched.

I would recommend you to run your program like that.

$ LD_LIBRARY_PATH=`pwd` ./your_program_binary_name

Upd: Also you have specified all of the libraries in OPENCV_LIB variable, the linker throws some of them out because your program doesn't use their symbols directly.

You can check my answer by adding some explicit calls to the functions which are defined in libopencv_features2d.so.406 and libopencv_flann.so.406 and then running ldd. If I'm right then all the libs would be resolved.

The linker optimizes library list because CMake call ld with -Wl,--as-needed parameters. You could also figure out how to turn off such behavior. I'm not a cmake guru to help you in that way.

Max Dmitrichenko
  • 330
  • 2
  • 8
  • You are right that these are indirect references which can be proven by using functions from these libs. But those have been there even for the old version of the OCV lib and everything was fine. The `--as-needed` flag has been enabled by default since Ubuntu 16, so I don't think that is the problem - disabling it is for sure the solution, but not a desirable one. The asumption is clear, I have only updated the version of the lib, nothing else has changed, so it should be working with any changes. Question is why? – Croolman Jul 22 '22 at 06:04
  • I mean - it might be as well solved by amending the `LD_LIBRARY_PATH`, but then again this wasn't needed before, why now it is? – Croolman Jul 22 '22 at 06:05
  • You have a perfect explanation of current behavior. Why it is was fine with the old version of the OCV libraries set is a question to the way they were built. May be they were linked with rpath though it is a bad style for libraries. Have you checked the old libraries for the RPATH with the command `objdump -x |grep RPATH` – Max Dmitrichenko Jul 22 '22 at 06:32
  • The new one does not have the `RPATH` set, the old one has, but only for `cuda` common paths, which are redundant. `RPATH /usr/local/cuda-10.1/lib64:/usr/local/cuda/lib64` – Croolman Jul 22 '22 at 08:21
  • Have you tried to put the old libraries back into `/LOCAL_FOLDER/OpenCV/lib/x64/`, relink your program with them if they have different suffix and see what `ldd` shows? – Max Dmitrichenko Jul 22 '22 at 12:43
  • It works with the old ones, but that was expected. It might be as well because between the versions of the lirbrary, some functions might have moved somewhere else transforming the direct dependencies into indirect. – Croolman Jul 23 '22 at 07:25
  • Well, you can quickly check the hypothesis of function migration. Take the old set of libraries. Delete all the libraries except for `libopencv_features2d.so.406` and `libopencv_flann.so.406`. Then run `ldd` on your program's binary. If the dependency is indirect than you won't see any references to these two libraries either resolved or not found. If the dependency is direct then you will see that linker have successfuly resolved them. – Max Dmitrichenko Jul 23 '22 at 22:01
  • Clever, the refs are not needed directly as well. So it means the old library itself is built "differently" (which was the clue all the time). Now it remains t ofind out what went wrong when building the new version of the lib. – Croolman Jul 25 '22 at 04:43
  • Sorry, but this is hardly possible without having sources of old and current version. At least we need compiler and linker flags of both – Max Dmitrichenko Jul 26 '22 at 11:12
  • It is impossible for me to get those, but since you have elaborated the most (I still havent succeeded, though), if noone will post an answer in the next hours, I will award you with the bounty. – Croolman Jul 27 '22 at 13:19
  • Oh! Thas wasn't a challange for the bounty for me. I've just had the same issue quite a time ago and wanted to share my finding with you. But thanks anyway ) – Max Dmitrichenko Jul 28 '22 at 12:29
  • It would have expired, and I cant get it back. So better to give than to lose completely – Croolman Jul 28 '22 at 12:36
1

The issue was that the new version of dependent library has the RUNPATH defined, but the other project/libraries using this library are using RPATH. When the RUNPATH is defined it ignores any defined RPATH, therefore transitive dependencies do ignore the RPATH. The solution was to remove the RUNPATH from the new library and add RPATH. The RPATH has been deprecated so if anyone facing this can use RUNPATH instead of RPATH they should do so.

The change from RUNPATH to RPATH was done with help of this answer.

Croolman
  • 1,103
  • 13
  • 40
  • Hm... but you've told in my answer that the old libraries do have rpath which points to CUDA directories. How this could help linker to libraries in local directory even in the absense of RUNPATH? – Max Dmitrichenko Jul 28 '22 at 12:26
  • The old ones have rpath which is why they worked. The new version has runpath defined instead of rpath. That's what broke the whole project. – Croolman Jul 28 '22 at 12:28
  • So, now you have your program binary which has rpath which points to the ORIGIN, and a local set of libraries which have rpath pointing to somewhere else, and linker still finds the indirect dependencies of your program binary in local dir? – Max Dmitrichenko Jul 28 '22 at 12:32
  • Not really, it points to origin, but the libraries from `LOCAL_FOLDER` are in `rpath` since cmake does that automatically for any library added with `link_directories` is what I have understood in the process. – Croolman Jul 28 '22 at 12:35