ld: Using -rpath,$ORIGIN inside a shared library (recursive)

Question

I just made a basic example of using ld's -rpath option with $ORIGIN here (see 2nd response for a working version). I'm trying to create an example where main.run links to foo.so, which in turn links to bar.so, all using rpath and $ORIGIN.

The run-time file-structure is:

• project/
  • lib/
    • dir/
      • sub/
        • bar.so
      • foo.so
  • run/
    • main.run (failing to build)

I'm building foo.so using:

g++ -c -o obj/foo.o src/foo.cpp -fPIC
g++ -shared -o lib/dir/foo.so obj/foo.o -Wl,-soname,foo.so -Wl,-rpath,'$ORIGIN/sub' -Llib/dir/sub -l:bar.so

Which builds fine. ldd lib/dir/foo.so can even find bar.so.

However, when I try to link main.run to foo.so, foo.so can't find bar.so.

I'm building main.so using:

g++ -c -o obj/main.o src/main.cpp
g++ -o run/main.run obj/main.o -Wl,-rpath,'$ORIGIN/../lib/dir' -Llib/dir -l:foo.so

This works fine if another version of foo.so is used that doesn't recursively link. (Uncomment lines in make.sh, in project below, to test).

However, using the normal foo.so I'm getting this error when building main.run:

/usr/bin/ld: warning: bar.so, needed by lib/dir/foo.so, not found (try using -rpath or -rpath-link)

So my questions are:

1. Does $ORIGIN within foo.so resolve to project/lib/dir (where foo.so is) or project/run (where main.run (the executable linking it) is)?
   ldd would seem to indicate that it's project/lib/dir, which would seem to be the best way (although I tried assuming both).
2. How do I get these to link (while preserving relocatability) - preferably without using -rpath-link.

You can download the project here. It's as simple as I can make it. 4 short sources and a script.
After extracting, just run ./make.sh from within project/.

Note: I'm using -l:. This shouldn't change anything except that the libraries are named like foo.so instead of libfoo.so, and lunk with -l:foo.so instead of -lfoo.

Answer 1

Well, I have something working. But I do not really understand why it works. This feels like a bug in ld to me.

I ran strace -f -o /var/tmp/strace.out -- g++ ... for the main.run compilation. The static linker is actually trying to open files whose literal name looks like "$ORIGIN/lib/dir/sub/bar.so", among 20-30 other things. (In other words, it is looking for an actual directory named $ORIGIN. Seriously.)

It also appears to be searching the -rpath-link path for the name "lib/dir/sub/bar.so", not just "bar.so". I have no clue why.

Anyway, this is the link for main.run that is working for me:

g++ -o run/main.run obj/main.o -Wl,-rpath,'$ORIGIN/../lib/dir' -Wl,-rpath-link,. -Llib/dir -l:foo.so

It is identical to yours but with -Wl,-rpath-link,. inserted.

[addendum]

OK I think I see what is going on. First, the static linker (GNU ld) simply does not honor $ORIGIN in the libraries it links against.

Second, the behavior when you use -lbar versus -l:bar.so is very different.

Run readelf -a on foo.so. In your build, it shows a dependency on "lib/dir/sub/bar.so". This is why setting the rpath-link to "." fixes the build of main.run; it causes the static linker to search "." for "lib/dir/sub/bar.so", which it finds.

If you rename bar.so to libbar.so, and link foo.so to use -lbar instead of -l:bar.so, the same readelf shows that foo.so now depends on "libbar.so" (with no path component). With that foo.so, you can get the main.run link to work using -Wl,-rpath-link,lib/dir/sub, as you would expect if you knew that the static linker simply does not honor $ORIGIN.

By the way, I do not see the -l:bar.so syntax documented anywhere in the GNU ld manual. Out of curiosity, how did you come up with it?

Assuming it is a supported feature, this looks a bit like a bug (-l:bar.so creating a dependency on lib/dir/sub/bar.so instead of just bar.so). You can either deal with this bug by setting rpath-link to '.' for main.run, or you can rename stuff in the usual way (libxxx.so).

Answer 2

From the ld-linux(8) manpage:

$ORIGIN and rpath

ld.so understands the string $ORIGIN (or equivalently ${ORIGIN}) in an rpath specification (DT_RPATH or DT_RUNPATH) to mean the directory containing the application executable. Thus, an application located in somedir/app could be compiled with gcc -Wl,-rpath,'$ORIGIN/../lib' so that it finds an associated shared library in somedir/lib no matter where somedir is located in the directory hierarchy. This facilitates the creation of "turn-key" applications that do not need to be installed into special directories, but can instead be unpacked into any directory and still find their own shared libraries.

Thus, in answer to your first question, there's only one value for $ORIGIN: project/run.

Therefore, the answer to your second question should be to use the following command to link foo.so:

g++ -shared -o lib/dir/foo.so obj/foo.o -Wl,-soname,foo.so -Wl,-rpath,'$ORIGIN/../lib/dir/sub' -Llib/dir/sub -l:bar.so

Answer 3

First, there are issues with $ sign expansion that might be causing problems. I'm building Python from source and I do this:

export LDFLAGS='-Wl,-rpath,\$${ORIGIN}/../lib -Wl,-rpath,\$${ORIGIN}/../usr/lib -Wl,--enable-new-dtags'

before running make. That works fine and it finds 1st level dependencies. Be careful with single and double quotes when dealing with this type of macro expansion issue.

Secondly, if you run objdump -x on a binary or a library, you can see the RPATH header that it actually contains. When I run objdump -x path/to/python |grep RPATH it shows me this.RPATH ${ORIGIN}/../lib:${ORIGIN}/../usr/lib`

I suggest that you check your binaries to see what is actually in the RPATH header. Unfortunately, I don't think that this will solve your problem. This is what I see when I run ldd path/to/python:

libpython2.7.so.1.0 => /data1/python27/bin/../lib/libpython2.7.so.1.0 (0x00002ad369d4f000)
libpthread.so.0 => /lib/libpthread.so.0 (0x00002ad36a12f000)
libdl.so.2 => /lib/libdl.so.2 (0x00002ad36a34d000)
libutil.so.1 => /lib/libutil.so.1 (0x00002ad36a551000)
libm.so.6 => /lib/libm.so.6 (0x00002ad36a754000)
libc.so.6 => /lib/libc.so.6 (0x00002ad36a9d8000)
/lib64/ld-linux-x86-64.so.2 (0x00002ad369b2d000)

As you can see, the first level dependency is correctly handled by rpath but the second level dependencies, i.e. the dependencies of libpython, revert to system libraries. And yes, libpython has the exact same RPATH header in its binary. I found your question while googling rpath recursive to try and resolve my problem of making a distro independent package.

Added later The rpath headers only change the FIRST path searched for libraries. If they are not found there, then the loader continues to search in the normal places. ldd only lists the actual path of the library that was found as a result of the search. When I copied these libraries to the rpath directory, then everything worked. Basically there is no tidy way to find all the dependencies and copy them, just ldd -v path/to/python and some parsing of that output.

Answer 4

Check my modified version of your make script. Basically, an additional -rpath-link without $ORIGIN should be used, since ld doesn't understand $ORIGIN at all.

As for your questions.

1. $ORIGIN only works during runtime, and it's w.r.t. each library. So different shared libraries have different $ORIGIN.
2. I'm afraid the best way is to add rpath-link, and this won't affect your portability, since they are relative, and won't exist in the final executable, as I have shown in my version of make.sh

Also, this is my own understanding of the whole linking stuff. I hope it helps.

Answer 5

I've been looking into this as well, and as best I can tell you need to use -rpath-link for any path that would use ORIGIN expansion. For example:

CC -shared (other flags) -R'$ORIGIN/../lib/' -o /buildpath/lib/libmylib1.so
CC -shared (other flags) -R'$ORIGIN/../lib/' -lmylib1 -o /buildpath/lib/libmylib2.so
# This fails to link 'somebinary'
CC (various flags) -R'$ORIGIN/../lib/' -lmylib2 -o /buildpath/bin/somebinary
# This works correctly
CC (various flags) -R'$ORIGIN/../lib/' -Wl,-rpath-link,/buildpath/lib/mylib1 -lmylib2 -o /buildpath/bin/somebinary
# The text above the carets to the right is a typo: ------------------^^^^^^
# I'm fairly sure it should read like this (though it has been awhile since I wrote this):
# (...) -Wl,-rpath-link,/buildpath/lib -lmylib1 (...)

ld will not expand $ORIGIN within paths specified using -rpath-link or paths it retrieves from a sub-dependency's RPATH. In the above example, mylib2 depends on mylib1; while linking somebinary, ld attempts to find mylib1 using the literal/unexpanded string $ORIGIN/../lib/ embedded in libmylib2.so. ld.so would at runtime, but not ld.

It also won't use paths specified with -L to find the sub-dependency librar(y|ies).

Answer 6

From my understanding this is an issue in ld (i.e. binutils) and how it resolve "Secondary Dependencies"

AFAIK started from binutils >= 2.30. rpath in dependencies are added to the search. i.e. ld ... main.exe find foo.so, then read the RPATH in foo.so thus find bar.so

Here my stack overflow question: Binutils Secondary Dependency Change

And here my investigation on various distro (inside docker container) to test various binutils version https://github.com/Mizux/SecondaryDependency

note: take a look a the travis-CI log...