Assume I have /foo/X.a and /foo/X.dylib, and LDFLAGS=-L/foo. What will AC_CHECK_LIB do? Is .dylib guaranteed to take precedence over .a? Vice versa? Random?
Assume I have /foo/Y.a and /bar/Y.a, and LDFLAGS=-L/foo:/bar? What will happen then?
Is there some way to manually establish precedence?
This problem is not addressed by autoconf. In the case that you have both /foo/X.a and /foo/X.dylib, the configure script will simply attempt to link with libX and will do whatever your toolchain does. If your linker uses X.a, then the configure script will do its tests using X.a. If your linker uses X.dylib, then the configure script will use that. The way to establish precedence will therefore depend on your toolchain (e.g. perhaps you can add -Bstatic to LDFLAGS).
Is there some way to manually establish precedence?
You are manually establishing precedence of library search by using the -L option.
Also what if I have /foo/Y.a and /bar/Y.a, and LDFLAGS=-L/foo:/bar What will happen then?
It will look in the directory /foo:/bar before the default library paths. It's not clear from your question if Y.a is actually supposed to be linked to anything.
What will AC_CHECK_LIB do? Is .dylib guaranteed to take precedence over .a? Vice versa? Random?
It won't be random, but will be influenced by other options given to configure (e.g. LDFLAGS, --disable-shared, etc.). All AC_CHECK_LIB does is plop in the name of the selected function into a main function (with appropriate libraries added) and see if it links.
