10

I am using conan in an enterprise environment where the operating system is rather old and has an old version of glibc (2.11). As a result, a lot of the pre-built binaries on conan.io do not end up working in my environment. However, conan doesn't know that and will happily download and install them on my system, resulting in link-time errors.

I have found that if I build from source I can get the libraries to work.

My desired behavior would be as follows:

  • The first time using conan install to install the library (e.g. it is not in my cache) then conan will build from source and place it in my cache, then use it.
  • On subsequent invocations of conan install, conan finds the cached library and uses that without having to rebuild from source.
  • I am invoking conan install as part of an automated build script, so I would like to not have to modify the invocation depending on if this is the first time the library is installed or not (but modifying a configuration file is fine).

I have had troubles obtaining this behavior in practice. Here are the challenges I have run into:

  • If I use conan install --build=thelibrary then conan will rebuild that library from source every time I invoke conan install --build=thelibrary, even if it is already present in my cache.
  • If I use conan install --build=missing, then Ican trick conan into building the library by setting some build options that do not have a pre-built binary associated with them.
    • This is fragile, as it only works for projects with enough build options that it is not tractable to create pre-built options for all combinations.
    • It also doesn't work if all the build options I need correspond to a pre-built binary.

Here is what I am looking for (and I assume exists but am not able to find):

  • Some setting I can place in my conanfile.txt (or some other configuration file) that tells conan to ignore pre-built binaries for a given library or libraries and instead build from source, but use the cached version if it is available.
    • This ideally should work without me having to tinker with build options.
  • I don't necessarily want to build all libraries from source, just the ones that won't run on my ancient OS, but if I have to settle for "all-or-nothing" I will take "all".

Is this possible with conan?

SethMMorton
  • 45,752
  • 12
  • 65
  • 86
  • You could consider using (if allowed) [ccache](https://ccache.dev/), and conan being open source, you technically can compile it on your old host. You just need to ask permission -to your boss- to spend a few hours doing so – Basile Starynkevitch Mar 05 '21 at 05:38

1 Answers1

1

glibc version is an old headache for Conan, because it's not part of settings, thus is not counted as part of package ID. The Conan Docker images are running Ubuntu, some of them are old, others are new. But there is a specific Docker image running CentOS6, which was created because of glibc 2.12 and could help with package generation.

For your specific case, we have few options:

  • Add glibc as part of settings, so Conan won't replace your package because of its package ID. As you should have more coworkers, you can use conan config command for settings distribution.

    # ~/.conan/settings.yml
glibc: [None, 2.11, ...]

    Adding it, you can update you profile too, making glibc=2.11 as a default setting.

  • Another alternative is package revisions feature, where you can lock a specific binary package for usage, which means, you want use that specific package. You just need to upload your generated package with glibc and use its binary package revision, e.g. lib/1.0@conan/stable#RREV:PACKAGE_ID#PREV

Also, answering your question:

Some setting I can place in my conanfile.txt (or some other configuration file) that tells conan to ignore pre-built binaries for a given library or libraries and instead build from source, but use the cached version if it is available.

Your cache is Conan first option, it will look for a pre-built package there first, if it's not available, it will look into your remotes, following a sorted order. Your request is not possible, first, because conanfile.txt doesn't support build policies, second, because conanfile.py only supports build all from sources, or build only missing.

My propose is, install an Artifactory instance, build what you need, upload your custom packages, and make it as your default remote.

I don't necessarily want to build all libraries from source, just the ones that won't run on my ancient OS, but if I have to settle for "all-or-nothing" I will take "all".

You can associate some package reference to a remote, running conan remote command. Let's say you want to download zlib/1.2.11 built with glibc-2.11 and it's available only in your organization remote:

$ conan remote add_ref zlib/1.2.11@org/stable my_org_repo
$ conan remote list_ref # only to validate, not mandatory
zlib/1.2.11@org/stable: my_org_repo

Now your specific package is associated to your organization. Conan still will look for that package your local cache first, but when not found, it will try to find at your Artifactory.

As you can see, your case could be solved easier using a new setting, instead of trying to hack build policies. As another alternative, you can replace glib setting by distro and its version.

uilianries
  • 3,363
  • 15
  • 28
  • Ah, so basically I had an XY problem. The glibc setting sounds like just the thing I want - I’ll give that a shot and report back. – SethMMorton Mar 05 '21 at 15:44
  • So I am trying to do this, but I am running into some issues. I do not see "glibc" listed as a documented option [here](https://docs.conan.io/en/latest/reference/config_files/settings.yml.html) or [here](https://docs.conan.io/en/latest/extending/custom_settings.html#custom-settings). When I try to add it to my settings or profile, `conan` keeps telling me it's an invalid setting or option. Can you provide the specific `conan config` and `conan profile` invocations I would want to use to add this option to my configuration? – SethMMorton Mar 05 '21 at 17:16
  • again, glic is not covered by Conan itself. It's a custom setting. Read https://docs.conan.io/en/latest/extending/custom_settings.html – uilianries Mar 05 '21 at 19:10
  • 2
    OK. So I did not have any luck with setting glibc. However, I **did** get the desired result with setting "distro" to `[SLES11]` in the `os: Linux:` section of `~/.conan/settings.yml`, then calling `conan profile update settings.os.distro=SLES11 default`. This caused everything except header-only libs to be built from source. Thank you for the pointers! – SethMMorton Mar 05 '21 at 20:02