39

To compile a software package on a workstation with many CPU cores (say 12), the configuration stage often takes much longer than the actual compilation stage because ./configure does the tests one by one, while make -j runs gcc as well as other commands in parallel.

I feel that it is a huge waste of resources to have the remaining 11 cores sitting idle most of the time waiting for the slow ./configure to complete. Why does it need to do the tests sequentially? Does each test depend on each other? I can be mistaken, but it looks like the majority of them are independent.

More importantly, are there any ways to speed up ./configure?

Edit: To illustrate the situation, here is an example with GNU Coreutils

cd /dev/shm
rm -rf coreutils-8.9
tar -xzf coreutils-8.9.tar.gz
cd coreutils-8.9
time ./configure
time make -j24

Results:

# For `time ./configure`
real    4m39.662s
user    0m26.670s
sys     4m30.495s
# For `time make -j24`
real    0m42.085s
user    2m35.113s
sys     6m15.050s

With coreutils-8.9, ./configure takes 6 times longer than make. Although ./configure use less CPU time (look at "user" & "sys" times), it takes much longer ("real") because it isn't parallelized. I have repeated the test a few times (with the relevant files probably staying in the memory cache) and the times are within 10%.

Carl B
  • 6,660
netvope
  • 5,475

6 Answers6

18

I recall discussions on the Autoconf mailing list about this issue from about 10 years ago, when most people actually only had one CPU core. But nothing has been done, and I suspect nothing will be done. It would be very hard to set up all the dependencies for parallel processing in configure, and do it in a way that is portable and robust.

Depending on your particular scenario, there might be a few ways to speed up the configure runs anyway. For example:

  • Use a faster shell. For example, consider using dash instead of bash as /bin/sh. (Note: Under Debian, dash is patched so that configure doesn't use it, because using it breaks a lot of configure scripts.)
  • If you run builds remotely (through ssh, for example), then I have found that the console output can be pretty slow. Consider calling configure -q.
  • If you repeatedly build the same project, consider using cache file. Call configure -C. See the Autoconf documentation for details.
  • If you build a lot of different projects, consider using a site file (config.site). Again, see the documentation.
  • Build several projects in parallel.
Peter Eisentraut
  • 7,480
4

There are many types of ./configure scripts. There are popular tools (autconf being one of them) to aide a developer in creating a ./configure script, but there's no rule that says every developer must use these tools, and then even among these tools, there can be wide variations in the way these scripts are constructed.

I'm not aware of any popular ./configure scripts that can be run in parallel. Most of the scripts built by popular tools do at least cache some or all of their results, so if you run it again (without doing a make clean first, anyway), it runs much faster the second time.

That's not to say it couldn't be done... but I suspect there's little motivation for the people working on autoconf, for instance, to do that, since for most packages, the configure phase is very quick relative to the actual compilation and linking phases.

Kalle Richter
  • 2,472
  • 6
  • 44
  • 64
Flimzy
  • 4,465
4

You have been smart in using ramdrive for the sourcetree to reside in, but think about it twice - what does configure do? It does its job by checking not just your sourcetree, but quite often also the system for availabilities of library, compilers, etc. In this case, the access problem sometimes resides in disk access - You will have it done much quicker if you have for example a SSD-based root file system.

bubu
  • 10,029
3

Your question might even be more relevant today as we have dozen-core CPUs with (pretty) low single core performance. Automated builds for Continuous Integration (CI) really waste lot's of CPU time / energy for every commit. Same with hopping between branches.

So review/read my hints about speeding up the thing at https://gitlab.com/gnuwget/wget2/wikis/Developer-hints:-Increasing-speed-of-GNU-toolchain.

"Why does it need to do the tests sequentially ? ..." There are in fact a few things that could be done in parallel, while others have to be sequential. Several things depend on the build environment - and the configure script itself is system independent. It doesn't even contain bashisms, so it works with a pure POSIX shell.

If you want to write portable software, there is no other build system like autotools. But if you don't mind about (wide) portability, avoid autotools - there is a plethora of fast and good-enough build tools.

Tim Ruehsen rockdaboot
  • 151
3

The hard drive is the bottleneck in this case. To speed up the build, build on a system with fast drives (read: low access time). There is a lot of fuss about SSD discs, but there was some criticism regarding them not affecting the compilation time in a positive way. That is to say building on SSD was not so much faster than on a decent sata drive. I can't recall where I read this becasue the article is a couple of years old.

Anyways... Untar to ram and build from there.

mkdir /tmp/tmp 
mount -t tmpfs -o size=400M tmpfs /tmp/tmp 
cd /tmp/tmp
tar xjf somesourcetarball-1.1.33.tar.bz2
Ярослав Рахматуллин
  • 11,633
3

If you're using the ondemand cpu governor, try using the performance one. This helps on the i7 and a8-3850 by 40-50%. Doesn't make much difference on the q9300.

On a quad core cpu, you might do 

for cpu in `seq 0 3`; do sudo cpufreq-set -g performance -c $cpu; done

(The -r option should make it so you don't have to do cpufreq-set for each core, but on my computers it doesn't work.)

The cache option helps even more, though.

Dan Kegel
  • 31