How to collect source files with CMake without globbing?

Question

The CMake documentation explicitly states that file(GLOB ...) is not recommended to collect source files for a build, but it doesn't mention what the recommended method actually is.

Specifying every source file manually sounds a little bit too manually to me. So, what is the right method to collect source files, if not file(GLOB ...)?

You could use GLOB from cmake in a script and have that add dump out the filenames. — Damian, Nov 27 '17 at 23:08
We dig manual labour so why not just typing in dozens (maybe more) of filenames? And we're sure no typos will be made, cause we'll visually check them 2-3 times. — mireazma, Mar 06 '21 at 11:30

score 26 · Accepted Answer · edited Jan 23 '13 at 20:43

26

Manual is indeed the recommended method. By recommending against using GLOB, the documentation is simply warning against a build system that depends on files present. For example, you want to add a test executable, so you create mytest.cpp. Oops. Now your library compilation breaks. The documentation for AUX_SOURCE_DIRECTORY (similar purpose as globbing for for source files) gives the following warning:

It is tempting to use this command to avoid writing the list of source files for a library or executable target. While this seems to work, there is no way for CMake to generate a build system that knows when a new source file has been added. Normally the generated build system knows when it needs to rerun CMake because the CMakeLists.txt file is modified to add a new source. When the source is just added to the directory without modifying this file, one would have to manually rerun CMake to generate a build system incorporating the new file.

If you're certain that you want all the contents of a directory, and don't plan on adding new ones, then by all means use a GLOB.

Also, don't forget listing files manually doesn't have to involve typing all the filenames. You could do, for example, ls *.cpp >> CMakeLists.txt, then use your editor to move the list of files to the correct place in the file.

edited Jan 23 '13 at 20:43

DLRdave

13,876
4
53
70

answered Jun 06 '12 at 13:21

Peter

14,559
35
55

Kind of depend on the IDE tbh. In Qt Creator, it always rerun CMake so using GLOB is not an issue. – Lap Jun 06 '12 at 15:14
2

`printf "add_library(xxx $(ls *.cpp))" >> CMakeLists.txt` should be nice too! – Alexander Oh Mar 13 '13 at 16:58
1

I don't think that your complication is a valid one, because test are usually in the test folder and should not clutter the source tree. – Arne Jan 17 '14 at 16:29
1

There's a difference between the complication not applying to many projects and the complication "not being valid". If these concerns don't apply to your project, then use GLOB and be happy. – Peter Jan 17 '14 at 16:41
So the only reason not to glob is that it takes more work to re-run cmake than to add a file in the CMakeLists.txt file? Isn't it easier to just run cmake than to have to look up the file name and manually add it to the file? The library compilation still breaks if I forget to add the file manually. – Björn Sundin Oct 28 '20 at 10:48
1

Some IDEs add files automatically to the CMakeLists.txt file, but isn't this a purely IDE dependent problem then? Doing it manually only makes life easier with some IDEs. For others, it makes it harder - for example VS Code with CMake tools. Correct me if I'm wrong. – Björn Sundin Oct 28 '20 at 10:55
The `ls` trick to get the list of file names is an excellent idea! – saxbophone Feb 03 '21 at 01:02

score 19 · Answer 2 · answered Feb 04 '14 at 19:43

I use GLOB for exactly that and every time I add a file I run

touch ../src/CMakeLists.txt

The next make command will re-scan the directories.

"There is no way for CMake to generate a build system that knows when a new source file has been added" Really? Okay, so tell it!

It's not 100% automatic but a damn sight easier than adding files manually.

score 5 · Answer 3 · answered May 25 '15 at 07:23

I use cog, a python module. Here is a sample to collect .cpp file:

The CMakeLists.txt:

set(SRC_FILES "")

# [[[cog
#    import cog, glob
#    for src in glob.glob('*.cpp'):
#       if "skeleton" in src: continue
#       cog.outl("SET(SRC_FILES ${SRC_FILES} %s)" % src)
# ]]]
# [[[end]]]

add_library(mylib STATIC  ${SRC_FILES})

And then, run:

python -m cogapp -r CMakeLists.txt

The CMakeLists.txt file will be updated in place.

For how to install cog and other usage, please read the article from the author.

Your solution is awesome! – Alen Wesker Feb 08 '22 at 10:18 — Alen Wesker, Feb 08 '22 at 10:18

palfi · Answer 4 · 2018-01-18T10:26:50.790

I use a conventional CMakeLists.txt and a python script to update it. I run the python script manually after adding files.

import os
import re       

def relFiles(base, sub):
    fullSub = os.path.join(base,sub)
    abs = [os.path.join(dp, f) for dp, dn, fn in os.walk(fullSub) for f in fn]
    return [os.path.relpath(f, base) for f in abs]

def updateAddLibrary(cmakelistsDir, subs):
    cmakelists = os.path.join(cmakelistsDir, "CMakeLists.txt")
    listings = [relFiles(cmakelistsDir, sub) for sub in subs]
    files = [f for listing in listings for f in listing] #flatten
    with open(cmakelists, 'r') as file:
        text = file.read()
    sources = "".join(["    %s\n" % f.replace('\\', '/') for f in files])
    text = re.sub(r"add_library\s*\(\s*([^\s\)]+).*?\)",
           r"add_library(\1\n%s)" % sources, 
           text, 1, re.DOTALL)
    with open(cmakelists, "w") as file:
        file.write(text)

dir = os.path.dirname(os.path.abspath(__file__))
updateAddLibrary(dir, ['inc','src'])

Example before:

...
add_library(MyLib
    inc/a.h
) 
...

after:

...
add_library(MyLib
    inc/a.h
    inc/sub/b.h
    src/a.cpp
)        
...

score 0 · Answer 5 · answered Feb 08 '22 at 10:40

Following douyw's answer, thank you for your answer.

Not a Cmake expert, don't want to be one, I spent a fxxking 3 hours trying to deal with GLOB(disabled) and aux_source_directory(Not even close to GLOB), and douyw save my life.

I add the recursive file walking, and it's working in my project:

Firstly, install the cogapp python module (python -m pip install cogapp)

set(SRC_FILES "")

# [[[cog
#   import cog, os
#   for root, _, files in os.walk(".", topdown=False):
#       for f in files: 
#           if not "_unittest" in f: continue
#           if not f.endswith(".cpp"): continue
#           cog.outl('SET(SRC_FILES ${SRC_FILES} "%s")' % os.path.join(root, f).replace('\\', '/'))
# ]]]
# [[[end]]]

run: python -m cogapp -r CMakeLists.txt

The upper lines add all "*_unittest.cpp" to the list
You can change the middle lines to make your own rule, just plain python. Using regex is better, but simple string searching can do the job on the above situation.
Notice the last line, it needs to replace // to change to the usable universal separator. You may generate your CMakeList.txt in windows.
You may replace SRC_FILES with whatever you want.
The python command may be run under Jenkins/TeamCity, triggered by svn/git commit. Then we can automatically add new files.

How to collect source files with CMake without globbing?

5 Answers5

Linked