Two step compilation procedure with gnu make

Question

I have a bunch of files (*.txt) that I use a python script to analyze and generate code (*.cpp) from. Once code generation is complete, I want the generated code to be compiled into object files (*.o).

I want to use gnu-make to set this up. My problem is that I can't figure out a working Makefile syntax to accomplish this.

Here are some additional details that makes this harder:

Since I don't know how many *.txt-files exists in the first place, I want to discover them using a shell find command.

files := $(shell find $(TXT_FILE_DIR) -type f -name '*.txt')

However, I still want gnu-make's fine dependency features to work here. That is, if any of the txt-files change, I want make to automatically discover these and only these changes, and generate new .cpp files for them and recompile them into .o files.

I am looking for a solution where make can run in parallel (and respect the -j flag) and call the python script that translates the *.txt files into *.cpp files.

My OS is Linux.

[new information]

Thank you Grexis, for your answer! However, because I have a somewhat odd mapping between src and dst-paths, I am actually looking for a different solution.

Here is an example of "the oddity" when translating my src and dst-paths:

$(TXT_DIR)/a/XXX.*XXX/b/c/file1.txt -> $(CPP_DIR)/a/YYY/b/c/file1.cpp $(TXT_DIR)/a/XXX.*XXX/b/c/file2.txt -> $(CPP_DIR)/a/YYY/b/c/file2.cpp $(TXT_DIR)/b/XXX.*XXX/c/d/file3.txt -> $(CPP_DIR)/b/YYY/c/d/file3.cpp

Notice here that the path is manipulated twice. Hence, both:

1. $(TXT_DIR) is replaced with $(CPP_DIR) and
2. The regular expression XXX.*XXX is replaced with the fixed string YYY

However, due to that its hard to do a multiple string replacement in gnu-make, e.g., gnu-make only supports a single % (stem) string replacement at a time, I think I have to use a more advanced gnu-make syntax. Because of this, I am looking for a way to define a "distinct recipe for each .txt -> .cpp pair". I have tried the following:

cleanPath = $(shell echo $(1) | sed s/XXX.*XXX/YYY/g) txt2cpp = $(subst $(TXT_DIR),$(CPP_DIR),$(patsubst %.txt,%.cpp,$(call cleanPath,$(1)))) txt2obj = $(subst $(TXT_DIR),$(OBJ_DIR),$(patsubst %.txt,%.o,$(call cleanPath,$(1)))) define analyse = $(2): $(1) python.py $(2) $(3) endef txt_files := $(shell find $(TXT_DIR) -type f -name '*.txt') cpps := $(foreach file,$(txt_files),$(eval $(call analyse,$(file),$(call txt2cpp,$(file)),$(call txt2obj,$(file))))) all: $(cpps)

However, it seems my python.py script is never called. Why is that?

If I enclose the calling to the python.py in a shell-command, e.g.,$(shell python.py $(2) $(3)), the python.py script is actually called. However, using a shell-command makes parallelism impossible. Unfortunately, not being able to execute the python.py script in parallel is not an option for me.

Answer 1

It's a bit unclear if the each text file creates exactly one C++ source with a matching name (i.e. 'foo.txt' gets processed into 'foo.cpp' and then 'foo.o').

Assuming that it is a direct relation, the feature you are looking for is called Pattern Rules.

TXT_FILES := $(wildcard $(TXT_FILE_DIR)/*.txt)
OBJ_FILES := $(patsubst $(TXT_FILE_DIR)/%.txt,$(OUT_DIR)/%.o,$(TXT_FILES))

main: $(OBJ_FILES)
    $(CXX) $(LDFLAGS) -o $@ $^

$(SRC_DIR)/%.cpp: $(TXT_FILE_DIR)/%.txt
    $(PYTHON) process.py $<

$(OUT_DIR)/%.o: $(SRC_DIR)/%.cpp
    $(CXX) -c $(CPPFLAGS) -o $@ $<

Breaking down the snippet above. $(TXT_FILES) is set to every *.txt file in $(TXT_FILE_DIR) with the wildcard operation in make. $(OBJ_FILES) uses pattern substitution with all the values in $(TXT_FILES). Every obj file is then added as a dependency to the main executable.

As explained in the Pattern Rules link documentation linked above, every .o file needs the associated .cpp file in $(SRC_DIR) (The automatic variables $@, $^, and $< are also documented in the link provided). In turn, every .cpp file requires a .txt with the same name.

You'll probably have to tweak the exact invocation of your python script though. If a specific output file needs to be specified, the $@ automatic variable should be used.

The section at the top for detecting all the source files was derived from this answer.