Using the line-profiler (in ipython) on compiled Cython code

Question

I read the answer to this question How to profile cython functions line-by-line, but I can't seem to get it to work with my setup.

I have a cumsum.pyx file:

# cython: profile=True
# cython: linetrace=True
# cython: binding=True
DEF CYTHON_TRACE = 1

def cumulative_sum(int n):
    cdef int s=0, i
    for i in range(n):
        s += i

    return s

I compiled it with:

cython cumsum.pyx
gcc cumsum.c $(pkg-config --cflags --libs python3) -o cumsum.so -shared -fPIC

Then I tried to profile it in ipython:

%load_ext line_profiler
from cumsum import cumulative_sum
%lprun -f cumulative_sum cumulative_sum(100)

I don't get an error message, only an empty profile:

Timer unit: 1e-06 s

Total time: 0 s
File: cumsum.pyx
Function: cumulative_sum at line 6

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
     6                                           def cumulative_sum(int n):
     7                                               cdef int s=0, i
     8                                               for i in range(n):
     9                                                   s += i
    10                                           
    11                                               return s

How can I get this to work?

PS: I use CMake, not setup.py, so I would appreciate a build system agnostic solution

score 1 · Answer 1 · answered Jan 29 '17 at 22:05

1

The documentation on Cythons "Profiling" already includes an example how to set the CYTHON_TRACE macro:

# distutils: define_macros=CYTHON_TRACE_NOGIL=1

instead of your DEF CYTHON_TRACE = 1.

It worked when I compiled it using %%cython:

%load_ext cython
%%cython

# cython: profile=True
# cython: linetrace=True
# cython: binding=True
# distutils: define_macros=CYTHON_TRACE_NOGIL=1

def cumulative_sum(int n):
    cdef int s=0, i
    for i in range(n):
        s += i
    return s

And showed the profiling:

%load_ext line_profiler
%lprun -f cumulative_sum cumulative_sum(100)
[...]
Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
     7                                           def cumulative_sum(int n):
     8         1            8      8.0      3.5      cdef int s=0, i
     9         1            3      3.0      1.3      for i in range(n):
    10       100          218      2.2     94.4          s += i
    11         1            2      2.0      0.9      return s

answered Jan 29 '17 at 22:05

MSeifert

145,886
38
333
352

Thanks. I tried it, but it doesn't solve the problem – user357269 Jan 29 '17 at 22:18
Could you be more specific. What's wrong? Did you use `cythonize()`? – MSeifert Jan 29 '17 at 22:19
Oh, sorry, nothing changed from my original post, the profile comes up empty. Does it work for you if you "pre-compile" it? – user357269 Jan 29 '17 at 22:21
Ah, maybe the distutils line doesn't do anything because I'm not using distutils – user357269 Jan 29 '17 at 22:23
@user357269 Yes, the `# distutils :`-line is (as per documentation) only working if you use `cythonize`. – MSeifert Jan 29 '17 at 22:25

score 0 · Accepted Answer · edited May 23 '17 at 12:33

Turns out the issue was that DEF CYTHON_TRACE = 1 doesn't actually set the right constant.

Workarounds include:

1. MSeifert's answer, using distutils

2. Changing the gcc line to

gcc cumsum.c $(pkg-config --cflags --libs python3) -o cumsum.so -shared -fPIC -DCYTHON_TRACE=1

3. Making an extra header trace.h and setting the constant there

 #define CYTHON_TRACE 1

along with adding the following to cumsum.pyx

cdef extern from "trace.h":
    pass

4. With CMake, adding

add_definitions(-DCYTHON_TRACE)

Using the line-profiler (in ipython) on compiled Cython code

2 Answers2