Profiling an application that uses reactors/websockets and threads

Question

Hi I wrote a Python program that should run unattended. What it basically does is fetching some data via http get requests in a couple of threads and fetching data via websockets and the autobahn framework. Running it for 2 days shows me that it has a growing memory demand and even stops without any notice. The documentation says I have to run the reactor as last line of code in the app.

I read that yappi is capable of profiling threaded applications Here is some pseudo code

from autobahn.twisted.websocket import  WebSocketClientFactory,connectWS

if __name__ == "__main__":
#setting up a thread

#start the thread
Consumer.start()

xfactory = WebSocketClientFactory("wss://url")
cex_factory.protocol = socket
## SSL client context: default
##
if factory.isSecure:
    contextFactory = ssl.ClientContextFactory()
else:
    contextFactory = None

connectWS(xfactory, contextFactory)

reactor.run()

The example from the yappi project site is the following:

import yappi
def a(): 
    for i in range(10000000): pass

yappi.start()
a()
yappi.get_func_stats().print_all()
yappi.get_thread_stats().print_all()

So I could put yappi.start() at the beginning and yappi.get_func_stats().print_all() plus yappi.get_thread_stats().print_all() after reactor.run() but since this code is never executed I will never get it executed.

So how do I profile a program like that ?

Regards

score 0 · Answer 1 · edited May 23 '17 at 12:04

It's possible to use twistd profilers by the following way:

twistd -n --profile=profiling_results.txt --savestats --profiler=hotshot your_app

hotshot is a default profiler, you are also able to use cprofile. Or you can run twistd from your python script by means of:

from twistd.scripts import run
run()

And add necessary parameters to script by sys.argv[1:1] = ["--profile=profiling_results.txt", ...] After all you can convert hotshot format to calltree by means of:

hot2shot2calltree profiling_results.txt > calltree_profiling

And open generated calltree_profiling file:

kcachegrind calltree_profiling

There is a project for profiling of asynchronous execution time twisted-theseus You can also try tool of pycharm: thread concurrency

There is a related question here sof You can also run your function by:

reactor.callWhenRunning(your_function, *parameters_list)

Or by reactor.addSystemEventTrigger() with event description and your profiling function call.

Profiling an application that uses reactors/websockets and threads

1 Answers1