8

I am creating a Django application that does various long computations with uploaded files. I don't want to make the user wait for the file to be handled - I just want to show the user a page reading something like 'file is being parsed'.

How can I make an asynchronous function call from a view?

Something that may look like that:

def view(request):
    ...
    if form.is_valid():
        form.save()
        async_call(handle_file)
    return render_to_response(...)
mshell_lauren
  • 5,171
  • 4
  • 28
  • 36
DataGreed
  • 13,245
  • 8
  • 45
  • 64
  • 1
    Duplicate: http://stackoverflow.com/questions/546017/how-do-i-run-another-script-in-python-without-waiting-for-it-to-finish, http://stackoverflow.com/questions/1081340/how-do-you-do-something-after-you-render-the-view-django – S.Lott Aug 30 '09 at 00:01
  • Even more similar: http://stackoverflow.com/questions/1239035/asynchronous-method-call-in-python – ire_and_curses Aug 30 '09 at 00:15

4 Answers4

8

Rather than trying to manage this via subprocesses or threads, I recommend you separate it out completely. There are two approaches: the first is to set a flag in a database table somewhere, and have a cron job running regularly that checks the flag and performs the required operation.

The second option is to use a message queue. Your file upload process sends a message on the queue, and a separate listener receives the message and does what's needed. I've used RabbitMQ for this sort of thing, but others are available.

Either way, your user doesn't have to wait for the process to finish, and you don't have to worry about managing subprocesses.

Daniel Roseman
  • 588,541
  • 66
  • 880
  • 895
  • oh, how haven't I thought of this! Thanks, that's really a good point. I almost forgot about cron at all. Thanks, that should solve the problem neatly :) – DataGreed Aug 30 '09 at 08:31
  • The issue is solved via cron (and a simple shedule python script on the debugging machine, that acts like a simple cron :) ) – DataGreed Sep 27 '09 at 20:13
  • 2
    Btw, if somone is reading the solution right now, i advise to try Celery. – DataGreed Jul 23 '11 at 20:14
2

I have tried to do the same and failed after multiple attempt due of the nature of django and other asynchronous call.

The solution I have come up which could be a bit over the top for you is to have another asynchronous server in the background processing messages queues from the web request and throwing some chunked javascript which get parsed directly from the browser in an asynchronous way (ie: ajax).

Everything is made transparent for the end user via mod_proxy setting.

Chmouel Boudjnah
  • 2,541
  • 3
  • 24
  • 28
  • thanks for the idea, but i guess, i just have to use cron to start a manage.py command that parses all queued files every couple of minutes, because the project isn't that huge and public to run a second server :) – DataGreed Aug 31 '09 at 06:23
0

Unless you specifically need to use a separate process, which seems to be the gist of the other questions S.Lott is indicating as duplicate of yours, the threading module from the Python standard library (documented here) may offer the simplest solution. Just make sure that handle_file is not accessing any globals that might get modified, nor especially modifying any globals itself; ideally it should communicate with the rest of your process only through Queue instances; etc, etc, all the usual recommendations about threading;-).

Alex Martelli
  • 854,459
  • 170
  • 1,222
  • 1,395
  • 1
    Thank you for your answer. Am i understanding it right, that the main application main quit even if the thread with some file handling is still running or not? Or the thread will be also terminated? If so, it is kind of... useless, because many webservers spawns manage.py process when needed and kill them if there are too much of them being not busy. I'm now really looking into just adding something like os.system('python handle_file.py path/to/file')... – DataGreed Aug 30 '09 at 00:34
  • just figured out, that it will still be a subprocess... guess, i just need to use subprocess.Popen – DataGreed Aug 30 '09 at 00:42
  • If you need a subprocess, you need a subprocess, and S.Lott was right (guess there's a reason the guy's got almost twice my rep;-). But, for completeness: a Python process terminates "naturally" when the last NON-DAEMON thread terminates, not before; so, threads whose jobs are important to terminate should be non-daemon (the default) OR explicitly .join'd by the main thread at process end, while "worker threads" doing background tasks should be daemon ones (and not .join'd to;-) iff they may be terminated harmlessly. – Alex Martelli Aug 30 '09 at 01:12
  • hmmm, tried ou subprocess, but it is still synchronous an the user has to wait till the page loads. – DataGreed Aug 30 '09 at 23:30
0

threading will break runserver if I'm not mistaken. I've had good luck with multiprocess in request handlers with mod_wsgi and runserver. Maybe someone can enlighten me as to why this is bad:

def _bulk_action(action, objs):
    # mean ponies here

def bulk_action(request, t):

    ...
    objs = model.objects.filter(pk__in=pks)

    if request.method == 'POST':
        objs.update(is_processing=True)

        from multiprocessing import Process
        p = Process(target=_bulk_action,args=(action,objs))
        p.start()

        return HttpResponseRedirect(next_url)

    context = {'t': t, 'action': action, 'objs': objs, 'model': model}
    return render_to_response(...)

http://docs.python.org/library/multiprocessing.html

New in 2.6

Skylar Saveland
  • 11,116
  • 9
  • 75
  • 91