1

I wish to:

  1. launch an async task (with something like django-celery or Gevent),
  2. wait a finite amount of time for task to finish (1 second),
  3. and then continue without killing the task.

I noticed this related thread https://stackoverflow.com/a/13001153/226800; however, in the accepted answer, when 'g' is destroyed does the task also get destroyed? I want to allow the task to run in the background even after the creator of the task has already returned, after waiting a while.

Community
  • 1
  • 1
brianray
  • 1,139
  • 1
  • 12
  • 16
  • I can't see what is missing from my answer; I can guarantee that it's factually correct—but you don't believe me? – Erik Kaplun Oct 05 '13 at 23:24
  • I think it needs cleaned up to be a 'correct' answer. The code you pasted in gist is actually more correct than your answer. I could not get anything (even after adding a while sleep) after task timed out but is continuing from the task in your example. – brianray Oct 07 '13 at 15:15
  • also, I want to clarify the question in that I really do want to completely return from the caller, If something like Gevent is used in context of a request/response cycle in Django, what happens to these events? Are they registered globally somehow? I really think http://stackoverflow.com/questions/9034091/how-to-check-task-status-in-celery might be my answer. Do you agree? – brianray Oct 07 '13 at 15:18
  • Are you talking about events or greenlets? if greenlets: then yes, greenlets are "registered" with the Gevent Hub and continue living indefinitely, until killed or until they terminate normally or because of an exception, or the entire process is killed. As to if Django might kill off the entire thread or process where the Gevent Hub resides in—yes, that might happen, but you need to Google "django gevent" and make sure you set everything up properly. – Erik Kaplun Oct 07 '13 at 15:27
  • Also, my Gist contains the exact same code as my answer, except in the Gist I'm using another greenlet to spawn the sub-greenlet. In my answer, the main greenlet is used implicitly, but it's still a greenlet, just like the main thread is a thread. And I didn't get the sentence that starts with *" I could not get anything (even after adding a while sleep) after ..."* – Erik Kaplun Oct 07 '13 at 15:28
  • ...all in all, I'm not entirely sure what you're after here: you don't seem to know much about how Gevent works, yet you make suggestions that I don't know what I'm saying about Gevent; also, it's not completely clear anymore as to what is it that you want to achieve anyway. – Erik Kaplun Oct 07 '13 at 15:29
  • Well when I ran the example and added: while True: sleep ... i never saw any more more output. I clearly have more to learn about Gevent--but I really think my accepted answer is I do not want to use Gevent for this problem. I gave you a +1; although, I do not think this is the best answer for my particular problem. I really think I need an event model that works in a different process on a higher level instead of a different thread in the same process. – brianray Oct 07 '13 at 15:52
  • Your question doesn't really specify those things at all, and it mentions Gevent as a possibility. But anyway: if you need your tasks to live in a separate process, then yes: pure Gevent won't do; either use Celery or run your own event processor (that might then use Gevent) in a separate process, communicating over whatever suitable (HTTP, TCP/sockets, ZeroMQ etc). Also, Gevent does not use threads, to be accurate. – Erik Kaplun Oct 07 '13 at 15:57
  • Btw see http://www.slideshare.net/mahendram/scaling-django-with-gevent and http://stackoverflow.com/questions/10964571/how-to-combine-django-plus-gevent-the-basics and just Google "django gevent". – Erik Kaplun Oct 07 '13 at 15:58

1 Answers1

1

EDIT: yes, if you call g.kill(), the task "contained" by g is also killed ASAP (i.e. at the first cooperative context switch) ...and, well, that's kind of the point, if you think of it.

In Gevent, this is how you'd do it:

import gevent

def task():
    for _ in range(10):
        print "processing"
        gevent.sleep(1.0)
    return 'result'

task_g = gevent.spawn(task)
res = task_g.join(timeout=3.0)
if res is None:
    print "task timed out but is continuing"
else:
    print "got result: ", res

If you instead prefer an exception based flow, you can use with_timeout; this also has the advantage that you can return None from your task and not have it confused with a timeout:

import gevent
from gevent.timeout import Timeout, with_timeout

def task():
    ...

task_g = gevent.spawn(task)

try:
    result = with_timeout(3.0, task_g.join)
except Timeout:
    print "timeout"
    result = None
else:
    print "got result:", result

Later you can still kill the task if it "totally" times out by task_g.kill().

Erik Kaplun
  • 37,128
  • 15
  • 99
  • 111
  • I thought when my program finishes the lifetime of my greenlet (wrapped by gevents) will also be terminated regardless of results. Not sure how this works in context of a running django process. do you know? – brianray Oct 02 '13 at 16:30
  • Using `subprocess`? I think it should get killed if the parent process gets killed. – Erik Kaplun Oct 02 '13 at 16:38
  • @brianray: Can I somehow improve/amend my answer for it to be acceptable? – Erik Kaplun Oct 02 '13 at 17:18
  • @brianray: Cannot open the link. But I'm quite sure I'm right: greenlets continue to be referenced by the Gevent main loop (known as Hub), so even if their creator is garbage collected, they keep running; see https://gist.github.com/eallik/6798544 for proof. – Erik Kaplun Oct 02 '13 at 18:44
  • here it is again http://pastebin.com/Pu5Vk5i9 But in your example the return from spawn does not fall off the stack – brianray Oct 02 '13 at 19:02
  • @brianray: What do you mean? In the gist I gave you, the variable `g2_g` falls of the stack in the frame of `g1`, which is where it was created. In fact, whenever you call `spawn` and throw away the return value, it "falls off" immediately, but it doesn't mean the spawned greenlet would be killed or anything. – Erik Kaplun Oct 02 '13 at 19:28