How to stop BaseHTTPServer.serve_forever() in a BaseHTTPRequestHandler subclass?

Question

I am running my HTTPServer in a separate thread (using the threading module which has no way to stop threads...) and want to stop serving requests when the main thread also shuts down.

The Python documentation states that BaseHTTPServer.HTTPServer is a subclass of SocketServer.TCPServer, which supports a shutdown method, but it is missing in HTTPServer.

The whole BaseHTTPServer module has very little documentation :(

Answer 1

Another way to do it, based on http://docs.python.org/2/library/basehttpserver.html#more-examples, is: instead of serve_forever(), keep serving as long as a condition is met, with the server checking the condition before and after each request. For example:

import CGIHTTPServer
import BaseHTTPServer

KEEP_RUNNING = True

def keep_running():
    return KEEP_RUNNING

class Handler(CGIHTTPServer.CGIHTTPRequestHandler):
    cgi_directories = ["/cgi-bin"]

httpd = BaseHTTPServer.HTTPServer(("", 8000), Handler)

while keep_running():
    httpd.handle_request()

Answer 2

I should start by saying that "I probably wouldn't do this myself, but I have in the past". The serve_forever (from SocketServer.py) method looks like this:

def serve_forever(self):
    """Handle one request at a time until doomsday."""
    while 1:
        self.handle_request()

You could replace (in subclass) while 1 with while self.should_be_running, and modify that value from a different thread. Something like:

def stop_serving_forever(self):
    """Stop handling requests"""
    self.should_be_running = 0
    # Make a fake request to the server, to really force it to stop.
    # Otherwise it will just stop on the next request.
    # (Exercise for the reader.)
    self.make_a_fake_request_to_myself()

Edit: I dug up the actual code I used at the time:

class StoppableRPCServer(SimpleXMLRPCServer.SimpleXMLRPCServer):

    stopped = False
    allow_reuse_address = True

    def __init__(self, *args, **kw):
        SimpleXMLRPCServer.SimpleXMLRPCServer.__init__(self, *args, **kw)
        self.register_function(lambda: 'OK', 'ping')

    def serve_forever(self):
        while not self.stopped:
            self.handle_request()

    def force_stop(self):
        self.server_close()
        self.stopped = True
        self.create_dummy_request()

    def create_dummy_request(self):
        server = xmlrpclib.Server('http://%s:%s' % self.server_address)
        server.ping()

Answer 3

The event-loops ends on SIGTERM, Ctrl+C or when shutdown() is called.

server_close() must be called after server_forever() to close the listening socket.

import http.server

class StoppableHTTPServer(http.server.HTTPServer):
    def run(self):
        try:
            self.serve_forever()
        except KeyboardInterrupt:
            pass
        finally:
            # Clean-up server (close socket, etc.)
            self.server_close()

Simple server stoppable with user action (SIGTERM, Ctrl+C, ...):

server = StoppableHTTPServer(("127.0.0.1", 8080),
                             http.server.BaseHTTPRequestHandler)
server.run()

Server running in a thread:

import threading

server = StoppableHTTPServer(("127.0.0.1", 8080),
                             http.server.BaseHTTPRequestHandler)

# Start processing requests
thread = threading.Thread(None, server.run)
thread.start()

# ... do things ...

# Shutdown server
server.shutdown()
thread.join()

Answer 4

In my python 2.6 installation, I can call it on the underlying TCPServer - it still there inside your HTTPServer:

TCPServer.shutdown


>>> import BaseHTTPServer
>>> h=BaseHTTPServer.HTTPServer(('',5555), BaseHTTPServer.BaseHTTPRequestHandler)
>>> h.shutdown
<bound method HTTPServer.shutdown of <BaseHTTPServer.HTTPServer instance at 0x0100D800>>
>>> 

Answer 5

I think you can use [serverName].socket.close()

Answer 6

In python 2.7, calling shutdown() works but only if you are serving via serve_forever, because it uses async select and a polling loop. Running your own loop with handle_request() ironically excludes this functionality because it implies a dumb blocking call.

From SocketServer.py's BaseServer:

def serve_forever(self, poll_interval=0.5):
    """Handle one request at a time until shutdown.

    Polls for shutdown every poll_interval seconds. Ignores
    self.timeout. If you need to do periodic tasks, do them in
    another thread.
    """
    self.__is_shut_down.clear()
    try:
        while not self.__shutdown_request:
            # XXX: Consider using another file descriptor or
            # connecting to the socket to wake this up instead of
            # polling. Polling reduces our responsiveness to a
            # shutdown request and wastes cpu at all other times.
            r, w, e = select.select([self], [], [], poll_interval)
            if self in r:
                self._handle_request_noblock()
    finally:
        self.__shutdown_request = False
        self.__is_shut_down.set()

Heres part of my code for doing a blocking shutdown from another thread, using an event to wait for completion:

class MockWebServerFixture(object):
    def start_webserver(self):
        """
        start the web server on a new thread
        """
        self._webserver_died = threading.Event()
        self._webserver_thread = threading.Thread(
                target=self._run_webserver_thread)
        self._webserver_thread.start()

    def _run_webserver_thread(self):
        self.webserver.serve_forever()
        self._webserver_died.set()

    def _kill_webserver(self):
        if not self._webserver_thread:
            return

        self.webserver.shutdown()

        # wait for thread to die for a bit, then give up raising an exception.
        if not self._webserver_died.wait(5):
            raise ValueError("couldn't kill webserver")

Answer 7

This is a simplified version of Helgi's answer for python 3.7:

import threading
import time
from http.server import ThreadingHTTPServer, SimpleHTTPRequestHandler


class MyServer(threading.Thread):
    def run(self):
        self.server = ThreadingHTTPServer(('localhost', 8000), SimpleHTTPRequestHandler)
        self.server.serve_forever()
    def stop(self):
        self.server.shutdown()


if __name__ == '__main__':
    s = MyServer()
    s.start()
    print('thread alive:', s.is_alive())  # True
    time.sleep(2)
    s.stop()
    print('thread alive:', s.is_alive())  # False

Answer 8

This method I use successfully (Python 3) to stop the server from the web application itself (a web page):

import http.server
import os
import re

class PatientHTTPRequestHandler(http.server.SimpleHTTPRequestHandler):
    stop_server = False
    base_directory = "/static/"
    # A file to use as an "server stopped user information" page.
    stop_command = "/control/stop.html"
    def send_head(self):
        self.path = os.path.normpath(self.path)
        if self.path == PatientHTTPRequestHandler.stop_command and self.address_string() == "127.0.0.1":
            # I wanted that only the local machine could stop the server.
            PatientHTTPRequestHandler.stop_server = True
            # Allow the stop page to be displayed.
            return http.server.SimpleHTTPRequestHandler.send_head(self)
        if self.path.startswith(PatientHTTPRequestHandler.base_directory):
            return http.server.SimpleHTTPRequestHandler.send_head(self)
        else:
            return self.send_error(404, "Not allowed", "The path you requested is forbidden.")

if __name__ == "__main__":
    httpd = http.server.HTTPServer(("127.0.0.1", 8080), PatientHTTPRequestHandler)
    # A timeout is needed for server to check periodically for KeyboardInterrupt
    httpd.timeout = 1
    while not PatientHTTPRequestHandler.stop_server:
        httpd.handle_request()

This way, pages served via base address http://localhost:8080/static/ (example http://localhost:8080/static/styles/common.css) will be served by the default handler, an access to http://localhost:8080/control/stop.html from the server's computer will display stop.html then stop the server, any other option will be forbidden.

Answer 9

I tried all above possible solution and ended up with having a "sometime" issue - somehow it did not really do it - so I ended up making a dirty solution that worked all the time for me:

If all above fails, then brute force kill your thread using something like this:

import subprocess
cmdkill = "kill $(ps aux|grep '<name of your thread> true'|grep -v 'grep'|awk '{print $2}') 2> /dev/null"
subprocess.Popen(cmdkill, stdout=subprocess.PIPE, shell=True)

Answer 10

import http.server
import socketserver
import socket as sck
import os
import threading


class myserver:
    def __init__(self, PORT, LOCATION):
        self.thrd = threading.Thread(None, self.run)
        self.Directory = LOCATION
        self.Port = PORT
        hostname = sck.gethostname()
        ip_address = sck.gethostbyname(hostname)
        self.url = 'http://' + ip_address + ':' + str(self.Port)
        Handler = http.server.SimpleHTTPRequestHandler
        self.httpd = socketserver.TCPServer(("", PORT), Handler)
        print('Object created, use the start() method to launch the server')
    def run(self):
        print('listening on: ' + self.url )
        os.chdir(self.Directory)
        print('myserver object started')        
        print('Use the objects stop() method to stop the server')
        self.httpd.serve_forever()
        print('Quit handling')

        print('Sever stopped')
        print('Port ' + str(self.Port) + ' should be available again.')


    def stop(self):
        print('Stopping server')
        self.httpd.shutdown()
        self.httpd.server_close()
        print('Need just one more request before shutting down'


    def start(self):
        self.thrd.start()

def help():
    helpmsg = '''Create a new server-object by initialising
NewServer = webserver3.myserver(Port_number, Directory_String)
Then start it using NewServer.start() function
Stop it using NewServer.stop()'''
    print(helpmsg)

Not a experience python programmer, just wanting to share my comprehensive solution. Mostly based on snippets here and there. I usually import this script in my console and it allows me to set up multiple servers for different locations using their specific ports, sharing my content with other devices on the network.

Answer 11

Here's a context-flavored version for Python 3.7+ which I prefer because it cleans up automatically and you can specify the directory to serve:

from contextlib import contextmanager
from functools import partial
from http.server import SimpleHTTPRequestHandler, ThreadingHTTPServer
from threading import Thread


@contextmanager
def http_server(host: str, port: int, directory: str):
    server = ThreadingHTTPServer(
        (host, port), partial(SimpleHTTPRequestHandler, directory=directory)
    )
    server_thread = Thread(target=server.serve_forever, name="http_server")
    server_thread.start()

    try:
        yield
    finally:
        server.shutdown()
        server_thread.join()


def usage_example():
    import time

    with http_server("127.0.0.1", 8087, "."):
        # now you can use the web server
        time.sleep(100)