nginx, node.js and socket.io - is there a working marriage?

Question

nginx is a killer static file server.

it can serve node.js, as in this example, but in a limited fashion.

but nginx is apparently unable to proxy websockets.

the only thing I found that might work is using HAProxy front end as per this article - but it's from October 6, 2011.

this has to be a common problem, but I'm not finding a very common solution.

Solution

(see https://github.com/bangkok-maco/barebone-node for complete solution and details)

ip testing schema:

127.0.0.12 - www.chat.nit - public, in /etc/hosts and haproxy
127.0.1.12 - internal nginx web server
127.0.2.12 - internal chat serving node.js socket.io

/etc/haproxy/haproxy.cfg:

global
 maxconn 4096
 nbproc 2
 daemon
 # user nobody
 log             127.0.0.1       local1 notice

defaults
 mode http

# listen on 127.0.0.12:80
frontend app
 bind 127.0.0.12:80
 mode tcp
 timeout client 86400000
 default_backend www_backend
 acl is_chat hdr_dom(Host) chat
 acl is_websocket path_beg /socket.io

 use_backend chat_socket_backend if is_websocket is_chat
 tcp-request inspect-delay 500ms
 tcp-request content accept if HTTP

# ngnix on 127.0.1.12:80
backend www_backend
 balance roundrobin
 option forwardfor
 mode http
 option httplog
 option httpclose
 timeout server 30000
 timeout connect 4000
 server w1 127.0.1.12:80 weight 1 maxconn 1024 check

# node (socket.io) on 127.0.2.12:80
backend chat_socket_backend
 balance roundrobin
 mode http
 option httplog
 option forwardfor
 timeout queue 5000
 timeout server 86400000
 timeout connect 86400000
 timeout check 1s
 no option httpclose
 option http-server-close
 option forceclose
 server s14 127.0.2.12:8000 weight 1 maxconn 1024 check

/etc/nginx/sites-enabled/www.chat.nit

server {
    listen   127.0.1.12:80;

    root /data/node/chat;
    index client.html;

    server_name www.chat.nit;

    # favicon.ico is in /images
    location = /favicon.ico$ { rewrite /(.*) /images/$1 last; }

    # standard includes
    location ^~ /(css|images|scripts)/ {
            try_files $uri =404;
    }

    # html page (only in root dir)
    location ~ ^/([-_a-z]+).html$ {
            try_files $uri =404;
    }

    error_page 404 /404.html;

    # redirect server error pages to the static page /50x.html
    #
    error_page 500 502 503 504 /50x.html;
    location = /50x.html {
            root /usr/share/nginx/www;
    }
}

chat (node.js): server.js

var app = require('http').createServer()
   , io = require('socket.io').listen(app);    
app.listen(8000,'127.0.2.12');

io.sockets.on('connection', function(socket) {
  ...
};

chat: client.html

<head>
  <script src="/scripts/socket.io/socket.io.js"></script>
  <script>
    var socket = io.connect('http://www.chat.nit:80'); 
    ...
  </script>
</head>

notes:

link socket.io client js into scripts/ directory

/.../scripts$ ln -s ../node_modules/socket.io/node_modules/socket.io-client/dist/ socket.io
/etc/default/haproxy (contrary to text, must set to work at all)

ENABLED=1
this version haproxy not logging. found kvz's write up on how to use rsyslogd via 127.0.0.1, but could not make it fly.
this solution is working - not sysadmin quality to be sure. (enhancements more than welcome.)

Actually HAProxy *is* working. I'm using it to proxy WebSockets and I'm very satisfied with it. From my experience proxying WebSockets (and other requests) is more important then "killer static file server", but actually it depends on your app. Also Node.js is not a bad static file server (for example with Express framework) - the technology is very similar to nginx (asynchronous one thread). — freakish, Jun 23 '12 at 13:10
can you post your HAProxy config as an answer? do you know if it's possible to split the output of HAProxy between nginx for non-ws and node.js for ws/wss? — cc young, Jun 23 '12 at 14:06
Yeah, splitting shouldn't be a problem. After all ws/wss can be recognized by `upgrade` header. — freakish, Jun 23 '12 at 14:40
You may be interested in checking out [tcp_proxy](https://github.com/yaoweibin/nginx_tcp_proxy_module) for Nginx, which can [proxy websockets](http://www.letseehere.com/reverse-proxy-web-sockets). — Michelle Tilley, Jun 23 '12 at 17:57

score 4 · Answer 1 · answered Mar 14 '13 at 10:32

4

It looks like that you can proxy WebSockets through nginx since v1.3.13

See http://nginx.org/en/docs/http/websocket.html for more details

answered Mar 14 '13 at 10:32

Art

23,747
29
89
101

score 2 · Accepted Answer · answered Jun 23 '12 at 14:39

Here's my (old and for testing purposes) HAProxy config for proxying WebSockets and normal HTTP requests.

global
    maxconn 4096
    nbproc 2
    daemon
    user nobody

defaults
    mode http

frontend app
    bind 0.0.0.0:8000
    mode tcp
    timeout client 86400000
    default_backend www_backend
    acl is_websocket path_beg /sockets

    use_backend socket_backend if is_websocket
    tcp-request inspect-delay 500ms
    tcp-request content accept if HTTP

backend www_backend
    balance roundrobin
    option forwardfor
    mode http
    option httplog
    option httpclose
    timeout server 30000
    timeout connect 4000
    server w1 localhost:8080 weight 1 maxconn 1024 check

backend socket_backend
    balance roundrobin
    mode http
    option httplog
    option forwardfor
    timeout queue 5000
    timeout server 86400000
    timeout connect 86400000
    timeout check 1s
    no option httpclose
    option http-server-close
    option forceclose
    server s1 localhost:8081 weight 1 maxconn 1024 check

Note that I am recognizing whether request is WS or not by looking at path (acl is_websocket path_beg /sockets line). This can be replaced with for example this:

acl is_websocket hdr(Upgrade) -i WebSocket

or this:

acl is_websocket hdr_beg(Host) -i ws

or both. Proxying to nginx with this config should work out of the box.

due to travel and natural ineptitude ;), took a little longer to test than I thought. bottom line - works great! thanks much!! have posted all pieces as appendix to question. — cc young, Jun 26 '12 at 12:30
Would you please have a look at what was posted? Appreciate any insight you might have. — cc young, Jun 26 '12 at 14:33