Python-socketio not emitting events sometimes on FastAPI Heroku server

Question

I have a FastAPI server running on Heroku, using python-socketio(5.3.0) to handle socket connections. I have an endpoint that is called when an event needs to be emitted to a certain socket. The problem I am facing is that the event doesn't emit every single time. It only emits some of the times. It's working perfectly on local.

FastAPI server with python-socketio:

import uvicorn
from fastapi import FastAPI
import socketio
from socketio.asyncio_namespace import AsyncNamespace
import json

notification_users = {}
notification_sockets = {}

class NotificationConnector(AsyncNamespace):

   async def on_connect(self, sid, *args, **kwargs):
     print("User connected: ", sid)

   async def on_disconnect(self, sid):
     user_id = chat_sockets.get(sid)
     print("User unsubscribed to chats: ", user_id)
     connection_list = chat_users.get(user_id)
     for socket in connection_list:
         if(socket==sid):
             connection_list.remove(sid)
    
     if connection_list == []:
         chat_users.pop(user_id)
     else:
         chat_users[user_id] = connection_list
    
     chat_sockets.pop(sid)

   async def on_subscribe(self, sid, data):
     user_id = str(data.get("userID"))
     print("User subscribed to notifications: ", user_id)
     if user_id in notification_users.keys():
         notification_users[user_id].append(sid)
     else:
         notification_users[user_id] = [sid]

     notification_sockets[sid] = user_id

fast_app = FastAPI()

sio = socketio.AsyncServer(
  async_mode='asgi',
  cors_allowed_origins='*',
  logger=True,
  engineio_logger=True
  )

sio.register_namespace(NotificationConnector("/notifications"))

app = socketio.ASGIApp(
  socketio_server=sio,
  other_asgi_app=fast_app,
  socketio_path='/socket.io/'
  )

@fast_app.post("/send")
async def send_notification(item: NotificationData):
  user_id = item.userId
  notification_content = json.dumps(item.notification)
  socket_list = notification_users.get(user_id,[])
  print(f"Socket list for {user_id} is {socket_list}")
  for socket in socket_list:
      print(f"Sending '{notification_content}' to socket Id: {socket}")
      await sio.emit('notifications', notification_content, room=socket, 
      namespace="/notifications")

return {"success":True},200

Are you running multiple workers with your application on heroku? Remember that separate workers = separate processes that are not sharing the memory, so if user connects to the socket, he will end up connected to a single worker and he will receive only events that are also triggered on the same worker. — GwynBleidD, Jun 09 '21 at 13:00
Yes, that was exactly what was happening. Changed that number of workers to 1 and that solved the issue. I probably shouldn't be using global variables for this. — itIsOkToAskStupidQuestions, Jun 09 '21 at 13:10
Yes, you'll probably need some external queue to handle that properly. — GwynBleidD, Jun 09 '21 at 14:11

score 1 · Accepted Answer · answered Jun 09 '21 at 13:08

The issue had nothing to do with python-socketio. It seems to be caused by how uvicorn deals with global variables. By default uvicorn creates multiple workers and each worker has its own copy of the global variables.

So, when a worker dealt with a /send POST request, it's copy of notification_users might not have had the information of the socket id to which the event needed to be emitted to.

Changing the number of workers to 1 fixed the issue. Will probably take a look at Redis instead of using global variables.

A better explanation of how uvicorn workers deal with global variables.

Python-socketio not emitting events sometimes on FastAPI Heroku server

1 Answers1