How i send big messages faster in python socket?

Question

I am trying to create some kind of screen share with python socket. The problem is that the images of my screen are very big (3,110,482‬ bytes) and it takes a lot of time for the socket to send them to the server. For making the sending more efficient I lowered the resolution of the images I am sending, but it is not enough. So I need to make the sending process more efficient.

Here is the function that takes images of my screen:

import numpy as np # for array manipulation
import pyautogui   # for recording images of the screen
import cv2         # for change the resolution of the images
from screeninfo import get_monitors # for getting the screen size

def get_screen_img(): # take an image of the screen and return it
    img = pyautogui.screenshot() # take a screenshot

    img = np.array(img) # convert to numpy array

    monitor = get_monitors()[0] # get info on the screen monitor
    # lowered the resolution by half
    img = cv2.resize(img, dsize=(monitor.width//2, monitor.height//2), interpolation=cv2.INTER_CUBIC)

    # do some manipulation for seeing the image right
    img = np.fliplr(img) # flip the image array
    img = np.rot90(img)  # rotate the image array in 90 degrees

    return img # return the image

Here is the function that sends the images:

import socket # for sending data
import pickle # for converting any kind of data to bytes

    def send_image(): # send a message
        # send the image and the type because I am sending more than just images so I need to tell the server what kind of info it gets
        msg = {"type": "image", "content": get_screen_img()}            

        msg = pickle.dumps(msg) # convert the message to bytes

    cSocket.send(msg) # send the msg

Edit: I am 99% sure that the problem is the size of the message. When I lowered the resolution more it works fine, but I need to send images in normal resolution.

Could you time execution of `get_screen_img()` and `Socket.send()`? From my understsnding, pyautogui is entirely written in Python but opencv, numpy, socket are written in C++ or C which are very fast. You may consider to send pre-created image a few times to test the actual average performance. — yoonghm, Mar 29 '20 at 08:29
You can't send faster than the receiver is receiving. Possibly the problem is at the other end. Or possibly the problem is in creating the data, not sending it? — user207421, Mar 29 '20 at 08:43
i am 99% sure that the problam is the size of the massege. when i lowered the resolution more all work fine. But i need to send images in normal resolution. — liad inon, Mar 29 '20 at 08:49
I check if the problem is to recive the massage but it wasn't. — liad inon, Mar 29 '20 at 08:53
Remote desktop tools tackle this problem by sending only the windows that have changed, and by caching heavily on the client, e.g. to make window moves fast. That's why static windows are fast in these tools, whereas animations or transparency effects are much slower. This requires considerable heavy lifting on both the client and the server ends, but you get pixel-perfect rendering at the receiver with little bandwidth use. The next best thing to try is to encode the whole screen into a video stream at the server and send that, like streaming tools such as [OBS](https://obsproject.com/) do. — Tomalak, Mar 29 '20 at 09:03
You can't send a byte array any faster than you're doing now. Show us the receiving code, and time the code that produces the bye array you're sending. Maybe you could compress the image, e.g. as a JPEG. NB The word is 'message', not 'massage'. — user207421, Mar 29 '20 at 09:06
You should tackle this as an algorithmic problem, not brute force. With a fixed bandwidth, there are only so many bytes you can send per time. Look into compression of individual pictures (removing duplicate data in a single picture), as well as compression of the stream (removing duplicate data between pictures), and logical compression (treating the screen not as an image but as windows with different requirements). — MisterMiyagi, Mar 29 '20 at 10:44

liad inon · Accepted Answer · 2020-03-29T11:19:19.520

2

So I find a solution. I compress the image with numpy and io like this:

img = get_screen_img()
compressed_img = io.BytesIO()
np.savez_compressed(compressed_img, img)

And decompress like this:

compressed_img.seek(0)
decompressed_img = np.load(compressed_img])['arr_0']

So if you have a efficiency problem with sending big messages with socket. I think that the best solution is to compress the message. You can also can compress with zlib or ather library.

Also in my case is work more faster if you dont send new frame if is the same to previous frame I send.

edited Mar 29 '20 at 11:19

answered Mar 29 '20 at 10:56

liad inon

243
1
12

1

You can make it even better/faster/more efficient with delta refreshes. With this technique you compress the change between the current screen and the last screen. Use numpy's ability to XOR arrays. For an example to see how I did it, see my answer here https://stackoverflow.com/questions/48950962/screen-sharing-in-python/67694466#67694466 – May 26 '21 at 21:33

How i send big messages faster in python socket?

1 Answers1