Diary #1 Send Screenshot from Python to HTML Canvas

 I wanted to control a computer from another computer in another room, so here we are. So far, I've just completed the proof of concept, but the idea needs refinement, and won't be nearly as efficient as TeamViewer/etc. But more convenient and requires less CPU than TeamViewer.

What it does:

• Server: Python. Listens as HTTP server. Upon a GET request, it will take a screenshot and return it.
• Client: HTML/JavaScript. Webpage with a button. When the button is pressed, it sends a GET request to the server to get a screenshot.

The screenshot data will be scaled down to 512 width & proportionate height. The color palette will be limited to 256 (to attempt to save image size). And the format of the image as sent to the client will be 'raw' RGB image data. To transfer it to the client via HTTP, we'll encode it in hex (doubling its size. This could be changed to base64 to shave off 75% of the added overhead).

The client will receive the data, decode it from hex to a byte array, then apply it to an ImageData object (changing the color format from RGB to RGBA). We will then put it in the context of the canvas at 0,0 x/y offset. 

Since this is an informal diary kind of thing, I'll just paste the relevant code bits:

Server (response to client's GET request after pressing button):

from math import floor # used for my 'byteAryToHex' fn
from PIL import Image # PIL fns for getting screenshot
from PIL import ImageGrab

                        img = ImageGrab.grab() # get whole-screen screenshot
                        size = 512,512
                        img.thumbnail(size) # resize img to thumbnail 512x512
                        img.convert("P", palette=Image.ADAPTIVE, colors=256) # simplify color palette to reduce size
                        bytes = img.tobytes('raw', 'RGB')
                        self.wfile.write(byteAryToHex(bytes).encode('utf-8'))

Client (after got GET response from server):

console.log("got img msg. loading img from it...");
                imgRaw = decodeHex(args[1]);
                // now read it into canvas (rgb to rgba)
                px = new ImageData(512,512);
                idx = 0;
                for(var xint = 0, len = imgRaw.length;xint < len;xint++){
                    px.data[idx++] = imgRaw[xint++];
                    px.data[idx++] = imgRaw[xint++];
                    px.data[idx++] = imgRaw[xint];
                    px.data[idx++] = 255;
                }
                c.putImageData(px, 0,0);

 

Also noteworthy pearl of frustration from my past:  Firefox blocks the response received by the client's GET request if the address of the request does not match the current web page (CORS). So the python server that's taking the screenshot also has to host an HTTP server and serve the webpage.

It is working in my system, and at any time I can get a screenshot of the computer's screen by pressing the button. Just getting that far took a few hours of  hours of frustration, after attempting to store it as the entire image's data (PNG/JPEG) as base64 in a data:image/png, but no success for reasons I couldn't understand and didn't want to spend time debugging. I went ahead and sent them as pixels from server->client, as I am far more familiar with drawing pixels into a canvas.