0

I am trying to screenshot a Microsoft Edge window using pywin32. This screenshot will then be used to a machine learning algorithm to play a game in Microsoft Edge. As you might guess, the program will be taking a screenshot multiple times, so I needed the screenshot to be fast as possible. To increase the speed, my program will resize the Microsoft Edge window to a small resolution (specifically, to 600 by 600). However, when the screenshot doesn't show the entire window even though I have moved it to a specified location.

My program:

import win32gui 
import win32ui 
import win32con 
import win32api  
from PIL import Image
import time



# grab a handle to the main desktop window 
hdesktop = win32gui.GetDesktopWindow() 

 
# determine the size of all monitors in pixels 
width = 600
height = 600 
left = 0 
top = 0 
 
# set window to correct location
print("You have 3 second to click the desired window!")
for i in range(3, 0, -1):
    print(i)
    time.sleep(1)
hwnd = win32gui.GetForegroundWindow()
win32gui.MoveWindow(hwnd, 0, 0, width, height, True)
 
# create a device context 
desktop_dc = win32gui.GetWindowDC(hdesktop) 
img_dc = win32ui.CreateDCFromHandle(desktop_dc) 
 
# create a memory based device context 
mem_dc = img_dc.CreateCompatibleDC() 
 
# create a bitmap object 
screenshot = win32ui.CreateBitmap() 
screenshot.CreateCompatibleBitmap(img_dc, width, height) 
mem_dc.SelectObject(screenshot) 
 
 
# copy the screen into our memory device context 
mem_dc.BitBlt((0, 0), (width, height), img_dc, (left, top),win32con.SRCCOPY) 
 

bmpinfo = screenshot.GetInfo()
bmpstr = screenshot.GetBitmapBits(True)
im = Image.frombuffer(
    'RGB',
    (bmpinfo['bmWidth'], bmpinfo['bmHeight']),
    bmpstr, 'raw', 'BGRX', 0, 1)

im.show()
# free our objects 
mem_dc.DeleteDC() 
win32gui.DeleteObject(screenshot.GetHandle())

My program first move and resizes the desired window (taken from win32gui.GetForegroundWindow()) by win32gui.MoveWindow(hwnd, 0, 0, width, height, True) Then, it tries to screenshot the window by taking the whole desktop window (hdesktop = win32gui.GetDesktopWindow() ) and then cropping it to the desired coordinates (mem_dc.BitBlt((0, 0), (width, height), img_dc, (left, top),win32con.SRCCOPY) ). I then convert the win32 screenshot to a PIL image so I could look at it. Note that the desired coordinates are the SAME coordinates used to move the window in the first place. However, when I try to run this program, the screenshot doesn't capture the entire window!

I have tried looking at the documentation of the MoveWindow and the BitBlt function, but I couldn't find the issue. The destination and source rectangle parameters is suppose to be (0,0), since of the MoveWindow function. The width and height parameters are the same. I also have tried experimenting with the bRepaint parameter, but it didn't make a difference.

Any Suggestions?

Eshaan Barkataki
  • 35
  • 1
  • 7
  • I would suggest using PIl's [`ImageGrab`](https://pillow.readthedocs.io/en/latest/reference/ImageGrab.html#imagegrab-module) module, which is very fast (and easy to use). – martineau May 15 '21 at 19:29
  • I considered that, but I heard that win32 was faster – Eshaan Barkataki May 15 '21 at 19:34
  • 1
    Premature optimization is the root of all evil… – martineau May 15 '21 at 19:37
  • I will need to optmizing everything since I am using it for a ML application – Eshaan Barkataki May 15 '21 at 19:58
  • 1
    Get it to work, then optimize it if necessary. – martineau May 15 '21 at 20:02
  • In order to get it work, it has to be optimized. If I miss a single frame then the AI will be dead in the game – Eshaan Barkataki May 15 '21 at 20:03
  • I've heard all the rationalizations before — see [Premature optimization](http://wiki.c2.com/?PrematureOptimization) and [When to optimize](https://en.wikipedia.org/wiki/Program_optimization#When_to_optimize). – martineau May 15 '21 at 20:17
  • Hmmm you have a point. I will consider screenshotting with ImageGrab. If it becomes too slow, then I guess I could optimize tons of other areas in my code. – Eshaan Barkataki May 15 '21 at 20:24
  • If it becomes too slow, ***profile*** it, then try to optimize the important parts. See [How can you profile a Python script?](https://stackoverflow.com/questions/582336/how-can-you-profile-a-python-script) – martineau May 15 '21 at 20:44
  • Nice, its good that there's already a tool that exists to profile code. I will definitely use that . – Eshaan Barkataki May 15 '21 at 20:50
  • There is also a third-party [line-profiler](https://pypi.org/project/line-profiler/) that can be quite useful when/if you get to that point. – martineau May 15 '21 at 20:54
  • Please read the discussion @martineau – Eshaan Barkataki May 16 '21 at 00:18
  • Perhaps the problem is the same issue as [win32gui MoveWindow() not aligned with left edge of screen](https://stackoverflow.com/questions/51694887/win32gui-movewindow-not-aligned-with-left-edge-of-screen). – martineau May 16 '21 at 06:48
  • First things first: You've decided to use Python, so you cannot possibly be interested in any sort of performance other than mediocre. As for your immediate problem, you'll have to make your application DPI-aware (see [High DPI Desktop Application Development on Windows](https://learn.microsoft.com/en-us/windows/win32/hidpi/high-dpi-desktop-application-development-on-windows)). Apart from that, retrieving (and setting) the true window size has been significantly more involved starting with Windows 8. – IInspectable May 16 '21 at 08:27
  • Oh ok, then I will start to use C++ (I am already familiar with the language). Thank you for the DPI-Aware documentation; I will begin reading it! – Eshaan Barkataki May 16 '21 at 14:01
  • What can I say? You're still young, so wasting a decade on C++ might not be that big of a deal. Though, for performance critical code, you'd probably want to choose FORTRAN. Either that, or Rust, which has the *potential* to rival FORTRAN in raw compute performance. The latter probably has a future, too. This might be a bit of a hyperbole, but the only practical use I have for my experience with C++ is **knowing** why Rust is all-around superior. Still, the above applies, you'll want to know *where* you are in the performance game, and how much (if any) headroom you have. – IInspectable May 16 '21 at 17:01
  • @IInspectable The reason why I choose C++ is because CUDA is written in it. Even though there are tons of wrappers for CUDA, using the original one is always best. If you don't know, CUDA enables programmers to use NVIDIA's GPUs, which are extremely popular among the deep learning community. In certain NVIDIA GPUs, there are tensor cores that could speed up deep learning tasks by a lot. The only way you can program those cores is by using the CUDA library. Using a programming language depends not only on the raw performance, but the libraries that support it and the tasks you could do with it. – Eshaan Barkataki May 16 '21 at 18:23
  • I also tried using `ctypes.windll.shcore.SetProcessDpiAwareness(1)` (trying python still, most of the ML code is already written in it, so I shall go all in :D ) before running `MoveWindow`, but the problem still occurs. It seems like it's resizing the width higher than the height, even though the width and height is suppose to be equal. I am on windows 10 – Eshaan Barkataki May 17 '21 at 02:07

1 Answers1

0

After experimenting with this question a little bit more, I finally found the problem. In the comments, I said that ctypes.windll.shcore.SetProcessDpiAwareness(1) doesn't work. However, it did. When I upscale the height and width, then the dimensions between the screenshot and the window fits perfectly. However, the reason why width and height doesn't work for smaller dimensions (I was originally setting width and height to 500) is because Microsoft Edge doesn't allow it. If the width goes within a certain threshold, then the actual width of the window would go to the smallest width Microsoft Edge wants it to be. An easy work around was the set the width and height to a larger resolution, and it worked!

Thank you so much for everyone in the comments, especially @IInspectable.

Eshaan Barkataki
  • 35
  • 1
  • 7