Actually, no need to laugh—it sounds to me like you're doing this the correct way already. Since you don't own the desktop, you shouldn't draw directly on it. Instead, you need to simulate it by overlaying a transparent form that you do own, and then drawing on that. Because you own the transparent overlay form, it's no problem to draw on it.
But beyond that, it sounds like you're just trying values randomly without a clear understanding of what they actually do. That's like throwing darts with your eyes closed. You won't have a very high hit count.
Let's start by understanding what your code does. The magic value 0x84 corresponds to the WM_NCHITTEST message, which is sent by Windows to a window to determine how mouse clicks on that window should be handled. In response to that message, you reply with one of the HT* values, given in the linked documentation. Each of those values has a particular meaning, also explained in the documentation. For example:
HTCAPTION (which has a value of 2) means that the clicked portion of the window should be treated as the window's caption/title bar. You know from using Windows that you can drag windows around on the screen using the title bar, so it makes sense that returning HTCAPTION in response to mouse clicks would allow your window to be draggable. You'll see this used on borderless forms (i.e., those with no title bar) to allow them to be movable.
HTTRANSPARENT (which has a value of -1) is another available value. This one's pretty simple. It just makes your window look transparent. It's like saying "don't mind me, there's no window here!" Mouse clicks are simply passed on to the window that lies below yours in the Z order as if you weren't there.
HTCLIENT (a value of 1) is the default result when the click occurs anywhere on the window's client area. You would return this (or simply call the default window procedure) when you want everything to work normally. Click events that return this value would go on to be processed normally by the framework, raising either the form's Click event, or getting passed on to child controls located on the form.
So, when you're not drawing, you probably want to return HTTRANSPARENT. When you are drawing, you probably want to return HTCLIENT so that your drawing code can see the mouse events and draw the result.
Fixing your code, then:
// Code for allowing clicking through of the form
protected override void WndProc(ref Message m)
{
const uint WM_NCHITTEST = 0x84;
const int HTTRANSPARENT = -1;
const int HTCLIENT = 1;
const int HTCAPTION = 2;
// ... or define an enum with all the values
if (m.Msg == WM_NCHITTEST)
{
// If it's the message we want, handle it.
if (penMode)
{
// If we're drawing, we want to see mouse events like normal.
m.Result = new IntPtr(HTCLIENT);
}
else
{
// Otherwise, we want to pass mouse events on to the desktop,
// as if we were not even here.
m.Result = new IntPtr(HTTRANSPARENT);
}
return; // bail out because we've handled the message
}
// Otherwise, call the base class implementation for default processing.
base.WndProc(ref m);
}