When DoDragDrop is not called on the mouse down event, mouse is captured and mouse up event is received. But when DoDragDrop is called, QueryContinueDrag is endlessly called and I don't get the mouse up event, even after I released the mouse.
You don't get a mouse up event because DoDragDrop() blocks your UI message loop until the drag operation is completed. So you need to use the input flags given to your QueryContinueDrag() implementation to decide whether to continue dragging, perform the drop, or abort the operation.
If you start the drag on a left mouse down, return S_OK if the grfKeyState parameter includes the MK_LBUTTON flag, and return DRAGDROP_S_DROP if the MK_LBUTTON flag is cleared. Return DRAGDROP_S_CANCEL if the fEscapePressed parameter is true. This is documented on MSDN:
IDropSource::QueryContinueDrag method
Parameters
fEscapePressed
Indicates whether the Esc key has been pressed since the previous call to QueryContinueDrag or to DoDragDrop if this is the first call to QueryContinueDrag. A TRUE value indicates the end user has pressed the escape key; a FALSE value indicates it has not been pressed.
grfKeyState
The current state of the keyboard modifier keys on the keyboard. Possible values can be a combination of any of the flags MK_CONTROL, MK_SHIFT, MK_ALT, MK_BUTTON, MK_LBUTTON, MK_MBUTTON, and MK_RBUTTON.
Return Value
This method can return the following values.
S_OK
The drag operation should continue. This result occurs if no errors are detected, the mouse button starting the drag-and-drop operation has not been released, and the Esc key has not been detected.
DRAGDROP_S_DROP
The drop operation should occur completing the drag operation. This result occurs if grfKeyState indicates that the key that started the drag-and-drop operation has been released.
DRAGDROP_S_CANCEL
The drag operation should be canceled with no drop operation occurring. This result occurs if fEscapePressed is TRUE, indicating the Esc key has been pressed.
Remarks
The DoDragDrop function calls QueryContinueDrag whenever it detects a change in the keyboard or mouse button state during a drag-and-drop operation. QueryContinueDrag must determine whether the drag-and-drop operation should be continued, canceled, or completed based on the contents of the parameters grfKeyState and fEscapePressed.
I get callbacks like QueryGetData(TYMED_HGLOBAL) or QueryGetData(TYMED_HGLOBAL, TYMED_ISTREAM, TYMED_GDI, TYMED_MFPICT, TYMED_ENHMF), but not for TYMED_FILE. Why isn't Notepad requesting that?
You can't use TYMED_FILE for CF_HDROP, you must use TYMED_HGLOBAL. And the content of the allocated HGLOBAL must be a DROPFILES struct followed by a double-null-terminated list of file paths. This is documented on MSDN:
Shell Clipboard Formats
CF_HDROP
This clipboard format is used when transferring the locations of a group of existing files. Unlike the other Shell formats, it is predefined, so there is no need to call RegisterClipboardFormat. The data consists of an STGMEDIUM structure that contains a global memory object. The structure's hGlobal member points to a DROPFILES structure as its hGlobal member.
The pFiles member of the DROPFILES structure contains an offset to a double null-terminated character array that contains the file names. If you are extracting a CF_HDROP format from a data object, you can use DragQueryFile to extract individual file names from the global memory object. If you are creating a CF_HDROP format to place in a data object, you will need to construct the file name array.
The file name array consists of a series of strings, each containing one file's fully qualified path, including the terminating NULL character. An additional null character is appended to the final string to terminate the array. For example, if the files c:\temp1.txt and c:\temp2.txt are being transferred, the character array looks like this:
c:\temp1.txt'\0'c:\temp2.txt'\0''\0'
Note
In this example, '\0' is used to represent the null character, not the literal characters that should be included.
If the object was copied to the clipboard as part of a drag-and-drop operation, the pt member of the DROPFILES structure contains the coordinates of the point where the object was dropped. You can use DragQueryPoint to extract the cursor coordinates.
If this format is present in a data object, an OLE drag loop simulates WM_DROPFILES functionality with non-OLE drop targets. This is important if your application is the source of a drag-and-drop operation on a Windows 3.1 system.