There are several ways to do this, all of which have their advantages and disadvantages, and googling will give you a lot of discussion about it.
The methods you describe have the weaknesses you mention.  Also you need to consider how unique you want your instance to be:
- per WindowStation 
- per machine 
I have a class that can be used to force a single instance per WindowStation for Windows Forms applications.  Basically it uses P/Invoke to do the following during startup, protecting against a race condition using a Mutex:
- Define a fixed unique string to identify your app (e.g. a GUID) 
- Obtain a Mutex whose name is derived from your unique string. 
- Call EnumWindows using P/Invoke to enumerate all main windows. 
- In the EnumWindows callback, call GetProp using P/Invoke on each window in turn, passing the window handle and your unique value as arguments.  If GetProp detects a window that has your unique key (i.e. another instance of your application), activate it (PostMessage(...SC_RESTORE...), SetForegroundWindow, SetFocus), then exit your new instance. 
- If you enumerate all windows without finding a previous instance, call SetProp using P/Invoke to set your unique value for your application's main form window handle. 
- Release the mutex and continue starting up. 
In my use case, this solution is better than simply holding a mutex for the lifetime of the process, because it gives a way to activate the previous instance if the user starts a second instance rather than just exiting.  Using GetProp/SetProp gives you a way of identifying your process instance without relying on the process name.