The main problem is that you use nginx.conf as the file name. You need the fully-qualified file name (with drive and directory). If the file resides in the same directory as your EXE, you should do
ShellExecute(Handle, nil,
  PChar(ExtractFilePath(Application.ExeName) + 'nginx.conf'),
  nil, nil, SW_SHOWNORMAL)
There is no need to set the directory, and you should normally use SW_SHOWNORMAL.
Also, this only works if the system running the application has the file associations set up properly for .conf files. If the system running the application opens .conf files with MS Paint, then the line above will start MS Paint. If there are no associations at all, the line won't work.
You can specify manually to use notepad.exe:
ShellExecute(Handle, nil, PChar('notepad.exe'),
  PChar(ExtractFilePath(Application.ExeName) + 'nginx.conf'),
  nil, SW_SHOWNORMAL)
Now we start notepad.exe and pass the file name as the first argument.
Third, you shouldn't use try..except the way you do now. The ShellExecute may fail for other reasons than 'invalid config path', and in any case, it won't raise an exception. Instead, consider
if FileExists(...) then
  ShellExecute(...)
else
  MessageBox(Handle, 'Invalid path to configuration file', 'Error', MB_ICONERROR)
Now, back to the main issue. My first code snippet only works if the system running your application happens to have an appropriate file association post for .conf files, while the second will always open Notepad. A better alternative might be to use the application used to open .txt files. David's answer gives an example of this.