Generally:
If you pass a command (-Command) or script file (-File) to execute to PowerShell's CLI (powershell.exe in Windows PowerShell, pwsh.exe in PowerShell [Core] v6+), PowerShell by default executes the command / script and then exits.
With a command or script specified for execution, you need to add the -NoExit switch if you want the new session to remain open.
Caveat: (Unless you call directly from within PowerShell with a script block), a positional argument - i.e., one neither preceded by -Command nor -File - is implicitly bound to:
-Command in Windows PowerShell
-File in PowerShell [Core] v6+.
Therefore, it's advisable to use the target parameter name explicitly.
With Start-Process (whose built-in alias on Windows - but not on Unix - is start):
Note: The primary use of Start-Process is to launch an independent process asynchronously in a new window. However, the latter isn't supported on Unix-like platforms, where Start-Process's utility is therefore limited.
start powershell { ECHO "hi" } happens to work from PowerShell (except that the window closes right after executing the command), but it's important to note that you cannot actually pass script blocks to Start-Process, only strings.
Start-Process accepts an executable name (implied -FilePath parameter), and an array of string arguments (implied -ArgumentList / -Args parameter).
If you pass a script block ({ ... }), it is automatically stringified, meaning that its literal string contents are used (stripped of the { and }) as the (only) -ArgumentList string value.
Thus, bypassing the unnecessary script-block creation, your command - with -NoExit applied as desired - should be (with explicitly named parameters; note that -Command and its argument must come last):
Start-Process -FilePath powershell -ArgumentList '-NoExit -Command ECHO "hi"'
Note:
While passing arguments individually, as an array to -ArgumentList is arguably conceptually cleaner, it is actually better to pass all arguments as a single string, using embedded quoting as necessary, due to a longstanding bug - see GitHub issue #5576.
Trying to open an interactive shell in a new window as a different user, via the -Credential parameter, is broken up to at least PowerShell 7.1, resulting in keyboard input getting blocked both in the new window and in the caller's window - see this answer for a workaround with runas.exe and GitHub issue #12129.