On vim, command-mode keys can be mapped through the ex command :map <key> <macro> and insert-mode keys can be mapped through :map! <key> <macro>. After mapped, the commands to remove the mapping from the command-mode keys and insert-mode keys are unmap <key> and unmap! <key> respectively.
This works well with command-mode keys, but with insert-mode keys the key expansion also works on the ex command line prompt: trying to type the key end up in the macro expansion taking place resulting in bad argument to the unmap! command (E474: Invalid Argument) or maybe the command might try to unmap some different key from the one intended (E31: No such mapping).
How can someone correctly remove a insert-mode mapped key on vim?