Here is my approach, which provides a consistent way of navigating between active terminals as well as jumping between the terminal and editor panes. You can try adding this to your keybindings.json directly but I would recommend you go through the keybinding UI (cmd+K cmd+S on a Mac) so you can review/manage conflicts etc.
With this I can use ctrl+x <arrow direction> to navigate to any visible editor or terminal. Once the cursor is in the terminal section you can use ctrl+x ctrl+up or ctrl+x ctrl+down to cycle through the active terminals (note that moving between on-screen split terminals is done with ctrl+x left or ctrl+x right).
cmd-J is still used to hide/show the terminal pane.
    {
        "key": "ctrl+x right",
        "command": "workbench.action.terminal.focusNextPane",
        "when": "terminalFocus"
    },
    {
        "key": "ctrl+x left",
        "command": "workbench.action.terminal.focusPreviousPane",
        "when": "terminalFocus"
    },
    {
        "key": "ctrl+x ctrl+down",
        "command": "workbench.action.terminal.focusNext",
        "when": "terminalFocus"
    },
    {
        "key": "ctrl+x ctrl+up",
        "command": "workbench.action.terminal.focusPrevious",
        "when": "terminalFocus"
    },
    {
        "key": "ctrl+x up",
        "command": "workbench.action.navigateUp"
    },
    {
        "key": "ctrl+x down",
        "command": "workbench.action.navigateDown"
    },
    {
        "key": "ctrl+x left",
        "command": "workbench.action.navigateLeft",
        "when": "!terminalFocus"
    },
    {
        "key": "ctrl+x right",
        "command": "workbench.action.navigateRight",
        "when": "!terminalFocus"
    },
This is a work in progress as I'm trying to mimic my emacs behavior in VS Code, but some key elements are missing, like being able to navigate to a terminal buffer by name (without using a mouse/dropdown). Overall I'm pretty surprised by the poor editor group/pane navigation UX in an otherwise amazing product, but working on getting used to it.