[Update] 2020-05-01
Now jest cli supports option --runTestsByPath, so that you can explicitly specify a file path rather than a pattern, which allows you to use \ on windows.
So the following launch.json should work:
{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "Jest Current File",
      "type": "node",
      "request": "launch",
      "args": [
        "node_modules/jest/bin/jest.js",
        "--runInBand",
        "--runTestsByPath",
        "${relativeFile}"
      ],
      "console": "integratedTerminal",
      "internalConsoleOptions": "neverOpen",
      "cwd": "${workspaceFolder}"
    }
  ]
}
The following is the original answer:
It seems that jest doesn't plan to work with \ and vscode doesn't plan to provide features to replace chars in Predefined variables.
But there are some workarounds:
- use - ${fileBasename}instead of- ${relativeFile}
 
- Or use input variables so that vscode prompt you to input custom test name when you debug. 
Here is an example launch.json for the above two workarounds.
{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "Jest Current FileBaseName",
      "type": "node",
      "request": "launch",
      "args": [
        "node_modules/jest/bin/jest.js",
        "--runInBand",
        "${fileBasename}"
      ],
      "console": "integratedTerminal",
      "internalConsoleOptions": "neverOpen",
      "cwd": "${workspaceRoot}"
    },
    {
      "name": "Jest Custom",
      "type": "node",
      "request": "launch",
      "args": [
        "node_modules/jest/bin/jest.js",
        "--runInBand",
        "${input:testName}"
      ],
      "console": "integratedTerminal",
      "internalConsoleOptions": "neverOpen",
      "cwd": "${workspaceRoot}"
    }
  ],
  "inputs": [
    {
      "type": "promptString",
      "id": "testName",
      "description": "The file or name you want to test",
      "default": "${fileBaseName}"
    }
  ]
}