How to split a command over multiple lines in appveyor.yml

Question

I would like to split a long build command across multiple lines in my appveyor.yml file, however I cannot get it to unwrap, so the build is failing when the first FOR command gets cut off and returns an error. I am not sure how to correctly split the lines in the .yml file so that they are reassembled inside Appveyor. How can this be done?

Here is a simplified version:

build_script:
- cmd: >-
    @echo off
    FOR %%P IN (x86,x64) DO ( ^
      FOR %%C IN (Debug,Release) DO ( ^
        msbuild ^
          /p:Configuration=%%C ^
          /p:Platform=%%P ... ^
        || EXIT 1 ^
      ) ^
    )

I want it to appear in AppVeyor as this:

@echo off
FOR %%P IN (x86,x64) DO ( FOR %%C IN (Debug,Release) DO ( msbuild /p:Configuration=%%C /p:Platform=%%P ... || EXIT 1 ) )

Extra spaces are unimportant, the important bit is that the line beginning with FOR until the final ) appears on the same line.

Note that in theory it would also be acceptable for Appveyor to see this:

@echo off
FOR %%P IN (x86,x64) DO ( ^
  FOR %%C IN (Debug,Release) DO ( ^
    msbuild ^
      /p:Configuration=%%C ^
      /p:Platform=%%P ... ^
    || EXIT 1 ^
  ) ^
)

As the Windows cmd.exe interpreter would then see the continuation markers (^) at the end of each line and treat them as one big long command, except that Appveyor does not appear to recognise the ^ marker so it sends each line to cmd.exe one at a time, instead of sending the whole multi-line block together.

This means the first option looks like the only viable solution, where the YAML is constructed such that the FOR line and everything after it is combined into a single line.

I have tried:

Single spacing with no extra characters at the end of each line. According to this guide, single-spaced YML lines are supposed to be unwrapped into a single line, but this does not happen with Appveyor.
Double-spaced lines with no extra characters at the end of each line. This is supposed to make each line a separate command, and indeed they are, as the first FOR command fails with error 255 because it is incomplete (only the FOR line is present and not the rest of the loop.)
Double-spaced lines terminated with ^. Appveyor only runs each line one at a time, so I get an error 255 on the first incomplete FOR command.
Single-spaced lines terminated with ^ as shown above. Same issue as double-spaced lines, error 255 from an incomplete FOR command.
Ending each line with && ^ does actually work when running separate commands (e.g. multiple msbuild statements), but this won't work with FOR loops because you can't have && without a command preceding it.

Is there a trick to splitting a single cmd command over multiple lines in appveyor.yml?

Jens A. Koch · Answer 1 · 2021-02-21T18:55:52.473

How to split a command over multiple lines in appveyor.yml?

Here are some some syntax examples for batch, cmd, ps.

I hope these examples save you some time...

Syntax Examples

Batch

# please note the & at EOL in the next example
install:
    # Install VULKAN_SDK 
    - if not exist %VULKAN_SDK% (
       curl -L --silent --show-error --output Vulkan_SDK_Installer.exe https://sdk.lunarg.com/sdk/download/%VULKAN_VERSION%/windows/VulkanSDK-%VULKAN_VERSION%-Installer.exe?Human=true &
       Vulkan_SDK_Installer.exe /S
    )
    - dir %VULKAN_SDK%

before_build:
  - |-
    set MINGW32_ARCH=i686-w64-mingw32
  - if exist %PREFIX% set NEEDDEPENDS=rem

  # Depends
  - |-
    %NEEDDEPENDS% mkdir %PREFIX%\include\SDL2
    %NEEDDEPENDS% mkdir %PREFIX%\lib
    %NEEDDEPENDS% cd %TEMP%
    %NEEDDEPENDS% appveyor DownloadFile https://sourceforge.net/projects/gnuwin32/files/gettext/0.14.4/gettext-0.14.4-lib.zip
    %NEEDDEPENDS% mkdir gettext-0.14.4-lib
    %NEEDDEPENDS% move gettext-0.14.4-lib.zip gettext-0.14.4-lib
    %NEEDDEPENDS% cd gettext-0.14.4-lib
    %NEEDDEPENDS% 7z x gettext-0.14.4-lib.zip > nul
    %NEEDDEPENDS% copy include\* %PREFIX%\include > nul
    %NEEDDEPENDS% copy lib\* %PREFIX%\lib > nul
    %NEEDDEPENDS% cd ..

deploy_script:
  # if tagged commit, build/upload wheel
  - IF "%APPVEYOR_REPO_TAG%"=="true" IF NOT "%TESTENV%"=="check" (
      pip install twine &&
      python setup.py register &&
      twine upload -u %PYPI_USER% -p %PYPI_PASS% dist/*
    )

CMD

before_build:
    - cmd: >-     

        mkdir build

        cd .\build

        set OpenBLAS_HOME=%APPVEYOR_BUILD_FOLDER%/%MXNET_OPENBLAS_DIR%

        set OpenCV_DIR=%APPVEYOR_BUILD_FOLDER%/%MXNET_OPENCV_DIR%/build

        cmake .. -DOPENCV_DIR=%OpenCV_DIR% -DUSE_CUDA=0 -DUSE_CUDNN=0 -DUSE_NVRTC=0 -DUSE_OPENCV=1 -DUSE_OPENMP=1 -DUSE_BLAS=open -DUSE_DIST_KVSTORE=0 -G "Visual Studio 12 2013 Win64"

PS

install:
    - ps: >-

        git submodule init

        git submodule update

        if (!(Test-Path ${env:MXNET_OPENBLAS_FILE})) {

            echo "Downloading openblas from ${env:MXNET_OPENBLAS_PKG} ..."

            appveyor DownloadFile "${env:MXNET_OPENBLAS_PKG}" -FileName ${env:MXNET_OPENBLAS_FILE} -Timeout 1200000
        }

install:
      - ps: |
          Add-Type -AssemblyName System.IO.Compression.FileSystem
          if (!(Test-Path -Path "C:\maven" )) {
            (new-object System.Net.WebClient).DownloadFile('https://repo1.maven.org/maven2/org/apache/maven/apache-maven/3.3.9/apache-maven-3.3.9-bin.zip', 'C:\maven-bin.zip')
            [System.IO.Compression.ZipFile]::ExtractToDirectory("C:\maven-bin.zip", "C:\maven")
          }

on_success:
  - ps: |
  if ($true)
  {
    Write-Host "Success"
  }

Thanks for the samples. Which one of these reassembles multiple `.yml` lines back into a single-line command inside Appveyor? — Malvineous, Sep 10 '16 at 23:41
Careful when using additional parentheses inside an `IF condition ()`, on any command. It messes Batch parsing and gives an error. I solved that by removing the parens. `setlocal enabledelayedexpansion` might help here too, but I haven't tried it. — Steve Chavez, Nov 06 '20 at 23:25

vaab · Answer 2 · 2018-12-14T08:06:22.060

Use double-quotes:

build_script:
- cmd: "
    @echo off
    FOR %%P IN (x86,x64) DO (
      FOR %%C IN (Debug,Release) DO (
        msbuild
          /p:Configuration=%%C
          /p:Platform=%%P ...
        || EXIT 1
      )
    )"

You can check yaml ref, and more precisely this example.

I was confronted to the same issue using appveyor (what a strange limitation !). Using double-quotes allowed me to leverage YAML subtlety to :

keep on my side an easy to read/write multi-line expression,
have a single-line value effectively stored application wise.

You can have a look where I use it and see how the build passes.

For other readers, please note, that the line will be collapsed, and as a consequence, need to be written with a special syntax to support that... you can't omit the || in the expression given by the OP for instance.

score 3 · Answer 3 · answered Jun 05 '16 at 21:55

3

CMD commands are always broken into separate lines and run one-by-one by wrapping into .cmd files. Place your code to build.cmd, commit to repo and then call as:

build_script:
- build.cmd

answered Jun 05 '16 at 21:55

Feodor Fitsner

2,354
16
13

So no way to have the commands on separate lines in `appveyor.yml` but have them unwrapped so that they are executed on a single line within Appveyor? I would much prefer to keep all the Appveyor-related content in a single file. – Malvineous Jun 06 '16 at 04:47
Nope, sorry. Currently, the only way is putting that code into a batch file. – Feodor Fitsner Jun 07 '16 at 18:23
SO each line is placed into a separate `.cmd` file? – Brecht Machiels Feb 15 '17 at 11:53
1

It is possible to execute multi-line PowerShell scripts (using `- ps: |`), so you convert your CMD script to PS and use that instead. – Brecht Machiels Feb 15 '17 at 13:20

score 3 · Answer 4 · answered Oct 06 '20 at 21:49

There's another prompt to consider other then batch and PS. msys2/mingw32/mingw64 are installed on appveyor VMs C:\msys64\usr\bin\bash, in addition to the "Git for Windows" version of mingw64.

Unfortunately, getting a multiline command in bash to work in appveyor was less then simple:

Example:

A piece of code like this:

cd /tmp
for server in $(grep '^Server' /etc/pacman.d/mirrorlist.msys | awk '{print $3}' | shuf | arch=x86_64 envsubst); do
  echo Trying ${server}
  curl --connect-timeout 10 -LO ${server}msys2-keyring-r21.b39fb11-1-any.pkg.tar.xz && break
done"

becomes

  - >-
    C:\msys64\usr\bin\bash -lc "
    cd /tmp;
    for server in $(grep $'\x5eServer' /etc/pacman.d/mirrorlist.msys | awk '{print $3}' | shuf | arch=x86_64 /usr/bin/envsubst); do
    :;  echo Trying ${server};
    :;  curl --connect-timeout 10 -LO ${server}msys2-keyring-r21.b39fb11-1-any.pkg.tar.xz && break;
    done"

Why is it so different?

A number of problems happen with a multiline bash string in batch in appveyor:

A multiline string in batch has to end in a ^. However doing this

# Literal newlines were not working
- |
  bash -c "echo Not ^
           echo good"

# Double quotes become a literal quote
# Equivalent: echo This is""echo now dangling
#             and "" is an escaped "
- |
  bash -c "echo This is"^
          "echo now dangling"

# Double double quote cancel out, but you're back
# to a literal newline, not working
- |
  bash -c "echo This does not"^
          """echo work either"

# The `|` notation does not seem to be useful here
# It just results in a string that won't run in appveyor 
# as we desired, when using literal newlines.

Folding needs to all be on the same indent, so no extra indenting allowed

# > vs >- vs >+ isn't important, it just strips extra newlines at the end
- >
  bash -c "echo This
  echo works"

# Indent means literal newline again
- >
  bash -c "echo Does not
           echo work"

# Says "This echo says", not "This" and "says"
- >-
  bash -c "
  echo This
  echo says"

# Basically: bash -c "echo This; echo works"
- >-
  bash -c "
  echo This;
  echo works"

In order to have proper indentation, we need to trick yaml with some dummy characters

# You cannot have ; after keywords then, do, etc...
- >-
  bash -c "
  if [ 1 == 1 ]; then
  ;  echo No;
  ;  echo good;
  fi

# Add in a dummy "true" + semicolon, as a no-op
- >-
  bash -c "
  if [ 1 == 1 ]; then
  :;  echo This works;
  fi

# Or if you prefer this style
- >-
  bash -c "if [ 1 == 1 ]; then
  :;         echo ""This also works"";
  :;       fi

The infamous caret. Apparently something expands ^ to ^^, and there was no good way around it

# echoes ^^
bash -c "echo ^"
bash -c "echo \^"
bash -c "echo \\^"

# echoes ^^^^ 
bash -c "echo ^^"

# Uses a hex notation to get around the issue
bash -c "echo $'\x5e'"

You have to be careful with some command in mingw64, as they introduce \r\n when in bash you only need a \n

# This is really using /mingw64/bin/envsubst in MINGW64 mode
bash -c "for x in $(echo $'${PWD}\n${OLDPWD}' | envsubst); do
         :;  echo ""${x}"" | xxd;
         done"

# Either need to have MSYSTEM set to MSYS, or
bash -c "for x in $(echo $'${PWD}\n${OLDPWD}' | /usr/bin/envsubst); do
         :;  echo ""${x}"" | xxd;
         done"

# or, as a last resort, use dos2unix
bash -c "for x in $(echo $'${PWD}\n${OLDPWD}' | envsubst | dos2unix); do
         :;  echo ""${x}"" | xxd;
         done"

Anthon · Answer 5 · 2016-06-04T16:17:33.030

If you know what appveyor expects (I don't), let's assume:

@echo off
FOR %%P IN (x86,x64) DO (
    FOR %%C IN (Debug,Release) DO ( msbuild /p:Configuration=%%C /p:Platform=%%P ... || EXIT 1 )
)

Then it is easy to generate the appropriate YAML by dumping it e.g. from Python:

import sys
import ruamel.yaml

appveyor_str = """\
@echo off
FOR %%P IN (x86,x64) DO (
    FOR %%C IN (Debug,Release) DO ( msbuild /p:Configuration=%%C /p:Platform=%%P ... || EXIT 1 )
)
"""

data = dict(build_script=[dict(cmd=appveyor_str)])

ruamel.yaml.round_trip_dump(data, sys.stdout)

gives you:

build_script:
- cmd: "@echo off\nFOR %%P IN (x86,x64) DO (\n    FOR %%C IN (Debug,Release) DO (\
    \ msbuild /p:Configuration=%%C /p:Platform=%%P ... || EXIT 1 )\n)\n"

(there are no spaces before any of the newlines in the above example)

Using the folded block style scalar (with >) gives you very little control over the folding of the scalar as you experienced. It is also impossible to escape sequences in a folded (or literal) block style scalar.

If your multiline string doesn't need escapes you can try to dump as a block style scalar:

import sys
import ruamel.yaml

appveyor_str = """\
@echo off
FOR %%P IN (x86,x64) DO (
    FOR %%C IN (Debug,Release) DO ( msbuild /p:Configuration=%%C /p:Platform=%%P ... || EXIT 1 )
)
"""

data = dict(build_script=[dict(cmd=ruamel.yaml.scalarstring.PreservedScalarString(appveyor_str))])

ruamel.yaml.round_trip_dump(data, sys.stdout)

which gives:

build_script:
- cmd: |
    @echo off
    FOR %%P IN (x86,x64) DO (
        FOR %%C IN (Debug,Release) DO ( msbuild /p:Configuration=%%C /p:Platform=%%P ... || EXIT 1 )
    )

(i.e. exactly what you put in)

If you right align everything, which is not as readable as you wish, and double newline after the first you get the required output:

import sys
import ruamel.yaml

yaml_str = """\
build_script:
- cmd: >-
    @echo off

    FOR %%P IN (x86,x64) DO (
    FOR %%C IN (Debug,Release) DO (
    msbuild
    /p:Configuration=%%C
    /p:Platform=%%P ...
    || EXIT 1
    )
    )
"""

data = ruamel.yaml.load(yaml_str)

print(data['build_script'][0]['cmd'])

gives:

@echo off
FOR %%P IN (x86,x64) DO ( FOR %%C IN (Debug,Release) DO ( msbuild /p:Configuration=%%C /p:Platform=%%P ... || EXIT 1 ) )

But you cannot indent (from the details of the folded block style scalar):

Lines starting with white space characters (more-indented lines) are not folded.

Unfortunately your output still mostly crams everything onto the same line, which is what I'm trying to avoid as it's difficult to read. I can put everything onto the one line manually in the `.yml` file but it's not at all easy to read, so I'm trying to figure out how to have real newlines in the `.yml` file. Appveyor expects everything on the same line (a newline terminates the command). — Malvineous, Jun 04 '16 at 13:08
@Malvineous I updated my answer with how to dump your appveyor string as literal block style scalar. If that is not working for you, can you please update your question to include the string as it should go into appveyor (especially wrt newlines and leading/trailing blanks) — Anthon, Jun 04 '16 at 14:06
Thanks for the update. I've updated the question to show what I'm after - the issue is that from `FOR` to the last `)` must all appear on the same line after the YAML is read, but in the `.yml` file I want them on different lines. — Malvineous, Jun 04 '16 at 15:23
I don't think you can get that with indenting, that confuses the unfolding. If appveyor strips your `^` you'll have to stick with left aligned code (see my update). — Anthon, Jun 04 '16 at 16:19

How to split a command over multiple lines in appveyor.yml

5 Answers5

Why is it so different?