How to share file(s) between YAML Stages in Azure DevOps

Question

I am trying to deploy my node.js code to Azure Function App using Azure DevOps. I have created the following Azure DevOps pipeline using YAML.

The problem I am facing is that on the deploy step, my pipeline is failing because it is not able to find the package. When I look at the logs, I believe during the clean up activity between jobs/stages, the folders are cleaned. I have tried using other predefined variables like Build.ArtifactStagingDirectory but none works.

trigger:
  - master

variables:
  azureServiceConnection: 'mySvcCon'
  azureFuncApp: myFApp

stages:
  - stage: Build_1
    displayName: 'Build Stage'
    jobs:
      - job: build
        displayName: 'Build Node.js app'
        pool:
          vmImage: 'Ubuntu-16.04'

        steps:
          - task: NodeTool@0
            displayName: 'Install Node.js'
            inputs:
              versionSpec: '8.x'

          - script: |
              npm install
            displayName: 'npm install and build'

          - task: CopyFiles@2
            displayName: 'Copy necessary files'
            inputs:
              SourceFolder: '$(System.DefaultWorkingDirectory)'
              Contents: |
                **/*
                !.vscode/**/*
              TargetFolder: '$(System.DefaultWorkingDirectory)/copied'

          - task: PublishBuildArtifacts@1
            displayName: 'Publish artifact'
            enabled: true
            inputs:
              PathtoPublish: '$(Build.ArtifactStagingDirectory)/copied'
              publishLocation: filePath
              targetPath: '$(System.DefaultWorkingDirectory)/publish'

  - stage: Deploy_2
    displayName: 'Deploy Stage'
    jobs:
      - job: Deploy
        displayName: 'Deploy to Function App'
        pool:
          vmImage: 'Ubuntu-16.04'

        steps:
          - task: AzureRMWebAppDeployment@4
            displayName: 'AzureRM Function App deploy'
            inputs:
              ConnectionType: 'AzureRM'
              ConnectedServiceName: $(azureServiceConnection)
              WebAppKind: 'Function App'
              WebAppName: $(azureFuncApp)
              Package: '$(System.DefaultWorkingDirectory)/publish'

How do I share my artifact between the stages? The same pipeline works if I put all steps in the same job. But I want to separate them out.

@ShaykiAbramczyk, I think Azure Pipelines now allows "deploy" as a task of the build pipeline. The advantage with having it all in the build pipeline will be there would be a single YAML file to manage, hence trying to take that route. — TechiRik, Apr 15 '19 at 18:59
No. build should to build. not to deploy. in the release pipeline you take the build artifacts (your package) and deploy it. — Shayki Abramczyk, Apr 16 '19 at 05:48
@ShaykiAbramczyk, I don't think that applies with [multi-stage pipelines](https://devblogs.microsoft.com/devops/whats-new-with-azure-pipelines/) — Krzysztof Czelusniak, Jul 18 '19 at 18:45
Might be of interest: https://github.com/MicrosoftDocs/azure-devops-docs/issues/5703 — J.G.Sebring, Feb 16 '21 at 14:15
@ShaykiAbramczyk, No you! It seems to me that MS is giving the flexibility of building and deploying single services using their new YAML style pipelines. In my opinion the "Releases" section should to coordinate the deployments of a multi-service system all at the same time. It's MUCH easier to maintain a build and release pipeline if both reside in code (aka, putting them both into the azure-pipelines.yml file) — st0ve, Feb 21 '21 at 15:24
@st0ve You all right! after MS added all the "release" features to the multi-stage YAML pipelines I really recommended to use one yaml for build & deploy :) — Shayki Abramczyk, Feb 21 '21 at 15:51
@TechiRik seems that my answer solves your issue for most people! Could you please mark it as accepted please? — ccoutinho, Aug 09 '23 at 10:02

score 31 · Answer 1 · edited May 28 '21 at 14:03

As mentioned in the answer and in some comments, it is now possible to download previously published artifacts.

In the code below, I am publishing the scripts folder as an artifact named dropScripts, which I have on the root of my solution. This way I can use the scripts contained in that folder in later stages of the pipeline. In another stage, I download the dropScripts artifact, and then I run a powershell script (script20.ps1), which was contained in the scripts folder.

stages:
- stage: 'Build'
  jobs: 
  - job: 'Build'
    pool:
      vmImage: 'ubuntu-16.04'
    steps:

    (...)

    - task: CopyFiles@2
      displayName: 'Copy powershell scripts for later use in the pipeline'
      inputs:
        contents: 'scripts/**'
        targetFolder: '$(Build.ArtifactStagingDirectory)'
    - publish: '$(Build.ArtifactStagingDirectory)/scripts'
      displayName: 'Publish powershell scripts for later use'
      artifact: dropScripts

- stage: DoSomethingWithTheScriptsStage
  dependsOn: Build
  jobs: 
  - job: DoSomethingWithTheScriptsJob
    pool: 
      vmImage: 'windows-2019'
    steps:
      - download: current
        artifact: dropScripts
      - task: PowerShell@2
        inputs:
          filePath: '$(Pipeline.Workspace)\dropScripts\script20.ps1'

the `download` step is just a shortcut for the `DownloadPipelineArtifact` task. — Sven, Oct 14 '21 at 19:22
$(Pipeline.Workspace) saved my day... had been using the wrong variable!!!!!! — 72GM, Feb 04 '22 at 14:04

SliverNinja - MSFT · Answer 2 · 2019-11-15T13:01:11.833

In general - creating artifacts is typically done by Build Pipeline while deploying artifacts is done in a Release Pipeline. There is definitely the opportunity to perform both actions in a single build pipeline depending on your usage. Combining especially makes sense when you are just getting started with Azure Pipelines as the ecosystem can be overwhelming with the amount of capabilities available. There is publicized work on merging the release capabilities into the build capabilities to simplify onboarding.

Separating the pipelines does give you the benefit of retrying deploy if the deployment failed the first time - it really depends how quick your build time is. Also supporting deployment of the same bits across environments is easier if you want to manually trigger environment or ringed release propagation. The list for separating build/deploy grows exponentially once you dig into some of the power-user features of release stages.

For your approach to work - you could leverage the dependsOn YAML element to link the subsequent jobs to have an output dependency.

Build Pipeline - Dependency Chaining

jobs:
- job: InitialA
  steps:
  - script: echo hello from initial A
- job: InitialB
  steps:
  - script: echo hello from initial B
- job: Subsequent
  dependsOn:
  - InitialA
  - InitialB
  steps:
  - script: echo hello from subsequent

Update 11/15/19

Devops recently released download task to consume files across CI/CD boundaries. Pipeline Artifacts can also be used to share files across stages now.

I don't see how this works - Job "Subsequent" runs after the other 2 definitely, but, how does this answer the OP in that it shares files? As far as I see "Subsequent" does a complete checkout on a fresh VM, so no files can be shared... — JamesDill, May 21 '19 at 06:57
@JamesDill - good call-out! The [docs have been updated with artifacts publish and download which shows how to share files between jobs - not just code](https://learn.microsoft.com/en-us/azure/devops/pipelines/process/phases?tabs=yaml&view=azure-devops#artifact-download). You can also [access dependent job variable state using `dependencies` variable](https://learn.microsoft.com/en-us/azure/devops/pipelines/process/phases?tabs=yaml&view=azure-devops#conditions). — SliverNinja - MSFT, May 21 '19 at 14:09
Is this still actual after MS introduced multi-stage pipelines? — Monsignor, Nov 14 '19 at 12:51
None of these links explain _How to share file(s) between YAML Stages in Azure DevOps_ as per the original question. The best they suggest is copy then publish in stage 1 then download in stage 2. Which is very convoluted if you generate multiple artifacts in different stages and need to publish in one step at the end — Nick.Mc, Jul 13 '22 at 07:30

score 2 · Answer 3 · answered Feb 01 '23 at 07:36

Publish and download pipeline Artifacts to share between stages

Using Azure Pipelines, you can download artifacts from earlier stages in your pipeline or from another pipeline. You can also publish your artifact to a file share or make it available as a pipeline artifact.

Thanks @ccoutinho and @Bingchen Li

https://learn.microsoft.com/en-us/azure/devops/pipelines/artifacts/pipeline-artifacts?view=azure-devops&tabs=yaml

trigger:
- main
stages:
- stage: build
  jobs:
  - job: run_build
    pool:
      vmImage: 'windows-latest'
    steps:
    - task: VSBuild@1
      inputs:
        solution: '**/*.sln'
        msbuildArgs: '/p:DeployOnBuild=true /p:WebPublishMethod=Package /p:PackageAsSingleFile=true /p:SkipInvalidConfigurations=true /p:DesktopBuildPackageLocation="$(build.artifactStagingDirectory)\WebApp.zip" /p:DeployIisAppPath="Default Web Site"'
        platform: 'Any CPU'
        configuration: 'Release'

    - task: CopyFiles@2
      displayName: 'Copy scripts'
      inputs:
        contents: 'scripts/**'
        targetFolder: '$(Build.ArtifactStagingDirectory)'

    - publish: '$(Build.ArtifactStagingDirectory)/scripts'
      displayName: 'Publish script'
      artifact: drop

- stage: test
  dependsOn: build
  jobs:
  - job: run_test
    pool:
      vmImage: 'windows-latest'
    steps:
    - download: current
      artifact: drop
    - task: PowerShell@2
      inputs:
        filePath: '$(Pipeline.Workspace)\drop\test.ps1'

do the intermediate artifacts stay available after the pipeline finishes? I am trying to achieve a clean artifact list after the pipeline finishes and I don't want to have the shared files to stay published — Matus, Feb 21 '23 at 11:18

Nick.Mc · Answer 4 · 2022-07-13T07:59:26.727

Going by this SO question and this thread https://github.com/MicrosoftDocs/azure-devops-docs/issues/5703 the only way to share files between stages is publish then download

I have seperate stages doing database drift scripts. I need to capture all of these and publish as an artifact but to get this to work I need to publish from each stage and download and publish again at the end in a big chain.

I ended up publishing differently named artifacts from each stage, so now I have a bunch of single file artifacts. Whatever works.

How to share file(s) between YAML Stages in Azure DevOps

4 Answers4

Build Pipeline - Dependency Chaining

Update 11/15/19