44

I have around 10 individual micro-services which are mostly cloud functions for various data processing jobs, which all live in a single github repository.

The goal is to trigger the selective deployment of these service to Google Cloud Functions, on push to a branch - when an individual function has been updated.

I must avoid the situation in which update of a single service causes the deployment of all the cloud functions.

My current repository structure:

/repo
--/service_A
----/function
----/notebook
--/service_B
----/function
----/notebook

On a side note, what are the pros/cons of using Github Actions VS Google Cloud Build for such automation?

dendog
  • 2,976
  • 5
  • 26
  • 63
  • do your `services` has dependencies on each other? if not , then this isn't a monorepo and you should change the title of your question to avoid confusing the topic of monorepos. – airtonix Apr 10 '23 at 01:25

5 Answers5

75

GitHub Actions supports monorepos with path filtering for workflows. You can create a workflow to selectively trigger when files on a specific path change.

https://help.github.com/en/articles/workflow-syntax-for-github-actions#onpushpull_requestpaths

For example, this workflow will trigger on a push when any files under the path service_A/ have changed (note the ** glob to match files in nested directories).

on:
  push:
    paths:
      - 'service_A/**'
Envek
  • 4,426
  • 3
  • 34
  • 42
peterevans
  • 34,297
  • 7
  • 84
  • 83
  • 2
    Ok thanks, that could work, but requires that each service has in essence its own workflow. – dendog Sep 27 '19 at 15:34
  • 2
    @dendog Yes, exactly. I think in many cases different services in a monorepo would be owned by different teams anyway, so it makes sense for each to have its own workflow. There could be a lot of duplicated workflow steps though, and I don't think GitHub Actions has any feature to share steps between workflows like, for example, CircleCI orbs. – peterevans Sep 27 '19 at 16:32
  • 2
    I know this is is an old thread now, be we have found you can do this by creating custom actions. For example, we host on IBM Cloud and use IBMCloud container registry, so we have a custom action for logging in to IBMCloud CLI which is used across multple jobs. – AndyRyan May 14 '21 at 15:52
  • 4
    downvoted because if this is how you're deploying apps our of your repo, then it's not a monorepo, it's just a repo with multiple apps. A monorepo is usually divided into two main areas: deliverables and dependencies. Main point here is that using path globs to your apps as the only means of detecting change means that you miss out on deploying changes in your dependencies. A better solution than path globbing is to use something that can combine what it knows about your git history and what it can determine your dependency graph looks like. Instead look at: https://nx.dev. – airtonix Jul 06 '22 at 01:29
  • Looks like there's a limit according to the docs, which should be kept in mind if relying on this: "Diffs are limited to 300 files. If there are files changed that aren't matched in the first 300 files returned by the filter, the workflow will not run. You may need to create more specific filters so that the workflow will run automatically." – Matt Welke Jul 31 '22 at 23:20
  • 2
    Please visit [monorepo.tools](https://monorepo.tools/), it helps to explain the actual benefits of monorepo - which are not creating a repetitive yaml for each project. – Missilexent Sep 08 '22 at 01:40
  • Can we nest .github folders? I am moving existing projects into the monorepo that already have their .github – Eric Burel Sep 22 '22 at 07:28
11

You could also run some script to discover which services were changed based on git diff and trigger corresponding job via GitHub REST API.

There could be two workflows main.yml and services.yml.

Main workflow will be configured to be started always on push and it will only start script to find out which services were changed. For each changed service repository dispatch event will be triggered with service name in payload.

Services workflow will be configured to be started on repository_dispatch and it will contain one job for each service. Jobs would have additional condition based on event payload.

See showcase with similar setup: https://github.com/zladovan/monorepo

Ján Čabala
  • 184
  • 1
  • 7
5

It's not a Monorepo

If you only have apps, then I'm sorry... but all you have is a repo of many apps.

A monorepo is a collection of packages that you can map a graph of dependencies between.

Aha, I have a monorepo

But if you have a collection of packges which depend on each other, then read on.

apps/
  one/
    depends:
      pkg/foo
  two/
    depends:
      pkg/bar
      pkg/foo
pkg/
  foo/
  bar/
  baz/

The answer is that you switch to a tool that can describe which packages have changed between the current git ref and some other git ref.

The following two examples runs the release npm script on each package that changed under apps/* and all the packges they would depend on.

I'm unsure if the pnpm method silently skips packages that don't have a release target/command/script.

Use NX Dev

Using NX.dev, it will work it out for you with its nx affected command.

  • you need a nx.json in the root of your monorepo
  • it assumes you're using the package.json approach with nx.dev, if you have project.json in each package, then the target would reside there.
  • your CI would then look like:
pnpx nx affected --target=release

Pnpm Filtering

Your other option is to switch to pnpm and use its filtering syntax:

pnpm --filter "...{apps/**}[origin/master]" release

Naive Path Filtering

If you just try and rely on "which paths" changed in this git commit, then you miss out on transient changes that affect the packages you actually want to deploy.

If you have a github action like:

on:
  push:
    paths:
      - 'app/**'

Then you won't ever get any builds for when you only push commits that change anything in pkg/**.

Other interesting github actions

airtonix
  • 4,772
  • 2
  • 38
  • 36
  • 3
    Honestly, I'm not sure why people down voted this answer, monorepo means so much more than having few projects or apps under the same root dir, you could share configurations and infrastructure definitions, define guide lines (such as eslint and prettier) and more. Id also recommend [turborepo](https://turborepo.org/) and [monorepo.tools](https://monorepo.tools/) to learn more about monorepo purpose – Missilexent Sep 08 '22 at 01:35
  • 1
    @Missilexent i think it's because CICD is a task that requires a lot of focus, many developers treat it as an afterthought so they feel that they can get away with just path filtering. It's all good, eventually they'll come back to my answer. – airtonix Sep 12 '22 at 01:39
  • @airtonix this answer is excellent, but I want more info! I like `nx affected` for CI, but the question title is about "Deploy". I'm looking for a way to determine what needs to be deployed with NPM workspaces after CI allowed reviewers to merge a feature branch. lerna seems overkill. I don't think nx has anything to help. – cyrf Oct 02 '22 at 02:37
  • @cyrf deploy === release. if you used to run some kind of path-sniffing-logic on some branch (master, release/*, environment/*) then you just run `nx affected --target=release`. You then need to ensure that your releasables have a "release" target that is responsible for deployment. This might be: push to another git repo, push to netlify, push to chromatic, push to vercel, build a docker container, push it to a deployment registry. Of course you can choose what ever target name you want to use, it doesn't have to be called "release". – airtonix Oct 02 '22 at 02:44
  • @cryf I'm using https://github.com/TheUnderScorer/nx-semantic-release with great success. Which then also gives me access to more release logic, ie: which commit should we consider from which version & changelogs are computed? might be a tag like `environment/${TARGET_STAGE}/payroll-app`. – airtonix Oct 02 '22 at 02:49
  • @airtonix I'll look into nx-semantic-release. I agree deploy===release, but only for apps, not for "a collection of packages which depend on each other". For packages, I want to make a release & do a publish. I want do this AFTER a feature branch is merged (not in CI). See 1) https://docs.github.com/en/repositories/releasing-projects-on-github/managing-releases-in-a-repository, and 2) https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-npm-registry#publishing-a-package. – cyrf Oct 02 '22 at 15:57
  • @cyrf in the context of a monorepo, everything is a package, (nx calls them all "projects") regardless if it's an "app" or a library . When you do the release is irrelevant to the use of nxdev. for example: i have several workflows in github that run on PRs that want to merge to master: `nx affected --target=lint` etc, and one that runs when pushes to master happen `nx affected --target=release`. What the "release" target looks can be up to the nx project (mine just run the semantic-release executor) and you can chain them with other targets of the same project or other projects. – airtonix Oct 03 '22 at 00:38
  • @cyrf to put some detail on semantics here a deployment === release. They're the same thing. I don't tell myself that only npm packages get released and only apps get deployed. A library deployment is: build, package, upload to distribution hub. an app deployment is: build, paclage, upload to server (which is really just a distribution hub for browsers). – airtonix Oct 03 '22 at 00:41
  • @Missilexent Maybe because not every project is a Node / JS / TS project. This answer is rather "If you have a Node monorepo like me, just use nx". Also, OP isn't talking about apps at all. – Dirk Lachowski Aug 08 '23 at 20:17
  • @DirkLachowski it doesn't matter what you call your deliverable that provides useful work to consumers. you have: distributables and internal deps. this is a monorepo regardless of your language engine.. in fact i still use nx.dev in my golang monorepos that have deliverables like apps, servers, etc.because there simply isn't another tool available that i can just drop in and get this amazingly reliable ability to know exactly what to build and deploy – airtonix Aug 10 '23 at 03:31
  • @airtonix Do you have by any chance a public repo showing that? Sounds interesting. – Dirk Lachowski Aug 16 '23 at 01:04
  • 1
    @DirkLachowski yep, I'm finalising this template https://github.com/airtonix/golang-monorepo-template – airtonix Aug 17 '23 at 09:14
2

Has Changed Path Action might be worth a try:

name: Conditional Deploy

on: push

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v2
        with:
          fetch-depth: 100

      - uses: marceloprado/has-changed-path@v1
        id: service_A_deployment
        with:
          paths: service_A

      - name: Deploy front
        if: steps.service_A_deployment.outputs.changed == 'true'
        run: /deploy-service_A.sh
pafede2
  • 1,626
  • 4
  • 23
  • 40
0

You can use dorny/paths-filter@v2 to execute a job in the workflow if some files in the subdirectory are changed. if you commit to frontend it will only execute the frontend job. and same for the backend. so only the required job in the workflow will run:

/
 frontend/
 backend/

YAML:

jobs:
  # JOB to run change detection
  changes:
    runs-on: ubuntu-latest
    # Required permissions
    permissions:
      pull-requests: read
    # Set job outputs to values from filter step
    outputs:
      backend: ${{ steps.filter.outputs.backend }}
      frontend: ${{ steps.filter.outputs.frontend }}
    steps:
    # For pull requests it's not necessary to checkout the code
    - uses: dorny/paths-filter@v2
      id: filter
      with:
        filters: |
          backend:
            - 'backend/**'
          frontend:
            - 'frontend/**'

  # JOB to build and test backend code
  backend:
    needs: changes
    if: ${{ needs.changes.outputs.backend == 'true' }}
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - ...

  # JOB to build and test frontend code
  frontend:
    needs: changes
    if: ${{ needs.changes.outputs.frontend == 'true' }}
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - ...

Learn more about it: https://github.com/dorny/paths-filter#examples

Ericgit
  • 6,089
  • 2
  • 42
  • 53