Monorepo-diff-buildkite-plugin

This Monorepo plugin will assist you in triggering pipelines, as well as run commands in your CI by watching folders in your monorepo.

Check out this post to learn more about How to set up Continuous Integration for monorepo using Buildkite.

Usefulness of the monorepo

A monorepo is a single, version-controlled code repository that houses multiple independent projects, offering benefits such as flexibility, streamlined management, and reduced tracking of changes and dependencies across multiple repositories.

This approach allows teams to:

• Reduce overhead associated with duplicating code for microservices.
• Easily maintain and monitor the entire codebase.

Check out the example monorepo source code.

Using the plugin

If the version number is not provided then the most recent version of the plugin will be used. Do not use version number as master or any branch names.

watch

It defines a list of paths or path to monitor for changes in the monorepo. It checks to see if there is a change to the subfolders specified in the path

path

A path or a list of paths to be watched, This part specifies which directory should be monitored. It can also be a glob pattern. For example specify path: "**/*.md" to match all markdown files. A list of paths can be provided to trigger the desired pipeline or run command or even do a pipeline upload.

When regex_paths: true is set on the watch block, paths are treated as regular expressions instead of globs.

skip_path

A path or a list of paths to be ignored, which can be an exact path, or a glob.

This is intended to be used in conjunction with path, and allows omitting specific paths from being matched.

When regex_paths: true is set, skip paths are also treated as regular expressions.

except_path

A path or a list of paths to prevent the paths listed to be matched, which can be an exact path, or a glob.

This is intended to be used in conjunction with path, and allows for creating exclusive matches with simpler rules when several files are modified in the same execution.

When regex_paths: true is set, except paths are also treated as regular expressions.

regex_paths

Set to true to treat path, skip_path, and except_path as regular expressions instead of globs. Uses regexp2 which supports full PCRE syntax including lookaheads and lookbehinds.

Regex matching is unanchored: a pattern matches if it occurs anywhere in the file path, not only at the start. For example, path: "src/.*" matches vendor/src/main.go as well as src/main.go. Anchor with ^ (and $ where needed) to match the full path, as in the example above.

This is useful when the paths you want to match would require many glob patterns to express. For example, to match all TypeScript/JavaScript source files under src/ while excluding test files, snapshots, and specific directories:

steps:
  - label: "Triggering pipelines"
    plugins:
      - monorepo-diff#v1.11.0:
          diff: "git diff --name-only HEAD~1"
          watch:
            - path: "^src/(?!pulumi|ci-generators|desktop|mobile|test)(?!.*\\.test\\.)(?!.*\\.snap$)(?!.*/__test__/)(?!.*/__mocks__/)(?!.*/__snapshots__/).*\\.[tj]sx?"
              regex_paths: true
              config:
                trigger: "frontend-pipeline"

Note: When regex_paths: true, all paths in that watch block must be valid regular expressions. Glob syntax (e.g. **) is not supported in regex mode.

For example, in the following configuration:

steps:
  - label: "Triggering pipelines"
    plugins:
      - monorepo-diff#v1.11.0:
          diff: "git diff --name-only HEAD~1"
          watch:
            - path: "**/*"
              skip_path: "folder/file"
              config:
                trigger: "pipeline-1"
            - path: "**/*"
              except_path: "folder/file"
              config:
                trigger: "pipeline-2"
            - path: "folder/file"
              config:
                trigger: "pipeline-3"

If a single execution modified folder/file only pipeline-3 will be triggered. But if any other file is modified as well (thus matching **/*), pipeline-1 will also be triggered, but not pipeline-2.

config

This is a sub-section that provides configuration for running commands or triggering another pipeline when changes occur in the specified path. Configuration supports 3 different step types.

• Trigger
• Command
• Group
• Conditionals

Step Validation

The plugin validates all step configurations before uploading the pipeline. Invalid steps are automatically skipped with a warning logged to the build output.

A valid step must have: - A command or commands field (for command steps), OR - A trigger field (for trigger steps), OR - A group field with either: - An action (command, commands, or trigger) directly on the group, OR - Valid nested steps

Invalid configurations that will be skipped:

# ❌ Empty step - no action defined
- path: "app/"
  config:
    label: "Deploy app"  # Only has a label, no command/trigger

# ❌ Empty group - no action and no nested steps
- path: "services/"
  config:
    group: "Deploy"
    # Missing: steps array or action

Valid configurations:

# ✅ Valid - has command
- path: "app/"
  config:
    label: "Deploy app"
    command: "echo deploying"

# ✅ Valid - group with nested steps
- path: "services/"
  config:
    group: "Deploy"
    steps:
      - command: "deploy.sh"

Plugins in Step Configurations

The plugin preserves plugins: blocks when specified in command step configurations. This allows you to use Buildkite plugins within your monorepo-watched steps.

Example

steps:
  - label: "Triggering pipelines"
    plugins:
      - monorepo-diff#v1.11.0:
          watch:
            - path: services/api/
              config:
                command: "npm test"
                plugins:
                  - artifacts#v1.9.4:
                      upload: "coverage/**/*"
                  - docker-compose#v5.12.1:
                      run: api
            - path: services/web/
              config:
                command: "yarn build"
                plugins:
                  - docker#v5.13.0:
                      image: "node:20"
                      workdir: /app

When changes are detected in the watched paths, the plugin generates steps that include the specified plugins. The plugins: blocks are preserved exactly as configured.

steps:
  - label: "Triggering pipelines"
    plugins:
      - monorepo-diff#v1.11.0:
          watch:
            - path: app/
              config:
                trigger: "app-deploy"
            - path: test/bin/
              config:
                command: "echo Make Changes to Bin"
            - path: docker/
              config:
                group: docker/**
                steps:  # Required: groups must have either 'steps' or an action
                  - plugins:
                      - docker#v5.13.0:
                          build: service
                          push: service
                  - command: docker/run-e2e-tests.sh

• Changes to the path app/ triggers the pipeline app-deploy
• Changes to the path test/bin will run the respective configuration command
• Changes to any file in the docker folder will run the steps in the group

⚠️ Warning : The user has to explictly state the paths they want to monitor or use wildcards. For instance if a user, is only watching path app/ changes made to app/bin will not trigger the configuration. This is because the subfolder /bin was not specified.

Example

steps:
  - label: "Triggering pipelines with plugin"
    plugins:
      - monorepo-diff#v1.11.0:
          watch:
            - path: test/.buildkite/
              config: # Required [trigger step configuration]
                trigger: test-pipeline # Required [trigger pipeline slug]
            - path:
                - app/
                - app/bin/service/
              config:
                trigger: "data-generator"
                label: ":package: Generate data"
                build:
                  meta_data:
                    release-version: "1.1"

• When changes are detected in the path test/.buildkite/ it triggers the pipeline test-pipeline
• If the changes are made to either app/ or app/bin/service/ it triggers the pipeline data-generator

Conditional Step Execution (if):

The plugin supports conditional execution of pipeline steps using the if key, matching Buildkite’s pipeline conditional syntax. The if key allows you to control when a step runs, based on branch names, environment variables, build metadata, or custom expressions.

Example

steps:
  - label: "Triggering pipelines with plugin"
    plugins:
      - monorepo-diff#v1.11.0:
          diff: git diff --name-only HEAD~1
          watch:
            - path: services/api
              config:
                trigger: deploy-api
                if: build.branch == 'main' || build.branch =~ /^release\//
            - path: services/web
              config:
                command: echo "Deploy Web"
                if: build.tag != null

In the example above,

• The deploy-api trigger will only run on the main branch or branches matching release/*.
• The web deployment command will run only if the build has a tag.

if also works at the group step level, controlling whether the entire group runs:

steps:
  - label: "Triggering pipelines with plugin"
    plugins:
      - monorepo-diff#v1.11.0:
          diff: git diff --name-only HEAD~1
          watch:
            - path: services/
              config:
                group: "Deploy Services"
                if: build.branch == 'main'
                steps:
                  - command: deploy-uat.sh
                    label: Deploy UAT
                  - command: deploy-prod.sh
                    label: Deploy Prod

diff (optional)

This will run the script provided to determine the folder changes. Depending on your use case, you may want to determine the point where the branch occurs https://stackoverflow.com/questions/1527234/finding-a-branch-point-with-git and perform a diff against the branch point.

Default: git diff --name-only HEAD~1

The diff command must produce newline-delimited output with one file path per line. This is the format that git diff --name-only produces by default. Newline-delimited output is required for filenames containing spaces to be parsed correctly.

Custom diff scripts should follow the same convention — print one path per line to standard output.

Sample output

README.md
lib/trigger.bash
directory/File Name With Spaces.md

Example scripts

diff: ./diff-against-last-successful-build.sh

#!/bin/bash

set -ueo pipefail

LAST_SUCCESSFUL_BUILD_COMMIT="$(aws s3 cp "${S3_LAST_SUCCESSFUL_BUILD_COMMIT_PATH}" - | head -n 1)"
git diff --name-only "$LAST_SUCCESSFUL_BUILD_COMMIT"

diff: ./diff-against-last-built-tag.sh

#!/bin/bash

set -ueo pipefail

LATEST_BUILT_TAG=$(git describe --tags --match foo-service-* --abbrev=0)
git diff --name-only "$LATEST_TAG"

Example

steps:
  - label: "Triggering pipelines"
    plugins:
      - monorepo-diff#v1.11.0:
          diff: "git diff --name-only HEAD~1"
          watch:
            - path: "bar-service/"
              config:
                command: "echo deploy-bar"
            - path: "foo-service/"
              config:
                trigger: "deploy-foo-service"

interpolation (optional)

This controls the pipeline interpolation on upload, and defaults to true. If set to false it adds --no-interpolation to the buildkite pipeline upload, to avoid trying to interpolate the commit message, which can cause failures.

default (optional)

A default config to run if no paths are matched, the config key is not required, so a default can be written with a config attribute or simple just a command or trigger.

Example

steps:
  - label: "Triggering pipelines"
    plugins:
      - monorepo-diff#v1.11.0:
          diff: "git diff --name-only HEAD~1"
          watch:
            - path: "bar-service/"
              config:
                command: "echo deploy-bar"
            - path: "foo-service/"
              config:
                trigger: "deploy-foo-service"
            - default:
                config: ## <-- Optional
                  command: echo "Hello, world!"

env (optional)

The object values provided in this configuration will be appended to env property of all steps or commands.

steps:
  - label: "Triggering pipelines"
    plugins:
      - monorepo-diff#v1.11.0:
          diff: "git diff --name-only HEAD~1"
          watch:
            - path: "foo-service/"
              config:
                trigger: "deploy-foo-service"
                label: "Triggered deploy"
                build:
                  message: "Deploying foo service"
                  env:
                    HELLO: 123
                    AWS_REGION: ~  # Null literal reads from $AWS_REGION

Environment Variables

Enviro