elastic-package

elastic-package is a command line tool, written in Go, used for developing Elastic packages. It can help you lint, format, test and build your packages. Learn about each of these and other features in Commands below.

Currently, elastic-package only supports packages of type Elastic Integrations.

Please review the integrations contributing guide to learn how to build and develop packages, understand the release procedure and explore the builder tools.

Getting started

Download latest release from the Releases page.

On macOS, use xattr -r -d com.apple.quarantine elastic-package after downloading to allow the binary to run.

Alternatively, you may use go install but you will not be able to use the elastic-package version command or check updates.

go install github.com/elastic/elastic-package@latest

Please make sure that you've correctly setup environment variables - $GOPATH and $PATH, and elastic-package is accessible from your $PATH.

Change directory to the package under development.

cd my-package

Run the help command and see available commands:

elastic-package help

Some sub-commands are Docker-based, check you also have Docker installed. In case you are using Podman Desktop, check this guide to make it compatible.

Development

Even though the project is "go-gettable", there is the Makefile present, which can be used to build, install, format the source code among others. Some examples of the available targets are:

make build - build the tool source

make clean - delete elastic-package binary and build folder

make format - format the Go code

make check - one-liner, used by CI to verify if source code is ready to be pushed to the repository

make install - build the tool source and move binary to $GOBIN

make gomod - ensure go.mod and go.sum are up to date

make update - update README.md file

make licenser - add the Elastic license header in the source code

To start developing, download and build the latest main of elastic-package binary:

git clone https://github.com/elastic/elastic-package.git
cd elastic-package
make build

When developing on Windows, please use the core.autocrlf=input or core.autocrlf=false option to avoid issues with CRLF line endings:

git clone --config core.autocrlf=input https://github.com/elastic/elastic-package.git
cd elastic-package
make build

This option can be also configured on existing clones with the following commands. Be aware that these commands will remove uncommited changes.

git config core.autocrlf input
git rm --cached -r .
git reset --hard

Testing with integrations repository

While working on a new branch, it is interesting to test these changes with all the packages defined in the integrations repository. This allows to test a much wider scenarios than the test packages that are defined in this repository.

This test can be triggered automatically directly from your Pull Request by adding a comment test integrations. Example: - Comment: https://github.com/elastic/elastic-package/pull/1335#issuecomment-1619721861 - Pull Request created in integrations repository: https://github.com/elastic/integrations/pull/6756

This comment triggers this Buildkite pipeline (Buildkite job).

This pipeline creates a new draft Pull Request in integration updating the required dependencies to test your own changes. As a new pull request is created, a CI job will be triggered to test all the packages defined in this repository. A new comment with the link to this new Pull Request will be posted in your package-spec Pull Request.

IMPORTANT: Remember to close this PR in the integrations repository once you close the package-spec Pull Request.

Usually, this process would require the following manual steps: 1. Create your elastic-package pull request and push all your commits 2. Get the SHA of the latest changeset of your PR: bash $ git show -s --pretty=format:%H 1131866bcff98c29e2c84bcc1c772fff4307aaca 3. Go to the integrations repository, and update go.mod and go.sum with that changeset: bash cd /path/to/integrations/repostiory go mod edit -replace github.com/elastic/elastic-package=github.com/<your_github_user>/elastic-package@1131866bcff98c29e2c84bcc1c772fff4307aaca go mod tidy 4. Push these changes into a branch and create a Pull Request - Creating this PR would automatically trigger a new build of the corresponding Buildkite pipeline.

Testing with Elastic serverless

While working on a branch, it might be interesting to test your changes using a project created in Elastic serverless, instead of spinning up a local Elastic stack. To do so, you can add a new comment while developing in your Pull request a comment like test serverless.

Adding that comment in your Pull Request will create a new build of this Buildkite pipeline. This pipeline creates a new Serverless project and run some tests with the packages defined in the test/packages/parallel folder. Currently, there are some differences with respect to testing with a local Elastic stack: - System tests are not executed. - Disabled comparison of results in pipeline tests to avoid errors related to GeoIP fields - Pipeline tests cannot be executed with coverage flags.

At the same time, this pipeline is going to be triggered daily to test the latest contents of the main branch with an Elastic serverless project.

Commands

elastic-package currently offers the commands listed below.

Some commands have a global context, meaning that they can be executed from anywhere and they will have the same result. Other commands have a package context; these must be executed from somewhere under a package's root folder and they will operate on the contents of that package.

For more details on a specific command, run elastic-package help <command>.

elastic-package help

Context: global

Use this command to get a listing of all commands available under elastic-package and a brief description of what each command does.

elastic-package completion

Context: global

Use this command to output shell completion information.

The command output shell completions information (for bash, zsh, fish and powershell). The output can be sourced in the shell to enable command completion.

Run elastic-package completion and follow the instruction for your shell.

elastic-package benchmark

Context: package

Use this command to run benchmarks on a package. Currently, the following types of benchmarks are available:

Pipeline Benchmarks

These benchmarks allow you to benchmark any Ingest Node Pipelines defined by your packages.

For details on how to configure pipeline benchmarks for a package, review the HOWTO guide.

Rally Benchmarks

These benchmarks allow you to benchmark an integration corpus with rally.

For details on how to configure rally benchmarks for a package, review the HOWTO guide.

Stream Benchmarks

These benchmarks allow you to benchmark ingesting real time data. You can stream data to a remote ES cluster setting the following environment variables:

ELASTIC_PACKAGE_ELASTICSEARCH_HOST=https://my-deployment.es.eu-central-1.aws.foundit.no
ELASTIC_PACKAGE_ELASTICSEARCH_USERNAME=elastic
ELASTIC_PACKAGE_ELASTICSEARCH_PASSWORD=changeme
ELASTIC_PACKAGE_KIBANA_HOST=https://my-deployment.kb.eu-central-1.aws.foundit.no:9243

System Benchmarks

These benchmarks allow you to benchmark an integration end to end.

For details on how to configure system benchmarks for a package, review the HOWTO guide.

elastic-package benchmark pipeline

Context: package

Run pipeline benchmarks for the package.

elastic-package benchmark rally

Context: package

Run rally benchmarks for the package (esrally needs to be installed in the path of the system).

elastic-package benchmark stream

Context: package

Run stream benchmarks for the package.

elastic-package benchmark system

Context: package

Run system benchmarks for the package.

elastic-package build

Context: package

Use this command to build a package.

Built packages are stored in the "build/" folder located at the root folder of the local Git repository checkout that contains your package folder. The command will also render the README file in your package folder if there is a corresponding template file present in "_dev/build/docs/README.md". All "_dev" directories under your package will be omitted. For details on how to generate and syntax of this README, see the HOWTO guide.

Built packages are served up by the Elastic Package Registry running locally (see "elastic-package stack"). If you want a local package to be served up by the local Elastic Package Registry, make sure to build that package first using "elastic-package build".

Built packages can also be published to the global package registry service.

When the package declares required input packages ("requires.input" in manifest.yml), the build downloads those input packages from the configured package registry (see "package_registry.base_url" in ~/.elastic-package/config.yml). The build then incorporates their policy and data stream templates, merges variable definitions into the integration manifest, bundles data stream field definitions, and resolves package: references on inputs and streams to the effective input types expected by Fleet. For details on using a local or custom registry during development, see the HOWTO guide.

If a required input package declares a "data_stream.dataset" var, it is dropped from the bundled data stream vars: an integration fixes its dataset by data stream name, so this var must not be left user-configurable. If the integration needs to declare "data_stream.dataset" itself, it can: either promoted to the input level ("policy_templates[].inputs[].vars"), or directly on the data stream. Both are bundled normally, and if both are declared, Fleet's usual variable-scope precedence applies at policy-render time, with the data-stream-level value taking precedence over the input-level one.

For details on how to enable dependency management, see the HOWTO guide.

elastic-package changelog

Context: package

Use this command to work with the changelog of the package.

You can use this command to modify the changelog following the expected format and good practices. This can be useful when introducing changelog entries for changes done by automated processes.

elastic-package changelog add

Context: package

Use this command to add an entry to the changelog file.

The entry added will include the given description, type and link. It is added on top of the last entry in the current version

Alternatively, you can start a new version indicating the specific version, or if it should be the next major, minor or patch version.

elastic-package check

Context: package

Use this command to verify if the package is correct in terms of formatting, validation and building.

It will execute the lint and build commands all at once, in that order.

elastic-package clean

Context: package

Use this command to clean resources used for building the package.

The command will remove built package files (in build/), files needed for managing the development stack (in ~/.elastic-package/stack/development) and stack service logs (in ~/.elastic-package/tmp/service_logs and ~/.elastic-package/profiles//service_logs/).

elastic-package create

Context: global

Use this command to create a new package or add more data streams.

The command can help bootstrap the first draft of a package using embedded package template. It can be used to extend the package with more data streams.

For details on how to create a new package, review the HOWTO guide.

elastic-package create data-stream

Context: global

Use this command to create a new data stream.

The command can extend the package with a new data stream using embedded data stream template and wizard.

elastic-package create package

Context: global

Use this command to create a new package.

The command can bootstrap the first draft of a package using embedded package template and wizard.

elastic-package dump

Context: global

Use this command as an exploratory tool to dump resources from Elastic Stack (objects installed as part of package and agent policies).

elastic-package dump agent-policies

Context: global

Use this command to dump agent policies created by Fleet as part of a package installation.

Use this command as an exploratory tool to dump agent policies as they are created by Fleet when installing a package. Dumped agent policies are stored in files as they are returned