github.com/broadinstitute/gatk @4.6.2.0

Please see the GATK website, where you can download a precompiled executable, read documentation, ask questions, and receive technical support. For GitHub basics, see here.

GATK 4

This repository contains the next generation of the Genome Analysis Toolkit (GATK). The contents of this repository are 100% open source and released under the Apache 2.0 license (see LICENSE.TXT).

GATK4 aims to bring together well-established tools from the GATK and Picard codebases under a streamlined framework, and to enable selected tools to be run in a massively parallel way on local clusters or in the cloud using Apache Spark. It also contains many newly developed tools not present in earlier releases of the toolkit.

Table of Contents

• Requirements
• Quick Start Guide
• Downloading GATK4
• Tools Included in Docker Image
• Building GATK4
• Running GATK4
  • Passing JVM options to gatk
  • Passing a configuration file to gatk
  • Running GATK4 with inputs on Google Cloud Storage
  • Running GATK4 Spark tools locally
  • Running GATK4 Spark tools on a Spark cluster
  • Running GATK4 Spark tools on Google Cloud Dataproc
  • Using R to generate plots
  • GATK Tab Completion for Bash
• For GATK Developers
  • General guidelines for GATK4 developers
  • Testing GATK4
  • Using Git LFS to download and track large test data
  • Creating a GATK project in the IntelliJ IDE
  • Setting up debugging in IntelliJ
  • Updating the Intellij project when dependencies change
  • Setting up profiling using JProfiler
  • Uploading Archives to Sonatype
  • Building GATK4 Docker images
  • Releasing GATK4
  • Generating GATK4 documentation
  • Generating GATK4 WDL Wrappers
  • Using Zenhub to track github issues
• Further Reading on Spark
• How to contribute to GATK
• Discussions
• Authors
• Citing GATK
• License

Requirements

• To run GATK:
  • Java 17 is needed to run or build GATK. We recommend one of the following:
    • Download the Eclipse Foundation's distribution of OpenJDK 17 from adoptium.net. Navigate to the release archive to find downloads for Java 17.
    • On Mac OS, you can install the Homebrew package manager and run brew install temurin@17 to install the Eclipse Foundation's OpenJDK 17.
  • Python 3.10.13, along with a set of additional Python packages, is required to run some tools and workflows (also required to run the gatk frontend script). See Python Dependencies for more information.
  • R 4.3.1 (needed for producing plots in certain tools)
• To build GATK:
  • A Java 17 JDK
  • Git 2.5 or greater
  • git-lfs 1.1.0 or greater. Required to download the large files used to build GATK, and test files required to run the test suite. Run git lfs install after downloading, followed by git lfs pull from the root of your git clone to download all of the large files, including those required to run the test suite. The full download is approximately 5 gigabytes. Alternatively, if you are just building GATK and not running the test suite, you can skip this step since the build itself will use git-lfs to download the minimal set of large lfs resource files required to complete the build. The test resources will not be downloaded, but this greatly reduces the size of the download.
  • Gradle 5.6. We recommend using the ./gradlew script which will download and use an appropriate gradle version automatically (see examples below).
  • R 4.3.1 (needed for running the test suite)
• Pre-packaged Docker images with all needed dependencies installed can be found on our dockerhub repository. This requires a recent version of the docker client, which can be found on the docker website.
• Python Dependencies:
  • GATK4 uses the Conda package manager to establish and manage the Python environment and dependencies required by Python-based GATK tools. This environment also includes the R dependencies used for plotting in some of the tools. The GATK Docker image comes with the gatk conda environment pre-configured and activated.
  • To establish the environment when not using the Docker image, a conda environment must first be "created", and then "activated":
    • First, make sure Miniconda or Conda is installed. We recommend installing Miniconda3-py310_23.10.0-1 from the miniconda download page, selecting the Linux or MacOS version of the installer as appropriate.
      • This is the same version of miniconda used by the official GATK docker image.
      • If you use a different version, you may run into issues.
      • If you have an ARM-based Mac, you must select the MacOSX-x86_64 installer, not the MacOSX-arm64 installer, and rely on Mac OS's built-in x86 emulation.
    • Set up miniconda:
      • Install miniconda to a location on your PATH such as /opt/miniconda, and then restart your shell: bash Miniconda3-py310_23.10.0-1-[YOUR_OS].sh -p /opt/miniconda -b
      • Disable conda auto-updates, which can cause compatibility issues with GATK: conda config --set auto_update_conda false
      • Enable the (much) faster libmamba solver to greatly speed up creation of the conda environment: conda config --set solver libmamba
    • To "create" the conda environment:
      • If running from a zip or tar distribution, run the command conda env create -f gatkcondaenv.yml to create the gatk environment.
      • If running from a cloned repository, run ./gradlew localDevCondaEnv. This generates the Python package archive and conda yml dependency file(s) in the build directory, and also creates (or updates) the local gatk conda environment.
    • To "activate" the conda environment (the conda environment must be activated within the same shell from which GATK is run):
      • Execute the shell command source activate gatk to activate the gatk environment.
    • See the Conda documentation for additional information about using and managing Conda environments.

Quick Start Guide

• Build the GATK: ./gradlew bundle (creates gatk-VERSION.zip in build/)
• Get help on running the GATK: ./gatk --help
• Get a list of available tools: ./gatk --list
• Run a tool: ./gatk PrintReads -I src/test/resources/NA12878.chr17_69k_70k.dictFix.bam -O output.bam
• Get help on a particular tool: ./gatk PrintReads --help

Downloading GATK4

You can download and run pre-built versions of GATK4 from the following places:

• A zip archive with everything you need to run GATK4 can be downloaded for each release from the github releases page. We also host unstable archives generated nightly in the Google bucket gs://gatk-nightly-builds.

• You can download a GATK4 docker image from our dockerhub repository. We also host unstable nightly development builds on this dockerhub repository.

  • Within the docker image, run gatk commands as usual from the default startup directory (/gatk).

Tools Included in Docker Image

Our docker image contains the following bioinformatics tools, which can be run by invoking the tool name from the command line: * bedtools (v2.30.0) * samtools (1.13) * bcftools (1.13) * tabix (1.13+ds)

We also include an installation of Python3 (3.10.13) with the following popular packages included: * numpy * scipy * pytorch * pymc3 * keras * scikit-learn * matplotlib * pandas * biopython * pyvcf * pysam

We also include an installation of R (4.3.1) with the following popular packages included: * data.table * dplyr * ggplot2

For more details on system packages, see the GATK Base Dockerfile and for more details on the Python3/R packages, see the Conda environment setup file. Versions for the Python3/R packages can be found there.

Building GATK4

• To do a full build of GATK4, first clone the GATK repository using "git clone", then run:

  ./gradlew bundle
  

Equivalently, you can just type:

    ./gradlew

* This creates a zip archive in the `build/` directory with a name like `gatk-VERSION.zip` containing a complete standalone GATK distribution, including our launcher `gatk`, both the local and spark jars, and this README.    
* You can also run GATK commands directly from the root of your git clone after running this command.
* Note that you *must* have a full git clone in order to build GATK, including the git-lfs files in `src/main/resources/large`. The zipped source code alone is not buildable.
* The large files under `src/main/resources/large/` are required to build GATK, since they are packaged inside the GATK jar and used by tools at runtime. These include things like ML models and native C/C++ libraries used for acceleration of certain tools.
* The large files under `src/test/resources/large/`, on the other hand, are only required by the test suite when running tests, and are not required to build GATK.

• Other ways to build:

  • ./gradlew installDist
    • Does a fast build that only lets you run GATK tools from inside your git clone, and locally only (not on a cluster). Good for developers!
  • ./gradlew installAll
    • Does a semi-fast build that only lets you run GATK tools from inside your git clone, but works both locally and on a cluster. Good for developers!
  • ./gradlew localJar
    • Builds only the GATK jar used for running tools locally (not on a Spark cluster). The resulting jar will be in build/libs with a name like gatk-package-VERSION-local.jar, and can be used outside of your git clone.
  • ./gradlew sparkJar
    • Builds only the GATK jar used for running tools on a Spark cluster (rather than locally). The resulting jar will be in build/libs with a name like gatk-package-VERSION-spark.jar, and can be used outside of your git clone.
    • This jar will not include Spark and Hadoop libraries, in order to allow the versions of Spark and Hadoop installed on your cluster to be used.

• To remove previous builds, run:

  ./gradlew clean
  

• For faster gradle operations, add org.gradle.daemon=true to your ~/.gradle/gradle.properties file. This will keep a gradle daemon running in the background and avoid the ~6s gradle start up time on every command.

• Gradle keeps a cache of dependencies used to build GATK. By default this goes in ~/.gradle. If there is insufficient free space in your home directory, you can change the location of the cache by setting the GRADLE_USER_HOME environment variable.

• The version number is automatically derived from the git history using git describe, you can override it by setting the versionOverride property. ( ./gradlew -DversionOverride=my_weird_version printVersion )

Running GATK4

• The standard way to run GATK4 tools is via the gatk wrapper script located in the root directory of a clone of this repository.
  • Requires Python 3.9 or greater
  • You need to have built the GATK as described in the Building GATK4 section above before running this script.
  • There are several ways gatk can be run:
    • Directly from the root of your git clone after building
    • By extracting the zip archive produced by ./gradlew bundle to a directory, and running gatk from there
    • Manually putting the gatk script within the same directory as fully-packaged GATK jars produced by ./gradlew localJar and/or ./gradlew sparkJar
    • Defining the environment variables GATK_LOCAL_JAR and GATK_SPARK_JAR, and setting them