TreeProfiler Overview

TreeProfiler is a command line tool designed to automate the annotation of large phylogenetic trees with corresponding metadata. It also facilitates the visual exploration of these annotations as phylogenetic profiles, making it a powerful resource for researchers working with complex biological data.

Key Features: - Automated Annotation that integrates metadata into phylogenetic tree, and summarizes annotation in internal nodes, including: - Categorical/Numerical metadata in TSV/CSV format - Taxonomic Annotation of NCBI/GTDB taxonomy database - Functional Annotation from eggnog-mapper output - Domain annotation from pfam/smart - Multiple Sequence Alignment annotation - Visual Exploration that allows for the detailed examination of annotated trees, aiding in the interpretation and presentation of data. - Analytic Methods for computing analysis from leaf nodes: - Ancestral Character Reconstruction for both discrete and continuous traits. - Phylogenetic Signal Delta Statistic] - Lineage Specificity Analysis

The official documentation of TreeProfiler is in https://dengzq1234.github.io/TreeProfiler/ where provides detailed instructions with examples.

If you have any doubts, please drop a line in issue or contact https://x.com/deng_ziqi

Full manuscript of TreeProfiler is in https://doi.org/10.1101/2023.09.21.558621

If you use TreeProfiler, please cite:

Ziqi Deng, Ana Hernández-Plaza, Jaime Huerta-Cepas. 
"TreeProfiler: A command-line tool for computing and visualizing phylogenetic profiles against large trees"  
bioRxiv (2023) doi: 10.1101/2023.09.21.558621

Table of Contents

• Introduction
• Installation
• Dependencies
• Quick install via pip
• Quick Start with example dataset
• Manual installation
  • Install ETE v4
  • Install TreeProfiler
• Input files
• Basic Usage
• Using TreeProfiler
• Parsing Input tree
  • Tree format
  • Tree parser
• treeprofiler-annotate, computing phylogenetic profiles and annotation
  • annotate, Annotate metadata into tree
  • Basic Metadata in TSV/CSV Format
  • Metadata TSV/CSV as a Numerical Data Matrix
  • Metadata Without Column Names
  • Missing Value Detection
  • Unmapped Tree leaf property detection
  • Handling Duplicated Leaf Names in Metadata
  • Specifying Data Types for Metadata Columns
  • Annotate Metadata to Internal Nodes
  • Taxonomic annotation
  • Using Different Taxonomic Databases from GTDB/NCBI
  • Identifying Taxon Names in Different Metadata Fields/Columns
  • Ignoring Unclassified Taxonomic Annotations
  • Functional Annotation
  • Annotating Multiple Sequence Alignments
  • Annotating EggNOG-Mapper Output and Domain information
  • Analytical Methods in TreeProfiler
  • Ancestral Character Reconstruction
  • Phylogenetic Signal Delta Statistic
  • Lineage Specificity Analysis
  • Output Formats for Annotated Trees
  • Use standard input and output formats
• treeprofiler-plot visualizing phylogenetic profiles and annotation
  • Interactive visualization interface
  • Basic options of visualizing layouts
  • Layouts for categorical data
  • Layouts for boolean data
  • Layouts for numerical data
  • Layouts for Analytical Data
  • Layouts for multiple sequence alignment
  • Layouts for eggnog-mapper pfam annotations
  • Layouts for eggnog-mapper smart annotations
  • Layouts for eggnog-mapper annotations
  • Layouts for Taxonomic data
  • Customize color in layout with color config

• Conditional query in annotated tree

  • Basic Query
  • Query in internal nodes
  • AND and OR conditions
  • Conditional query with config file
  • Conditional limit based on taxonomic level

• Demo1 Explore GTDB taxonomic tree with metadata and habitat information of progenome3

Introduction

TreeProfiler is command-line tool for profiling metadata table into phylogenetic tree with descriptive analysis and output visualization

Installation

Dependencies

TreeProfiler requires - Python version >= 3.10 - ETE Toolkit v4 - biopython >= 1.8 - selenium >= 4.24 - scipy >= 1.8.0 - matplotlib >= 3.4 - pymc >= 5.0.0 - numpy == 1.24.4 - aesara - pastml (custom) - webdriver_manager - pyvirtualdisplay

Quick install via pip

# create environment
conda create -n treeprofiler python=3.10
conda activate treeprofiler

# Install ETE Toolkit v4 for treeprofiler
pip install "git+https://github.com/etetoolkit/ete.git@ete4_treeprofiler"

# Install custom pastml package for ete4
pip install "git+https://github.com/dengzq1234/pastml.git@pastml2ete4" 

# Install TreeProfiler tool via pip
pip install TreeProfiler

# Or install directly from github
pip install https://github.com/compgenomicslab/TreeProfiler/archive/main.zip
# or development mode for latestest update
pip install git+https://github.com/compgenomicslab/TreeProfiler@dev-repo

Quick Start with examples dataset

TreeProfiler provide various example dataset for testing in examples/ or https://github.com/compgenomicslab/TreeProfiler/tree/main/examples, each directory consists a demo script *_demo.sh for quick starting different functions in TreeProfiler which alreadyh as annotate-plot pipeline of example data. User can fast explore different example tree with different visualizations. Here is the demonstration:

# execute demo script of example1
cd examples/basic_example1/
sh ./example1_demo.sh

Annotate example tree with two metadata tables
start parsing...
Time for parse_csv to run:  0.001968860626220703
Time for load_metadata_to_tree to run:  0.0003094673156738281
Time for merge annotations to run:  0.05160331726074219
Time for annotate_taxa to run:  4.76837158203125e-07
Visualize properties categorical data random_type in rectangle_layout, numerical data sample1, sample2 in heatmap_layout and barplot_layout.
Current trees in memory: 0
Added tree example with id 0.
 * Serving Flask app 'ete4.smartview.gui.server' (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: on
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)

As the session starts in local server http://127.0.0.1:5000, annotated tree and selected properties are visualized at the interactive session. Here is detailed introduction of interactive session of visualization(here)

Check other tutorial scripts

# display demo script of each example
./examples/basic_example1/example1_demo.sh
./examples/automatic_query/highlight_demo.sh
./examples/automatic_query/collapse_demo.sh
./examples/automatic_query/prune_demo.sh
./examples/basic_example2/example2_demo.sh
./examples/taxonomy_example/ncbi/ncbi_demo.sh
./examples/taxonomy_example/gtdb/gtdb_demo.sh
./examples/pratical_example/progenome3/progenome_demo.sh
./examples/pratical_example/gtdb_r202/gtdbv202full_demo.sh
./examples/pratical_example/gtdb_r202/gtdbv202lite_demo.sh
./examples/pratical_example/emapper/emapper_demo.sh

Manual installation

Install ETE v4

Quick way

pip install https://github.com/etetoolkit/ete/archive/ete4.zip

For local development To install ETE in a local directory to help with the development, you can:

• Clone this repository (git clone https://github.com/etetoolkit/ete.git)
• Install dependecies
• If you are using conda: conda install -c conda-forge cython bottle brotli pyqt numpy<2.0
• Otherwise, you can install them with pip install <dependencies>
• Build and install ete4 from the repository's root directory: pip install -e .

(In Linux there may be some cases where the gcc library must be installed, which can be done with conda install -c conda-forge gcc_linux-64)

Install TreeProfiler

Install dependencies

# install BioPython, selenium, scipy via conda
conda install -c conda-forge biopython selenium scipy matplotlib pymc aesara
# or pip
pip install biopython selenium scipy matplotlib pymc aesara webdriver_manager pyvirtualdisplay

Install TreeProfiler

# install TreeProfiler
git clone https://github.com/compgenomicslab/TreeProfiler
cd TreeProfiler/
python setup.py install

Or

# install directly
pip install https://github.com/compgenomicslab/TreeProfiler/archive/main.zip

Input files

TreeProfiler takes following file types as input

Input	Filetype
Tree	newick, ete
Metadata	tar.gz, tsv

• ete format is a novel format developed to solve the situation we encounter in the previous step, annotated tree can be recover easily with all the annotated data without changing the data type. Besides, the ete format optimized the tree file size after mapped with its associated data. Hence it's very handy for programers in their own script. At this moment we can only view the ete format in treeprofiler, but we will make the ete format more universal to other phylogenetic software.
• Metadata input could be single or multiple files, either tar.gz compressed file(s) which contains multiple .tsv or plain .tsv file(s).

Basic Usage

TreeProfiler has two main subcommand: - annotate - plot

The first one annotate is used to annotate your input tree and corresponding metadata, TreeProfiler will map all the metadata into corresponding tree node. In this step, annotated tree will be generated in newick and ete format

treeprofiler annotate --tree tree.nw --input-type newick --metadata metadata.tsv --outdir ./

The second subcommand plot is used to visualize tree with associated metadata. By default, treeprofiler will launch an interactive session at localhost for user to explore input tree.

treeprofiler plot --tree tree_annotated.nw --input-type newick 

or

treeprofiler plot --tree tree_annotated.ete --input-type ete 

Using TreeProfiler

In this Tutorial we will use TreeProfiler and demostrate basic usage with data in examples/

Parsing Input tree

Tree format

TreeProfiler accpept input tree in .nw or .ete by putting --input-type {newick,ete} flag to identify. By default, TreeProfiler will automatically detech the format of tree. The difference between .nw and .ete:

• newick format is more universal and be able to used in different other phylogenetic software although associated data of tree nodes will be considered as plain text.

• ete format is a novel format developed to solve the situation we encounter in the previous step, annotated tree can be recover easily with all the annotated data without changing the data type. Besides, the ete format optimized the tree file size after mapped with its associated data. Hence it's very handy for programers in their own script. At this moment we can only view the ete format in treeprofiler, but we will make the ete format more universal to other phylogenetic software. Hence using ete format in plot subcommand is highly reccomended

Tree parser

TreeProfiler provides argument --internal {name,support} to specify newick tree when it include values in internal node. [default: support]

newick	leaves	internal_node value	internal_parser
(A:0.5, B:0.5)Internal_C:0.5;	A, B	Internal_C	name
(A:0.5, B:0.5)0.99:0.5;	A, B	0.99	support

treeprofiler-annotate computing phylogenetic profiles and annotation

annotate, Annotate metadata into tree

TreeProfiler annotate subcommand is the step that annotate input metadata to target tree. As a result, itwill generate the following