MCPcopy Index your code
hub / github.com/DTStack/Taier

github.com/DTStack/Taier @v1.4.0

Chat with this repo
repository ↗ · DeepWiki ↗ · release v1.4.0 ↗ · + Follow
20,042 symbols 60,632 edges 2,279 files 3,970 documented · 20%
What it actually does AI analysis from the code graph — generated when you open this
loading…
README
     <a href="https://dtstack.github.io/Taier/" target="_blank" rel="noopener noreferrer">
       <img src="https://github.com/DTStack/Taier/raw/v1.4.0/website/static/img/logo.svg" width="20%" height="20%" alt="Taier Logo" />
    </a>

Taier

A distributed dispatching system

<a href="https://dtstack.github.io/Taier/">官网</a> |
<a href="https://dtstack.github.io/Taier/docs/guides/introduction/">官方文档</a>

中文 | English

介绍

Taier 太阿，是中国古代十大名剑之一

Taier 是一个开源的分布式 DAG 调度系统，专注不同任务的提交和调度。旨在降低 ETL 开发成本，解决任务之间复杂的依赖关系和提交、调度、运维带来的上手成本

在 Taier 上进行 ETL 开发，不用关心任务错综复杂的依赖关系与底层的大数据平台的架构实现，将工作的重心更多地聚焦在业务之中

Taier 提供了一个提交、调度、运维、指标信息展示的一站式大数据开发平台

功能

核心功能如下:

  • 分布式扩展
  • 可视化 DAG 配置
  • IDE 式开发平台
  • 自定义扩展任务插件
  • 向导、脚本多种模式
  • 上下游依赖调度
  • 支持实时、离线任务
  • 支持对接不同版本的 Hadoop
  • 支持Flink Standalone
  • 对集群环境 0 侵入
  • 多租户多集群隔离
  • 支持 Kerberos 认证
  • 任务多版本支持
  • 自定义参数替换
  • 集群资源实时监控
  • 数据指标实时获取
  • 任务资源限制

架构

architecture

快速开始

docker 启动

$ wget https://github.com/DTStack/Taier/blob/master/docker-compose.yml
$ docker-compose up -d

在线预览地址

开发任务

main

任务类型

Tasks Documentation
工作流 文档
数据同步 文档
实时采集 文档
Flink 文档
Shell 文档
Python 文档
Spark SQL 文档
Hive SQL 文档
Flink SQL 文档
OceanBase SQL 文档
ClickHouse SQL 文档
Doris SQL 文档
TiDB SQL 文档
MySQL SQL 文档
Vertica SQL 文档
Postgre SQL 文档
SqlServer SQL 文档
Greenplum SQL 文档
MaxCompute SQL 文档
GaussDB SQL 文档
自定义扩展 文档

问题反馈

在使用上有遇到 bug 或者优化点，强烈建议你提 issue 我们将及时修复

联系我们

  • Slack https://join.slack.com/t/slack-p437975/shared_invite/zt-1iw5x1fw7-A6rVolqjP2z8V09~WwFUiA

贡献代码

我该如何贡献？

参考文档

贡献者

License

Taier is under the Apache 2.0 license. See the LICENSE file for details.

Extension points exported contracts — how you extend this code

IDownload (Interface)
@author jiangbo @date 2018/7/6 16:08 [49 implementers]
taier-data-develop/src/main/java/com/dtstack/taier/develop/utils/develop/common/IDownload.java
ID (Interface)
ID接口,内置实体主键ID @description: @author: liuxx @date: 2021/3/15 [74 implementers]
taier-dao/src/main/java/com/dtstack/taier/dao/domain/ID.java
Sort (Interface)
@Auther: dazhi @Date: 2022/3/11 4:54 PM @Email: dazhi@dtstack.com @Description: 排序接口 [15 implementers]
taier-scheduler/src/main/java/com/dtstack/taier/scheduler/server/Sort.java
EngineResourceInfo (Interface)
company: www.dtstack.com author: toutian create: 2018/11/1 [19 implementers]
taier-worker/taier-worker-plugin/base/src/main/java/com/dtstack/taier/base/resource/EngineResourceInfo.java
CallBack (Interface)
Created by sishu.yss on 2017/8/28. [11 implementers]
taier-worker/taier-worker-api/src/main/java/com/dtstack/taier/pluginapi/callback/CallBack.java
IDownloader (Interface)
下载器 @author ：wangchuan date：Created in 下午5:05 2022/9/23 company: www.dtstack.com [82 implementers]
taier-datasource/taier-datasource-api/src/main/java/com/dtstack/taier/datasource/api/downloader/IDownloader.java
SingleParamFunc (Interface)
单入参函数 @author ：wangchuan date：Created in 下午5:42 2022/3/16 company: www.dtstack.com [11 implementers]
taier-datasource/taier-datasource-plugin/taier-datasource-plugin-common/src/main/java/com/dtstack/taier/datasource/plugin/common/function/SingleParamFunc.java
ICustomMetric (Interface)
@author ：wangchuan date：Created in 下午3:57 2021/4/16 company: www.dtstack.com [10 implementers]
taier-common/src/main/java/com/dtstack/taier/common/metric/stream/prometheus/ICustomMetric.java

Core symbols most depended-on inside this repo

get
called by 2040
taier-datasource/taier-datasource-api/src/main/java/com/dtstack/taier/datasource/api/client/IRestful.java
put
called by 1523
taier-datasource/taier-datasource-api/src/main/java/com/dtstack/taier/datasource/api/client/IRestful.java
add
called by 1471
taier-scheduler/src/main/java/com/dtstack/taier/scheduler/server/queue/GroupPriorityQueue.java
format
called by 1269
taier-data-develop/src/main/java/com/dtstack/taier/develop/sql/formate/Formatter.java
append
called by 765
taier-worker/taier-worker-api/src/main/java/com/dtstack/taier/pluginapi/io/UnsafeStringWriter.java
info
called by 716
taier-datasource/taier-datasource-plugin/taier-datasource-plugin-es/src/main/java/org/elasticsearch/client/RequestConverters.java
getString
called by 696
taier-datasource/taier-datasource-plugin/taier-datasource-plugin-common/src/main/java/com/dtstack/taier/datasource/plugin/common/utils/MathUtil.java
isBlank
called by 611
taier-common/src/main/java/com/dtstack/taier/common/util/Strings.java

Shape

Method 16,266
Class 1,991
Function 1,105
Interface 431
Enum 249

Languages

Java91%
TypeScript9%

Modules by API surface

taier-ui/src/api/index.ts162 symbols
taier-datasource/taier-datasource-plugin/taier-datasource-plugin-spark/src/main/java/org/apache/orc/impl/TreeReaderFactory.java158 symbols
taier-datasource/taier-datasource-plugin/taier-datasource-plugin-hive/src/main/java/org/apache/orc/impl/TreeReaderFactory.java158 symbols
taier-data-develop/src/main/java/com/dtstack/taier/develop/vo/develop/result/DevelopTaskGetTaskByIdResultVO.java145 symbols
taier-data-develop/src/main/java/com/dtstack/taier/develop/vo/develop/query/DevelopTaskResourceParamVO.java121 symbols
taier-data-develop/src/main/java/com/dtstack/taier/develop/vo/develop/query/DevelopScheduleTaskResultVO.java115 symbols
taier-data-develop/src/main/java/com/dtstack/taier/develop/vo/develop/query/DevelopScheduleTaskVO.java113 symbols
taier-worker/taier-worker-plugin/hadoop/yarn3-hdfs3-hadoop3/src/main/java/org/apache/hadoop/security/UserGroupInformation.java98 symbols
taier-worker/taier-worker-plugin/hadoop/yarn2-hdfs2-hadoop2/src/main/java/org/apache/hadoop/security/UserGroupInformation.java98 symbols
taier-worker/taier-worker-plugin/spark/yarn3-hdfs3-spark210/spark-yarn-client/src/main/java/org/apache/hadoop/security/UserGroupInformation.java97 symbols
taier-worker/taier-worker-plugin/spark/yarn2-hdfs2-spark210/spark-yarn-client/src/main/java/org/apache/hadoop/security/UserGroupInformation.java97 symbols
taier-worker/taier-worker-plugin/spark/yarn2-hdfs2-spark210-core/spark-yarn-client-core/src/main/java/org/apache/hadoop/security/UserGroupInformation.java97 symbols

Datastores touched

(mysql)Database · 1 repos
metastoreDatabase · 1 repos
(mongodb)Database · 1 repos
dbNameDatabase · 1 repos
taierDatabase · 1 repos

For agents

$ claude mcp add Taier \
  -- python -m otcore.mcp_server <graph>

⬇ download graph artifact

Ask about this repo answers extend the page