MCPcopy Index your code
hub / github.com/alibaba/otter

github.com/alibaba/otter @otter-4.2.18

repository ↗ · DeepWiki ↗ · release otter-4.2.18 ↗ · + Follow
7,683 symbols 24,880 edges 931 files 1,842 documented · 24%
README

最新更新

  • otter已在阿里云推出商业化版本 数据传输服务DTS, 开通即用,免去部署维护的昂贵使用成本。DTS针对阿里云RDS、DRDS等产品进行了适配,解决了Binlog日志回收,主备切换、VPC网络切换等场景下的同步高可用问题。同时,针对RDS进行了针对性的性能优化。出于稳定性、性能及成本的考虑,强烈推荐阿里云用户使用DTS产品。DTS产品使用文档
  • DTS支持阿里云RDS&DRDS的Binlog日志实时订阅,现推出首月0折体验,限时限量,立即体验>>>

    环境搭建 & 打包

    环境搭建:

    1. 进入$otter_home目录
    2. 执行:mvn clean install
    3. 导入maven项目。如果eclipse下报"Missing artifact com.oracle:ojdbc14:jar:10.2.0.3.0",修改$otter_home/pom.xml中"${user.dir}/lib/ojdbc14-10.2.0.3.0.jar"为绝对路径,比如"d:/lib/ojdbc14-10.2.0.3.0.jar"

    打包:

    1. 进入$otter_home目录
    2. 执行:mvn clean install -Dmaven.test.skip -Denv=release
    3. 发布包位置:$otter_home/target

    项目背景

       阿里巴巴B2B公司,因为业务的特性,卖家主要集中在国内,买家主要集中在国外,所以衍生出了杭州和美国异地机房的需求,同时为了提升用户体验,整个机房的架构为双A,两边均可写,由此诞生了otter这样一个产品。

       otter第一版本可追溯到04~05年,此次外部开源的版本为第4版,开发时间从2011年7月份一直持续到现在,目前阿里巴巴B2B内部的本地/异地机房的同步需求基本全上了otte4。

    目前同步规模:

    1. 同步数据量6亿
    2. 文件同步1.5TB(2000w张图片)
    3. 涉及200+个数据库实例之间的同步
    4. 80+台机器的集群规模

    项目介绍

    名称:otter ['ɒtə(r)]

    译意: 水獭,数据搬运工

    语言: 纯java开发

    定位: 基于数据库增量日志解析,准实时同步到本机房或异地机房的mysql/oracle数据库. 一个分布式数据库同步系统

    工作原理

    原理描述:

    1. 基于Canal开源产品,获取数据库增量日志数据。 什么是Canal, 请点击

    2. 典型管理系统架构,manager(web管理)+node(工作节点)

          a. manager运行时推送同步配置到node节点

          b. node节点将同步状态反馈到manager上

    3. 基于zookeeper,解决分布式状态调度的,允许多node节点之间协同工作.

    什么是canal?

    otter之前开源的一个子项目,开源链接地址:http://github.com/alibaba/canal

    Introduction

    See the page for introduction: Introduction.

    QuickStart

    See the page for quick start: QuickStart.

    AdminGuide

    See the page for admin deploy guide : AdminGuide

    相关文档

    See the page for 文档: 相关PPT&PDF

    常见问题

    See the page for FAQ: FAQ

    版本相关:

    1. 建议版本:4.2.15 (otter开源版本从内部演变而来,所以初始版本直接从4.x开始)

    2. 下载发布包:download

    3. maven依赖 : 暂无

    相关开源

    1. 阿里巴巴mysql数据库binlog的增量订阅&消费组件:http://github.com/alibaba/canal
    2. 阿里巴巴去Oracle数据迁移同步工具(目标支持MySQL/DRDS):http://github.com/alibaba/yugong

    问题反馈

    注意:canal&otter QQ讨论群已经建立,群号:161559791 ,欢迎加入进行技术讨论。

    1. qq交流群: 161559791

    2. 邮件交流: jianghang115@gmail.com

    3. 新浪微博: agapple0002

    4. 报告issue:issues

    【招聘】阿里巴巴中间件团队招聘JAVA高级工程师
    岗位主要为技术型内容(非业务部门),阿里中间件整个体系对于未来想在技术上有所沉淀的同学还是非常有帮助的
    工作地点:杭州、北京均可. ps. 阿里待遇向来都是不错的,有意者可以QQ、微博私聊. 
    具体招聘内容:https://job.alibaba.com/zhaopin/position_detail.htm?positionId=32666
    

    Extension points exported contracts — how you extend this code

    EventProcessor (Interface)
    业务自定义处理过程 @author jianghang 2012-6-25 下午02:26:36 @version 4.1.0 [11 implementers]
    shared/etl/src/main/java/com/alibaba/otter/shared/etl/extend/processor/EventProcessor.java
    ComputeFunction (Interface)
    cache失效后重新计算的函数 [29 implementers]
    shared/common/src/main/java/com/alibaba/otter/shared/common/utils/cache/RefreshMemoryMirror.java
    EventType (Interface)
    通讯事件类型 @author jianghang 2011-9-9 下午04:02:13 [8 implementers]
    shared/communication/src/main/java/com/alibaba/otter/shared/communication/core/model/EventType.java
    ArbitrateEvent (Interface)
    具体的仲裁行为接口 @author jianghang 2011-8-9 下午04:39:20 [7 implementers]
    shared/arbitrate/src/main/java/com/alibaba/otter/shared/arbitrate/impl/ArbitrateEvent.java
    GenericService (Interface)
    @author simon 2011-10-31 上午10:34:17 [11 implementers]
    manager/biz/src/main/java/com/alibaba/otter/manager/biz/common/baseservice/GenericService.java
    ActionProtected (Interface)
    基于url的匹配 @author jianghang 2011-11-11 下午12:30:43 @version 4.0.0 [14 implementers]
    manager/web/src/main/java/com/alibaba/otter/manager/web/webx/valve/auth/action/ActionProtected.java
    OtterExtractor (Interface)
    组装数据,有多种来源,mysql,oracle,store,file等. [7 implementers]
    node/etl/src/main/java/com/alibaba/otter/node/etl/extract/extractor/OtterExtractor.java
    NodeTaskListener (Interface)
    在nodeTask发生变化时,广播通知下 @author jianghang 2012-4-20 下午10:45:17 @version 4.0.2 [11 implementers]
    node/common/src/main/java/com/alibaba/otter/node/common/config/NodeTaskListener.java

    Core symbols most depended-on inside this repo

    add
    called by 591
    shared/common/src/main/java/com/alibaba/otter/shared/common/utils/compile/model/JdkCompilerClassLoader.java
    get
    called by 568
    node/etl/src/main/java/com/alibaba/otter/node/etl/common/pipe/Pipe.java
    put
    called by 561
    node/etl/src/main/java/com/alibaba/otter/node/etl/common/pipe/Pipe.java
    size
    called by 343
    shared/arbitrate/src/main/java/com/alibaba/otter/shared/arbitrate/impl/setl/TerminArbitrateEvent.java
    push
    called by 319
    node/etl/src/main/java/com/alibaba/otter/node/etl/common/jmx/StageAggregation.java
    error
    called by 257
    node/etl/src/main/java/com/alibaba/otter/node/etl/load/loader/interceptor/LoadInterceptor.java
    getPipelineId
    called by 236
    node/etl/src/main/java/com/alibaba/otter/node/etl/model/protobuf/BatchProto.java
    call
    called by 209
    shared/communication/src/main/java/com/alibaba/otter/shared/communication/core/model/Callback.java

    Shape

    Method 6,162
    Class 803
    Function 530
    Interface 141
    Enum 47

    Languages

    Java93%
    TypeScript7%

    Modules by API surface

    node/etl/src/main/java/com/alibaba/otter/node/etl/model/protobuf/BatchProto.java662 symbols
    shared/common/src/main/java/com/alibaba/otter/shared/common/model/config/pipeline/PipelineParameter.java112 symbols
    manager/deployer/src/main/resources/webapp/js/flot/jquery.flot.js63 symbols
    manager/deployer/src/main/resources/webapp/js/flot/jquery.flot.min.js59 symbols
    shared/common/src/main/java/com/alibaba/otter/shared/common/utils/zookeeper/ZkClientx.java57 symbols
    shared/push/src/main/java/com/alibaba/otter/common/push/datasource/media/MediaPushDataSource.java49 symbols
    manager/deployer/src/main/resources/webapp/js/flot/jquery.js49 symbols
    manager/deployer/src/main/resources/webapp/js/flot/jquery.min.js44 symbols
    shared/etl/src/main/java/com/alibaba/otter/shared/etl/model/EventData.java42 symbols
    node/etl/src/main/java/com/alibaba/otter/node/etl/OtterController.java42 symbols
    shared/common/src/main/java/com/alibaba/otter/shared/common/model/config/parameter/SystemParameter.java41 symbols
    node/etl/src/main/java/com/alibaba/otter/node/etl/load/loader/db/DbLoadAction.java38 symbols

    Dependencies from manifests, versioned

    cglib:cglib-nodep2.2 · 1×
    ch.qos.logback:logback-classic1.1.3 · 1×
    ch.qos.logback:logback-core1.1.3 · 1×
    com.101tec:zkclient0.10 · 1×
    com.alibaba.citrus:citrus-webx-all
    com.alibaba.fastsql:fastsql2.0.0_preview_973 · 1×
    com.alibaba.otter:canal.instance.manager
    com.alibaba.otter:canal.parse
    com.alibaba.otter:canal.server
    com.alibaba.otter:manager.biz
    com.alibaba.otter:manager.web
    com.alibaba.otter:node.canal

    Datastores touched

    (mysql)Database · 1 repos
    erosaDatabase · 1 repos
    otterDatabase · 1 repos
    retlDatabase · 1 repos
    srfDatabase · 1 repos

    For agents

    $ claude mcp add otter \
      -- python -m otcore.mcp_server <graph>

    ⬇ download graph artifact