hub / github.com/ShifuML/guagua

github.com/ShifuML/guagua @guagua-0.7.8

Chat with this repo

repository ↗ · DeepWiki ↗ · release guagua-0.7.8 ↗ · + Follow

1,639 symbols 4,482 edges 202 files 643 documented · 39%

What it actually does AI analysis from the code graph — generated when you open this

loading…

README

Guagua

An iterative computing framework on both Hadoop MapReduce and Hadoop YARN.

News

Guagua 0.7.7 is released with a lot of improvements. Check our [changes] (https://github.com/ShifuML/guagua/blob/master/CHANGES.txt#L19)

Conference

QCON Shanghai 2014 Slides

Getting Started

Please visit Guagua wiki site for tutorials.

What is Guagua?

Guagua, a sub-project of Shifu, is a distributed, pluggable and scalable iterative computing framework based on Hadoop MapReduce and YARN.

This graph shows the iterative computing process for Guagua.

Guagua Process

Typical use cases for Guagua are distributed machine learning model training based on Hadoop. By using Guagua, we implement distributed neural network algorithm which can reduce model training time from days to hours on 1TB data sets. Distributed neural network algorithm is based on Encog and Guagua. Any details please check our example source code.

Google Group

Please join Guagua group if questions, bugs or anything else.

Copyright and License

Extension points exported contracts — how you extend this code

GuaguaRecordReader (Interface)

GuaguaRecordReader is used for consistent interface to iterate data through FileSplit provided. The typical impl [13 implementers]

guagua-core/src/main/java/ml/shifu/guagua/io/GuaguaRecordReader.java

Bytable (Interface)

Bytable defines master and worker results. Bytable is coped from Hadoop io Writable. Hadoop Writabl [10 implementers]

guagua-core/src/main/java/ml/shifu/guagua/io/Bytable.java

WorkerComputable (Interface)

WorkerComputable defines worker computation for any guagua application. All guagua application should define its [8 implementers]

guagua-core/src/main/java/ml/shifu/guagua/worker/WorkerComputable.java

WorkerInterceptor (Interface)

WorkerInterceptor defines hook points for each guagua application. Four hooks are defined like hooks before and [14 implementers]

guagua-core/src/main/java/ml/shifu/guagua/worker/WorkerInterceptor.java

MasterInterceptor (Interface)

MasterInterceptor defines hook points for each guagua application. Four hooks are defined like hooks before and [14 implementers]

guagua-core/src/main/java/ml/shifu/guagua/master/MasterInterceptor.java

Core symbols most depended-on inside this repo

get

called by 175

guagua-core/src/main/java/ml/shifu/guagua/util/StringUtils.java

append

called by 129

guagua-core/src/main/java/ml/shifu/guagua/util/AppendList.java

size

called by 94

guagua-core/src/main/java/ml/shifu/guagua/util/AppendList.java

getZooKeeper

called by 82

guagua-core/src/main/java/ml/shifu/guagua/coordinator/zk/GuaguaZooKeeper.java

called by 77

guagua-core/src/main/java/ml/shifu/guagua/io/GuaguaRecordReader.java

add

called by 62

guagua-yarn/src/main/java/ml/shifu/guagua/yarn/util/InputSplitUtils.java

add

called by 56

guagua-mapreduce/src/main/java/ml/shifu/guagua/mapreduce/GuaguaInputFormat.java

getProps

called by 55

guagua-core/src/main/java/ml/shifu/guagua/worker/WorkerContext.java

Shape

Method 1,411

Class 208

Interface 18

Enum 2

Languages

Java100%

Modules by API surface

guagua-yarn/src/main/java/ml/shifu/guagua/yarn/GuaguaAppMaster.java85 symbols

guagua-core/src/main/java/ml/shifu/guagua/BasicCoordinator.java61 symbols

guagua-core/src/main/java/ml/shifu/guagua/master/NettyMasterCoordinator.java43 symbols

guagua-yarn/src/main/java/ml/shifu/guagua/yarn/GuaguaYarnClient.java41 symbols

guagua-core/src/main/java/ml/shifu/guagua/master/GuaguaMasterService.java37 symbols

guagua-yarn/src/main/java/ml/shifu/guagua/yarn/util/InputSplitUtils.java33 symbols

guagua-yarn/src/main/java/ml/shifu/guagua/yarn/GuaguaYarnTask.java32 symbols

guagua-core/src/main/java/ml/shifu/guagua/worker/GuaguaWorkerService.java31 symbols

guagua-mapreduce/src/main/java/ml/shifu/guagua/mapreduce/GuaguaInputFormat.java30 symbols

guagua-core/src/main/java/ml/shifu/guagua/master/MasterContext.java26 symbols

guagua-core/src/main/java/ml/shifu/guagua/master/AbstractMasterCoordinator.java25 symbols

guagua-core/src/main/java/ml/shifu/guagua/coordinator/zk/ZooKeeperUtils.java25 symbols

For agents

$ claude mcp add guagua \
  -- python -m otcore.mcp_server <graph>

⬇ download graph artifact