hub / github.com/GoogleCloudPlatform/DataflowTemplates

github.com/GoogleCloudPlatform/DataflowTemplates @main

26,896 symbols 162,241 edges 2,864 files 5,882 documented · 22%

What it actually does AI analysis from the code graph — generated when you open this

loading…

README

Google Cloud Dataflow Template Pipelines

These Dataflow templates are an effort to solve simple, but large, in-Cloud data tasks, including data import/export/backup/restore and bulk API operations, without a development environment. The technology under the hood which makes these operations possible is the Google Cloud Dataflow service combined with a set of Apache Beam SDK templated pipelines.

Google is providing this collection of pre-implemented Dataflow templates as a reference and to provide easy customization for developers wanting to extend their functionality.

Status

Note on Default Branch

As of November 18, 2021, our default branch is now named main. This does not affect forks. If you would like your fork and its local clone to reflect these changes you can follow GitHub's branch renaming guide.

Getting Help

Have a bug or a feature request? Please file a GitHub issue.

Template Pipelines

Extension points exported contracts — how you extend this code

SourceSchemaScanner (Interface)

Interface for scanning information schema of source databases. Different implementations will handle different source da [10 …

v2/spanner-common/src/main/java/com/google/cloud/teleport/v2/spanner/sourceddl/SourceSchemaScanner.java

RetryStrategy (Interface)

An interface used to control if we retry the statements when a SQLException occurs. If {@link RetryStrategy#appl [92 implementers]

v2/datastream-common/src/main/java/com/google/cloud/teleport/v2/datastream/io/CdcJdbcIO.java

IDsToSpSourceConnector (Interface)

Interface for Datastream source connectors. Encapsulates all source-specific logic for processing change events and mana [6 …

v2/datastream-to-spanner/src/main/java/com/google/cloud/teleport/v2/templates/source/IDsToSpSourceConnector.java

UnifiedTypeMapping (Interface)

Interface to convert a source schema to Schema Avro Schema implemented by providers of various source types, lik [38 implementers]

v2/sourcedb-to-spanner/src/main/java/com/google/cloud/teleport/v2/reader/io/schema/typemapping/UnifiedTypeMapping.java

IShardIdFetcher (Interface)

The interface to get the shard identifier. [10 implementers]

v2/spanner-migrations-sdk/src/main/java/com/google/cloud/teleport/v2/spanner/utils/IShardIdFetcher.java

IDMLGenerator (Interface)

Interface for generating DML statements. This interface provides a contract for implementing classes to define the l [9 …

v2/spanner-to-sourcedb/src/main/java/com/google/cloud/teleport/v2/templates/dbutils/dml/IDMLGenerator.java

PreloadAction (Interface)

Interface for running preload Actions. Before the pipeline loads, PCollections are not available. [6 implementers]

v2/googlecloud-to-neo4j/src/main/java/com/google/cloud/teleport/v2/neo4j/actions/preload/PreloadAction.java

TemplateBlock (Interface)

TemplateBlock interface, implementing classes will be used as blocks in a Template. [7 implementers]

metadata/src/main/java/com/google/cloud/teleport/metadata/auto/TemplateBlock.java

Core symbols most depended-on inside this repo

put

called by 7365

v2/sourcedb-to-spanner/src/main/java/com/google/cloud/teleport/v2/reader/io/jdbc/rowmapper/JdbcMappings.java

get

called by 2942

v2/cdc-data-generator/src/main/java/com/google/cloud/teleport/v2/templates/mysql/MySqlSchemaFetcher.java

apply

called by 2692

v2/datastream-common/src/main/java/com/google/cloud/teleport/v2/datastream/io/CdcJdbcIO.java

called by 2449

plugins/core-plugin/src/test/java/com/google/cloud/teleport/plugin/sample/AtoBOk.java

called by 2338

v1/src/main/java/com/google/cloud/teleport/spanner/FileShard.java

set

called by 2036

v2/spanner-common/src/main/java/com/google/cloud/teleport/v2/spanner/ddl/IndexColumn.java

addParameter

called by 1923

v1/src/main/java/com/google/cloud/teleport/spanner/ddl/Udf.java

name

called by 1865

it/google-cloud-platform/src/main/java/org/apache/beam/it/gcp/artifacts/Artifact.java

Shape

Method 22,997

Class 3,227

Interface 321

Function 221

Enum 99

Struct 18

Route 7

TypeAlias 6

Languages

Java96%

Python3%

Go1%

TypeScript1%

Modules by API surface

v1/src/main/java/com/google/cloud/teleport/spanner/spannerio/SpannerIO.java236 symbols

python/src/test/python/bigquery-anomaly-detection/pipeline_test.py207 symbols

v2/elasticsearch-common/src/main/java/com/google/cloud/teleport/v2/elasticsearch/utils/ElasticsearchIO.java166 symbols

v2/googlecloud-to-googlecloud/src/main/java/com/google/cloud/teleport/v2/utils/FirestoreConverters.java156 symbols

v2/sourcedb-to-spanner/src/main/java/org/apache/beam/sdk/io/localcassandra/CassandraIO.java146 symbols

v1/src/main/java/com/google/cloud/teleport/templates/common/DatastoreConverters.java143 symbols

v2/spanner-to-sourcedb/src/test/java/com/google/cloud/teleport/v2/templates/source/cassandra/CassandraTypeHandlerTest.java101 symbols

v2/common/src/main/java/com/google/cloud/teleport/v2/transforms/ErrorConverters.java98 symbols

v2/googlecloud-to-neo4j/src/test/java/com/google/cloud/teleport/v2/neo4j/model/helpers/JobSpecIndexTest.java97 symbols

v2/datastream-common/src/main/java/com/google/cloud/teleport/v2/datastream/io/CdcJdbcIO.java96 symbols

v2/common/src/main/java/com/google/cloud/teleport/v2/transforms/BigQueryConverters.java94 symbols

v2/common/src/main/java/com/google/cloud/teleport/v2/transforms/CsvConverters.java90 symbols

Datastores touched

(mysql)Database · 1 repos

(mongodb)Database · 1 repos

sampledbDatabase · 1 repos

dbDatabase · 1 repos

your-dbDatabase · 1 repos

my-dbDatabase · 1 repos

testdbDatabase · 1 repos

postgresDatabase · 1 repos

For agents

$ claude mcp add DataflowTemplates \
  -- python -m otcore.mcp_server <graph>

⬇ download graph artifact