hub / github.com/GoogleCloudPlatform/DataflowTemplates

github.com/GoogleCloudPlatform/DataflowTemplates @main

26,896 symbols 162,241 edges 2,864 files 5,882 documented · 22% ● updated today2026-08-25-00_RC00 · 2026-08-26★ 1,31019 open issues

Browse by type

⨍ Functions 23,218◇ Types & classes 3,671↳ Endpoints 7

What it actually does AI analysis from the code graph — generated when you open this

loading…

README

Google Cloud Dataflow Template Pipelines

These Dataflow templates are an effort to solve simple, but large, in-Cloud data tasks, including data import/export/backup/restore and bulk API operations, without a development environment. The technology under the hood which makes these operations possible is the Google Cloud Dataflow service combined with a set of Apache Beam SDK templated pipelines.

Google is providing this collection of pre-implemented Dataflow templates as a reference and to provide easy customization for developers wanting to extend their functionality.

Status

Note on Default Branch

As of November 18, 2021, our default branch is now named main. This does not affect forks. If you would like your fork and its local clone to reflect these changes you can follow GitHub's branch renaming guide.

Getting Help

Have a bug or a feature request? Please file a GitHub issue.

Template Pipelines

Extension points exported contracts — how you extend this code

SourceSchemaScanner (Interface)

Interface for scanning information schema of source databases. Different implementations will handle different source da [10 …

v2/spanner-common/src/main/java/com/google/cloud/teleport/v2/spanner/sourceddl/SourceSchemaScanner.java

RetryStrategy (Interface)

An interface used to control if we retry the statements when a SQLException occurs. If {@link RetryStrategy#appl [92 implementers]

v2/datastream-common/src/main/java/com/google/cloud/teleport/v2/datastream/io/CdcJdbcIO.java

IDsToSpSourceConnector (Interface)

Interface for Datastream source connectors. Encapsulates all source-specific logic for processing change events and mana [6 …

v2/datastream-to-spanner/src/main/java/com/google/cloud/teleport/v2/templates/source/IDsToSpSourceConnector.java

UnifiedTypeMapping (Interface)

Interface to convert a source schema to Schema Avro Schema implemented by providers of various source types, lik [38 implementers]

v2/sourcedb-to-spanner/src/main/java/com/google/cloud/teleport/v2/reader/io/schema/typemapping/UnifiedTypeMapping.java

IShardIdFetcher (Interface)

The interface to get the shard identifier. [10 implementers]

v2/spanner-migrations-sdk/src/main/java/com/google/cloud/teleport/v2/spanner/utils/IShardIdFetcher.java

IDMLGenerator (Interface)

Interface for generating DML statements. This interface provides a contract for implementing classes to define the l [9 …

v2/spanner-to-sourcedb/src/main/java/com/google/cloud/teleport/v2/templates/dbutils/dml/IDMLGenerator.java

browse all types & interfaces →

Core symbols most depended-on inside this repo

↓ 470 filesRetryStrategy

An interface used to control if we retry the statements when a SQLException occurs. If RetryStrategy#apply(SQLException) returns …

v2/datastream-common/src/main/java/com/google/cloud/teleport/v2/datastream/io/CdcJdbcIO.java

↓ 410 filesPubSubValueFormatter

Interface which allows extracting data from raw JSON object and preparing it for BigQuery consumption.

v2/googlecloud-to-googlecloud/src/main/java/com/google/cloud/teleport/v2/templates/bigtablechangestreamstopubsub/schemautils/PubSubValueFormatter.java

↓ 506 filesSSLOptionsFactory

v2/sourcedb-to-spanner/src/main/java/com/google/cloud/teleport/v2/source/cassandra/reader/io/cassandra/iowrapper/SSLOptionsProvider.java

↓ 467 filesConnectionProvider

v2/cdc-data-generator/src/main/java/com/google/cloud/teleport/v2/templates/mysql/MySqlSchemaFetcher.java

↓ 261 filesArtifact

Represents a single artifact. An "artifact" is an entity in object storage, file storage, or block storage. Artifacts should be …

it/google-cloud-platform/src/main/java/org/apache/beam/it/gcp/artifacts/Artifact.java

↓ 249 filesTemplateTestBase

Base class for Templates. It wraps around tests that extend it to stage the Templates when -DspecPath isn't provided. It is …

it/google-cloud-platform/src/main/java/org/apache/beam/it/gcp/TemplateTestBase.java

browse all functions →

Shape

Method 22,997

Class 3,227

Interface 321

Function 221

Enum 99

Struct 18

Route 7

TypeAlias 6

Languages

Java96%

Python3%

Go1%

TypeScript1%

Modules by API surface

v1/src/main/java/com/google/cloud/teleport/spanner/spannerio/SpannerIO.java236 symbols

python/src/test/python/bigquery-anomaly-detection/pipeline_test.py207 symbols

v2/elasticsearch-common/src/main/java/com/google/cloud/teleport/v2/elasticsearch/utils/ElasticsearchIO.java166 symbols

v2/googlecloud-to-googlecloud/src/main/java/com/google/cloud/teleport/v2/utils/FirestoreConverters.java156 symbols

v2/sourcedb-to-spanner/src/main/java/org/apache/beam/sdk/io/localcassandra/CassandraIO.java146 symbols

v1/src/main/java/com/google/cloud/teleport/templates/common/DatastoreConverters.java143 symbols

v2/spanner-to-sourcedb/src/test/java/com/google/cloud/teleport/v2/templates/source/cassandra/CassandraTypeHandlerTest.java101 symbols

v2/common/src/main/java/com/google/cloud/teleport/v2/transforms/ErrorConverters.java98 symbols

v2/googlecloud-to-neo4j/src/test/java/com/google/cloud/teleport/v2/neo4j/model/helpers/JobSpecIndexTest.java97 symbols

v2/datastream-common/src/main/java/com/google/cloud/teleport/v2/datastream/io/CdcJdbcIO.java96 symbols

v2/common/src/main/java/com/google/cloud/teleport/v2/transforms/BigQueryConverters.java94 symbols

v2/common/src/main/java/com/google/cloud/teleport/v2/transforms/CsvConverters.java90 symbols

Datastores touched

(mysql)Database · 1 repos

(mongodb)Database · 1 repos

sampledbDatabase · 1 repos

dbDatabase · 1 repos

your-dbDatabase · 1 repos

my-dbDatabase · 1 repos

testdbDatabase · 1 repos

postgresDatabase · 1 repos

For agents

$ claude mcp add DataflowTemplates \
  -- python -m otcore.mcp_server <graph>

⬇ download graph artifact