github.com/estuary/flow @v0.6.10

| Docs home | Free account | Data platform comparison reference | Contact us

<img src ="site/static/img/estuary-new.png"
 width="250"/>

Build millisecond-latency, scalable, future-proof data pipelines in minutes.

Estuary is the Right-Time Data Platform that integrates all of the systems you use to produce, process, and consume data.

Estuary unifies today's batch and streaming paradigms so that your systems – current and future – are synchronized around the same datasets, updating in milliseconds.

With an Estuary pipeline, you:

• 📷 Capture data from your systems, services, and SaaS into collections: millisecond-latency datasets that are stored as regular files of JSON data, right in your cloud storage bucket.

• 🎯 Materialize a collection as a view within another system, such as a database, key/value store, Webhook API, or pub/sub service.

• 🌊 Derive new collections by transforming from other collections, using the full gamut of stateful stream workflow, joins, and aggregations — in real time.

Publish your data flows in Estuary's shared SaaS environment. Or use a private or BYOC deployment for enterprise-ready security.

Get started

Ready to try out Estuary? Sign up for free to get started! 🚀

Have questions? We'd love to hear from you:

• Join our Slack Community
• Reach out directly

Using Estuary

Estuary combines a low-code UI for essential workflows and a CLI for fine-grain control over your pipelines. Together, the two interfaces comprise Estuary's unified platform. You can switch seamlessly between them as you build and refine your pipelines, and collaborate with a wider breadth of data stakeholders.

• The UI-based web application is at dashboard.estuary.dev.
• Install the flowctl CLI using these instructions.

➡️ Sign up for a free Estuary account here.

See the BSL license for information on using Estuary outside the managed offering.

Resources

• 📖 Estuary documentation

• 🧐 Examples and tutorials

  • Blog tutorials
  • Docs tutorials
  • Create a Basic Dataflow
  • PostgreSQL CDC Streaming to Snowflake
  • Real-time CDC with MongoDB
  • GitHub examples
    • See our example projects and demos
    • Many examples/ in this repo cover derivations, reductions, and other flow.yaml examples

Support

The best (and fastest) way to get support from the Estuary team is to join the community on Slack.

You can also email us.

Connectors

Captures and materializations use connectors: plug-able components that integrate Flow with external data systems. Estuary's in-house connectors focus on high-scale technology systems and change data capture (think databases, pub-sub, and filestores).

Estuary can run Airbyte community connectors using airbyte-to-flow, allowing us to support a greater variety of SaaS systems.

See our website for the full list of currently supported connectors.

If you don't see what you need, request it here.

How does it work?

Estuary builds on a real-time streaming broker created by the same founding team called Gazette.

Because of this, Estuary's collections are both a batch dataset – they're stored as a structured "data lake" of general-purpose files in cloud storage – and a stream, able to commit new documents and forward them to readers within milliseconds. New use cases read directly from cloud storage for high-scale backfills of history, and seamlessly transition to low-latency streaming on reaching the present.

• Learn more about how Gazette works here.

• Learn more about how Estuary works here.

What makes Estuary so fast?

Estuary mixes a variety of architectural techniques to achieve great throughput without adding latency:

• Optimistic pipelining, using the natural back-pressure of systems to which data is committed.
• Leveraging reduce annotations to group collection documents by key wherever possible, in memory, before writing them out.
• Co-locating derivation states (registers) with derivation compute: registers live in an embedded RocksDB that's replicated for durability and machine re-assignment. They update in memory and only write out at transaction boundaries.
• Vectorizing the work done in external Remote Procedure Calls (RPCs) and even process-internal operations.
• Marrying the development velocity of Go with the raw performance of Rust, using a zero-copy CGO service channel.