<img width="341" height="426" src="https://github.com/andkret/Cookbook/raw/main/images/CookbookCover.jpg" alt="Data Engineering Cookbook">
















<a href="https://github.com/andkret/Cookbook/raw/main/sections/01-Introduction.md">What is this Book?</a>&nbsp;&nbsp;&nbsp;

How to Contribute    YouTube    <a Twitter    Amazon Shop

If You Like This Book & Need More Help

Check out my Data Engineering Academy at LearnDataEngineering.com trusted by almost 2,000 students!

Visit learndataengineering.com: Click Here

• Learn Data Engineering with our online Academy
• Perfect for becoming a Data Engineer or add Data Engineering to your skillset
• Proven process based on years of experience and hundreds of hours of personal coaching
• Over 30 prepared courses on the most important techniques, fundamental tools and platforms plus our
• Associate Data Engineer Certification
• Academy Discord server with over 1,000 members

Support This Book For Free!

• Amazon: Click Here buy whatever you like from Amazon using this link* (Also check out my complete podcast gear and books)

Here's what's new:

Find the change log with all recent updates here: SEE UPDATES

Contents:

• Introduction
• Basic Engineering Skills
• Advanced Engineering Skills
• Free Hands On Courses / Tutorials‚
• Case Studies
• Best Practices Cloud Platforms
• 130+ Data Sources Data Science
• 1001 Interview Questions
• Recommended Books, Courses, and Podcasts

• Updates

• How To Contribute

• Support What You Like
• Important Links

Full Table Of Contents:

Introduction

• What is this Cookbook
• Data Engineers
• My Data Science Platform Blueprint
• Connect
• Buffer
• Processing Framework
• Store
• Visualize
• Who Companies Need
• How to Learn Data Engineering
  • Andreas on the Super Data Science Podcast
  • Building Blocks to Learn Data Engineering
• Roadmap for Beginners
  • Roadmap for Data Analysts
  • Roadmap for Data Scientists
  • Roadmap for Software Engineers
• Data Engineers Skills Matrix
• How to Become a Senior Data Engineer

Basic Engineering Skills

• Learn To Code
• Get Familiar With Git
• Agile Development
• Why is agile so important?
• Agile rules I learned over the years
• Agile Frameworks
  • Scrum
  • OKR
• Software Engineering Culture
• Learn how a Computer Works
• Data Network Transmission
• Security and Privacy
• SSL Public and Private Key Certificates
• JSON Web Tokens
• GDPR regulations
• Linux
• OS Basics
• Shell scripting
• Cron Jobs
• Packet Management
• Docker
• What is Docker and How it Works
  • Don't Mess Up Your System
  • Preconfigured Images
  • Take it With You
  • Kubernetes Container Deployment
  • How to Create Start and Stop a Container
  • Docker Micro Services
  • Kubernetes
  • Why and How To Do Docker Container Orchestration
  • Userful Docker Commands
• The Cloud
• IaaS vs PaaS vs SaaS
• AWS Azure IBM Google IBM
• Cloud vs On-Premises
• Security
• Hybrid Clouds
• Security Zone Design
• How to secure a multi layered application
• Cluster security with Kerberos

Advanced Engineering Skills

• Data Science Platform
• Why a Good Data Platform Is Important
• Big Data vs Data Science and Analytics
• The 4 Vs of Big Data
• Why Big Data
  • Planning is Everything
  • The Problem with ETL
  • Scaling Up
  • Scaling Out
  • When not to Do Big Data
• 81 Platform & Pipeline Design Questions
• Data Source Questions
• Goals and Destination Questions
• Connect
• REST APIs
  • API Design
  • Implemenation Frameworks
  • Security
• Apache Nifi
• Logstash
• Buffer
• Apache Kafka
  • Why a Message Queue Tool?
  • Kafka Architecture
  • Kafka Topics
  • Kafka and Zookeeper
  • How to Produce and Consume Messages
  • Kafka Commands
• Apache Redis Pub-Sub
• AWS Kinesis
• Google Cloud PubSub
• Processing Frameworks
  • Lambda and Kappa Architecture
  • Batch Processing
  • Stream Processing
    • Three Methods of Streaming
    • At Least Once
    • At Most Once
    • Exactly Once
    • Check The Tools
  • Should You do Stream or Batch Processing
  • Is ETL still relevant for Analytics?
• MapReduce
  • How Does MapReduce Work
  • MapReduce
  • MapReduce Example
  • MapReduce Limitations
• Apache Spark
  • What is the Difference to MapReduce?
  • How Spark Fits to Hadoop
  • Spark vs Hadoop
  • Spark and Hadoop a Perfect Fit
  • Spark on YARn
  • My Simple Rule of Thumb
  • Available Languages
  • Spark Driver Executor and SparkContext
  • Spark Batch vs Stream processing
  • How Spark uses Data From Hadoop
  • What are RDDs and How to Use Them
  • SparkSQL How and Why to Use It
  • What are Dataframes and How to Use Them
  • Machine Learning on Spark (TensorFlow)
  • MLlib
  • Spark Setup
  • Spark Resource Management
• AWS Lambda
• Apache Flink
• Elasticsearch
• Apache Drill
• StreamSets
• Store
  • Analytical Data Stores
    • Data Warehouse vs Data Lake
    • Snowflake and dbt
  • Transactional Data Stores
    • SQL Databases
    • PostgreSQL DB
    • Database Design
    • SQL Queries
    • Stored Procedures
    • ODBC/JDBC Server Connections
  • NoSQL Stores
    • HBase KeyValue Store
    • [HDFS Document Store](sect