JOCKO

Kafka/distributed commit log service in Go.

Goals of this project:

• Implement Kafka in Go
• Protocol compatible with Kafka so Kafka clients and services work with Jocko
• Make operating simpler
• Distribute a single binary
• Use Serf for discovery, Raft for consensus (and remove the need to run ZooKeeper)
• Smarter configuration settings
  • Able to use percentages of disk space for retention policies rather than only bytes and time kept
  • Handling size configs when you change the number of partitions or add topics
• Learn a lot and have fun

TODO

• [x] Producing
• [x] Fetching
• [x] Partition consensus and distribution
• [ ] Protocol
  • [x] Produce
  • [x] Fetch
  • [x] Metadata
  • [x] Create Topics
  • [x] Delete Topics
  • [ ] Consumer group
• [x] Discovery
• [ ] API versioning
• [ ] Replication [current task]
• [ ] Tests [current task]

Reading

• How Jocko's built-in service discovery and consensus works
• How Jocko's (and Kafka's) storage internals work

Project Layout

├── broker        broker subsystem
├── cmd           commands
│   └── jocko     command to run a Jocko broker and manage topics
├── commitlog     low-level commit log implementation
├── examples      examples running/using Jocko
│   ├── cluster   example booting up a 3-broker Jocko cluster
│   └── sarama    example producing/consuming with Sarama
├── protocol      golang implementation of Kafka's protocol
├── raft          wrapper around Hashicorp's Raft lib to handle consensus
├── serf          wrapper around Hashicorp's Serf lib to handle service discovery
├── prometheus    wrapper around Prometheus' client lib to handle metrics
├── server        API subsystem
└── testutil      test utils
    └── mock      mocks of the various subsystems

License

MIT

• travisjeffery.com
• GitHub @travisjeffery
• Twitter @travisjeffery
• Medium @travisjeffery