s3grep - Fast Parallel grep for S3

I use grep almost daily, but often deal with unstructured content and logs on S3...so I wrote an easy way to grep S3!

s3grep is a parallel CLI tool for searching logs and unstructured content in Amazon S3 buckets. It supports .gz decompression, progress bars, and robust error handling—making it ideal for cloud-native log analysis.

---

Features

• Parallel, concurrent search across S3 objects
• Supports plain text and .gz compressed files
• Progress bars for files and bytes processed
• Case-sensitive and insensitive search
• Line number output option
• Graceful handling of binary files and decompression errors
• Colorized match highlighting

---

Installation

Pre-built Binaries

For now, just git clone and cargo build. 😁

Pre-built binaries (will be) available on the Releases page.

---

Usage

s3grep --pattern "ERROR" --bucket my-logs-bucket --prefix logs/ --concurrent-tasks 16

CLI Options

Flag	Description
-p, --pattern	Search pattern (required)
-b, --bucket	S3 bucket name (required)
-z, --prefix	S3 prefix to search in (default: "")
-c, --concurrent-tasks	Number of concurrent tasks (default: 8)
-i, --case-sensitive	Case sensitive search
-q, --quiet	Hide progress bar
-n, --line-number	Show line numbers in output

---

Example

s3grep --pattern "timeout" --bucket my-bucket --prefix logs/2025/06/ --concurrent-tasks 12 --line-number

---

Testing

Integration tests use Localstack to mock S3. See CONTRIBUTING.md for details.

---

Contributing

Contributions are welcome! Please see CONTRIBUTING.md for guidelines.

---

License

This project is licensed under the MIT License.

---

Acknowledgments

• Inspired by daily use of grep and the need for cloud-native log search.
• Built with Rust, aws-sdk-rust, and Localstack for testing.