cncf-fuzzing

This repository is related to fuzzing of CNCF projects. It holds fuzzers as well as documentation on fuzzing.

See the blog post Improving security by fuzzing the cncf landscape for details.

Are you a CNCF project and would like to have it fuzzed? Please create an issue on this repository requesting to be fuzzed and we will assist.

Fuzzing handbook

CNCF has a Fuzzing Handbook which introduces fuzzing in a pragmatic manner, and also has a chapter that goes into details with how to use OSS-Fuzz to establish a continuous fuzzing set up.

The handbook is available here

CNCF projects and fuzzing

Fuzzing is a technique for automating stress testing of applications and it can be used to find reliability and security issues. The technique is traditionally used by security researchers to find security vulnerabilities, however, fuzzing has become more integrated into the software development lifecycle and is increasingly being used by developers.

CNCF projects that use fuzzing include: - Argo - Containerd - CRI-O - Envoy - Fluent-bit - FluxCD and full report, section 5 - Kubernetes - Linkerd2-proxy - Prometheus - RunC - Vitess - Distribution

Talks on CNCF fuzzing: - Finding Bugs and Vulnerabilities Automatically - Fuzzing the CNCF Landscape, Cloud Native SecurityCon, 2022 - Securing Fluent Bit by Way of Fuzzing, FluentCon, 2022 - Fuzz Testing of Envoy - Adi Peleg & Teju Nareddy - Lightning Talk: Securing Envoy: Catching Vulnerabilities With Continuous Fuzz Testing - Teju Nareddy

Dedicated fuzzing audit reports: - Fluent Bit - Argo - etcd - linkerd2-proxy - Envoy - Vitess

Integrate fuzzing into your project

Integrating fuzzing into a project takes a lot effort and is often done over a long period of time. Fuzzing can be integrated into your project with various levels of maturity. There are three essential tasks when integrating fuzzing into your project: - Develop fuzzers - Execute the fuzzers - Analyse crashes

The following describes three common steps in integrating fuzzing into your project.

1) Local fuzzing set up

The first step in integrating fuzzing into a project is to develop a set of fuzz drivers for your project. The specific fuzzer you need to use depends on the programming language of your project. The following list provides links to common fuzzers for various languages: - C/C++: libFuzzer - Rust: Cargo-fuzz - Go: Go-fuzz and native go fuzzing - Python: Atheris fuzzer - Java: jazzer fuzzer

The specific purpose of a fuzz driver vary greatly. In essence, they are closely related to unit tests and the difference is the fuzz driver takes a random input which is used to enforce diverse code execution of the target code. Common goals of a fuzz driver include:

• Execute large amounts of code to achieve high code coverage
• Execute a specific complex piece of code, e.g. parsing routines
• Execute code relative to the threat model of project

This step is usually the most time-consuming and making it possible to write fuzz drivers for your project can sometimes be a large effort. However, once you have fuzz drivers for you project you should be able to run these locally and observe results.

2) Integrate continuous fuzzing with OSS-Fuzz

Once you have developed a local fuzzing set up for your project, the next step is to run the fuzzers in a continuous manner. Modern fuzzers rely on genetic algorithms to build up an input corpus, which, in a simplified manner, means that the fuzzer by nature increases it’s quality in proportion to how long it has run. Continuously running a fuzzer is thus important to ensure high quality of the fuzzing and continuous fuzzing is also important in order to capture bugs that may occur as a project progresses.

OSS-Fuzz is a service for running fuzzers continuously for open source projects. OSS-Fuzz comes with a convenient management infrastructure with a dashboard as well as bug-tracking features, which makes managing running of the fuzzers easy. We recommend integrating with OSS-Fuzz, and several CNCF projects are integrated already.

3) Integrate fuzzing into CI

Fuzzing can be integrated in your CI, e.g. a GitHub action, such that the fuzzers run for a short amount of time on pull requests and/or push actions. This is in many ways similar to running tests as part of your CI to ensure regressions don’t occur. Once you have integrated with OSS-Fuzz, you can get CI integration by way of CIFuzz for free.

What results to expect

Fuzzing works best with projects that have high code complexity, e.g. parsers, decoders, etc. but can be used in many other projects. You can fuzz projects in many languages and the type of bug you will find depends on which language your project is written in.

• Envoy has invested significantly in fuzzing and OSS-Fuzz has reported more than 700 bugs as well as 81 security relevant bugs
• Fluent-bit has been fuzzed for slightly more than a year, and OSS-Fuzz has reported more than 100 reliability issues and more than 50 security issues.

For an example where fuzzing was determined to have limited effects consider Cloud custodian. Cloud custodian is a project written in Python and is very horizontal in its architecture in that it does not have deep code complexities. This is an example where fuzzing will have limited results as discussed in detail in a PR on the Cloud Custodian repository. However, Cloud Custodian still benefited from fuzzing finding a bug in the code of Cloud Custodian where fuzzing could be applied, but, in comparison to the other projects mentioned above Cloud Custodian is not integrated into OSS-Fuzz.

The following list indicates some common software properties that means your code is likely to benefit from fuzzing - High code complexity - Deep code paths - Accepts untrusted input - If a reliability or reliability issue occur then it can have significant consequences for systems - Is used as a library by other applications - Projects in memory unsafe languages should have a high priority for being fuzzed (but fuzzing is not exclusive to memory unsafe languages)

Fuzzing hours

We have monthly fuzzing hours on every second Friday of each month at 2PM GMT+1.

URL: https://us05web.zoom.us/j/86181802116 Passcode: LBw6fY