BGPKIT Parser

This readme is generated from the library's doc comments using cargo-readme. Please refer to the Rust docs website for the full documentation

BGPKIT Parser aims to provide the most ergonomic MRT/BGP/BMP message parsing Rust API.

BGPKIT Parser has the following features: - performant: comparable to C-based implementations like bgpdump or bgpreader. - actively maintained: we consistently introduce feature updates and bug fixes, and support most of the relevant BGP RFCs. - ergonomic API: a three-line for loop can already get you started. - battery-included: ready to handle remote or local, bzip2 or gz data files out of the box

Getting Started

Add bgpkit-parser to your Cargo.toml.

Parse a BGP MRT file in three lines:

use bgpkit_parser::BgpkitParser;

for elem in BgpkitParser::new("http://archive.routeviews.org/route-views4/bgpdata/2022.01/UPDATES/updates.20220101.0000.bz2").unwrap() {
    println!("{}", elem);
}

Examples

The examples below are organized by complexity. For complete runnable examples, check out the examples folder.

Basic Examples

Parsing a Single MRT File

Let's say we want to print out all the BGP announcements/withdrawal from a single MRT file, either located remotely or locally. Here is an example that does so.

use bgpkit_parser::BgpkitParser;
let parser = BgpkitParser::new("http://archive.routeviews.org/bgpdata/2021.10/UPDATES/updates.20211001.0000.bz2").unwrap();
for elem in parser {
    println!("{}", elem)
}

Yes, it is this simple!

Counting BGP Messages

You can use iterator methods for quick analysis. For example, counting the number of announcements/withdrawals in a file:

use bgpkit_parser::BgpkitParser;
let url = "http://archive.routeviews.org/bgpdata/2021.10/UPDATES/updates.20211001.0000.bz2";
let count = BgpkitParser::new(url).unwrap().into_iter().count();
println!("total: {}", count);

Output:

total: 255849

Intermediate Examples

Filtering BGP Messages

BGPKIT Parser has a built-in [Filter] mechanism to efficiently filter messages. Add filters when creating the parser to only process matching [BgpElem]s.

Available filter types: See the [Filter] enum documentation for all options.

use bgpkit_parser::BgpkitParser;

/// Filter by IP prefix
let parser = BgpkitParser::new("http://archive.routeviews.org/bgpdata/2021.10/UPDATES/updates.20211001.0000.bz2").unwrap()
    .add_filter("prefix", "211.98.251.0/24").unwrap();

for elem in parser {
    println!("{}", elem);
}

Common filters: - prefix: Match a specific IP prefix - origin_asn: Match origin AS number - peer_asn: Match peer AS number - peer_ip: Match peer IP address - type: Filter by announcement (a) or withdrawal (w) - as_path: Match AS path with regex

Negative filters: Most filters support negation by prefixing the filter value with !. For example: - origin_asn = !13335: Match elements where origin AS is NOT 13335 - prefix = !211.98.251.0/24: Match elements where prefix is NOT 211.98.251.0/24 - peer_ip = !192.0.2.1: Match elements where peer IP is NOT 192.0.2.1

Note: Timestamp filters (ts_start, ts_end) do not support negation.

use bgpkit_parser::BgpkitParser;

// Filter out all elements from AS 13335 (get everything EXCEPT AS 13335)
let parser = BgpkitParser::new("http://archive.routeviews.org/bgpdata/2021.10/UPDATES/updates.20211001.0000.bz2").unwrap()
    .add_filter("origin_asn", "!13335").unwrap();

for elem in parser {
    println!("{}", elem);
}

Parsing Multiple MRT Files with BGPKIT Broker

BGPKIT Broker library provides search API for all RouteViews and RIPE RIS MRT data files. Using the broker's Rust API (bgpkit-broker), we can easily compile a list of MRT files that we are interested in for any time period and any data type (update or rib). This allows users to gather information without needing to know about the locations of specific data files.

The example below shows a relatively more interesting example that does the following: - find all BGP archive data created on time 1634693400 - filter to only BGP updates files - find all announcements originated from AS13335 - print out the total count of the announcements

use bgpkit_parser::{BgpkitParser, BgpElem};

let broker = bgpkit_broker::BgpkitBroker::new()
    .ts_start("1634693400")
    .ts_end("1634693400")
    .page(1);

for item in broker.into_iter().take(2) {
    log::info!("downloading updates file: {}", &item.url);
    let parser = BgpkitParser::new(item.url.as_str()).unwrap();

    log::info!("parsing updates file");
    // iterating through the parser. the iterator returns `BgpElem` one at a time.
    let elems = parser
        .into_elem_iter()
        .filter_map(|elem| {
            if let Some(origins) = &elem.origin_asns {
                if origins.contains(&13335.into()) {
                    Some(elem)
                } else {
                    None
                }
            } else {
                None
            }
        })
        .collect::<Vec<BgpElem>>();
    log::info!("{} elems matches", elems.len());
}

Error Handling

BGPKIT Parser returns Result types for operations that may fail. Here are common scenarios and how to handle them:

Handling Parser Creation Errors

use bgpkit_parser::BgpkitParser;

// The URL might be invalid or unreachable
match BgpkitParser::new("http://example.com/data.mrt.bz2") {
    Ok(parser) => {
        for elem in parser {
            println!("{}", elem);
        }
    }
    Err(e) => {
        eprintln!("Failed to create parser: {}", e);
        // Common causes:
        // - Invalid URL or file path
        // - Network connection issues
        // - Unsupported compression format
    }
}

Handling Filter Errors

use bgpkit_parser::BgpkitParser;

let mut parser = BgpkitParser::new("http://example.com/data.mrt.bz2").unwrap();

// Filter addition can fail with invalid input
match parser.add_filter("prefix", "invalid-prefix") {
    Ok(_) => println!("Filter added successfully"),
    Err(e) => {
        eprintln!("Invalid filter: {}", e);
        // Common causes:
        // - Invalid IP prefix format
        // - Invalid AS number
        // - Unknown filter type
    }
}

Robust Production Code

use bgpkit_parser::BgpkitParser;

fn process_mrt_file(url: &str) -> Result<usize, Box<dyn std::error::Error>> {
    let parser = BgpkitParser::new(url)?
        .add_filter("origin_asn", "13335")?;

    let mut count = 0;
    for elem in parser {
        // Process element
        count += 1;
    }

    Ok(count)
}

// Usage
match process_mrt_file("http://example.com/updates.bz2") {
    Ok(count) => println!("Processed {} elements", count),
    Err(e) => eprintln!("Error: {}", e),
}

Advanced Examples

Parsing Real-time Data Streams

BGPKIT Parser provides parsing for real-time data streams, including RIS-Live and BMP/OpenBMP messages.

Parsing Messages From RIS-Live

Here is an example of handling RIS-Live message streams. After connecting to the websocket server, we need to subscribe to a specific data stream. In this example, we subscribe to the data stream from one collector (rrc21). We can then loop and read messages from the websocket.

RIS Live's JSON fields expose only a subset of BGP attributes. To parse the original BGP wire message instead, request includeRaw and use parse_ris_live_message_raw. The older parse_ris_live_message function remains available for parsing RIS Live's JSON-projected fields.

use bgpkit_parser::{parse_ris_live_message_raw, RisLiveClientMessage, RisSubscribe};
use tungstenite::{connect, Message};

const RIS_LIVE_URL: &str = "ws://ris-live.ripe.net/v1/ws/?client=rust-bgpkit-parser";

/// This is an example of subscribing to RIS-Live's streaming data from one host (`rrc21`).
///
/// For more RIS-Live details, check out their documentation at https://ris-live.ripe.net/manual/
fn main() {
    // connect to RIPE RIS Live websocket server
    let (mut socket, _response) =
        connect(RIS_LIVE_URL)
            .expect("Can't connect to RIS Live websocket server");

    // subscribe to messages from one collector and request hex-encoded raw BGP messages
    let msg = RisSubscribe::new().host("rrc21").include_raw(true).to_json_string();
    socket.send(Message::Text(msg.into())).unwrap();

    loop {
        let msg = socket.read().expect("Error reading message").to_string();
        if let Ok(elems) = parse_ris_live_message_raw(msg.as_str()) {
            for elem in elems {
                println!("{}", elem);
            }
        }
    }
}

Parsing OpenBMP Messages From RouteViews Kafka Stream

RouteViews provides a real-time Kafka stream of the OpenBMP data received from their collectors. Below is a partial example of how we handle the raw bytes received from the Kafka stream. For full examples, check out the examples folder on GitHub.

use bgpkit_parser::parser::bmp::messages::*;
use bgpkit_parser::parser::utils::*;
use bgpkit_parser::{Elementor, parse_openbmp_header, parse_bmp_msg};

let bytes = &m.value;
let mut data = Bytes::from(bytes.clone());
let header = parse_openbmp_header(&mut data).unwrap();
let bmp_msg = parse_bmp_msg(&mut data);
match bmp_msg {
    Ok(msg) => {
        let timestamp = header.timestamp;
        let per_peer_header = msg.per_peer_header.unwrap();
        match msg.message_body {
            BmpMessageBody::RouteMonitoring(m) => {
                for elem in Elementor::bgp_to_elems(
                    m.bgp_message,
                    timestamp,
                    &per_peer_header.peer_ip,
                    &per_peer_header.peer_asn
                )
                {
                    info!("{}", elem);
                }
            }
            _ => {}
        }
    }
    Err(_e) => {
        let hex = hex::encode(bytes);
        error!("{}", hex);
    }
}

Encoding: Archiving Filtered MRT Records

The example will download one MRT file from RouteViews, filter out all the BGP messages that are not originated from AS3356, and write the filtered MRT records to disk. Then it re-parses the filtered MRT file and prints out the number of BGP messages.

use bgpkit_parser::Elementor;
use itertools::Itertools;
use std::io::Write;

let mut updates_encoder = bgpkit_parser::encoder::MrtUpdatesEncoder::new();

bgpkit_parser::BgpkitParser::new(
    "http://archive.routeviews.org/bgpdata/2023.10/UPDATES/updates.20231029.2015.bz2",
).unwrap()
    .add_filter("origin_asn", "3356").unwrap()
    .into_iter()
    .for_each(|elem| {
        updates_encoder.process_elem(&elem);
    });

let mut mrt_writer = oneio::get_writer("as3356_mrt.gz").unwrap();
mrt_writer.write_all(updates_encoder.export_bytes().as_ref()).unwrap();
drop(mrt_writer);

FAQ & Troubleshooting

Common Issues

Parser creation fails with "unsupported compression"

Problem: The file uses an unsupported compression format.

Solution: BGPKIT Parser natively supports .bz2 and .gz compression. For other formats, decompress the file first or use the oneio crate which supports additional formats.

Out of memory when parsing large files

Problem: Collecting all elements into a vector exhausts available memory.

Solution: Use streaming iteration instead of collecting:

// ❌ Don't do this for large files
let all_elems: Vec<_> = parser.into_iter().collect();

// ✅ Process iteratively
for elem in parser {
    // Process one element at a time
    process(elem);
}

Slow performance on network files

Problem: Remote file parsing is slower than expected.

Solution: - Use the --cache-dir option in CLI to cache downloaded files - In library code, download the file first with appropriate buffering - Consider processing files in parallel if dealing with multiple files

Missing or incomplete BGP attributes

Problem: Some [BgpElem] fields are None when you expect values.

Solution: Not all BGP messages contain all attributes. Check the MRT format and BGP message type: - Withdrawals typically don't have AS paths or communities - Some collectors may not export certain attributes - Use pattern matching to handle Option types properly

Performance Tips

Use filters early

Apply filters during parser creation to avoid processing unwanted data: ```rust // ✅ Efficient - filters during parsing let parser = BgpkitParser::new(url)? .add_filter("prefix", "1.1.1.0/24")?;

// ❌ Less effi