Multi Layer Archive (MLA)

MLA is an archive file format with the following features:

• Support for compression (based on rust-brotli)
• Support for authenticated encryption with asymmetric keys (AES256-GCM with an ECIES schema over Curve25519, based on Rust-Crypto aes-ctr and DalekCryptography x25519-dalek)
• Effective, architecture agnostic and portable (written entirely in Rust)
• Small memory footprint during archive creation
• Streamable archive creation:
• An archive can be built even over a data-diode
• A file can be added through chunks of data, without initially knowing the final size
• File chunks can be interleaved (one can add the beginning of a file, start a second one, and then continue adding the first file's parts)
• Archive files are seekable, even if compressed or encrypted. A file can be accessed in the middle of the archive without reading from the beginning
• If truncated, archives can be repaired. Files which were still in the archive, and the beginning of the ones for which the end is missing, will be recovered
• Arguably less prone to bugs, especially while parsing an untrusted archive (Rust safety)

Repository

This repository contains:

• mla: the Rust library implementing MLA reader and writer
• mlar: a Rust utility wrapping mla for common actions (create, list, extract, ...)
• curve25519-parser: a Rust library for parsing DER/PEM public and private Ed25519 keys and X25519 keys (as made by openssl)
• mla-fuzz-afl : a Rust utility to fuzz mla
• bindings : bindings for other languages
• .github: Continuous Integration needs

Quick command-line usage

Here are some commands to use mlar in order to work with archives in MLA format.

# Generate an X25519 key pair {key, key.pub} (OpenSSL could also be used)
mlar keygen key

# Create an archive with some files, using the public key
mlar create -p key.pub -o my_archive.mla /etc/os-release /etc/issue

# List the content of the archive, using the private key
mlar list -k key -i my_archive.mla

# Extract the content of the archive into a new directory
# In this example, this creates two files:
# extracted_content/etc/issue and extracted_content/etc/os-release
mlar extract -k key -i my_archive.mla -o extracted_content

# Display the content of a file in the archive
mlar cat -k key -i my_archive.mla /etc/os-release

# Convert the archive to a long-term one, removing encryption and using the best
# and slower compression level
mlar convert -k key -i my_archive.mla -o longterm.mla -l compress -q 11

# Create an archive with multiple recipient
mlar create -p archive.pub -p client1.pub -o my_archive.mla ...

mlar can be obtained:

• through Cargo: cargo install mlar
• using the latest release for supported operating systems

Quick API usage

• Create an archive, with compression and encryption:

use curve25519_parser::parse_openssl_25519_pubkey;
use mla::config::ArchiveWriterConfig;
use mla::ArchiveWriter;

const PUB_KEY: &[u8] = include_bytes!("samples/test_x25519_pub.pem");

fn main() {
    // Load the needed public key
    let public_key = parse_openssl_25519_pubkey(PUB_KEY).unwrap();

    // Create an MLA Archive - Output only needs the Write trait
    let mut buf = Vec::new();
    // Default is Compression + Encryption, to avoid mistakes
    let mut config = ArchiveWriterConfig::default();
    // The use of multiple public keys is supported
    config.add_public_keys(&vec![public_key]);
    // Create the Writer
    let mut mla = ArchiveWriter::from_config(&mut buf, config).unwrap();

    // Add a file
    mla.add_file("filename", 4, &[0, 1, 2, 3][..]).unwrap();

    // Complete the archive
    mla.finalize().unwrap();
}

• Add files part per part, in a "concurrent" fashion:

...
// A file is tracked by an id, and follows this API's call order:
// 1. id = start_file(filename);
// 2. append_file_content(id, content length, content (impl Read))
// 2-bis. repeat 2.
// 3. end_file(id)

// Start a file and add content
let id_file1 = mla.start_file("fname1").unwrap();
mla.append_file_content(id_file1, file1_part1.len() as u64, file1_part1.as_slice()).unwrap();
// Start a second file and add content
let id_file2 = mla.start_file("fname2").unwrap();
mla.append_file_content(id_file2, file2_part1.len() as u64, file2_part1.as_slice()).unwrap();
// Add a file as a whole
mla.add_file("fname3", file3.len() as u64, file3.as_slice()).unwrap();
// Add new content to the first file
mla.append_file_content(id_file1, file1_part2.len() as u64, file1_part2.as_slice()).unwrap();
// Mark still opened files as finished
mla.end_file(id_file1).unwrap();
mla.end_file(id_file2).unwrap();

• Read files from an archive

use curve25519_parser::parse_openssl_25519_privkey;
use mla::config::ArchiveReaderConfig;
use mla::ArchiveReader;
use std::io;

const PRIV_KEY: &[u8] = include_bytes!("samples/test_x25519_archive_v1.pem");
const DATA: &[u8] = include_bytes!("samples/archive_v1.mla");

fn main() {
    // Get the private key
    let private_key = parse_openssl_25519_privkey(PRIV_KEY).unwrap();

    // Specify the key for the Reader
    let mut config = ArchiveReaderConfig::new();
    config.add_private_keys(&[private_key]);

    // Read from buf, which needs Read + Seek
    let buf = io::Cursor::new(DATA);
    let mut mla_read = ArchiveReader::from_config(buf, config).unwrap();

    // Get a file
    let mut file = mla_read
        .get_file("simple".to_string())
        .unwrap() // An error can be raised (I/O, decryption, etc.)
        .unwrap(); // Option(file), as the file might not exist in the archive

    // Get back its filename, size, and data
    println!("{} ({} bytes)", file.filename, file.size);
    let mut output = Vec::new();
    std::io::copy(&mut file.data, &mut output).unwrap();

    // Get back the list of files in the archive:
    for fname in mla_read.list_files().unwrap() {
        println!("{}", fname);
    }
}

:warning: Filenames are Strings, which may contain path separator (/, \, .., etc.). Please consider this while using the API, to avoid path traversal issues.

Using MLA with others languages

Bindings are available for:

• C/CPP

Design

As the name spoils it, an MLA archive is made of several, independent, layers. The following section introduces the design ideas behind MLA. Please refer to FORMAT.md for a more formal description.

Layers

Each layer acts as a Unix PIPE, taking bytes in input and outputting in the next layer. A layer is made of:

• a Writer, implementing the Write trait. It is responsible for emitting bytes while creating a new archive
• a Reader, implementing both Read and Seek traits. It is responsible for reading bytes while reading an archive
• a FailSafeReader, implementing only the Read trait. It is responsible for reading bytes while repairing an archive

Layers are made with the repairable property in mind. Reading them must never need information from the footer, but a footer can be used to optimize the reading. For example, accessing a file inside the archive can be optimized using the footer to seek to the file beginning, but it is still possible to get information by reading the whole archive until the file is found.

Layers are optional, but their order is enforced. Users can choose to enable or disable them. Current order is the following:

1. File storage abstraction (not a layer)
2. Raw layer (mandatory)
3. Compression layer
4. Encryption layer
5. Position layer (mandatory)
6. Stored bytes

Overview

+----------------+-------------------------------------------------------------------------------------------------------------+
| Archive Header |                                                                                                             | => Final container (File / Buffer / etc.)
+------------------------------------------------------------------------------------------------------------------------------+
                 +-------------------------------------------------------------------------------------------------------------+
                 |                                                                                                             | => Raw layer
                 +-------------------------------------------------------------------------------------------------------------+
                 +-----------+---------+------+---------+------+---------------------------------------------------------------+
                 | E. header | Block 1 | TAG1 | Block 2 | TAG2 | Block 3 | TAG3 | ...                                          | => Encryption layer
                 +-----------+---------+------+---------+------+---------------------------------------------------------------+
                             |         |      |         |      |         |      |                                              |
                             +-------+--      --+-------       -----------      ----+---------+------+---------+ +-------------+
                             | Blk 1 |          | Blk 2                             | Block 3 | ...  | Block n | |    Footer   | => Compression Layer
                             +-------+--      --+-------       -----------      ----+---------+------+---------+ +-------------+
                            /         \                                                             /           \
                           /           \                                                           /             \
                          /             \                                                         /               \
                         +-----------------------------------------------------------------------------------------+
                         |                                                                                         |             => Position layer
                         +-----------------------------------------------------------------------------------------+
                         +-------------+-------------+-------------+-------------+-----------+-------+-------------+
                         | File1 start | File1 data1 | File2 start | File1 data2 | File1 end |  ...  | Files index |             => Files information and content
                         +-------------+-------------+-------------+-------------+-----------+-------+-------------+

Layers description

Raw Layer

Implemented in RawLayer* (i.e. RawLayerWriter, RawLayerReader and RawLayerFailSafeReader).

This is the simplest layer. It is required to provide an API between layers and final output worlds. It is also used to keep the position of data's start.

Position Layer

Implemented in PositionLayer*.

Similar to the RawLayer, this is a very simple, utility, layer. It keeps track of how many bytes have been written to the sub-layers.

For instance, it is required by the file storage layer to keep track of the position in the flow of files, for indexing purpose.

Encryption Layer

Implemented in EncryptionLayer*.

This layer encrypts data using the symmetric authenticated encryption with associated data (AEAD) algorithm AES-GCM 256, and encrypts the symmetric key using an ECIES schema based on Curve25519.

The ECIES schema is extended to support multiple public keys: a public key is generated and then used to perform n Diffie-Hellman exchanges with the n users public keys. The generated public key is also recorded in the header (to let the user replay the DH exchange). Once derived according to ECIES, we get n keys. These keys are then used to encrypt a common key k, and the resulting n ciphertexts are stored in the layer header. This key k will later be used for the symmetric encryption of the archive.

In addition to the key, a nonce (8 bytes) is also generated per archive. A fixed associated data is used.

The generation uses OsRng from crate rand, that uses getrandom() from crate getrandom. getrandom provides implementations for many systems, listed here. On Linux it uses the getrandom() syscall and falls back on /dev/urandom. On Windows it uses the RtlGenRandom API (available since Windows XP/Windows Server 2003).

In order to be "better safe than sorry", a ChaChaRng is seeded from the bytes generated by OsRng in order to build a CSPRNG(Cryptographically Secure PseudoRandom Number Generator). This ChaChaRng provides the actual bytes used in keys and nonces generations.

The layer data is then made of several encrypted blocks, each with a constant size except for the last one. Each block is encrypted with an IV including the base nonce and a counter. This construction is close to the STREAM one, except for the last_block bit. The choice has been made not to use it, because: * At the time of writing, the archive writer does not know that the current block is the last one. Therefore, it cannot use a specific IV. To circu