@/bitstream

• Isomorphic: Works in Node.js and in the browser
• Zero-dependency: No runtime dependencies
• Battle-hardened: Used to implement media standards at Astronaut Labs
• Comprehensive testing: 90.38% coverage and growing!
• Performant: Elements use generators (not promises) internally to maximize performance
• Flexible: Supports both imperative and declarative styles
• Modern: Ships as ES modules (with CommonJS fallback)

Highly performant Typescript library for reading and writing to "bitstreams": tightly packed binary streams containing fields of varying lengths of bits. This package lets you treat a series of bytes as a series of bits without needing to manage which bytes the desired bit-fields fall within. Bitstreams are most useful when implementing network protocols and data formats (both encoders and decoders).

Motivation

Astronaut Labs is building a next-generation broadcast technology stack centered around Node.js and Typescript. That requires implementing a large number of binary specifications. We needed a way to do this at scale while ensuring accuracy, quality and comprehensiveness. We also see value in making our libraries open source so they can serve as approachable reference implementations for other implementors and increase competition in our industry. The best way to do that is to be extremely comprehensive in the way we build these libraries. Other implementations tend to skip "irrelevant" details or take shortcuts, we strive to avoid this to produce the most complete libraries possible, even if we don't need every detail for our immediate needs.

Installation

npm install @astronautlabs/bitstream

Libraries using Bitstream

The following libraries are using Bitstream. They are great examples of what you can do with it! If you would like your library to be listed here, please send a pull request!

• https://github.com/astronautlabs/st2010
• https://github.com/astronautlabs/st291
• https://github.com/astronautlabs/scte35
• https://github.com/astronautlabs/scte104
• https://github.com/astronautlabs/rfc8331

BitstreamReader: Reading from bitstreams imperatively

import { BitstreamReader } from '@astronautlabs/bitstream';

let reader = new BitstreamReader();

reader.addBuffer(Buffer.from([0b11110000, 0b10001111]));

await reader.read(2); // == 0b11
await reader.read(3); // == 0b110
await reader.read(4); // == 0b0001
await reader.read(7); // == 0b0001111

The above will read the values as unsigned integers in big-endian (network byte order) format.

Asynchronous versus Synchronous

All read operations come in two flavors, asynchronous and synchronous. For instance to read an unsigned integer asynchronously, use read(). For this and other asynchronous read operations, resolution of the resulting promise is delayed until enough data is available to complete the operation. Note that there can be only one asynchronous read operation in progress at a time for a given BitstreamReader object.

The synchronous method for reading unsigned integers is readSync(). When using synchronous methods, there must be enough bytes available to the reader (via addBuffer()) to read the desired number of bits. If this is not the case, an exception is thrown. You can check how many bits are available using the isAvailable() method:

if (reader.isAvailable(10)) {
    // 10 bits are available
    let value = reader.readSync(10);
}

Alternatively, you can use .assure() to wait until the desired number of bits are available. Again, there can only be one pending call to read() or assure() at a time. This allows you to "batch" synchronous reads in a single "await" operation.

await reader.assure(13);
let value1 = reader.readSync(3);
let value2 = reader.readSync(10);

Reading signed integers

Use the readSigned / readSignedSync methods to read a signed two's complement integer.

Reading floating-point integers [IEEE 754]

Use the readFloat / readFloatSync methods to read an IEEE 754 floating point value. The bit length passed must be either 32 (32-bit single-precision) or 64 (64-bit double-precision).

Reading Strings

Use the readString / readStringSync methods to read string values.

await reader.readString(10); // read a fixed length string with 10 characters.

By default readString() will cut off the string at the first character with value 0 (ie, the string is considered null-terminated) and stop reading. You can disable this behavior so that the returned string always contains all bytes that were in the bitstream:

await reader.readString(10, { nullTerminated: false });

The default text encoding is UTF-8 (utf-8). You can read a string using any text encoding supported by the platform you are on. For Node.js these are the encodings supported by Buffer. On the web, only utf-8 is available (see documentation for TextEncoder/TextDecoder).

await reader.readString(10, { encoding: 'utf16le' })

Important: In cases like above where you are using encodings where a character spans multiple bytes (including UTF-8), the length given to readString() is always the number of bytes not the number of characters. It is easy to make mistaken assumptions in this regard.

BitstreamWriter: Writing to bitstreams imperatively

import { BitstreamWriter } from '@astronautlabs/bitstream';

let writer = new BitstreamWriter(writableStream, bufferLength);

writer.write(2, 0b10);
writer.write(10, 0b1010101010);
writer.write(length, value);

writableStream can be any object which has a write(chunk : Uint8Array) method (see exported Writable interface).

Examples of writables you can use include: - Node.js' writable streams (Writable from the stream package) - WritableStreamDefaultWriter from the WHATWG Streams specification in the browser - Any custom object which implements the Writable interface

The bufferLength parameter determines how many bytes will be buffered before the buffer will be flushed out to the passed writable stream. This parameter is optional, the default (and minimum value) is 1 (one byte per buffer).

Writing unsigned integers

writer.write(2, 0b01);
writer.write(2, 0b1101);
writer.write(2, 0b1111101); // 0b01 will be written

Note: Any bits in value above the length'th bit will be ignored, so all of the above are equivalent.

Writing signed integers

Use the writeSigned() method to write signed two's complement integers.

Writing floating point values

Use the writeFloat() method to write IEEE 754 floating point values. Only lengths of 32 (for 32-bit single-precision) and 64 (for 64-bit double-precision) are supported.

Writing strings

Use the writeString() method to write string values. The default encoding is UTF-8 (utf-8). Any other encoding supported by the platform can be used (ie those supported by Buffer on Node.js). On the web, only utf-8 is supported (see TextEncoder / TextDecoder).

Writing byte arrays

Use the writeBuffer() method to write byte arrays. On Node.js you can also pass Buffer.

BitstreamElement: declarative structural serialization

Efficient structural (de)serialization can be achieved by building subclasses of the BitstreamElement class.

Deserialization (reading)

You can declaratively specify elements of bitstreams, then read and write them to bitstream readers/writers as needed. To do this, extend the BitstreamElement class:

import { BitstreamElement, Field } from '@astronautlabs/bitstream';

class MyElement extends BitstreamElement {
    @Field(2) field1 : number;
    @Field(4) field2 : number;
    @Field(3) field3 : number;
    @Field(1) field4 : number;
    @Field(1) field5 : boolean;
}

Then, read from a BitstreamReader using:

let element = await MyElement.read(bitstreamReader);

Or, deserialize from a Buffer using:

let element = await MyElement.deserialize(buffer);

Number fields

• null and undefined are written as 0 during serialization
• Numbers are treated as big-endian unsigned integers by default. Decimal portions of numbers are truncated.
  • Use the { number: { format: 'signed' }} option to use signed two's complement integer. Decimals are truncated
  • Use the { number: { format: 'float' }} option to use IEEE 754 floating point. Decimals are retained
• An error is thrown when trying to serialize NaN or infinite values (except when using the float format)

Boolean fields

If you specify type boolean for a field, the integer value 0 will be deserialized to false and all other values will be deserialized as true. When booleans are serialized, 0 is used for false and 1 is used for true.

You can customize this behavior using the boolean field options (ie @Field(8, { boolean: { ... } })):

• true: The numeric value to use for true (default 1)
• false: The numeric value to use for false (default 0)
• undefined: The numeric value to use when writing undefined (default 0)
• mode: How to handle novel inputs when reading:
  • "true-unless": The value is true unless the numeric value chosen for 'false' is observed (default mode). For example 0 is false, 1 is true, 100 is true
  • "false-unless": The value is false unless the numeric value chosen for 'true' is observed. For example 0 is false, 1 is true, 100 is false
  • "undefined": The value is true if the numeric value for 'true' is observed, false if the numeric value for 'false' is observed and undefined otherwise. For example 0 is false, 1 is true, 100 is undefined.

If none of these options fit your use case, you can write a custom Serializer.

String fields

Elements can also handle serializing fixed-length strings. For example, to represent a fixed-length 5-byte string, set the length to 5 (note that this differs from other types where the length is specified in bits).

Note: Null-termination is the default here, make sure to set nullTerminated to false if you do not desire that behavior.

    @Field(5) stringField : string;

If you wish to control the encoding options, use the string options group:

    @Field(5, { string: { encoding: 'ascii', nullTerminated: false } }) 
    stringField : string;

For information about the available encodings, see "Reading Strings" above.

Byte array fields (Buffers)

You can represent a number of bytes as a byte array:

    @Field(10*8) read10BytesIntoBuffer : Uint8Array;

When running on Node.js you can also specify Buffer as the type instead:

    @Field(10*8) read10BytesIntoBuffer : Buffer;

Nested elements

You can also nest element classes:

class PartElement extends BitstreamElement {
    @Field(3) a : number;
    @Field(5) b : number;
}

class WholeElement extends BitstreamElement {
    @Field() part1 : PartElement;
    @Field() part2 : PartElement;
}

Arrays

Many kinds of arrays are supported.

Array with number values (unsigned/signed/float)

Arrays with number elements are natively supported for convenience.

For example, to read 10 8-bit unsigned integers:

class BufferElement extends BitstreamElement {
    @Field(10, { array: { type: Number, elementLength: 8 }})
    array : number[];
}

Important: - You must specify the array.type option when using elements. Typescript cannot convey the element type via reflection, only that the field is of type Array which is not enough information to deserialize the array. - When using arrays of numbers you must specify the array.elementLength option for the library to know how many bits each array item represents within the bitstream. For other array item types such as BitstreamElement the length is already known and array.elementLength is ignored

You can read signed integers and floating point values instead of unsigned integers by specifying the number.format option (see Number Fields for more information).

When handling arrays, the length parameter of @Field() is used to represent the item count for the array by default. Alternatively (and mainly for historical reasons) you can also set the options.array.count property.

Derived lengths

As with all fields, you can represent the number of items dynamically with a determinant fucntion, allowing the array length to be dependent on any value already read before the field being read. For more information about this, see the section on "Dynamic Lengths" below.

You can pair this with the writtenValue feature to ensure that the correct length is always written to the bitstream.

Array with preceding count

You can specify that a preceding count field should be read to determine what the length of the array is in the countFieldLength option. If this option is not provided or undefined, no preceding count field is read.

Array with element values

You can also use arrays of elements:

class ItemElement extends BitstreamElement {
    @Field(3) a : number;
    @Field(5) b : number;
}

class ListElement extends BitstreamElement {
    @Field({ array: { type: PartElement, countFieldLength: 8 }}) 
    parts : PartElement[];
}

The above will expect an 8-bit "count" field (indicates how many items are in the