starpc

Streaming Protobuf RPC with bidirectional streaming over any multiplexed transport.

Table of Contents

• Features
• Documentation
• Installation
• Quick Start
• Examples
• Protobuf Definition
• Go Implementation
• TypeScript Implementation
• Rust Implementation
• Debugging
• Development Setup
• Support

Features

• Full Proto3 services support for TypeScript, Go, and Rust
• Bidirectional streaming in browsers via WebSocket or WebRTC
• Built on libp2p streams with @chainsafe/libp2p-yamux
• Efficient multiplexing of RPCs over single connections
• Zero-reflection Go via protobuf-go-lite
• Lightweight TypeScript via protobuf-es-lite
• Async Rust via prost and tokio
• Sub-stream support via rpcstream

Documentation

• Getting Started with TypeScript
• Getting Started with Go
• Getting Started with Rust

Installation

# Clone the template project
git clone -b starpc https://github.com/aperturerobotics/protobuf-project
cd protobuf-project

# Install dependencies (any of these work)
bun install
npm install
pnpm install

# Generate TypeScript and Go code
bun run gen

Quick Start

1. Start with the protobuf-project template repository (starpc branch)
2. Add your .proto files to the project
3. Run bun run gen to generate TypeScript and Go code
4. Implement your services using the examples below

Protobuf

The following examples use the echo protobuf sample.

syntax = "proto3";
package echo;

// Echoer service returns the given message.
service Echoer {
  // Echo returns the given message.
  rpc Echo(EchoMsg) returns (EchoMsg);
  // EchoServerStream is an example of a server -> client one-way stream.
  rpc EchoServerStream(EchoMsg) returns (stream EchoMsg);
  // EchoClientStream is an example of client->server one-way stream.
  rpc EchoClientStream(stream EchoMsg) returns (EchoMsg);
  // EchoBidiStream is an example of a two-way stream.
  rpc EchoBidiStream(stream EchoMsg) returns (stream EchoMsg);
}

// EchoMsg is the message body for Echo.
message EchoMsg {
  string body = 1;
}

Go

This example demonstrates both the server and client:

// construct the server
echoServer := &echo.EchoServer{}
mux := srpc.NewMux()
if err := echo.SRPCRegisterEchoer(mux, echoServer); err != nil {
    t.Fatal(err.Error())
}
server := srpc.NewServer(mux)

// create an in-memory connection to the server
openStream := srpc.NewServerPipe(server)

// construct the client
client := srpc.NewClient(openStream)

// construct the client rpc interface
clientEcho := echo.NewSRPCEchoerClient(client)
ctx := context.Background()
bodyTxt := "hello world"
out, err := clientEcho.Echo(ctx, &echo.EchoMsg{
    Body: bodyTxt,
})
if err != nil {
    t.Fatal(err.Error())
}
if out.GetBody() != bodyTxt {
    t.Fatalf("expected %q got %q", bodyTxt, out.GetBody())
}

TypeScript

For Go <-> TypeScript interop, see the integration test. For TypeScript <-> TypeScript, see the e2e test.

Rust

Add the dependencies to your Cargo.toml:

[dependencies]
starpc = "0.49"
prost = "0.14"
tokio = { version = "1", features = ["rt", "macros"] }

[build-dependencies]
starpc = { version = "0.49", features = ["build"] }
prost-build = "0.14"

Create a build.rs to generate code from your proto files:

fn main() -> Result<(), Box<dyn std::error::Error>> {
    starpc::build::configure()
        .compile_protos(&["proto/echo.proto"], &["proto"])?;
    Ok(())
}

Implement and use your service:

use starpc::{Client, Mux, Server, SrpcClient};
use std::sync::Arc;

// Include generated code
mod proto {
    include!(concat!(env!("OUT_DIR"), "/echo.rs"));
}

use proto::*;

// Implement the server trait
struct EchoServer;

#[starpc::async_trait]
impl EchoerServer for EchoServer {
    async fn echo(&self, request: EchoMsg) -> starpc::Result<EchoMsg> {
        Ok(request) // Echo back the message
    }
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Create server
    let mux = Arc::new(Mux::new());
    mux.register(Arc::new(EchoerHandler::new(EchoServer)))?;
    let server = Server::with_arc(mux);

    // Create an in-memory connection for demonstration
    let (client_stream, server_stream) = tokio::io::duplex(64 * 1024);

    // Spawn server handler
    tokio::spawn(async move {
        let _ = server.handle_stream(server_stream).await;
    });

    // Create client
    let opener = starpc::client::transport::SingleStreamOpener::new(client_stream);
    let client = SrpcClient::new(opener);
    let echoer = EchoerClientImpl::new(client);

    // Make RPC call
    let response = echoer.echo(&EchoMsg { body: "Hello!".into() }).await?;
    println!("Response: {}", response.body);

    Ok(())
}

See the echo example for a complete working example.

WebSocket Example

Connect to a WebSocket server:

import { WebSocketConn } from 'starpc'
import { EchoerClient } from './echo/index.js'

const ws = new WebSocket('ws://localhost:8080/api')
const conn = new WebSocketConn(ws, 'outbound')
const client = conn.buildClient()
const echoer = new EchoerClient(client)

const result = await echoer.Echo({ body: 'Hello world!' })
console.log('result:', result.body)

In-Memory Example

Server and client with an in-memory pipe:

import { pipe } from 'it-pipe'
import { createHandler, createMux, Server, StreamConn } from 'starpc'
import { EchoerDefinition, EchoerServer } from './echo/index.js'

// Create server with registered handlers
const mux = createMux()
const echoer = new EchoerServer()
mux.register(createHandler(EchoerDefinition, echoer))
const server = new Server(mux.lookupMethod)

// Create client and server connections, pipe together
const clientConn = new StreamConn()
const serverConn = new StreamConn(server)
pipe(clientConn, serverConn, clientConn)

// Build client and make RPC calls
const client = clientConn.buildClient()
const echoerClient = new EchoerClient(client)

// Unary call
const result = await echoerClient.Echo({ body: 'Hello world!' })
console.log('result:', result.body)

// Client streaming
import { pushable } from 'it-pushable'
const stream = pushable({ objectMode: true })
stream.push({ body: 'Message 1' })
stream.push({ body: 'Message 2' })
stream.end()
const response = await echoerClient.EchoClientStream(stream)
console.log('response:', response.body)

// Server streaming
for await (const msg of echoerClient.EchoServerStream({ body: 'Hello' })) {
  console.log('server msg:', msg.body)
}

Debugging

Enable debug logging in TypeScript using the DEBUG environment variable:

# Enable all starpc logs
DEBUG=starpc:* node app.js

# Enable specific component logs
DEBUG=starpc:stream-conn node app.js

Attribution

protoc-gen-go-starpc is a heavily modified version of protoc-gen-go-drpc. Check out drpc as well - it's compatible with grpc, twirp, and more.

Uses vtprotobuf to generate Protobuf marshal/unmarshal code for Go.

protoc-gen-es-starpc is a modified version of protoc-gen-connect-es. Uses protobuf-es-lite (fork of protobuf-es) for TypeScript.

Development Setup

MacOS

The aptre CLI handles all protobuf generation without requiring additional homebrew packages. Just run bun run gen after installing bun.

Support

• GitHub Issues
• Discord
• Matrix