Wingtips - Give Your Distributed Systems a Dapper Footprint

[![Maven Central][maven_central_img]][maven_central] [![Build][gh_action_build_img]][gh_action_build] [![Code Coverage][codecov_img]][codecov] [![License][license img]][license]

Wingtips is a distributed tracing solution for Java 7 and greater based on the Google Dapper paper.

There are a few modules associated with this project:

• wingtips-core - The core library providing the majority of the distributed tracing functionality.
• wingtips-java8 - Provides several Java 8 helpers, particularly around helping tracing and MDC information to hop threads in asynchronous/non-blocking use cases.
• wingtips-servlet-api - A plugin for Servlet based applications for integrating distributed tracing with a simple Servlet Filter. Supports Servlet 2.x and Servlet 3 (async request) environments.
• wingtips-zipkin2 - A plugin providing easy Zipkin integration by converting Wingtips spans to Zipkin spans and sending them to a Zipkin server.
• wingtips-spring - A plugin to help with Wingtips distributed tracing in older Spring environments (i.e. not Spring WebFlux). This is mostly for the older non-WebFlux HTTP clients - for Spring MVC serverside support, see wingtips-servlet-api, and for Spring WebFlux (both serverside and clientside) see wingtips-spring-webflux.
• wingtips-spring-webflux - A plugin to help with Wingtips distributed tracing in Spring WebFlux environments.
• wingtips-spring-boot - A plugin to help with Wingtips distributed tracing in Spring Boot environments using Spring Web MVC (Servlet-based). For Spring Boot 2 WebFlux environments see wingtips-spring-boot2-webflux instead.
• wingtips-spring-boot2-webflux - A plugin to help with Wingtips distributed tracing in Spring Boot 2 WebFlux environments (not Servlet-based Web MVC environments - for that see wingtips-spring-boot).
• wingtips-zipkin2-spring-boot - A plugin to help with Wingtips distributed tracing in Spring Boot environments using Spring Web MVC (Servlet-based) that also utilize Zipkin. For Spring Boot 2 + WebFlux + Zipkin environments see wingtips-zipkin2-spring-boot2-webflux instead.
• wingtips-zipkin2-spring-boot2-webflux - A plugin to help with Wingtips distributed tracing in Spring Boot 2 WebFlux environments that also utilize Zipkin (not Servlet-based Web MVC environments - for that see wingtips-zipkin2-spring-boot).
• wingtips-apache-http-client - A plugin to help with Wingtips distributed tracing when using Apache's HttpClient.
• wingtips-jersey2 - A plugin for Jersey 2 based applications. This is intended to be used in conjunction with the RequestTracingFilter from wingtips-servlet-api.

If you prefer hands-on exploration rather than readmes, the sample applications provide concrete examples of using Wingtips that are simple, compact, and straightforward.

Table of Contents

• Overview
  • What is a Distributed Trace Made Of?
• Quickstart and Usage
  • Generic Application Pseudo-Code
  • Generic Application Pseudo-Code Explanation
    • Is your application running in a Servlet-based framework?
  • Simplified Span Management using Java try-with-resources Statements
  • Output and Logging
    • Automatically attaching trace information to all log messages
  • Nested Sub-Spans and the Span Stack
    • Using sub-spans to surround downstream calls
  • Propagating Distributed Traces Across Network or Application Boundaries
  • Adjusting Behavior and Execution Options
    • Sampling
    • Notification of span lifecycle events
    • Changing serialized representation of Spans for the logs
  • Span Tags
    • HTTP Span Tag and Naming Strategies and Adapters
    • Default HTTP Tags
  • Custom Timestamped Span Annotations
• Usage in Reactive Asynchronous Nonblocking Scenarios
• Using Distributed Tracing to Help with Debugging Issues/Errors/Problems
• Integrating With Other Distributed Tracing Tools
• Sample Applications
• License

Overview

Distributed tracing is a mechanism to track requests through a network of distributed systems in order to create transparency and shed light on the sometimes complex interactions and behavior of those systems. For example in a cloud-based microservice architecture a single request can touch dozens or hundreds of servers as it spreads out in a tree-like fashion. Without distributed tracing it would be very difficult to identify which part(s) of such a complex system were causing performance issues - either in general or for that request in particular.

Distributed tracing provides the capability for near-realtime monitoring or historical analysis of interactions between servers given the necessary tools to collect and interpret the traces. It allows you to easily see where applications are spending their time - e.g. is it mostly in-application work, or is it mostly waiting for downstream network calls?

Distributed tracing can also be used in some cases for error debugging and problem investigations with some caveats. See the relevant section for more information on this topic.

What is a Distributed Trace Made Of?

• Every distributed trace contains a TraceID that represents the entire request across all servers/microservices it touches. These TraceIDs are generated as probabilistically unique 64-bit integers (longs). In practice these IDs are passed around as unsigned longs in lowercase hexadecimal string format, but conceptually they are just random 64-bit integers.
• Each unit of work that you want to track in the distributed trace is defined as a Span. Spans are usually broken down into overall-request-time for a given request in a given server/microservice, and downstream-call-time for each downstream call made for that request from the server/microservice. Spans are identified by a SpanID, which is also a pseudorandomly generated 64-bit long.
• Spans can have Parent Spans, which is how we build a tree of the request's behavior as it branches across server boundaries. The ParentSpanID is set to the parent Span's SpanID.
• All Spans contain the TraceID of the overall trace they are attached to, whether or not they have Parent Spans.
• Spans include a SpanName, which is a more human readable indication of what the span was, e.g. GET_/some/endpoint for the overall span for a REST request, or downstream-POST_https://otherservice.com/other/endpoint for the span performing a downstream call to another service.
• Spans contain timing info - a start timestamp and a duration value.

See the Output and Logging section for an example of what a span looks like when it is logged.

Quickstart and Usage

Generic Application Pseudo-Code

NOTE: The following pseudo-code only applies to thread-per-request frameworks and scenarios. For asynchronous non-blocking scenarios see this section.

import com.nike.wingtips.Span;
import com.nike.wingtips.Tracer;

// ======As early in the request cycle as possible======
try {
    // Determine if a parent span exists by inspecting the request (e.g. request headers)
    Span parentSpan = extractParentSpanFromRequest(request);

    // Start the overall request span (which becomes the "current span" for this thread unless/until a sub-span is created)
    if (parentSpan == null)
        Tracer.getInstance().startRequestWithRootSpan("newRequestSpanName");
    else
        Tracer.getInstance().startRequestWithChildSpan(parentSpan, "newRequestSpanName");

    // It's recommended that you include the trace ID of the overall request span in the response headers
    addTraceIdToResponseHeaders(response, Tracer.getInstance().getCurrentSpan());

    // Execute the normal request logic
    doRequestLogic();
}
finally {
    // ======As late in the request/response cycle as possible======
    Tracer.getInstance().completeRequestSpan(); // Completes the overall request span and logs it to SLF4J
}

Generic Application Pseudo-Code Explanation

In a typical usage scenario you'll want to call one of the Tracer.getInstance().startRequest...() methods as soon as possible when a request enters your application, and you'll want to call Tracer.getInstance().completeRequestSpan() as late as possible in the request/response cycle. In between these two calls the span that was started (the "overall-request-span") is considered the "current span" for this thread and can be retrieved if necessary by calling Tracer.getInstance().getCurrentSpan().

The extractParentSpanFromRequest() method is potentially different for different applications, however for HTTP-based frameworks the pattern is usually the same - look for and extract distributed-tracing-related information from the HTTP headers and use that information to create a parent span. There is a utility method that performs this work for you: HttpRequestTracingUtils.fromRequestWithHeaders(RequestWithHeaders, List<String>). You simply need to provide the HTTP request wrapped by an implementation of RequestWithHeaders and the list of user ID header keys for your application (if any) and it will do the rest using the standard distributed tracing header key constants found in TraceHeaders. See the javadocs for those classes and methods for more information and usage instructions.

NOTE: Given the thread-local nature of this library you'll want to make sure the span completion call is in a finally block or otherwise guaranteed to be called no matter what (even if the request fails with an error) to prevent problems when subsequent requests are processed on the same thread. The Tracer class does its best to recover from incorrect thread usage scenarios and log information about what happened but the best solution is to prevent the problems from occurring in the first place. See the section below on try-with-resources for some tips on foolproof ways to safely complete your spans.

Is your application running in a Servlet-based framework?

If your application is running in a Servlet environment (e.g. Spring Boot w/ Spring MVC, raw Spring MVC, Jersey, raw Servlets, etc) then this entire lifecycle can be handled by a Servlet Filter. We've created one for you that's ready to drop in and go - see the wingtips-servlet-api Wingtips plugin module library for details. That plugin module is also a good resource to see how the code for a production-ready implementation of this library might look.

Simplified Span Management using Java try-with-resources Statements

Spans support Java try-with-resources statements to help guarantee proper usage in blocking/non-asynchronous scenarios (for asynchronous scenarios please refer to the asynchronous usage section of this readme). As previously mentioned, Spans that are not properly completed can lead to incorrect distributed tracing information showing up, and the try-with-resources statements guarantee that spans are completed appropriately. Here are some examples (note: there are some important tradeoffs you should consider before using this feature):

Overall request span using try-with-resources

try(Span requestSpan = Tracer.getInstance().startRequestWith*(...)) {
    // Traced blocking code for overall request (not asynchronous) goes here ...
}
// No finally block needed to properly complete the overall request span

Subspan using try-with-resources

try (Span subspan = Tracer.getInstance().startSubSpan(...)) {
    // Traced blocking code for subspan (not asynchronous) goes here ...
}
// No finally block needed to properly complete the subspan

Warning about error handling when using try-with-resources to autoclose spans

The try-with-resources feature t