Shaderfrog GLSL Compiler

The Shaderfrog GLSL compiler is an open source GLSL 1.00 and 3.00 parser and preprocessor that compiles back to GLSL. Both the parser and preprocessor can preserve comments and whitespace.

The parser is built with a PEG grammar, via the Peggy Javascript library. The PEG grammars for both the preprocessor and parser are available on Github.

See the state of this library for limitations and goals of this compiler.

Table of Contents

• Installation
• Parsing
• The program type
• Scope
• Errors
• Preprocessing
• Manipulating and Searching ASTs
• Visitors
• Utility Functions
• What are "parsing" and "preprocessing"?
• State of this library
• Limitations of the Parser and Preprocessor
• Local Development

Usage

Installation

npm install --save @shaderfrog/glsl-parser

Parsing

import { parser, generate } from '@shaderfrog/glsl-parser';

// To parse a GLSL program's source code into an AST:
const program = parser.parse('float a = 1.0;');

// To turn a parsed AST back into a source program
const transpiled = generate(program);

The parser accepts an optional second options argument:

parser.parse('float a = 1.0;', options);

Where options is:

type ParserOptions = {
  // Hide warnings. If set to false or not set, then the parser logs warnings
  // like undefined functions and variables. If `failOnWarn` is set to true,
  // warnings will still cause the parser to raise an error. Defaults to false.
  quiet: boolean;

  // An optional string representing the origin of the GLSL, for debugging and
  // error messages. For example, "main.js". If the parser raises an error, the
  // grammarSource shows up in the error.source field. If you format the error
  // (see the errors section), the grammarSource shows up in the formatted error
  // string. Defaults to undefined.
  grammarSource: string;

  // If true, sets location information on each AST node, in the form of
  // { column: number, line: number, offset: number }. Defaults to false.
  includeLocation: boolean;

  // If true, causes the parser to raise an error instead of log a warning.
  // The parser does limited type checking, and things like undeclared variables
  // are treated as warnings. Defaults to false.
  failOnWarn: boolean;
}

The program type

parser.parse() returns a Program, which is a special AST Node:

interface Program {
  // Hard coded AST node type of "program"
  type: 'program';
  // The AST itself is an array of the top level statements
  program: AstNode[];
  // All of the scopes found during parsing
  scopes: Scope[];
  // Leading whitespace of the source code
  wsStart?: string;
  // Trailing whitespace of the source code
  wsEnd?: string;
}

Scope

parse() returns a Program, which has a scopes array on it. A scope looks like:

type Scope = {
  name: string;
  parent?: Scope;
  bindings: ScopeIndex;
  types: TypeScopeIndex;
  functions: FunctionScopeIndex;
  location?: LocationObject;
}

The name of a scope is either "global", the name of the function that introduced the scope, or in anonymous blocks, "{". In each scope, bindings represents variables, types represents user-created types (structs in GLSL), and functions represents functions.

For bindings and types, the scope index looks like:

type ScopeIndex = {
  [name: string]: {
    declaration?: AstNode;
    references: AstNode[];
  }
}

Where name is the name of the variable or type. declaration is the AST node where the variable was declared. In the case the variable is used without being declared, declaration won't be present. If you set the failOnWarn parser option to true, the parser will throw an error when encountering an undeclared variable, rather than allow a scope entry without a declaration.

For functions, the scope index is slightly different:

type FunctionScopeIndex = {
  [name: string]: {
    [signature: string]: {
      returnType: string;
      parameterTypes: string[];
      declaration?: FunctionNode;
      references: AstNode[];
    }
  }
};

Where name is the name of the function, and signature is a string representing the function's return and parameter types, in the form of "returnType: paramType1, paramType2, ..." or "returnType: void" in the case of no arguments. Each signature in this index represents an "overloaded" function in GLSL, as in:

void someFunction(int x) {};
void someFunction(int x, int y) {};

With this source code, there will be two entries under name, one for each overload signature. The references are the uses of that specific overloaded version of the function. references also contains the function prototypes for the overloaded function, if present.

In the case there is only one declaration for a function, there will still be a single entry under name with the function's signature.

⚠️ Caution! This parser does very limited type checking. This leads to a known case where a function call can match to the wrong overload in scope:

void someFunction(float, float);
void someFunction(bool, bool);
someFunction(true, true); // This will be attributed to the wrong scope entry

The parser doesn't know the type of the operands in the function call, so it matches based on the name and arity of the functions.

See also Utility-Functions for renaming scope references.

Errors

If you have invalid GLSL, the parser throws a GlslSyntaxError. The parser uses Peggy under the hood, so GlslSyntaxError is a convenience type alias for a peggy.SyntaxError.

import { parser, GlslSyntaxError } from '@shaderfrog/glsl-parser';

let error: GlslSyntaxError | undefined;
try {
  // Line without a semicolon
  c.parse(`float a`);
} catch (e) {
  error = e as GlslSyntaxError;
}

If you want to check if a caught error is an instanceof a GlslSyntaxError, then you need to use the Peggy SyntaxError error object, which lives on the parser:

console.log(error instanceof parser.SyntaxError)
// true

The only error the parser intentionally throws is a GlslSyntaxError. You should be safe to cast it to a GlslSyntaxError with as in Typescript.

The error message string is automatically generated by Peggy:

console.log(error.message)
// 'Expected ",", ";", "=", or array specifier but end of input found'

The error object includes the location of the error. It is not printed in the error message by default.

console.log(error.location)
/*
{
  source: undefined,
  start: { offset: 7, line: 1, column: 8 },
  end: { offset: 7, line: 1, column: 8 }
}
*/

Note the source field on the error object is the grammarSource string provided to the parser options, which is undefined by default. If you pass in a grammarSource to parser.parse(), it shows up in the error object. This is meant to help you track which source file you're parsing, for example you could enter "myfile.glsl" as an argument to parser.parse() so that the error includes that your source GLSL came from your application's myfile.glsl file.

The error object also has a fairly confusing format() method, which comes from the underlying Peggy error object. It produces an ASCII formatted string with arrows and underlines. The source option passed to .format() must match your grammarSource in parser options (which is undefined by default). This API is awkward and I might override it in future versions of the parser.

console.log(error.format([{ text, source: undefined }])
/*
Error: Expected ",", ";", "=", or array specifier but "f" found.
  --> undefined:2:1
  |
2 | float b
  | ^
*/

Preprocessing

The parser also ships with a preprocess() function.

The preprocessor takes in a program source code string and produces a preprocessed program source code string. If you want to access and manipulate the AST produced by preprocessing, see the next sections.

import preprocess from '@shaderfrog/glsl-parser/preprocessor';

// Preprocess a program
console.log(preprocess(`
  #define a 1
  float b = a;
`, options));

Where options is:

type PreprocessorOptions = {
  // Don't strip comments before preprocessing
  preserveComments: boolean,
  // Macro definitions to use when preprocessing
  defines: {
    SOME_MACRO_NAME: 'macro body'
  },
  // A list of callbacks evaluated for each node type, and returns whether or not
  // this AST node is subject to preprocessing
  preserve: {
    ast_node_name: (path) => boolean
  }
}

A preprocessed program string can be handed off to the main GLSL parser. Preprocessing is optional, but a program string may not be valid until it is preprocessed.

If you want more control over preprocessing, the preprocess function above is a convenience method for approximately the following:

import {
  preprocessAst,
  preprocessComments,
  generate,
  parser,
} from '@shaderfrog/glsl-parser/preprocessor';

// Remove comments before preprocessing
const commentsRemoved = preprocessComments(`float a = 1.0;`);

// Parse the source text into an AST
const ast = parser.parse(commentsRemoved);

// Then preprocess it, expanding #defines, evaluating #ifs, etc
preprocessAst(ast);

// Then convert it back into a program string, which can be passed to the
// core glsl parser
const preprocessed = generate(ast);

Manipulating and Searching ASTs

Visitors

The Shaderfrog parser provides a AST visitor function for manipulating and searching an AST. The visitor API loosely follows the Babel visitor API. A visitor object looks like:

const visitors = {
  function_call: {
    enter: (path) => {},
    exit: (path) => {},
  }
}

Where every key in the object is a node type, and every value is an object with optional enter and exit functions. What's passed to each function is not the AST node itself, instead it's a "path" object, which gives you information about the node's parents, methods to manipulate the node, and the node itself. The path object:

{
  // Properties:

  // The node itself
  node: AstNode;
  // The parent of this node
  parent: AstNode | null;
  // The parent path of this path
  parentPath: Path | null;
  // The key of this node in the parent object, if node parent is an object
  key: string | null;
  // The index of this node in the parent array, if node parent is an array
  index: number | null;

  // Methods:

  // Don't visit any children of this node
  skip: () => void;
  // Stop traversal entirely
  stop: () => void;
  // Remove this node from the AST
  remove: () => void;
  // Replace this node with another AST node. See replaceWith() documentation.
  replaceWith: (replacer: any) => void;
  // Search for parents of this node's parent using a test function
  findParent: (test: (p: Path) => boolean) => Path | null;
}

Visit an AST by calling the visit method with an AST and visitors:

import { visit } from '@shaderfrog/glsl-parser/ast';

visit(ast, visitors);

The visit function doesn't return a value. If you want to collect data from the AST, use a variable in the outer scope to collect data. For example:

let numberOfFunctionCalls = 0;
visit(ast, {
  function_call: {
    enter: (path) => {
      numberOfFunctionCalls += 1;
    },
  }
});
console.log('There are ', numberOfFunctionCalls, 'function calls');

You can also visit the preprocessed AST with visitPreprocessedAst. Visitors follow the same convention outlined above.

import {
  parser,
  visitPreprocessedAst,
} from '@shaderfrog/glsl-parser/preprocessor';

// Parse the source text into an AST
const ast = parser.parse(`float a = 1.0;`);
visitPreprocessedAst(ast, visitors);

Stopping traversal

To skip all children of a node, call path.skip().

To stop traversal entirely, call path.stop() in either enter() or exit(). No future enter() nor exit() callbacks will fire.

Visitor .replaceWith() Behavior

When you visit a node and call path.replaceWith(otherNode) inside the visitor's enter() method: 1. otherNode and its children are visited by the same visitors. 2. The exit() function of the visitor is not called.

Notes: - Calling .replaceWith() in a visitor's exit() method is undefined behavior. - Replacing a node with the same type can cause infinite recursion, as the visitor will continue to visit the replaced node of the same type. You must handle this case manually.

These rules apply to all visitors, both GLSL AST visitors and preprocessor AST visitors.

Utility Functions

Rename variables / identifiers in a program

You can rename bindings (aka variables), functions, and types (aka structs) with renameBindings, renameFunctions, and renameTypes respectively.

The signature for these methods:

const renameBindings = (
  // The scope to rename the bindings in. ast.scopes[0] is the global scope.
  // Passing this ast.scopes[0] renames all global variables
  bindings: ScopeIndex,

  // The rename function. This is called once per scope entry with the original
  // name in the scope, to generate the renamed variable. 
  mangle: (name: string) => string
): ScopeIndex

These scope renaming functions,