[![logo][logo-url]][reflex-url]

[![build status][ci-image]][ci-url] [![license][bsd-3-image]][bsd-3-url] [![CodeProject][codeproject-image]][codeproject-url]

A high-performance C++ regex library and lexical analyzer generator with Unicode support.

Two example use cases:

1. A RE/flex-generated tokenizer is used by the Tiger Compiler.
2. The RE/flex C++ regex engines are used by ugrep.

The RE/flex lexical analyzer generator extends Flex++ with Unicode support, indent/dedent anchors, POSIX regex lazy quantifiers, word boundaries, functions for lex and syntax error reporting, lexer rule execution performance profiling, and other new features.

Only RE/flex supports backtrack-free regex lazy matching in linear time using an advanced DFA transformation algorithm (invented by Dr. Robert van Engelen.)

RE/flex is faster than Flex and much faster than regex libraries such as Boost.Regex, C++11 std::regex, PCRE2 and RE2. For example, tokenizing a 2 KB representative C source code file into 244 tokens takes only 8.7 microseconds:

Command / Function	Software	Time (μs)
reflex --fast --noindent	RE/flex 3.4.1	8.7
reflex --fast	RE/flex 3.4.1	8.9
flex -+ --full	Flex 2.5.35	9.8
boost::spirit::lex::lexertl::actor_lexer::iterator_type	Boost.Spirit.Lex 1.82.0	10.7
reflex --full	RE/flex 3.4.1	20.6
pcre2_jit_match()	PCRE2 (jit) 10.42	60.8
hs_compile_multi(), hs_scan()	Hyperscan 5.4.2	129
reflex -m=boost-perl	Boost.Regex 1.82.0	205
RE2::Consume()	RE2 (pre-compiled) 2023-09-01	218
reflex -m=boost	Boost.Regex POSIX 1.82.0	392
pcre2_match()	PCRE2 10.42	500
RE2::Consume()	RE2 POSIX (pre-compiled) 2023-09-01	534
flex -+	Flex 2.5.35	3759
pcre2_dfa_match()	PCRE2 POSIX (dfa) 10.42	4029
regcomp(), regexec()	GNU C POSIX.2 regex	4932
std::cregex_iterator()	C++11 std::regex	6490

Note: performance in elapsed time (lower is better) in microseconds for 1000 to 10000 benchmark runs using Mac OS X 12.6.9 with clang 12.0.0 -O2, 2.9 GHz Intel Core i7, 16 GB 2133 MHz LPDDR3. Hyperscan disqualifies as a scanner due to its "All matches reported" semantics resulting in 1915 matches for this test, and due to its event handler requirements. Download the tests

The performance table is indicative of the impact on performance when using PCRE2 and Boost.Regex with RE/flex. PCRE2 and Boost.Regex are optional libraries integrated with RE/flex for Perl matching because of their efficiency. By default, RE/flex uses DFA-based extended regular expression matching in linear time, the fastest method (as shown in the table).

The RE/flex matcher tracks line numbers, column numbers, and indentations, whereas Lex and Flex do not (option noyylineno) and neither do the other regex matchers in the table (except PCRE2 and Boost.Regex when used with RE/flex). Tracking this information incurs some overhead. RE/flex also automatically decodes UTF-8/16/32 input and accepts std::istream, strings, and wide strings as input.

Features

• Includes many examples, such as a mini C compiler to Java bytecode, a tokenizer for C/C++ source code, a tokenizer for Python source code, a tokenizer for Java source code, Lua, JSON, XML, YAML, and more.
• Compatible with Flex and Bison to eliminate a learning curve, making a transition from Flex++ to RE/flex frustration-free.
• Auto-generates code that integrates seamlessly with Bison Reentrant, Bison-Bridge, Bison-Locations, Bison 3.0 C++ interface %skeleton "lalr1.cc" and Bison Complete Symbols.
• Generates code and includes methods for lexical and syntax error reporting and recovery.
• The generated scanner source code is structured and easy to understand.
• Fully supports Unicode and Unicode properties \p{C}, including Unicode identifier matching for C++11, Java, C#, and Python source code.
• Auto-detects UTF-8/16/32 input to match Unicode patterns.
• Supports file encodings ISO-8859-1 through ISO-8859-15, CP 1250 through 1258, CP 437, CP 850, CP 858, KOI8, MACROMAN, EBCDIC, and custom code pages.
• Generates scanners for lexical analysis on files, C++ streams, (wide) strings, and memory such as mmap files.
• Indent/nodent/dedent anchors to match indentation levels to tokenize.
• Lazy quantifiers for POSIX regex matching, i.e. no hacks are needed to work around greedy repetitions.
• Word boundary anchors.
• Freespace mode option to improve readability of lexer specifications.
• %class and %init to customize the generated Lexer classes.
• %include to modularize lexer specifications.
• Multiple lexer classes can be combined and used in one application, e.g. by multiple threads in a thread-safe manner.
• Configurable Lexer class generation to customize the interface for various parsers, including Yacc and Bison.
• Generates Graphviz files to visualize FSMs with the Graphviz dot tool.
• Includes an extensible hierarchy of pattern matcher engines, with a choice of regex engines, including the RE/flex regex engine, PCRE2, and Boost.Regex.
• The RE/flex regex library makes C++11 std::regex, PCRE2, and Boost.Regex much easier to use for pattern matching on (wide) strings, files, and streams.
• IEEE POSIX P1003.2 standard compliant like Lex and Flex (but generates C++).
• Extensive documentation in the online [User Guide][manual-url].
• Lots of other improvements over Flex++, such as yypush_buffer_state saves the scanner state (line, column, and indentation positions), not just the input buffer; no input buffer length limit (Flex has a 16KB limit); line() returns the current line (e.g. for error reporting).

Note: PCRE2 and Boost.Regex are not dependencies, they can be used as optional regex engines in addition to the RE/flex regex engine.

Installation

Windows

Use reflex/bin/reflex.exe from the command line or add a Custom Build Step in MSVC++ as follows:

1. select the project name in Solution Explorer then Property Pages from the Project menu (see also custom-build steps in Visual Studio);

2. add an extra path to the reflex/include folder in the Include Directories under VC++ Directories, which should look like $(VC_IncludePath);$(WindowsSDK_IncludePath);C:\Users\YourUserName\Documents\reflex\include (this assumes the reflex source package is in your Documents folder).

3. enter "C:\Users\YourUserName\Documents\reflex\bin\win32\reflex.exe" --header-file "C:\Users\YourUserName\Documents\mylexer.l" in the Command Line property under Custom Build Step (this assumes mylexer.l is in your Documents folder);

4. enter lex.yy.h lex.yy.cpp in the Outputs property;

5. specify Execute Before as PreBuildEvent.

If you are using specific reflex options such as --flex then add these in step 3.

Before compiling your program with MSVC++, drag the folders reflex/lib and reflex/unicode to the Source Files in the Solution Explorer panel of your project. Next, run reflex.exe simply by compiling your project (which may fail, but that is OK for now as long as we executed the custom build step to run reflex.exe). Drag the generated lex.yy.h (if present) and lex.yy.cpp files to the Source Files. Now you are all set!

In addition, the reflex/vs directory contains batch scripts to build projects with MS Visual Studio C++.

MacOS

On macOS systems you can use homebrew to install RE/flex with brew install re-flex. Or use MacPorts to install RE/flex with sudo port install re-flex.

NetBSD

On NetBSD systems you can use the standard NetBSD package installer (pkgsrc): http://cdn.netbsd.org/pub/pkgsrc/current/pkgsrc/devel/RE-flex/README.html

Quick install

First clone the code:

$ git clone https://github.com/Genivia/RE-flex

Then simply do a quick clean build, assuming your environment is pretty much standard:

$ ./clean.sh
$ ./build.sh

This compiles the reflex tool and installs it locally in reflex/bin. For local use of RE/flex in your project, you can add this location to your $PATH variable to enable the new reflex command:

$ export PATH=$PATH:/your_path_to_reflex/reflex/bin

Note that the libreflex.a and libreflex.so libraries are saved locally in reflex/lib. Link against the library when you use the RE/flex regex engine in your code, such as:

$ c++ <options and .o/.cpp files> -L/your_path_to_reflex/reflex/lib -lreflex

or you could statically link libreflex.a with:

$ c++ <options and .o/.cpp files> /your_path_to_reflex/reflex/lib/libreflex.a

Also note that the RE/flex header files that you will need to include in your project are locally located in include/reflex.

To install the man page, the header files in /usr/local/include/reflex, the library in /usr/local/lib and the reflex command in /usr/local/bin:

$ sudo ./allinstall.sh

Configure and make

The configure script accepts configuration and installation options. To view these options, run:

$ ./configure --help

Run configure and make:

$ ./configure && make

To build the examples also:

$ ./configure --enable-examples && make

After this successfully completes, you can optionally run make install to install the reflex command and the libreflex library:

$ sudo make install

Unfortunately, cloning from Git does not preserve timestamps which means that you may run into "WARNING: 'aclocal-1.15' is missing on your system." To work around this problem, run:

$ autoreconf -fi
$ ./configure && make

The above builds the library with SSE/AVX optimizations applied. To disable AVX optimizations:

$ ./configure --disable-avx && make

To disable both SSE2 and AVX optimizations:

$ ./configure --disable-sse2 && make

Optional libraries to install

• To use PCRE2 as a regex engine with the RE/flex library and scanner generator, install [PCRE2][pcre-url] and link your code with -lpcre2-8.

• To use Boost.Regex as a regex engine with the RE/flex library and scanner generator, install [Boost][boost-url] and link your code with -lboost_regex or -lboost_regex-mt.

• To visualize the FSM graphs generated with reflex option --graphs-file, install [Graphviz dot][dot-url].

Improved Vim syntax highlighting

Copy the lex.vim file to ~/.vim/syntax/ to enjoy improved syntax highlighting for both Flex and RE/flex.

Usage

There are two ways you can use this project:

1. as a scanner generator for C++, similar to Flex;
2. as a flexible regex library API for C++.

For the first use case, use the reflex tool on the command line on a lexer specification:

$ reflex --flex --bison --graphs-file lexspec.l

This generates a scanner for Bison from the lexer specification lexspec.l and saves the finite state machine (FSM) as a Graphviz .gv file that can be visualized with the [Graphviz dot][dot-url] tool:

$ dot -Tpdf reflex.INITIAL.gv > reflex.INITIAL.pdf
$ open reflex.INITIAL.pdf

![Visualize DFA graphs with Graphviz dot][FSM-url]

Several examples are included to get you started. See the [manual][manual-url] for more details.

For the second use case, use the RE/flex matcher API classes to start pattern search, matching, splitting and scanning on strings, wide strings, files, and streams.

You can select matchers that are based on different regex engines:

• RE/flex regex: #include <reflex/matcher.h> and use reflex::Matcher;
• RE/flex fuzzy regex for approximate matching: #include <reflex/fuzzymatcher.h> and use reflex::FuzzyMatcher
• PCRE2: #include <reflex/pcre2matcher.h> and use reflex::PCRE2Matcher or reflex::PCRE2UTFMatcher.
• Boost.Regex: #include <reflex/boostmatcher.h> and use reflex::BoostMatcher or reflex::BoostPosixMatcher;
• C++11 std::regex: #include <reflex/stdmatcher.h> and use reflex::StdMatcher or reflex::StdPosixMatcher.

Each matcher may differ in regex syntax features (see the full documentation), but they all share the same methods and iterators, such as:

• matches() returns nonzero if the whole input from start to end matches the specified pattern;
• find() search input and returns nonzero if a match was found, can be repeated;
• scan() scan input and returns nonzero if input at current position matches, can be repeated;
• split() returns nonzero for a split of the input at the next match, can be repeated;
• find.begin()...find.end() a filter iterator, iterates with find();
• scan.begin()...scan.end() a tokenizer iterator, iterates with scan();
• split.begin()...split.end() a splitter iterator, iterates with split().

The input matched and searched may be a string, a wide string, a file, or a stream. Searching is incremental, meaning that the input is not buffered as a whole in memory, but rather buffered in parts in a sliding window of a few KB. The window size may grow to fit a pattern match. UTF-16/32 file input with a UTF BOM is automatically normalized and matched as UTF-8.

For example, using Boost.Regex (alternatively use PCRE2 reflex::PCRE2Matcher or reflex::PCRE2UTFMatcher to match Unicode UTF-8 input):

```{.cpp}

include <reflex/boo