Introduction

Introduction

The Wave C++ preprocessor library is a Standards conformant implementation of the mandated C99/C++ preprocessor functionality packed behind a simple to use interface, which integrates well with the well known idioms of the Standard Template Library (STL).

The Wave C++ preprocessor is not a monolithic application, it's rather a modular library, which exposes mainly a context object and an iterator interface. The context object helps to configure the actual preprocessing process (as search path's, predefined macros, etc.). The exposed iterators are generated by this context object too. Iterating over the sequence defined by the two iterators will return the preprocessed tokens, which are built on the fly from the given input stream.

The C++ preprocessor iterator itself is fed by a C++ lexer iterator, which implements an abstract interface. The C++ lexers packaged with the Wave library may be used standalone, too, and are not tied to the C++ preprocessor iterator at all.

To make the C++ preprocessing library modular, the C++ lexer is held completely separate and independent from the preprocessor. To prove this concept, two different, but functionally identical C++ lexers were implemented. Additionally there is implemented a IDL lexer, which allows to use the preprocessor library as the lexing component of a IDL oriented tool. All these lexers implement the mentioned abstract interface, so that the C++ preprocessor iterator may be used with all of them. The abstraction of the lexer from the preprocessor iterator library was done to allow plugging in different lexers without the need to reimplement the preprocessor. This will allow for benchmarking and specific fine tuning of the process of preprocessing itself.

The first of these C++ lexers is implemented with the help of the well known Re2C [3] tool, which generates C code from given regular expressions. The lexers generated with Re2C are known to be very fast, because they are not table driven but directly code the token building logic (very similar to hand coded lexers).

The second of these C++ lexers is built around a table driven lexer, where the DFA tables (discrete finite automaton tables) are generated from regular expressions with the help of a Spirit-based lexer generating framework named Slex [5]. The Slex is fed during runtime with the token definitions (regular expressions) and generates the resulting DFA table. This table is used to combine the input characters into corresponding lexemes (tokens). The generated DFA table can be saved to disk to avoid the generation process at program startup.

Wave may be used for preprocessing IDL files too, since the token set needed for the IDL language is very similar to the C++ token set. That's the reason, why the Wave preprocessor library contains also an IDL lexer. The IDL lexer is also based on the Re2C tool, but recognizes a different set of tokens. So this lexer does not recognize any keywords (except true and false, which are needed by the preprocessor itself). This is needed because there exist different IDL language flavours, where identifiers of one flavour may be keywords of others - Ok, this requires postponement of keyword identification until after the preprocessing, but allows to use Wave for all of the IDL derivatives.

It is possible to build other C++ lexers if needed. Currently there are plans to adapt the Spirit C++ lexer example cpplexer [6], which is completely based on static Spirit[4] grammars.

Both of the included lexers and the library itself are able to act in a C99 compliant mode. In this mode the lexers reject C++-only tokens ('::', '->*', '.*' and the alternate keywords such as 'and', etc.). The preprocessor additionally handles placemarkers (empty macro arguments) and variadics (macros with variable parameter counts). As an extension to the C++ Standard, the library can be enabled to handle placemarkers and variadics in C++ mode too.

Last updated: Sunday, December 2, 2007 19:03