pest

After trying to use flex/bison, some 20 years ago, for parsing a programming language, I quickly switched to hand writting lexer and parser. The primary motivation is that those tools tend to have more drawbacks than benefits. However, I believe that it is important to constantly reassess your views, and check if tooling has improved.

While there are dozens of available generators, I don’t have the time to evaluate all of them. When converting GQLite from C++ to Rust, I settled on pest, it appeared to be very popular with 195M downloads on crates.io, it has an extension for VSCode and it is maintained. I also liked that the syntax of the grammar makes is closed to the BNF Notation, which makes it easier to read.

While pest is popular it fell short for GQLite. The core issue: it doesn’t generate a proper parse tree. The resulting data structure is cumbersome, lacks compile-time guarantees, and loses critical parsing information. For a complex language like OpenCypher, this became a major obstacle. It might be suitable for smaller language, but it becomes a major pain with larger language like OpenCypher. The main problem is that the structure can be simplified to:

struct Node
{
  rule: Rule,
  children: Vec<Node>, 
}

Where Rule is an enum that specify the rule. So essentially, pest translate the text source input into an other structure, that then need to be parsed into your final structure. In theory, the children part of the structure is constant for each rule. However, if you want to write safe Rust code, you shouldn’t make that assumption. There has been a number of effort for pest to provide a parse tree (pest-ast, pest 3) but none have delivered yet.

logos/nom

So after deciding to drop pest, I was left with the choice of trying an other tool, or writting a parser from scratch. The original C++ parser was easy to write, as over the years, I have developed my own toolbox for parsing in C++. Since, it is the first parser I wrote in Rust, I had nothing to start with. I looked around to see what is available, and a popular choice in the Rust ecosystem is nom with 422M downloads on crates.io.

The nice thing about nom is that it gives you the primitives to work with and write your hand-crafted parser. By default, it works on stream of characters, while I was developing the parser this felt awkward. At least for a language like OpenCypher, it is more natural to look at it as a stream of token. So I needed a lexer, and after looking for what is available I settled for logos, it is well documented, well maintained, and fast.

The parsing was done on pokec_small_import.cypher, a 5MB big, 131716 lines long creation query. You could argue that a different choice of query would give different results. However, when it comes to OpenCypher the only long queries are the queries for creating nodes. And that is the only time parsing speed has an affect, as can be seen on the benchmark results.

The main new feature is the postgres backend which is currently implemented as a client-side library, with server-side extension still on the todo list. This pave the way with integration into kDB. This new backend is on par with other backends in term of supported OpenCypher queries.

A major change is that only the redb backend is enabled by default in the gqlitedb rust crate. However all three backends are enabled for the python, ruby and C/C++ bindings.

You can check our installation instructions. And the up-to-date changelog for the query enginge.

It also includes all the fix from the 1.3.x series, including improvement to the GQL GUI Browser, and fixes to query engine.

You can check our installation instructions. And the up-to-date changelog for the query enginge.

It also includes all the fix from the 1.2.x series, including the GQL GUI Browser, improvements to aggregation, support for multi-threaded environments.

You can check our installation instructions. And the up-to-date changelog for the query enginge.

This release also include the first attempt at looking to support multithreading, through an experimental ConnecitonServer.

The roadmap includes support for a Redb and SQLite backends, improvement to the Ruby/Python bindings, continuous improvement to the support of OpenCypher.

You can check our installation instructions.

You can check our installation instructions.

GQLite 1.0 can be used in C, C++, Ruby, Python and Crystal program. QML bindings are coming via the Cyqlops library.

You can check our installation instructions.