Why yet another query engine

Couldn’t you achieve the same design tenets via other query engines like SparkSQL? Seems that SparkSQL can do the same as long as the underlying data storage can be exposed as a DataFrame (similar to PartiQL Data Model).


Or why not participate in something like Apache Calcite?

The open source provides a lightweight reference engine for PartiQL, so that the community can easily check compatibility of syntax, semantics and also use the parser, etc.

I can give a little bit of background in addition to what @yannip says. We needed something really small and light to explore and demonstrate what we wanted to do. SparkSQL and Calcite are great projects and I’d love to figure out how PartiQL can fit with them some day. Spark SQL in particular has a very traditional schema required (i.e. only statically typed) implementation. Calcite (at least the many moons ago that I looked into it) was similar. The dynamic typing (schema optional) and every type being first-class are examples a big enough differences that I thought just implementing something completely from the ground up was warranted for this purpose rather than having to refactor a code base serving a different purpose.

It also happens that the library happens to be a easy to embed, so it has practical usage beyond a pure reference, but the goal of the codebase wasn’t/isn’t necessarily to be a grand database engine or replace the ones out there.

Hope that helps at least provides some context.