RNTuple Introduction

RNTuple (for n-tuple and nested tuple) is the experimental evolution of TTree columnar data storage. RNTuple introduces new interfaces that aim to be more robust. In particular, the new interfaces are type-safe through the use of templates, and the ownership is well-defined through the use of smart pointers. For instance

tree->Branch("px", &Category, "px/F");

becomes

auto px = model->MakeField<float>("px");
// px is std::shared_ptr<float>

The physical layout changes slightly from big endian to little endian so that it matches the in-memory layout on most modern architectures. Combined with a clear separation of offset/index data and payload data for collections, uncompressed RNTuple data can be directly mapped to memory without further copies.

Goals

RNTuple shall investigate improvements of the TTree I/O in the following ways

1. More speed
   • Improve mapping to vectorized and parallel hardware
   • For types known at compile / JIT time: generate optimized code
   • Optimized for simple types (float, int, and vectors of them)
   • Better memory control: work with a fixed budget of pre-defined I/O buffers
   • Naturally thread-safe and asynchronous interfaces
2. More robust interfaces
   • Compile-time type safety by default
   • Decomposition into layers: logical layer, primitives layer, storage layer
   • Separation of data model and live data
   • Self-contained I/O code to support creation of a standalone I/O library

Concepts

At the logical layer, the user defines a data model using the RNTupleModel class. The data model is a collection of serializable C++ types with associated names, similar to branches in a TTree. The data model can contain (nested) collections, e.g., a type can be std::vector<std::vector<float>>.

Each serializable type is represented by a field, concretely by a templated version of RField, e.g. RField<double>. A field can generate or adopt an associated value, which represents a memory location storing a value of the given C++ type. These distinguished memory locations are the destinations and sources for the deserialization and serialization.

The (de-)serialization is a mapping from the C++ type to the more simple column type system. A column contains an arbitrary number of fixed-sized elements of a well-defined set of types: integers and floats of different bit sizes. A C++ type may be mapped to multiple columns. For instance, an std::vector<float> maps to two columns, an offset column indicating the size of the vector per entry, and a payload column with the float data.

Columns are partitioned into pages (roughly: TTree baskets) of a few kB – a few tens of kB each. The physical layer (only) needs to provide the means to store and retrieve pages. The physical layer is decoupled from the high-level C++ logic. The physical layer implements an abstract page storage interface, so that dedicated implementations for key-value stores and other storage systems are conceivable. At this point, the only provided backend stores the pages in ROOT files.

RNTuples are further grouped into clusters, which are, like TTree clusters, self-contained blocks of consecutive entries. Clusters provide a unit of writing and will provide the means for parallel writing of data in a future version of RNTuple.

Related classes