Logo ROOT  
Reference Guide
 
Loading...
Searching...
No Matches
ROOT::Experimental::RNTupleDS Class Referencefinal

The RDataSource implementation for RNTuple.

It lets RDataFrame read RNTuple data.

An RDataFrame that reads RNTuple data can be constructed using FromRNTuple().

For each column containing an array or a collection, a corresponding column #colname is available to access colname.size() without reading and deserializing the collection values.

Definition at line 46 of file RNTupleDS.hxx.

Classes

struct  REntryRangeDS
 The PrepareNextRanges() method populates the fNextRanges list with REntryRangeDS records. More...
 
struct  RFieldInfo
 Holds useful information about fields added to the RNTupleDS. More...
 

Public Member Functions

 RNTupleDS (ROOT::Experimental::RNTuple *ntuple)
 
 RNTupleDS (std::string_view ntupleName, const std::vector< std::string > &fileNames)
 
 RNTupleDS (std::string_view ntupleName, std::string_view fileName)
 
 ~RNTupleDS ()
 
void Finalize () final
 Convenience method called after concluding an event-loop.
 
void FinalizeSlot (unsigned int slot) final
 Convenience method called at the end of the data processing associated to a slot.
 
const std::vector< std::string > & GetColumnNames () const final
 Returns a reference to the collection of the dataset's column names.
 
std::unique_ptr< ROOT::Detail::RDF::RColumnReaderBaseGetColumnReaders (unsigned int, std::string_view, const std::type_info &) final
 If the other GetColumnReaders overload returns an empty vector, this overload will be called instead.
 
std::vector< std::pair< ULong64_t, ULong64_t > > GetEntryRanges () final
 Return ranges of entries to distribute to tasks.
 
std::string GetLabel () final
 Return a string representation of the datasource type.
 
std::size_t GetNFiles () const final
 Returns the number of files from which the dataset is constructed.
 
std::string GetTypeName (std::string_view colName) const final
 Type of a column as a string, e.g.
 
bool HasColumn (std::string_view colName) const final
 Checks if the dataset has a certain column.
 
void Initialize () final
 Convenience method called before starting an event-loop.
 
void InitSlot (unsigned int slot, ULong64_t firstEntry) final
 Convenience method called at the start of the data processing associated to a slot.
 
bool SetEntry (unsigned int, ULong64_t) final
 Advance the "cursors" returned by GetColumnReaders to the selected entry for a particular slot.
 
void SetNSlots (unsigned int nSlots) final
 Inform RDataSource of the number of processing slots (i.e.
 
- Public Member Functions inherited from ROOT::RDF::RDataSource
virtual ~RDataSource ()=default
 
template<typename T >
std::vector< T ** > GetColumnReaders (std::string_view columnName)
 Called at most once per column by RDF.
 

Protected Member Functions

Record_t GetColumnReadersImpl (std::string_view name, const std::type_info &) final
 type-erased vector of pointers to pointers to column values - one per slot
 
- Protected Member Functions inherited from ROOT::RDF::RDataSource
virtual std::string AsString ()
 

Private Member Functions

 RNTupleDS (std::unique_ptr< ROOT::Experimental::Internal::RPageSource > pageSource)
 
void AddField (const RNTupleDescriptor &desc, std::string_view colName, DescriptorId_t fieldId, std::vector< RFieldInfo > fieldInfos)
 Provides the RDF column "colName" given the field identified by fieldID.
 
void ExecStaging ()
 The main function of the fThreadStaging background thread.
 
void PrepareNextRanges ()
 Populates fNextRanges with the next set of entry ranges.
 
void StageNextSources ()
 Starting from fNextFileIndex, opens the next fNSlots files.
 

Private Attributes

std::vector< std::vector< Internal::RNTupleColumnReader * > > fActiveColumnReaders
 List of column readers returned by GetColumnReaders() organized by slot.
 
std::vector< std::string > fColumnNames
 
std::vector< std::string > fColumnTypes
 
std::vector< REntryRangeDSfCurrentRanges
 Basis for the ranges returned by the last GetEntryRanges() call.
 
std::condition_variable fCvStaging
 Signal for the state information of fIsReadyForStaging and fHasNextSources.
 
std::unordered_map< ROOT::Experimental::DescriptorId_t, std::string > fFieldId2QualifiedName
 Connects the IDs of active proto fields and their subfields to their fully qualified name (a.b.c.d).
 
std::vector< std::string > fFileNames
 
std::unordered_map< ULong64_t, std::size_t > fFirstEntry2RangeIdx
 Maps the first entries from the ranges of the last GetEntryRanges() call to their corresponding index in the fCurrentRanges vectors.
 
bool fHasNextSources = false
 Is true when the staging thread has populated the next batch of files to fStagingArea.
 
bool fIsReadyForStaging = false
 Is true when the staging thread should start working.
 
std::mutex fMutexStaging
 Protects the shared state between the main thread and the I/O thread.
 
std::size_t fNextFileIndex = 0
 Index into fFileNames to the next file to process.
 
std::vector< REntryRangeDSfNextRanges
 Basis for the ranges populated by the PrepareNextRanges() call.
 
unsigned int fNSlots = 0
 
std::string fNTupleName
 The data source may be constructed with an ntuple name and a list of files.
 
std::unique_ptr< RNTupleDescriptorfPrincipalDescriptor
 A clone of the first pages source's descriptor.
 
std::vector< std::unique_ptr< ROOT::Experimental::RFieldBase > > fProtoFields
 We prepare a prototype field for every column.
 
ULong64_t fSeenEntries = 0
 The number of entries so far returned by GetEntryRanges()
 
std::vector< std::unique_ptr< ROOT::Experimental::Internal::RPageSource > > fStagingArea
 The staging area is relevant for chains of files, i.e.
 
bool fStagingThreadShouldTerminate = false
 Is true when the I/O thread should quit.
 
std::thread fThreadStaging
 The background thread that runs StageNextSources()
 

Friends

class Internal::RNTupleColumnReader
 

Additional Inherited Members

- Protected Types inherited from ROOT::RDF::RDataSource
using Record_t = std::vector< void * >
 

#include <ROOT/RNTupleDS.hxx>

Inheritance diagram for ROOT::Experimental::RNTupleDS:
[legend]

Constructor & Destructor Documentation

◆ RNTupleDS() [1/4]

ROOT::Experimental::RNTupleDS::RNTupleDS ( std::unique_ptr< ROOT::Experimental::Internal::RPageSource pageSource)
explicitprivate

Definition at line 372 of file RNTupleDS.cxx.

◆ RNTupleDS() [2/4]

ROOT::Experimental::RNTupleDS::RNTupleDS ( std::string_view  ntupleName,
std::string_view  fileName 
)

Definition at line 409 of file RNTupleDS.cxx.

◆ RNTupleDS() [3/4]

ROOT::Experimental::RNTupleDS::RNTupleDS ( ROOT::Experimental::RNTuple ntuple)

Definition at line 414 of file RNTupleDS.cxx.

◆ RNTupleDS() [4/4]

ROOT::Experimental::RNTupleDS::RNTupleDS ( std::string_view  ntupleName,
const std::vector< std::string > &  fileNames 
)

Definition at line 419 of file RNTupleDS.cxx.

◆ ~RNTupleDS()

ROOT::Experimental::RNTupleDS::~RNTupleDS ( )
default

Member Function Documentation

◆ AddField()

void ROOT::Experimental::RNTupleDS::AddField ( const RNTupleDescriptor desc,
std::string_view  colName,
DescriptorId_t  fieldId,
std::vector< RFieldInfo fieldInfos 
)
private

Provides the RDF column "colName" given the field identified by fieldID.

For records and collections, AddField recurses into the sub fields. The fieldInfos argument is a list of objects holding info about the fields of the outer collection(s) (w.r.t. fieldId). For instance, if fieldId refers to an std::vector<Jet>, with struct Jet { float pt; float eta; }; AddField will recurse into Jet.pt and Jet.eta and provide the two inner fields as std::vector<float> each.

Definition at line 228 of file RNTupleDS.cxx.

◆ ExecStaging()

void ROOT::Experimental::RNTupleDS::ExecStaging ( )
private

The main function of the fThreadStaging background thread.

Definition at line 454 of file RNTupleDS.cxx.

◆ Finalize()

void ROOT::Experimental::RNTupleDS::Finalize ( )
finalvirtual

Convenience method called after concluding an event-loop.

See Initialize for more details.

Reimplemented from ROOT::RDF::RDataSource.

Definition at line 716 of file RNTupleDS.cxx.

◆ FinalizeSlot()

void ROOT::Experimental::RNTupleDS::FinalizeSlot ( unsigned int  )
finalvirtual

Convenience method called at the end of the data processing associated to a slot.

Parameters
[in]slotThe data processing slot wihch needs to be finalized This method might be called multiple times per thread per event-loop.

Reimplemented from ROOT::RDF::RDataSource.

Definition at line 673 of file RNTupleDS.cxx.

◆ GetColumnNames()

const std::vector< std::string > & ROOT::Experimental::RNTupleDS::GetColumnNames ( ) const
inlinefinalvirtual

Returns a reference to the collection of the dataset's column names.

Implements ROOT::RDF::RDataSource.

Definition at line 162 of file RNTupleDS.hxx.

◆ GetColumnReaders()

std::unique_ptr< ROOT::Detail::RDF::RColumnReaderBase > ROOT::Experimental::RNTupleDS::GetColumnReaders ( unsigned int  ,
std::string_view  ,
const std::type_info &   
)
finalvirtual

If the other GetColumnReaders overload returns an empty vector, this overload will be called instead.

Parameters
[in]slotThe data processing slot that needs to be considered
[in]nameThe name of the column for which a column reader needs to be returned
[in]tidA type_info At least one of the two must return a non-empty/non-null value.

Reimplemented from ROOT::RDF::RDataSource.

Definition at line 434 of file RNTupleDS.cxx.

◆ GetColumnReadersImpl()

RDF::RDataSource::Record_t ROOT::Experimental::RNTupleDS::GetColumnReadersImpl ( std::string_view  name,
const std::type_info &   
)
finalprotectedvirtual

type-erased vector of pointers to pointers to column values - one per slot

Implements ROOT::RDF::RDataSource.

Definition at line 427 of file RNTupleDS.cxx.

◆ GetEntryRanges()

std::vector< std::pair< ULong64_t, ULong64_t > > ROOT::Experimental::RNTupleDS::GetEntryRanges ( )
finalvirtual

Return ranges of entries to distribute to tasks.

They are required to be contiguous intervals with no entries skipped. Supposing a dataset with nEntries, the intervals must start at 0 and end at nEntries, e.g. [0-5],[5-10] for 10 entries. This function will be invoked repeatedly by RDataFrame as it needs additional entries to process. The same entry range should not be returned more than once. Returning an empty collection of ranges signals to RDataFrame that the processing can stop.

Implements ROOT::RDF::RDataSource.

Definition at line 584 of file RNTupleDS.cxx.

◆ GetLabel()

std::string ROOT::Experimental::RNTupleDS::GetLabel ( )
inlinefinalvirtual

Return a string representation of the datasource type.

The returned string will be used by ROOT::RDF::SaveGraph() to represent the datasource in the visualization of the computation graph. Concrete datasources can override the default implementation.

Reimplemented from ROOT::RDF::RDataSource.

Definition at line 166 of file RNTupleDS.hxx.

◆ GetNFiles()

std::size_t ROOT::Experimental::RNTupleDS::GetNFiles ( ) const
inlinefinalvirtual

Returns the number of files from which the dataset is constructed.

Reimplemented from ROOT::RDF::RDataSource.

Definition at line 161 of file RNTupleDS.hxx.

◆ GetTypeName()

std::string ROOT::Experimental::RNTupleDS::GetTypeName ( std::string_view  colName) const
finalvirtual

Type of a column as a string, e.g.

GetTypeName("x") == "double". Required for jitting e.g. df.Filter("x>0").

Parameters
[in]colNameThe name of the column

Implements ROOT::RDF::RDataSource.

Definition at line 683 of file RNTupleDS.cxx.

◆ HasColumn()

bool ROOT::Experimental::RNTupleDS::HasColumn ( std::string_view  colName) const
finalvirtual

Checks if the dataset has a certain column.

Parameters
[in]colNameThe name of the column

Implements ROOT::RDF::RDataSource.

Definition at line 689 of file RNTupleDS.cxx.

◆ Initialize()

void ROOT::Experimental::RNTupleDS::Initialize ( )
finalvirtual

Convenience method called before starting an event-loop.

This method might be called multiple times over the lifetime of a RDataSource, since users can run multiple event-loops with the same RDataFrame. Ideally, Initialize should set the state of the RDataSource so that multiple identical event-loops will produce identical results.

Reimplemented from ROOT::RDF::RDataSource.

Definition at line 694 of file RNTupleDS.cxx.

◆ InitSlot()

void ROOT::Experimental::RNTupleDS::InitSlot ( unsigned int  ,
ULong64_t   
)
finalvirtual

Convenience method called at the start of the data processing associated to a slot.

Parameters
[in]slotThe data processing slot wihch needs to be initialized
[in]firstEntryThe first entry of the range that the task will process. This method might be called multiple times per thread per event-loop.

Reimplemented from ROOT::RDF::RDataSource.

Definition at line 662 of file RNTupleDS.cxx.

◆ PrepareNextRanges()

void ROOT::Experimental::RNTupleDS::PrepareNextRanges ( )
private

Populates fNextRanges with the next set of entry ranges.

Moves files from the staging area as necessary and aligns ranges with cluster boundaries for scheduling the tail of files. Upon return, the fNextRanges list is ordered. It has usually fNSlots elements; fewer if there is not enough work to give at least one cluster to every slot.

Definition at line 489 of file RNTupleDS.cxx.

◆ SetEntry()

bool ROOT::Experimental::RNTupleDS::SetEntry ( unsigned int  slot,
ULong64_t  entry 
)
inlinefinalvirtual

Advance the "cursors" returned by GetColumnReaders to the selected entry for a particular slot.

Parameters
[in]slotThe data processing slot that needs to be considered
[in]entryThe entry which needs to be pointed to by the reader pointers Slots are adopted to accommodate parallel data processing. Different workers will loop over different ranges and will be labelled by different "slot" values. Returns true if the entry has to be processed, false otherwise.

Implements ROOT::RDF::RDataSource.

Definition at line 177 of file RNTupleDS.hxx.

◆ SetNSlots()

void ROOT::Experimental::RNTupleDS::SetNSlots ( unsigned int  nSlots)
finalvirtual

Inform RDataSource of the number of processing slots (i.e.

worker threads) used by the associated RDataFrame. Slots numbers are used to simplify parallel execution: RDataFrame guarantees that different threads will always pass different slot values when calling methods concurrently.

Implements ROOT::RDF::RDataSource.

Definition at line 739 of file RNTupleDS.cxx.

◆ StageNextSources()

void ROOT::Experimental::RNTupleDS::StageNextSources ( )
private

Starting from fNextFileIndex, opens the next fNSlots files.

Calls LoadStructure() on the opened files. The very first file is already available from the constructor.

Definition at line 472 of file RNTupleDS.cxx.

Friends And Related Symbol Documentation

◆ Internal::RNTupleColumnReader

friend class Internal::RNTupleColumnReader
friend

Definition at line 47 of file RNTupleDS.hxx.

Member Data Documentation

◆ fActiveColumnReaders

std::vector<std::vector<Internal::RNTupleColumnReader *> > ROOT::Experimental::RNTupleDS::fActiveColumnReaders
private

List of column readers returned by GetColumnReaders() organized by slot.

Used to reconnect readers to new page sources when the files in the chain change.

Definition at line 97 of file RNTupleDS.hxx.

◆ fColumnNames

std::vector<std::string> ROOT::Experimental::RNTupleDS::fColumnNames
private

Definition at line 93 of file RNTupleDS.hxx.

◆ fColumnTypes

std::vector<std::string> ROOT::Experimental::RNTupleDS::fColumnTypes
private

Definition at line 94 of file RNTupleDS.hxx.

◆ fCurrentRanges

std::vector<REntryRangeDS> ROOT::Experimental::RNTupleDS::fCurrentRanges
private

Basis for the ranges returned by the last GetEntryRanges() call.

Definition at line 101 of file RNTupleDS.hxx.

◆ fCvStaging

std::condition_variable ROOT::Experimental::RNTupleDS::fCvStaging
private

Signal for the state information of fIsReadyForStaging and fHasNextSources.

Definition at line 113 of file RNTupleDS.hxx.

◆ fFieldId2QualifiedName

std::unordered_map<ROOT::Experimental::DescriptorId_t, std::string> ROOT::Experimental::RNTupleDS::fFieldId2QualifiedName
private

Connects the IDs of active proto fields and their subfields to their fully qualified name (a.b.c.d).

This enables the column reader to rewire the field IDs when the file changes (chain), using the fully qualified name as a search key in the descriptor of the other page sources.

Definition at line 92 of file RNTupleDS.hxx.

◆ fFileNames

std::vector<std::string> ROOT::Experimental::RNTupleDS::fFileNames
private

Definition at line 64 of file RNTupleDS.hxx.

◆ fFirstEntry2RangeIdx

std::unordered_map<ULong64_t, std::size_t> ROOT::Experimental::RNTupleDS::fFirstEntry2RangeIdx
private

Maps the first entries from the ranges of the last GetEntryRanges() call to their corresponding index in the fCurrentRanges vectors.

This is necessary because the returned ranges get distributed arbitrarily onto slots. In the InitSlot method, the column readers use this map to find the correct range to connect to.

Definition at line 106 of file RNTupleDS.hxx.

◆ fHasNextSources

bool ROOT::Experimental::RNTupleDS::fHasNextSources = false
private

Is true when the staging thread has populated the next batch of files to fStagingArea.

Definition at line 117 of file RNTupleDS.hxx.

◆ fIsReadyForStaging

bool ROOT::Experimental::RNTupleDS::fIsReadyForStaging = false
private

Is true when the staging thread should start working.

Definition at line 115 of file RNTupleDS.hxx.

◆ fMutexStaging

std::mutex ROOT::Experimental::RNTupleDS::fMutexStaging
private

Protects the shared state between the main thread and the I/O thread.

Definition at line 111 of file RNTupleDS.hxx.

◆ fNextFileIndex

std::size_t ROOT::Experimental::RNTupleDS::fNextFileIndex = 0
private

Index into fFileNames to the next file to process.

Definition at line 82 of file RNTupleDS.hxx.

◆ fNextRanges

std::vector<REntryRangeDS> ROOT::Experimental::RNTupleDS::fNextRanges
private

Basis for the ranges populated by the PrepareNextRanges() call.

Definition at line 102 of file RNTupleDS.hxx.

◆ fNSlots

unsigned int ROOT::Experimental::RNTupleDS::fNSlots = 0
private

Definition at line 99 of file RNTupleDS.hxx.

◆ fNTupleName

std::string ROOT::Experimental::RNTupleDS::fNTupleName
private

The data source may be constructed with an ntuple name and a list of files.

Definition at line 63 of file RNTupleDS.hxx.

◆ fPrincipalDescriptor

std::unique_ptr<RNTupleDescriptor> ROOT::Experimental::RNTupleDS::fPrincipalDescriptor
private

A clone of the first pages source's descriptor.

Definition at line 60 of file RNTupleDS.hxx.

◆ fProtoFields

std::vector<std::unique_ptr<ROOT::Experimental::RFieldBase> > ROOT::Experimental::RNTupleDS::fProtoFields
private

We prepare a prototype field for every column.

If a column reader is actually requested in GetColumnReaders(), we move a clone of the field into a new column reader for RDataFrame. Only the clone connects to the backing page store and acquires I/O resources. The field IDs are set in the context of the first source and used as keys in fFieldId2QualifiedName.

Definition at line 88 of file RNTupleDS.hxx.

◆ fSeenEntries

ULong64_t ROOT::Experimental::RNTupleDS::fSeenEntries = 0
private

The number of entries so far returned by GetEntryRanges()

Definition at line 100 of file RNTupleDS.hxx.

◆ fStagingArea

std::vector<std::unique_ptr<ROOT::Experimental::Internal::RPageSource> > ROOT::Experimental::RNTupleDS::fStagingArea
private

The staging area is relevant for chains of files, i.e.

when fFileNames is not empty. In this case, files are opened in the background in batches of size fNSlots and kept in the staging area. The first file (chains or no chains) is always opened on construction in order to process the schema. For all subsequent files, the corresponding page sources in the staging area only executed LoadStructure(), i.e. they should have a compressed buffer of the meta-data available. Concretely:

  1. We open the first file on construction to read the schema and then move the corresponding page source in the staging area.
  2. On Initialize(), we start the I/O background thread, which in turn opens the first batch of files.
  3. At the beginning of GetEntryRanges(), we a) wait for the I/O thread to finish, b) call PrepareNextRanges() in the main thread to move the page sources from the staging area into fNextRanges; this will also call Attach() on the page sources (i.e., deserialize the meta-data), and c) trigger staging of the next batch of files in the I/O background thread.
  4. On Finalize(), the I/O background thread is stopped.

Definition at line 81 of file RNTupleDS.hxx.

◆ fStagingThreadShouldTerminate

bool ROOT::Experimental::RNTupleDS::fStagingThreadShouldTerminate = false
private

Is true when the I/O thread should quit.

Definition at line 119 of file RNTupleDS.hxx.

◆ fThreadStaging

std::thread ROOT::Experimental::RNTupleDS::fThreadStaging
private

The background thread that runs StageNextSources()

Definition at line 109 of file RNTupleDS.hxx.

Libraries for ROOT::Experimental::RNTupleDS:

The documentation for this class was generated from the following files: