The RDataSource implementation for RNTuple.
It lets RDataFrame read RNTuple data.
An RDataFrame that reads RNTuple data can be constructed using FromRNTuple().
For each column containing an array or a collection, a corresponding column #colname
is available to access colname.size()
without reading and deserializing the collection values.
Definition at line 74 of file RNTupleDS.hxx.
Classes | |
struct | REntryRangeDS |
The PrepareNextRanges() method populates the fNextRanges list with REntryRangeDS records. More... | |
struct | RFieldInfo |
Holds useful information about fields added to the RNTupleDS. More... | |
Public Member Functions | |
RNTupleDS (const RNTupleDS &)=delete | |
RNTupleDS (RNTupleDS &&)=delete | |
RNTupleDS (std::string_view ntupleName, const std::vector< std::string > &fileNames) | |
RNTupleDS (std::string_view ntupleName, std::string_view fileName) | |
~RNTupleDS () final | |
ROOT::RDF::RSampleInfo | CreateSampleInfo (unsigned int, const std::unordered_map< std::string, ROOT::RDF::Experimental::RSample * > &) const final |
void | Finalize () final |
Convenience method called after concluding an event-loop. | |
void | FinalizeSlot (unsigned int slot) final |
Convenience method called at the end of the data processing associated to a slot. | |
const std::vector< std::string > & | GetColumnNames () const final |
Returns a reference to the collection of the dataset's column names. | |
std::unique_ptr< ROOT::Detail::RDF::RColumnReaderBase > | GetColumnReaders (unsigned int, std::string_view, const std::type_info &) final |
If the other GetColumnReaders overload returns an empty vector, this overload will be called instead. | |
std::vector< std::pair< ULong64_t, ULong64_t > > | GetEntryRanges () final |
Return ranges of entries to distribute to tasks. | |
std::string | GetLabel () final |
Return a string representation of the datasource type. | |
std::size_t | GetNFiles () const final |
Returns the number of files from which the dataset is constructed. | |
std::string | GetTypeName (std::string_view colName) const final |
Type of a column as a string, e.g. | |
bool | HasColumn (std::string_view colName) const final |
Checks if the dataset has a certain column. | |
void | Initialize () final |
Convenience method called before starting an event-loop. | |
void | InitSlot (unsigned int slot, ULong64_t firstEntry) final |
Convenience method called at the start of the data processing associated to a slot. | |
RNTupleDS & | operator= (const RNTupleDS &)=delete |
RNTupleDS & | operator= (RNTupleDS &&)=delete |
bool | SetEntry (unsigned int, ULong64_t) final |
Advance the "cursors" returned by GetColumnReaders to the selected entry for a particular slot. | |
void | SetNSlots (unsigned int nSlots) final |
Inform RDataSource of the number of processing slots (i.e. | |
![]() | |
RDataSource ()=default | |
RDataSource (const RDataSource &)=delete | |
RDataSource (RDataSource &&)=delete | |
virtual | ~RDataSource ()=default |
template<typename T > | |
std::vector< T ** > | GetColumnReaders (std::string_view columnName) |
Called at most once per column by RDF. | |
RDataSource & | operator= (const RDataSource &)=delete |
RDataSource & | operator= (RDataSource &&)=delete |
virtual void | SetGlobalEntryRange (std::pair< ULong64_t, ULong64_t > entryRange) |
Restrict processing to a [begin, end) range of entries. | |
Protected Member Functions | |
Record_t | GetColumnReadersImpl (std::string_view name, const std::type_info &) final |
type-erased vector of pointers to pointers to column values - one per slot | |
![]() | |
virtual std::string | AsString () |
virtual std::unique_ptr< ROOT::Detail::RDF::RColumnReaderBase > | CreateColumnReader (unsigned int slot, std::string_view col, const std::type_info &tid, TTreeReader *) |
Creates a column reader for the requested column. | |
virtual std::string | DescribeDataset () |
virtual const std::vector< std::string > & | GetColumnNamesNoDuplicates () const |
virtual const std::vector< std::string > & | GetTopLevelFieldNames () const |
virtual std::string | GetTypeNameWithOpts (std::string_view colName, bool) const |
virtual void | InitializeWithOpts (const std::set< std::string > &) |
virtual void | ProcessMT (ROOT::Detail::RDF::RLoopManager &) |
virtual void | RunFinalChecks (bool) const |
Private Member Functions | |
RNTupleDS (std::string_view ntupleName, const std::vector< std::string > &fileNames, const std::pair< ULong64_t, ULong64_t > &range) | |
RNTupleDS (std::unique_ptr< ROOT::Internal::RPageSource > pageSource) | |
void | AddField (const ROOT::RNTupleDescriptor &desc, std::string_view colName, ROOT::DescriptorId_t fieldId, std::vector< RFieldInfo > fieldInfos, bool convertToRVec=true) |
Provides the RDF column "colName" given the field identified by fieldID. | |
void | ExecStaging () |
The main function of the fThreadStaging background thread. | |
ROOT::RFieldBase * | GetFieldWithTypeChecks (std::string_view fieldName, const std::type_info &tid) |
void | PrepareNextRanges () |
Populates fNextRanges with the next set of entry ranges. | |
void | StageNextSources () |
Starting from fNextFileIndex , opens the next fNSlots files. | |
Private Attributes | |
std::vector< std::vector< ROOT::Internal::RDF::RNTupleColumnReader * > > | fActiveColumnReaders |
List of column readers returned by GetColumnReaders() organized by slot. | |
std::unordered_map< std::size_t, std::vector< std::unique_ptr< ROOT::RFieldBase > > > | fAlternativeProtoFields |
Columns may be requested with types other than with which they were initially added as proto fields. | |
std::vector< std::string > | fColumnNames |
std::vector< std::string > | fColumnTypes |
std::vector< REntryRangeDS > | fCurrentRanges |
Basis for the ranges returned by the last GetEntryRanges() call. | |
std::condition_variable | fCvStaging |
Signal for the state information of fIsReadyForStaging and fHasNextSources. | |
std::unordered_map< ROOT::DescriptorId_t, std::string > | fFieldId2QualifiedName |
Connects the IDs of active proto fields and their subfields to their fully qualified name (a.b.c.d). | |
std::vector< std::string > | fFileNames |
std::unordered_map< ULong64_t, std::size_t > | fFirstEntry2RangeIdx |
Maps the first entries from the ranges of the last GetEntryRanges() call to their corresponding index in the fCurrentRanges vectors. | |
bool | fHasNextSources = false |
Is true when the staging thread has populated the next batch of files to fStagingArea. | |
bool | fIsReadyForStaging = false |
Is true when the staging thread should start working. | |
std::mutex | fMutexStaging |
Protects the shared state between the main thread and the I/O thread. | |
std::size_t | fNextFileIndex = 0 |
Index into fFileNames to the next file to process. | |
std::vector< REntryRangeDS > | fNextRanges |
Basis for the ranges populated by the PrepareNextRanges() call. | |
std::string | fNTupleName |
The data source may be constructed with an ntuple name and a list of files. | |
std::vector< std::pair< ULong64_t, ULong64_t > > | fOriginalRanges |
ROOT::RNTupleDescriptor | fPrincipalDescriptor |
A clone of the first pages source's descriptor. | |
std::vector< std::unique_ptr< ROOT::RFieldBase > > | fProtoFields |
We prepare a prototype field for every column. | |
ULong64_t | fSeenEntriesNoGlobalRange = 0 |
The number of entries seen so far in GetEntryRanges() | |
std::vector< std::size_t > | fSlotsToRangeIdxs |
One element per slot, corresponding to the current range index for that slot, as filled by InitSlot. | |
std::vector< std::unique_ptr< ROOT::Internal::RPageSource > > | fStagingArea |
The staging area is relevant for chains of files, i.e. | |
bool | fStagingThreadShouldTerminate = false |
Is true when the I/O thread should quit. | |
std::thread | fThreadStaging |
The background thread that runs StageNextSources() | |
Friends | |
ROOT::RDataFrame | ROOT::Internal::RDF::FromRNTuple (std::string_view ntupleName, const std::vector< std::string > &fileNames, const std::pair< ULong64_t, ULong64_t > &range) |
class | ROOT::Internal::RDF::RNTupleColumnReader |
Additional Inherited Members | |
![]() | |
using | Record_t = std::vector<void *> |
![]() | |
std::optional< std::pair< ULong64_t, ULong64_t > > | fGlobalEntryRange {} |
unsigned int | fNSlots {} |
#include <ROOT/RNTupleDS.hxx>
|
explicitprivate |
Definition at line 377 of file RNTupleDS.cxx.
|
explicitprivate |
Definition at line 427 of file RNTupleDS.cxx.
ROOT::RDF::RNTupleDS::RNTupleDS | ( | std::string_view | ntupleName, |
std::string_view | fileName ) |
Definition at line 413 of file RNTupleDS.cxx.
ROOT::RDF::RNTupleDS::RNTupleDS | ( | std::string_view | ntupleName, |
const std::vector< std::string > & | fileNames ) |
Definition at line 419 of file RNTupleDS.cxx.
|
delete |
|
finaldefault |
|
private |
Provides the RDF column "colName" given the field identified by fieldID.
For records and collections, AddField recurses into the sub fields. The fieldInfos argument is a list of objects holding info about the fields of the outer collection(s) (w.r.t. fieldId). For instance, if fieldId refers to an std::vector<Jet>
, with
AddField will recurse into Jet.pt
and Jet.eta
and provide the two inner fields as ROOT::VecOps::RVec<float>
each.
In case the field is a collection of type ROOT::VecOps::RVec
, std::vector
or std::array
, its corresponding column is added as a ROOT::VecOps::RVec
. Otherwise, the collection field's on-disk type is used. Note, however, that inner record members of such collections will still be added as ROOT::VecOps::RVec
(e.g., std::set<Jet> will be added as a
std::set, but
Jet.[pt|eta] will be added as `ROOT::VecOps::RVec<float>).
Definition at line 237 of file RNTupleDS.cxx.
|
finalvirtual |
Reimplemented from ROOT::RDF::RDataSource.
Definition at line 889 of file RNTupleDS.cxx.
|
private |
The main function of the fThreadStaging background thread.
Definition at line 507 of file RNTupleDS.cxx.
|
finalvirtual |
Convenience method called after concluding an event-loop.
See Initialize for more details.
Reimplemented from ROOT::RDF::RDataSource.
Definition at line 847 of file RNTupleDS.cxx.
|
finalvirtual |
Convenience method called at the end of the data processing associated to a slot.
[in] | slot | The data processing slot wihch needs to be finalized This method might be called multiple times per thread per event-loop. |
Reimplemented from ROOT::RDF::RDataSource.
Definition at line 797 of file RNTupleDS.cxx.
|
inlinefinalvirtual |
Returns a reference to the collection of the dataset's column names.
Implements ROOT::RDF::RDataSource.
Definition at line 224 of file RNTupleDS.hxx.
|
finalvirtual |
If the other GetColumnReaders overload returns an empty vector, this overload will be called instead.
[in] | slot | The data processing slot that needs to be considered |
[in] | name | The name of the column for which a column reader needs to be returned |
[in] | tid | A type_info At least one of the two must return a non-empty/non-null value. |
Reimplemented from ROOT::RDF::RDataSource.
Definition at line 489 of file RNTupleDS.cxx.
|
finalprotectedvirtual |
type-erased vector of pointers to pointers to column values - one per slot
Implements ROOT::RDF::RDataSource.
Definition at line 435 of file RNTupleDS.cxx.
|
finalvirtual |
Return ranges of entries to distribute to tasks.
They are required to be contiguous intervals with no entries skipped. Supposing a dataset with nEntries, the intervals must start at 0 and end at nEntries, e.g. [0-5],[5-10] for 10 entries. This function will be invoked repeatedly by RDataFrame as it needs additional entries to process. The same entry range should not be returned more than once. Returning an empty collection of ranges signals to RDataFrame that the processing can stop.
Implements ROOT::RDF::RDataSource.
Definition at line 640 of file RNTupleDS.cxx.
|
private |
Definition at line 441 of file RNTupleDS.cxx.
|
inlinefinalvirtual |
Return a string representation of the datasource type.
The returned string will be used by ROOT::RDF::SaveGraph() to represent the datasource in the visualization of the computation graph. Concrete datasources can override the default implementation.
Reimplemented from ROOT::RDF::RDataSource.
Definition at line 228 of file RNTupleDS.hxx.
|
inlinefinalvirtual |
Returns the number of files from which the dataset is constructed.
Reimplemented from ROOT::RDF::RDataSource.
Definition at line 223 of file RNTupleDS.hxx.
|
finalvirtual |
Type of a column as a string, e.g.
GetTypeName("x") == "double"
. Required for jitting e.g. df.Filter("x>0")
.
[in] | colName | The name of the column |
Implements ROOT::RDF::RDataSource.
Definition at line 807 of file RNTupleDS.cxx.
|
finalvirtual |
Checks if the dataset has a certain column.
[in] | colName | The name of the column |
Implements ROOT::RDF::RDataSource.
Definition at line 820 of file RNTupleDS.cxx.
|
finalvirtual |
Convenience method called before starting an event-loop.
This method might be called multiple times over the lifetime of a RDataSource, since users can run multiple event-loops with the same RDataFrame. Ideally, Initialize
should set the state of the RDataSource so that multiple identical event-loops will produce identical results.
Reimplemented from ROOT::RDF::RDataSource.
Definition at line 825 of file RNTupleDS.cxx.
Convenience method called at the start of the data processing associated to a slot.
[in] | slot | The data processing slot wihch needs to be initialized |
[in] | firstEntry | The first entry of the range that the task will process. This method might be called multiple times per thread per event-loop. |
Reimplemented from ROOT::RDF::RDataSource.
Definition at line 774 of file RNTupleDS.cxx.
|
private |
Populates fNextRanges with the next set of entry ranges.
Moves files from the staging area as necessary and aligns ranges with cluster boundaries for scheduling the tail of files. Upon return, the fNextRanges list is ordered. It has usually fNSlots elements; fewer if there is not enough work to give at least one cluster to every slot.
Definition at line 544 of file RNTupleDS.cxx.
Advance the "cursors" returned by GetColumnReaders to the selected entry for a particular slot.
[in] | slot | The data processing slot that needs to be considered |
[in] | entry | The entry which needs to be pointed to by the reader pointers Slots are adopted to accommodate parallel data processing. Different workers will loop over different ranges and will be labelled by different "slot" values. Returns true if the entry has to be processed, false otherwise. |
Implements ROOT::RDF::RDataSource.
Definition at line 243 of file RNTupleDS.hxx.
|
finalvirtual |
Inform RDataSource of the number of processing slots (i.e.
worker threads) used by the associated RDataFrame. Slots numbers are used to simplify parallel execution: RDataFrame guarantees that different threads will always pass different slot values when calling methods concurrently.
Reimplemented from ROOT::RDF::RDataSource.
Definition at line 870 of file RNTupleDS.cxx.
|
private |
Starting from fNextFileIndex
, opens the next fNSlots
files.
Calls LoadStructure()
on the opened files. The very first file is already available from the constructor.
Definition at line 525 of file RNTupleDS.cxx.
|
friend |
|
friend |
Definition at line 75 of file RNTupleDS.hxx.
|
private |
List of column readers returned by GetColumnReaders() organized by slot.
Used to reconnect readers to new page sources when the files in the chain change.
Definition at line 131 of file RNTupleDS.hxx.
|
private |
Columns may be requested with types other than with which they were initially added as proto fields.
For example, a column with a ROOT::RVec<float>
proto field may instead be requested as a std::vector<float>
. In case this happens, we create an alternative proto field and store it here, with the original index in fProtoFields
as key. A single column can have more than one alternative proto fields.
Definition at line 122 of file RNTupleDS.hxx.
|
private |
Definition at line 127 of file RNTupleDS.hxx.
|
private |
Definition at line 128 of file RNTupleDS.hxx.
|
private |
Basis for the ranges returned by the last GetEntryRanges() call.
Definition at line 135 of file RNTupleDS.hxx.
|
private |
Signal for the state information of fIsReadyForStaging and fHasNextSources.
Definition at line 151 of file RNTupleDS.hxx.
|
private |
Connects the IDs of active proto fields and their subfields to their fully qualified name (a.b.c.d).
This enables the column reader to rewire the field IDs when the file changes (chain), using the fully qualified name as a search key in the descriptor of the other page sources.
Definition at line 126 of file RNTupleDS.hxx.
|
private |
Definition at line 93 of file RNTupleDS.hxx.
|
private |
Maps the first entries from the ranges of the last GetEntryRanges() call to their corresponding index in the fCurrentRanges vectors.
This is necessary because the returned ranges get distributed arbitrarily onto slots. In the InitSlot method, the column readers use this map to find the correct range to connect to.
Definition at line 140 of file RNTupleDS.hxx.
Is true when the staging thread has populated the next batch of files to fStagingArea.
Definition at line 155 of file RNTupleDS.hxx.
Is true when the staging thread should start working.
Definition at line 153 of file RNTupleDS.hxx.
|
private |
Protects the shared state between the main thread and the I/O thread.
Definition at line 149 of file RNTupleDS.hxx.
|
private |
Index into fFileNames to the next file to process.
Definition at line 111 of file RNTupleDS.hxx.
|
private |
Basis for the ranges populated by the PrepareNextRanges() call.
Definition at line 136 of file RNTupleDS.hxx.
|
private |
The data source may be constructed with an ntuple name and a list of files.
Definition at line 92 of file RNTupleDS.hxx.
Definition at line 142 of file RNTupleDS.hxx.
|
private |
A clone of the first pages source's descriptor.
Definition at line 89 of file RNTupleDS.hxx.
|
private |
We prepare a prototype field for every column.
If a column reader is actually requested in GetColumnReaders(), we move a clone of the field into a new column reader for RDataFrame. Only the clone connects to the backing page store and acquires I/O resources. The field IDs are set in the context of the first source and used as keys in fFieldId2QualifiedName.
Definition at line 117 of file RNTupleDS.hxx.
|
private |
The number of entries seen so far in GetEntryRanges()
Definition at line 133 of file RNTupleDS.hxx.
|
private |
One element per slot, corresponding to the current range index for that slot, as filled by InitSlot.
Definition at line 144 of file RNTupleDS.hxx.
|
private |
The staging area is relevant for chains of files, i.e.
when fFileNames is not empty. In this case, files are opened in the background in batches of size fNSlots
and kept in the staging area. The first file (chains or no chains) is always opened on construction in order to process the schema. For all subsequent files, the corresponding page sources in the staging area only executed LoadStructure()
, i.e. they should have a compressed buffer of the meta-data available. Concretely:
Initialize()
, we start the I/O background thread, which in turn opens the first batch of files.GetEntryRanges()
, we a) wait for the I/O thread to finish, b) call PrepareNextRanges()
in the main thread to move the page sources from the staging area into fNextRanges
; this will also call Attach()
on the page sources (i.e., deserialize the meta-data), and c) trigger staging of the next batch of files in the I/O background thread.Finalize()
, the I/O background thread is stopped. Definition at line 110 of file RNTupleDS.hxx.
Is true when the I/O thread should quit.
Definition at line 157 of file RNTupleDS.hxx.
|
private |
The background thread that runs StageNextSources()
Definition at line 147 of file RNTupleDS.hxx.