The RDataSource implementation for RNTuple.

An RDataFrame that reads RNTuple data can be constructed using FromRNTuple().

For each column containing an array or a collection, a corresponding column #colname is available to access colname.size() without reading and deserializing the collection values.

Definition at line 43 of file RNTupleDS.hxx.

Classes
struct	REntryRangeDS
	The PrepareNextRanges() method populates the fNextRanges list with REntryRangeDS records. More...

struct	RFieldInfo
	Holds useful information about fields added to the RNTupleDS. More...

Public Member Functions
	RNTupleDS (ROOT::Experimental::RNTuple *ntuple)

	RNTupleDS (std::string_view ntupleName, const std::vector< std::string > &fileNames)

	RNTupleDS (std::string_view ntupleName, std::string_view fileName)

	~RNTupleDS ()

void	Finalize () final
	Convenience method called after concluding an event-loop.

void	FinalizeSlot (unsigned int slot) final
	Convenience method called at the end of the data processing associated to a slot.

const std::vector< std::string > &	GetColumnNames () const final
	Returns a reference to the collection of the dataset's column names.

std::unique_ptr< ROOT::Detail::RDF::RColumnReaderBase >	GetColumnReaders (unsigned int, std::string_view, const std::type_info &) final
	If the other GetColumnReaders overload returns an empty vector, this overload will be called instead.

std::vector< std::pair< ULong64_t, ULong64_t > >	GetEntryRanges () final
	Return ranges of entries to distribute to tasks.

std::string	GetLabel () final
	Return a string representation of the datasource type.

std::string	GetTypeName (std::string_view colName) const final
	Type of a column as a string, e.g.

bool	HasColumn (std::string_view colName) const final
	Checks if the dataset has a certain column.

void	Initialize () final
	Convenience method called before starting an event-loop.

void	InitSlot (unsigned int slot, ULong64_t firstEntry) final
	Convenience method called at the start of the data processing associated to a slot.

bool	SetEntry (unsigned int slot, ULong64_t entry) final
	Advance the "cursors" returned by GetColumnReaders to the selected entry for a particular slot.

void	SetNSlots (unsigned int nSlots) final
	Inform RDataSource of the number of processing slots (i.e.

Public Member Functions inherited from ROOT::RDF::RDataSource
virtual	~RDataSource ()=default

template<typename T >
std::vector< T ** >	GetColumnReaders (std::string_view columnName)
	Called at most once per column by RDF.

Protected Member Functions
Record_t	GetColumnReadersImpl (std::string_view name, const std::type_info &) final
	type-erased vector of pointers to pointers to column values - one per slot

Protected Member Functions inherited from ROOT::RDF::RDataSource
virtual std::string	AsString ()

Private Member Functions
	RNTupleDS (std::unique_ptr< ROOT::Experimental::Internal::RPageSource > pageSource)

void	AddField (const RNTupleDescriptor &desc, std::string_view colName, DescriptorId_t fieldId, std::vector< RFieldInfo > fieldInfos)
	Provides the RDF column "colName" given the field identified by fieldID.

void	PrepareNextRanges ()
	Populates fNextRanges with the next set of entry ranges.

Private Attributes
std::vector< std::vector< Internal::RNTupleColumnReader * > >	fActiveColumnReaders
	List of column readers returned by GetColumnReaders() organized by slot.

std::vector< std::string >	fColumnNames

std::vector< std::string >	fColumnTypes

std::vector< REntryRangeDS >	fCurrentRanges
	Basis for the ranges returned by the last GetEntryRanges() call.

std::unordered_map< ROOT::Experimental::DescriptorId_t, std::string >	fFieldId2QualifiedName
	Connects the IDs of active proto fields and their subfields to their fully qualified name (a.b.c.d).

std::vector< std::string >	fFileNames

std::unordered_map< ULong64_t, std::size_t >	fFirstEntry2RangeIdx
	Maps the first entries from the ranges of the last GetEntryRanges() call to their corresponding index in the fCurrentRanges vectors.

std::size_t	fNextFileIndex = 0
	Index into fFileNames to the next file to process.

std::vector< REntryRangeDS >	fNextRanges
	Basis for the ranges populated by the PrepareNextRanges() call.

unsigned int	fNSlots = 0

std::string	fNTupleName
	The data source may be constructed with an ntuple name and a list of files.

std::unique_ptr< RNTupleDescriptor >	fPrincipalDescriptor
	A clone of the first pages source's descriptor.

std::unique_ptr< Internal::RPageSource >	fPrincipalSource
	The first source is used to extract the schema and build the prototype fields.

std::vector< std::unique_ptr< ROOT::Experimental::RFieldBase > >	fProtoFields
	We prepare a prototype field for every column.

ULong64_t	fSeenEntries = 0
	The number of entries so far returned by GetEntryRanges()

Friends
class	Internal::RNTupleColumnReader

Additional Inherited Members
Protected Types inherited from ROOT::RDF::RDataSource
using	Record_t = std::vector< void * >

#include <ROOT/RNTupleDS.hxx>

Inheritance diagram for ROOT::Experimental::RNTupleDS:

[legend]

Constructor & Destructor Documentation

◆ RNTupleDS() [1/4]

ROOT::Experimental::RNTupleDS::RNTupleDS ( std::unique_ptr< ROOT::Experimental::Internal::RPageSource > pageSource )

explicitprivate

Definition at line 375 of file RNTupleDS.cxx.

◆ RNTupleDS() [2/4]

ROOT::Experimental::RNTupleDS::RNTupleDS	(	std::string_view	ntupleName,
		std::string_view	fileName
	)

Definition at line 384 of file RNTupleDS.cxx.

◆ RNTupleDS() [3/4]

ROOT::Experimental::RNTupleDS::RNTupleDS ( ROOT::Experimental::RNTuple * ntuple )

Definition at line 389 of file RNTupleDS.cxx.

◆ RNTupleDS() [4/4]

ROOT::Experimental::RNTupleDS::RNTupleDS	(	std::string_view	ntupleName,
		const std::vector< std::string > &	fileNames
	)

Definition at line 394 of file RNTupleDS.cxx.

◆ ~RNTupleDS()

ROOT::Experimental::RNTupleDS::~RNTupleDS ( )

default

Member Function Documentation

◆ AddField()

void ROOT::Experimental::RNTupleDS::AddField	(	const RNTupleDescriptor &	desc,
		std::string_view	colName,
		DescriptorId_t	fieldId,
		std::vector< RFieldInfo >	fieldInfos
	)

private

Provides the RDF column "colName" given the field identified by fieldID.

For records and collections, AddField recurses into the sub fields. The fieldInfos argument is a list of objects holding info about the fields of the outer collection(s) (w.r.t. fieldId). For instance, if fieldId refers to an std::vector<Jet>, with struct Jet { float pt; float eta; }; AddField will recurse into Jet.pt and Jet.eta and provide the two inner fields as std::vector<float> each.

Definition at line 231 of file RNTupleDS.cxx.

◆ Finalize()

void ROOT::Experimental::RNTupleDS::Finalize ( )

finalvirtual

Convenience method called after concluding an event-loop.

See Initialize for more details.

Reimplemented from ROOT::RDF::RDataSource.

Definition at line 633 of file RNTupleDS.cxx.

◆ FinalizeSlot()

void ROOT::Experimental::RNTupleDS::FinalizeSlot ( unsigned int )

finalvirtual

Convenience method called at the end of the data processing associated to a slot.

Parameters

[in] slot The data processing slot wihch needs to be finalized This method might be called multiple times per thread per event-loop.

Reimplemented from ROOT::RDF::RDataSource.

Definition at line 599 of file RNTupleDS.cxx.

◆ GetColumnNames()

const std::vector< std::string > & ROOT::Experimental::RNTupleDS::GetColumnNames ( ) const

inlinefinalvirtual

Returns a reference to the collection of the dataset's column names.

Implements ROOT::RDF::RDataSource.

Definition at line 127 of file RNTupleDS.hxx.

◆ GetColumnReaders()

std::unique_ptr< ROOT::Detail::RDF::RColumnReaderBase > ROOT::Experimental::RNTupleDS::GetColumnReaders	(	unsigned int	,
		std::string_view	,
		const std::type_info &
	)

finalvirtual

If the other GetColumnReaders overload returns an empty vector, this overload will be called instead.

Parameters

[in]	slot	The data processing slot that needs to be considered
[in]	name	The name of the column for which a column reader needs to be returned
[in]	tid	A type_info At least one of the two must return a non-empty/non-null value.

Reimplemented from ROOT::RDF::RDataSource.

Definition at line 408 of file RNTupleDS.cxx.

◆ GetColumnReadersImpl()

RDF::RDataSource::Record_t ROOT::Experimental::RNTupleDS::GetColumnReadersImpl	(	std::string_view	name,
		const std::type_info &
	)

finalprotectedvirtual

type-erased vector of pointers to pointers to column values - one per slot

Implements ROOT::RDF::RDataSource.

Definition at line 401 of file RNTupleDS.cxx.

◆ GetEntryRanges()

std::vector< std::pair< ULong64_t, ULong64_t > > ROOT::Experimental::RNTupleDS::GetEntryRanges ( )

finalvirtual

Return ranges of entries to distribute to tasks.

They are required to be contiguous intervals with no entries skipped. Supposing a dataset with nEntries, the intervals must start at 0 and end at nEntries, e.g. [0-5],[5-10] for 10 entries. This function will be invoked repeatedly by RDataFrame as it needs additional entries to process. The same entry range should not be returned more than once. Returning an empty collection of ranges signals to RDataFrame that the processing can stop.

Implements ROOT::RDF::RDataSource.

Definition at line 532 of file RNTupleDS.cxx.

◆ GetLabel()

std::string ROOT::Experimental::RNTupleDS::GetLabel ( )

inlinefinalvirtual

Return a string representation of the datasource type.

The returned string will be used by ROOT::RDF::SaveGraph() to represent the datasource in the visualization of the computation graph. Concrete datasources can override the default implementation.

Reimplemented from ROOT::RDF::RDataSource.

Definition at line 131 of file RNTupleDS.hxx.

◆ GetTypeName()

std::string ROOT::Experimental::RNTupleDS::GetTypeName ( std::string_view colName ) const

finalvirtual

Type of a column as a string, e.g.

GetTypeName("x") == "double". Required for jitting e.g. df.Filter("x>0").

Parameters

[in] colName The name of the column

Implements ROOT::RDF::RDataSource.

Definition at line 609 of file RNTupleDS.cxx.

◆ HasColumn()

bool ROOT::Experimental::RNTupleDS::HasColumn ( std::string_view colName ) const

finalvirtual

Checks if the dataset has a certain column.

Parameters

[in] colName The name of the column

Implements ROOT::RDF::RDataSource.

Definition at line 615 of file RNTupleDS.cxx.

◆ Initialize()

void ROOT::Experimental::RNTupleDS::Initialize ( )

finalvirtual

Convenience method called before starting an event-loop.

This method might be called multiple times over the lifetime of a RDataSource, since users can run multiple event-loops with the same RDataFrame. Ideally, Initialize should set the state of the RDataSource so that multiple identical event-loops will produce identical results.

Reimplemented from ROOT::RDF::RDataSource.

Definition at line 620 of file RNTupleDS.cxx.

◆ InitSlot()

void ROOT::Experimental::RNTupleDS::InitSlot	(	unsigned int	,
		ULong64_t
	)

finalvirtual

Convenience method called at the start of the data processing associated to a slot.

Parameters

[in]	slot	The data processing slot wihch needs to be initialized
[in]	firstEntry	The first entry of the range that the task will process. This method might be called multiple times per thread per event-loop.

Reimplemented from ROOT::RDF::RDataSource.

Definition at line 588 of file RNTupleDS.cxx.

◆ PrepareNextRanges()

void ROOT::Experimental::RNTupleDS::PrepareNextRanges ( )

private

Populates fNextRanges with the next set of entry ranges.

Opens files from the chain as necessary and aligns ranges with cluster boundaries for scheduling the tail of files. Upon return, the fNextRanges list is ordered. It has usually fNSlots elements; fewer if there is not enough work to give at least one cluster to every slot.

Definition at line 434 of file RNTupleDS.cxx.

◆ SetEntry()

bool ROOT::Experimental::RNTupleDS::SetEntry	(	unsigned int	slot,
		ULong64_t	entry
	)

finalvirtual

Advance the "cursors" returned by GetColumnReaders to the selected entry for a particular slot.

Parameters

[in]	slot	The data processing slot that needs to be considered
[in]	entry	The entry which needs to be pointed to by the reader pointers Slots are adopted to accommodate parallel data processing. Different workers will loop over different ranges and will be labelled by different "slot" values. Returns true if the entry has to be processed, false otherwise.

Implements ROOT::RDF::RDataSource.

Definition at line 428 of file RNTupleDS.cxx.

◆ SetNSlots()

void ROOT::Experimental::RNTupleDS::SetNSlots ( unsigned int nSlots )

finalvirtual

Inform RDataSource of the number of processing slots (i.e.

worker threads) used by the associated RDataFrame. Slots numbers are used to simplify parallel execution: RDataFrame guarantees that different threads will always pass different slot values when calling methods concurrently.

Implements ROOT::RDF::RDataSource.

Definition at line 642 of file RNTupleDS.cxx.

Friends And Related Symbol Documentation

◆ Internal::RNTupleColumnReader

friend class Internal::RNTupleColumnReader

friend

Definition at line 44 of file RNTupleDS.hxx.

Member Data Documentation

◆ fActiveColumnReaders

std::vector<std::vector<Internal::RNTupleColumnReader *> > ROOT::Experimental::RNTupleDS::fActiveColumnReaders

private

List of column readers returned by GetColumnReaders() organized by slot.

Used to reconnect readers to new page sources when the files in the chain change.

Definition at line 81 of file RNTupleDS.hxx.

◆ fColumnNames

std::vector<std::string> ROOT::Experimental::RNTupleDS::fColumnNames

private

Definition at line 77 of file RNTupleDS.hxx.

◆ fColumnTypes

std::vector<std::string> ROOT::Experimental::RNTupleDS::fColumnTypes

private

Definition at line 78 of file RNTupleDS.hxx.

◆ fCurrentRanges

std::vector<REntryRangeDS> ROOT::Experimental::RNTupleDS::fCurrentRanges

private

Basis for the ranges returned by the last GetEntryRanges() call.

Definition at line 85 of file RNTupleDS.hxx.

◆ fFieldId2QualifiedName

std::unordered_map<ROOT::Experimental::DescriptorId_t, std::string> ROOT::Experimental::RNTupleDS::fFieldId2QualifiedName

private

Connects the IDs of active proto fields and their subfields to their fully qualified name (a.b.c.d).

This enables the column reader to rewire the field IDs when the file changes (chain), using the fully qualified name as a search key in the descriptor of the other page sources.

Definition at line 76 of file RNTupleDS.hxx.

◆ fFileNames

std::vector<std::string> ROOT::Experimental::RNTupleDS::fFileNames

private

Definition at line 65 of file RNTupleDS.hxx.

◆ fFirstEntry2RangeIdx

std::unordered_map<ULong64_t, std::size_t> ROOT::Experimental::RNTupleDS::fFirstEntry2RangeIdx

private

Maps the first entries from the ranges of the last GetEntryRanges() call to their corresponding index in the fCurrentRanges vectors.

This is necessary because the returned ranges get distributed arbitrarily onto slots. In the InitSlot method, the column readers use this map to find the correct range to connect to.

Definition at line 90 of file RNTupleDS.hxx.

◆ fNextFileIndex

std::size_t ROOT::Experimental::RNTupleDS::fNextFileIndex = 0

private

Index into fFileNames to the next file to process.

Definition at line 66 of file RNTupleDS.hxx.

◆ fNextRanges

std::vector<REntryRangeDS> ROOT::Experimental::RNTupleDS::fNextRanges

private

Basis for the ranges populated by the PrepareNextRanges() call.

Definition at line 86 of file RNTupleDS.hxx.

◆ fNSlots

unsigned int ROOT::Experimental::RNTupleDS::fNSlots = 0

private

Definition at line 83 of file RNTupleDS.hxx.

◆ fNTupleName

std::string ROOT::Experimental::RNTupleDS::fNTupleName

private

The data source may be constructed with an ntuple name and a list of files.

Definition at line 64 of file RNTupleDS.hxx.

◆ fPrincipalDescriptor

std::unique_ptr<RNTupleDescriptor> ROOT::Experimental::RNTupleDS::fPrincipalDescriptor

private

A clone of the first pages source's descriptor.

Definition at line 61 of file RNTupleDS.hxx.

◆ fPrincipalSource

std::unique_ptr<Internal::RPageSource> ROOT::Experimental::RNTupleDS::fPrincipalSource

private

The first source is used to extract the schema and build the prototype fields.

The page source is used to extract a clone of the descriptor to fPrincipalDescriptor. Afterwards it is moved into the first REntryRangeDS.

Definition at line 59 of file RNTupleDS.hxx.

◆ fProtoFields

std::vector<std::unique_ptr<ROOT::Experimental::RFieldBase> > ROOT::Experimental::RNTupleDS::fProtoFields

private

We prepare a prototype field for every column.

If a column reader is actually requested in GetColumnReaders(), we move a clone of the field into a new column reader for RDataFrame. Only the clone connects to the backing page store and acquires I/O resources. The field IDs are set in the context of the first source and used as keys in fFieldId2QualifiedName.

Definition at line 72 of file RNTupleDS.hxx.

◆ fSeenEntries

ULong64_t ROOT::Experimental::RNTupleDS::fSeenEntries = 0

private

The number of entries so far returned by GetEntryRanges()

Definition at line 84 of file RNTupleDS.hxx.

Libraries for ROOT::Experimental::RNTupleDS:

[legend]

The documentation for this class was generated from the following files:

tree/dataframe/inc/ROOT/RNTupleDS.hxx
tree/dataframe/src/RNTupleDS.cxx

Classes

Public Member Functions

Protected Member Functions

Private Member Functions

Private Attributes

Friends

Additional Inherited Members

Constructor & Destructor Documentation

◆ RNTupleDS() [1/4]

◆ RNTupleDS() [2/4]

◆ RNTupleDS() [3/4]

◆ RNTupleDS() [4/4]

◆ ~RNTupleDS()

Member Function Documentation

◆ AddField()

◆ Finalize()

◆ FinalizeSlot()

◆ GetColumnNames()

◆ GetColumnReaders()

◆ GetColumnReadersImpl()

◆ GetEntryRanges()

◆ GetLabel()

◆ GetTypeName()

◆ HasColumn()

◆ Initialize()

◆ InitSlot()

◆ PrepareNextRanges()

◆ SetEntry()

◆ SetNSlots()

Friends And Related Symbol Documentation

◆ Internal::RNTupleColumnReader

Member Data Documentation

◆ fActiveColumnReaders

◆ fColumnNames

◆ fColumnTypes

◆ fCurrentRanges

◆ fFieldId2QualifiedName

◆ fFileNames

◆ fFirstEntry2RangeIdx

◆ fNextFileIndex

◆ fNextRanges

◆ fNSlots

◆ fNTupleName

◆ fPrincipalDescriptor

◆ fPrincipalSource

◆ fProtoFields

◆ fSeenEntries