Logo ROOT  
Reference Guide
ROOT::RDF::RDataSource Class Referenceabstract

RDataSource defines an API that RDataFrame can use to read arbitrary data formats.

A concrete RDataSource implementation (i.e. a class that inherits from RDataSource and implements all of its pure methods) provides an adaptor that RDataFrame can leverage to read any kind of tabular data formats. RDataFrame calls into RDataSource to retrieve information about the data, retrieve (thread-local) readers or "cursors" for selected columns and to advance the readers to the desired data entry.

The sequence of calls that RDataFrame (or any other client of a RDataSource) performs is the following:

RDataSource implementations must support running multiple event-loops consecutively (although sequentially) on the same dataset.

  • SetNSlots() is called once per RDataSource object, typically when it is associated to a RDataFrame.
  • GetColumnReaders() can be called several times, potentially with the same arguments, also in-between event-loops, but not during an event-loop.
  • GetEntryRanges() will be called several times, including during an event loop, as additional ranges are needed. It will not be called concurrently.
  • Initialise() and Finalise() are called once per event-loop, right before starting and right after finishing.
  • InitSlot(), SetEntry(), and FinaliseSlot() can be called concurrently from multiple threads, multiple times per event-loop.

Definition at line 105 of file RDataSource.hxx.

Public Member Functions

virtual ~RDataSource ()=default
 
virtual void Finalise ()
 Convenience method called after concluding an event-loop. More...
 
virtual void FinaliseSlot (unsigned int)
 Convenience method called at the end of the data processing associated to a slot. More...
 
virtual const std::vector< std::string > & GetColumnNames () const =0
 Returns a reference to the collection of the dataset's column names. More...
 
template<typename T >
std::vector< T ** > GetColumnReaders (std::string_view columnName)
 Called at most once per column by RDF. More...
 
virtual std::vector< std::pair< ULong64_t, ULong64_t > > GetEntryRanges ()=0
 Return ranges of entries to distribute to tasks. More...
 
virtual std::string GetLabel ()
 Return a string representation of the datasource type. More...
 
virtual std::string GetTypeName (std::string_view) const =0
 Type of a column as a string, e.g. More...
 
virtual bool HasColumn (std::string_view) const =0
 Checks if the dataset has a certain column. More...
 
virtual void Initialise ()
 Convenience method called before starting an event-loop. More...
 
virtual void InitSlot (unsigned int, ULong64_t)
 Convenience method called at the start of the data processing associated to a slot. More...
 
virtual bool SetEntry (unsigned int slot, ULong64_t entry)=0
 Advance the "cursors" returned by GetColumnReaders to the selected entry for a particular slot. More...
 
virtual void SetNSlots (unsigned int nSlots)=0
 Inform RDataSource of the number of processing slots (i.e. More...
 

Protected Types

using Record_t = std::vector< void * >
 

Protected Member Functions

virtual std::string AsString ()
 
virtual Record_t GetColumnReadersImpl (std::string_view name, const std::type_info &)=0
 type-erased vector of pointers to pointers to column values - one per slot More...
 

#include <ROOT/RDataSource.hxx>

Inheritance diagram for ROOT::RDF::RDataSource:
[legend]

Member Typedef Documentation

◆ Record_t

using ROOT::RDF::RDataSource::Record_t = std::vector<void *>
protected

Definition at line 108 of file RDataSource.hxx.

Constructor & Destructor Documentation

◆ ~RDataSource()

virtual ROOT::RDF::RDataSource::~RDataSource ( )
virtualdefault

Member Function Documentation

◆ AsString()

virtual std::string ROOT::RDF::RDataSource::AsString ( )
inlineprotectedvirtual

◆ Finalise()

virtual void ROOT::RDF::RDataSource::Finalise ( )
inlinevirtual

Convenience method called after concluding an event-loop.

See Initialise for more details.

Reimplemented in ROOT::RDF::RCsvDS.

Definition at line 205 of file RDataSource.hxx.

◆ FinaliseSlot()

virtual void ROOT::RDF::RDataSource::FinaliseSlot ( unsigned int  )
inlinevirtual

Convenience method called at the end of the data processing associated to a slot.

Parameters
[in]slotThe data processing slot wihch needs to be finalised This method might be called multiple times per thread per event-loop.

Definition at line 199 of file RDataSource.hxx.

◆ GetColumnNames()

virtual const std::vector< std::string > & ROOT::RDF::RDataSource::GetColumnNames ( ) const
pure virtual

◆ GetColumnReaders()

template<typename T >
std::vector< T ** > ROOT::RDF::RDataSource::GetColumnReaders ( std::string_view  columnName)
inline

Called at most once per column by RDF.

Return vector of pointers to pointers to column values - one per slot.

Template Parameters
TThe type of the data stored in the column
Parameters
[in]columnNameThe name of the column

These pointers are veritable cursors: it's a responsibility of the RDataSource implementation that they point to the "right" memory region.

Definition at line 147 of file RDataSource.hxx.

◆ GetColumnReadersImpl()

virtual Record_t ROOT::RDF::RDataSource::GetColumnReadersImpl ( std::string_view  name,
const std::type_info &   
)
protectedpure virtual

◆ GetEntryRanges()

virtual std::vector< std::pair< ULong64_t, ULong64_t > > ROOT::RDF::RDataSource::GetEntryRanges ( )
pure virtual

Return ranges of entries to distribute to tasks.

They are required to be contiguous intervals with no entries skipped. Supposing a dataset with nEntries, the intervals must start at 0 and end at nEntries, e.g. [0-5],[5-10] for 10 entries. This function will be invoked repeatedly by RDataFrame as it needs additional entries to process. The same entry range should not be returned more than once. Returning an empty collection of ranges signals to RDataFrame that the processing can stop.

Implemented in ROOT::Internal::RDF::RNumpyDS< ColumnTypes >, ROOT::RDF::RCsvDS, ROOT::RDF::RLazyDS< ColumnTypes >, ROOT::RDF::RTrivialDS, ROOT::Experimental::RNTupleDS, ROOT::RDF::RSqliteDS, and ROOT::RDF::RArrowDS.

◆ GetLabel()

virtual std::string ROOT::RDF::RDataSource::GetLabel ( )
inlinevirtual

Return a string representation of the datasource type.

The returned string will be used by ROOT::RDF::SaveGraph() to represent the datasource in the visualization of the computation graph. Concrete datasources can override the default implementation.

Reimplemented in ROOT::Internal::RDF::RNumpyDS< ColumnTypes >, ROOT::RDF::RCsvDS, ROOT::RDF::RLazyDS< ColumnTypes >, ROOT::RDF::RTrivialDS, ROOT::RDF::RSqliteDS, and ROOT::RDF::RArrowDS.

Definition at line 211 of file RDataSource.hxx.

◆ GetTypeName()

virtual std::string ROOT::RDF::RDataSource::GetTypeName ( std::string_view  ) const
pure virtual

Type of a column as a string, e.g.

GetTypeName("x") == "double". Required for jitting e.g. df.Filter("x>0").

Parameters
[in]columnNameThe name of the column

Implemented in ROOT::Internal::RDF::RNumpyDS< ColumnTypes >, ROOT::RDF::RCsvDS, ROOT::RDF::RLazyDS< ColumnTypes >, ROOT::Experimental::RNTupleDS, ROOT::RDF::RSqliteDS, ROOT::RDF::RArrowDS, and ROOT::RDF::RTrivialDS.

◆ HasColumn()

virtual bool ROOT::RDF::RDataSource::HasColumn ( std::string_view  ) const
pure virtual

Checks if the dataset has a certain column.

Parameters
[in]columnNameThe name of the column

Implemented in ROOT::Internal::RDF::RNumpyDS< ColumnTypes >, ROOT::RDF::RCsvDS, ROOT::RDF::RLazyDS< ColumnTypes >, ROOT::RDF::RTrivialDS, ROOT::Experimental::RNTupleDS, ROOT::RDF::RSqliteDS, and ROOT::RDF::RArrowDS.

◆ Initialise()

virtual void ROOT::RDF::RDataSource::Initialise ( )
inlinevirtual

Convenience method called before starting an event-loop.

This method might be called multiple times over the lifetime of a RDataSource, since users can run multiple event-loops with the same RDataFrame. Ideally, Initialise should set the state of the RDataSource so that multiple identical event-loops will produce identical results.

Reimplemented in ROOT::Internal::RDF::RNumpyDS< ColumnTypes >, ROOT::RDF::RLazyDS< ColumnTypes >, ROOT::RDF::RTrivialDS, ROOT::Experimental::RNTupleDS, ROOT::RDF::RSqliteDS, and ROOT::RDF::RArrowDS.

Definition at line 184 of file RDataSource.hxx.

◆ InitSlot()

virtual void ROOT::RDF::RDataSource::InitSlot ( unsigned int  ,
ULong64_t   
)
inlinevirtual

Convenience method called at the start of the data processing associated to a slot.

Parameters
[in]slotThe data processing slot wihch needs to be initialised
[in]firstEntryThe first entry of the range that the task will process. This method might be called multiple times per thread per event-loop.

Reimplemented in ROOT::RDF::RArrowDS.

Definition at line 192 of file RDataSource.hxx.

◆ SetEntry()

virtual bool ROOT::RDF::RDataSource::SetEntry ( unsigned int  slot,
ULong64_t  entry 
)
pure virtual

Advance the "cursors" returned by GetColumnReaders to the selected entry for a particular slot.

Parameters
[in]slotThe data processing slot that needs to be considered
[in]entryThe entry which needs to be pointed to by the reader pointers Slots are adopted to accommodate parallel data processing. Different workers will loop over different ranges and will be labelled by different "slot" values. Returns true if the entry has to be processed, false otherwise.

Implemented in ROOT::Internal::RDF::RNumpyDS< ColumnTypes >, ROOT::RDF::RCsvDS, ROOT::RDF::RLazyDS< ColumnTypes >, ROOT::RDF::RTrivialDS, ROOT::Experimental::RNTupleDS, ROOT::RDF::RSqliteDS, and ROOT::RDF::RArrowDS.

◆ SetNSlots()

virtual void ROOT::RDF::RDataSource::SetNSlots ( unsigned int  nSlots)
pure virtual

Inform RDataSource of the number of processing slots (i.e.

worker threads) used by the associated RDataFrame. Slots numbers are used to simplify parallel execution: RDataFrame guarantees that different threads will always pass different slot values when calling methods concurrently.

Implemented in ROOT::Internal::RDF::RNumpyDS< ColumnTypes >, ROOT::RDF::RCsvDS, ROOT::RDF::RLazyDS< ColumnTypes >, ROOT::RDF::RTrivialDS, ROOT::Experimental::RNTupleDS, ROOT::RDF::RSqliteDS, and ROOT::RDF::RArrowDS.


The documentation for this class was generated from the following file: