Logo ROOT  
Reference Guide
ROOT::RDF::RArrowDS Class Referencefinal

RDataFrame data source class to interface with Apache Arrow.

The RArrowDS implements a proxy RDataSource to be able to use Apache Arrow tables with RDataFrame.

A RDataFrame that adapts an arrow::Table class can be constructed using the factory method ROOT::RDF::MakeArrowDataFrame, which accepts one parameter:

  1. An arrow::Table smart pointer.

The types of the columns are derived from the types in the associated arrow::Schema.

Definition at line 22 of file RArrowDS.hxx.

Public Member Functions

 RArrowDS (std::shared_ptr< arrow::Table > table, std::vector< std::string > const &columns)
 Constructor to create an Arrow RDataSource for RDataFrame. More...
 
 ~RArrowDS ()
 Destructor. More...
 
const std::vector< std::string > & GetColumnNames () const override
 Returns a reference to the collection of the dataset's column names. More...
 
std::vector< std::pair< ULong64_t, ULong64_t > > GetEntryRanges () override
 Return ranges of entries to distribute to tasks. More...
 
std::string GetLabel () override
 Return a string representation of the datasource type. More...
 
std::string GetTypeName (std::string_view colName) const override
 Type of a column as a string, e.g. More...
 
bool HasColumn (std::string_view colName) const override
 Checks if the dataset has a certain column. More...
 
void Initialise () override
 Convenience method called before starting an event-loop. More...
 
void InitSlot (unsigned int slot, ULong64_t firstEntry) override
 Convenience method called at the start of the data processing associated to a slot. More...
 
bool SetEntry (unsigned int slot, ULong64_t entry) override
 Advance the "cursors" returned by GetColumnReaders to the selected entry for a particular slot. More...
 
void SetNSlots (unsigned int nSlots) override
 Inform RDataSource of the number of processing slots (i.e. More...
 
- Public Member Functions inherited from ROOT::RDF::RDataSource
virtual ~RDataSource ()=default
 
virtual void Finalise ()
 Convenience method called after concluding an event-loop. More...
 
virtual void FinaliseSlot (unsigned int)
 Convenience method called at the end of the data processing associated to a slot. More...
 
virtual const std::vector< std::string > & GetColumnNames () const =0
 Returns a reference to the collection of the dataset's column names. More...
 
template<typename T >
std::vector< T ** > GetColumnReaders (std::string_view columnName)
 Called at most once per column by RDF. More...
 
virtual std::vector< std::pair< ULong64_t, ULong64_t > > GetEntryRanges ()=0
 Return ranges of entries to distribute to tasks. More...
 
virtual std::string GetLabel ()
 Return a string representation of the datasource type. More...
 
virtual std::string GetTypeName (std::string_view) const =0
 Type of a column as a string, e.g. More...
 
virtual bool HasColumn (std::string_view) const =0
 Checks if the dataset has a certain column. More...
 
virtual void Initialise ()
 Convenience method called before starting an event-loop. More...
 
virtual void InitSlot (unsigned int, ULong64_t)
 Convenience method called at the start of the data processing associated to a slot. More...
 
virtual bool SetEntry (unsigned int slot, ULong64_t entry)=0
 Advance the "cursors" returned by GetColumnReaders to the selected entry for a particular slot. More...
 
virtual void SetNSlots (unsigned int nSlots)=0
 Inform RDataSource of the number of processing slots (i.e. More...
 

Private Member Functions

std::vector< void * > GetColumnReadersImpl (std::string_view name, const std::type_info &type) override
 This needs to return a pointer to the pointer each value getter will point to. More...
 

Private Attributes

std::vector< std::string > fColumnNames
 
std::vector< std::pair< ULong64_t, ULong64_t > > fEntryRanges
 
std::vector< std::pair< size_t, size_t > > fGetterIndex
 
size_t fNSlots = 0U
 
std::shared_ptr< arrow::Table > fTable
 
std::vector< std::unique_ptr< ROOT::Internal::RDF::TValueGetter > > fValueGetters
 

Additional Inherited Members

- Protected Types inherited from ROOT::RDF::RDataSource
using Record_t = std::vector< void * >
 
- Protected Member Functions inherited from ROOT::RDF::RDataSource
virtual std::string AsString ()
 
virtual Record_t GetColumnReadersImpl (std::string_view name, const std::type_info &)=0
 type-erased vector of pointers to pointers to column values - one per slot More...
 

#include <ROOT/RArrowDS.hxx>

Inheritance diagram for ROOT::RDF::RArrowDS:
[legend]

Constructor & Destructor Documentation

◆ RArrowDS()

ROOT::RDF::RArrowDS::RArrowDS ( std::shared_ptr< arrow::Table >  inTable,
std::vector< std::string > const &  inColumns 
)

Constructor to create an Arrow RDataSource for RDataFrame.

Parameters
[in]tablethe arrow Table to observe.
[in]columnsthe name of the columns to use In case columns is empty, we use all the columns found in the table

Definition at line 387 of file RArrowDS.cxx.

◆ ~RArrowDS()

ROOT::RDF::RArrowDS::~RArrowDS ( )

Destructor.

Definition at line 459 of file RArrowDS.cxx.

Member Function Documentation

◆ GetColumnNames()

const std::vector< std::string > & ROOT::RDF::RArrowDS::GetColumnNames ( ) const
overridevirtual

Returns a reference to the collection of the dataset's column names.

Implements ROOT::RDF::RDataSource.

Definition at line 463 of file RArrowDS.cxx.

◆ GetColumnReadersImpl()

std::vector< void * > ROOT::RDF::RArrowDS::GetColumnReadersImpl ( std::string_view  name,
const std::type_info &  type 
)
overrideprivatevirtual

This needs to return a pointer to the pointer each value getter will point to.

Implements ROOT::RDF::RDataSource.

Definition at line 569 of file RArrowDS.cxx.

◆ GetEntryRanges()

std::vector< std::pair< ULong64_t, ULong64_t > > ROOT::RDF::RArrowDS::GetEntryRanges ( )
overridevirtual

Return ranges of entries to distribute to tasks.

They are required to be contiguous intervals with no entries skipped. Supposing a dataset with nEntries, the intervals must start at 0 and end at nEntries, e.g. [0-5],[5-10] for 10 entries. This function will be invoked repeatedly by RDataFrame as it needs additional entries to process. The same entry range should not be returned more than once. Returning an empty collection of ranges signals to RDataFrame that the processing can stop.

Implements ROOT::RDF::RDataSource.

Definition at line 468 of file RArrowDS.cxx.

◆ GetLabel()

std::string ROOT::RDF::RArrowDS::GetLabel ( )
overridevirtual

Return a string representation of the datasource type.

The returned string will be used by ROOT::RDF::SaveGraph() to represent the datasource in the visualization of the computation graph. Concrete datasources can override the default implementation.

Reimplemented from ROOT::RDF::RDataSource.

Definition at line 594 of file RArrowDS.cxx.

◆ GetTypeName()

std::string ROOT::RDF::RArrowDS::GetTypeName ( std::string_view  ) const
overridevirtual

Type of a column as a string, e.g.

GetTypeName("x") == "double". Required for jitting e.g. df.Filter("x>0").

Parameters
[in]columnNameThe name of the column

Implements ROOT::RDF::RDataSource.

Definition at line 474 of file RArrowDS.cxx.

◆ HasColumn()

bool ROOT::RDF::RArrowDS::HasColumn ( std::string_view  ) const
overridevirtual

Checks if the dataset has a certain column.

Parameters
[in]columnNameThe name of the column

Implements ROOT::RDF::RDataSource.

Definition at line 492 of file RArrowDS.cxx.

◆ Initialise()

void ROOT::RDF::RArrowDS::Initialise ( )
overridevirtual

Convenience method called before starting an event-loop.

This method might be called multiple times over the lifetime of a RDataSource, since users can run multiple event-loops with the same RDataFrame. Ideally, Initialise should set the state of the RDataSource so that multiple identical event-loops will produce identical results.

Reimplemented from ROOT::RDF::RDataSource.

Definition at line 588 of file RArrowDS.cxx.

◆ InitSlot()

void ROOT::RDF::RArrowDS::InitSlot ( unsigned int  ,
ULong64_t   
)
overridevirtual

Convenience method called at the start of the data processing associated to a slot.

Parameters
[in]slotThe data processing slot wihch needs to be initialised
[in]firstEntryThe first entry of the range that the task will process. This method might be called multiple times per thread per event-loop.

Reimplemented from ROOT::RDF::RDataSource.

Definition at line 510 of file RArrowDS.cxx.

◆ SetEntry()

bool ROOT::RDF::RArrowDS::SetEntry ( unsigned int  slot,
ULong64_t  entry 
)
overridevirtual

Advance the "cursors" returned by GetColumnReaders to the selected entry for a particular slot.

Parameters
[in]slotThe data processing slot that needs to be considered
[in]entryThe entry which needs to be pointed to by the reader pointers Slots are adopted to accommodate parallel data processing. Different workers will loop over different ranges and will be labelled by different "slot" values. Returns true if the entry has to be processed, false otherwise.

Implements ROOT::RDF::RDataSource.

Definition at line 501 of file RArrowDS.cxx.

◆ SetNSlots()

void ROOT::RDF::RArrowDS::SetNSlots ( unsigned int  nSlots)
overridevirtual

Inform RDataSource of the number of processing slots (i.e.

worker threads) used by the associated RDataFrame. Slots numbers are used to simplify parallel execution: RDataFrame guarantees that different threads will always pass different slot values when calling methods concurrently.

Implements ROOT::RDF::RDataSource.

Definition at line 553 of file RArrowDS.cxx.

Member Data Documentation

◆ fColumnNames

std::vector<std::string> ROOT::RDF::RArrowDS::fColumnNames
private

Definition at line 26 of file RArrowDS.hxx.

◆ fEntryRanges

std::vector<std::pair<ULong64_t, ULong64_t> > ROOT::RDF::RArrowDS::fEntryRanges
private

Definition at line 25 of file RArrowDS.hxx.

◆ fGetterIndex

std::vector<std::pair<size_t, size_t> > ROOT::RDF::RArrowDS::fGetterIndex
private

Definition at line 29 of file RArrowDS.hxx.

◆ fNSlots

size_t ROOT::RDF::RArrowDS::fNSlots = 0U
private

Definition at line 27 of file RArrowDS.hxx.

◆ fTable

std::shared_ptr<arrow::Table> ROOT::RDF::RArrowDS::fTable
private

Definition at line 24 of file RArrowDS.hxx.

◆ fValueGetters

std::vector<std::unique_ptr<ROOT::Internal::RDF::TValueGetter> > ROOT::RDF::RArrowDS::fValueGetters
private

Definition at line 30 of file RArrowDS.hxx.


The documentation for this class was generated from the following files: