RDataFrame data source class for reading CSV files.
The RCsvDS class implements a CSV file reader for RDataFrame.
A RDataFrame that reads from a CSV file can be constructed using the factory method ROOT::RDF::FromCSV, which accepts five parameters:
true
). If false
, header names will be automatically generated as Col0, Col1, ..., ColN.The type of columns that do not appear in the map is inferred from the data. The supported types are:
true
and false
; specified with 'O'.These are some formatting rules expected by the RCsvDS implementation:
The current implementation of RCsvDS reads the entire CSV file content into memory before RDataFrame starts processing it. Therefore, before creating a CSV RDataFrame, it is important to check both how much memory is available and the size of the CSV file.
RCsvDS can handle empty cells and also allows the usage of the special keywords "NaN" and "nan" to indicate nan
values. If the column is of type double, these cells are stored internally as nan
. Empty cells and explicit nan
-s inside columns of type Long64_t/bool are stored as zeros.
Definition at line 45 of file RCsvDS.hxx.
Classes | |
struct | ROptions |
Options that control how the CSV file is parsed. More... | |
Public Member Functions | |
RCsvDS (const RCsvDS &)=delete | |
RCsvDS (RCsvDS &&)=delete | |
RCsvDS (std::string_view fileName, bool readHeaders=true, char delimiter=',', Long64_t linesChunkSize=-1LL, std::unordered_map< std::string, char > &&colTypes={}) | |
Constructor to create a CSV RDataSource for RDataFrame. | |
RCsvDS (std::string_view fileName, const ROptions &options) | |
Constructor to create a CSV RDataSource for RDataFrame. | |
~RCsvDS () final | |
Destructor. | |
void | Finalize () final |
Convenience method called after concluding an event-loop. | |
const std::vector< std::string > & | GetColumnNames () const final |
Returns a reference to the collection of the dataset's column names. | |
std::unique_ptr< ROOT::Detail::RDF::RColumnReaderBase > | GetColumnReaders (unsigned int slot, std::string_view colName, const std::type_info &tid) final |
If the other GetColumnReaders overload returns an empty vector, this overload will be called instead. | |
std::vector< std::pair< ULong64_t, ULong64_t > > | GetEntryRanges () final |
Return ranges of entries to distribute to tasks. | |
std::string | GetLabel () final |
Return a string representation of the datasource type. | |
std::size_t | GetNFiles () const final |
Returns the number of files from which the dataset is constructed. | |
std::string | GetTypeName (std::string_view colName) const final |
Type of a column as a string, e.g. | |
bool | HasColumn (std::string_view colName) const final |
Checks if the dataset has a certain column. | |
RCsvDS & | operator= (const RCsvDS &)=delete |
RCsvDS & | operator= (RCsvDS &&)=delete |
bool | SetEntry (unsigned int slot, ULong64_t entry) final |
Advance the "cursors" returned by GetColumnReaders to the selected entry for a particular slot. | |
void | SetNSlots (unsigned int nSlots) final |
Inform RDataSource of the number of processing slots (i.e. | |
![]() | |
RDataSource ()=default | |
RDataSource (const RDataSource &)=delete | |
RDataSource (RDataSource &&)=delete | |
virtual | ~RDataSource ()=default |
virtual void | FinalizeSlot (unsigned int) |
Convenience method called at the end of the data processing associated to a slot. | |
template<typename T > | |
std::vector< T ** > | GetColumnReaders (std::string_view columnName) |
Called at most once per column by RDF. | |
virtual void | Initialize () |
Convenience method called before starting an event-loop. | |
virtual void | InitSlot (unsigned int, ULong64_t) |
Convenience method called at the start of the data processing associated to a slot. | |
RDataSource & | operator= (const RDataSource &)=delete |
RDataSource & | operator= (RDataSource &&)=delete |
virtual void | SetGlobalEntryRange (std::pair< ULong64_t, ULong64_t > entryRange) |
Restrict processing to a [begin, end) range of entries. | |
Protected Member Functions | |
std::string | AsString () final |
![]() | |
virtual std::unique_ptr< ROOT::Detail::RDF::RColumnReaderBase > | CreateColumnReader (unsigned int slot, std::string_view col, const std::type_info &tid, TTreeReader *) |
Creates a column reader for the requested column. | |
virtual ROOT::RDF::RSampleInfo | CreateSampleInfo (unsigned int, const std::unordered_map< std::string, ROOT::RDF::Experimental::RSample * > &) const |
virtual std::string | DescribeDataset () |
virtual const std::vector< std::string > & | GetColumnNamesNoDuplicates () const |
virtual const std::vector< std::string > & | GetTopLevelFieldNames () const |
virtual std::string | GetTypeNameWithOpts (std::string_view colName, bool) const |
virtual void | InitializeWithOpts (const std::set< std::string > &) |
virtual void | ProcessMT (ROOT::Detail::RDF::RLoopManager &) |
virtual void | RunFinalChecks (bool) const |
Private Types | |
using | ColType_t = char |
Private Member Functions | |
void | Construct () |
void | FillHeaders (const std::string &) |
void | FillRecord (const std::string &, Record_t &) |
void | FreeRecords () |
void | GenerateHeaders (size_t) |
std::vector< void * > | GetColumnReadersImpl (std::string_view, const std::type_info &) final |
type-erased vector of pointers to pointers to column values - one per slot | |
ColType_t | GetType (std::string_view colName) const |
void | InferColTypes (std::vector< std::string > &) |
void | InferType (const std::string &, unsigned int) |
std::vector< std::string > | ParseColumns (const std::string &) |
size_t | ParseValue (const std::string &, std::vector< std::string > &, size_t) |
bool | Readln (std::string &line) |
void | RewindToData () |
void | ValidateColTypes (std::vector< std::string > &) const |
Private Attributes | |
std::vector< std::deque< bool > > | fBoolEvtValues |
std::vector< std::vector< void * > > | fColAddresses |
std::set< std::string > | fColContainingEmpty |
std::unordered_map< std::string, ColType_t > | fColTypes |
std::list< ColType_t > | fColTypesList |
std::unique_ptr< ROOT::Internal::RRawFile > | fCsvFile |
std::int64_t | fDataLineNumber = 0 |
std::uint64_t | fDataPos = 0 |
std::vector< std::vector< double > > | fDoubleEvtValues |
ULong64_t | fEntryRangesRequested = 0ULL |
std::vector< std::string > | fHeaders |
std::int64_t | fLineNumber = 0 |
std::vector< std::vector< Long64_t > > | fLong64EvtValues |
std::int64_t | fMaxLineNumber = -1 |
ROptions | fOptions |
ULong64_t | fProcessedLines = 0ULL |
std::vector< Record_t > | fRecords |
std::vector< std::vector< std::string > > | fStringEvtValues |
Static Private Attributes | |
static const std::unordered_map< ColType_t, std::string > | fgColTypeMap |
static const TRegexp | fgDoubleRegex1 |
static const TRegexp | fgDoubleRegex2 |
static const TRegexp | fgDoubleRegex3 |
static const TRegexp | fgFalseRegex |
static const TRegexp | fgIntRegex |
static const TRegexp | fgTrueRegex |
Additional Inherited Members | |
![]() | |
using | Record_t = std::vector<void *> |
![]() | |
std::optional< std::pair< ULong64_t, ULong64_t > > | fGlobalEntryRange {} |
unsigned int | fNSlots {} |
#include <ROOT/RCsvDS.hxx>
|
private |
Definition at line 75 of file RCsvDS.hxx.
Constructor to create a CSV RDataSource for RDataFrame.
[in] | fileName | Path or URL of the CSV file. |
[in] | options | File parsing settings |
Definition at line 434 of file RCsvDS.cxx.
ROOT::RDF::RCsvDS::RCsvDS | ( | std::string_view | fileName, |
bool | readHeaders = true, | ||
char | delimiter = ',', | ||
Long64_t | linesChunkSize = -1LL, | ||
std::unordered_map< std::string, char > && | colTypes = {} ) |
Constructor to create a CSV RDataSource for RDataFrame.
[in] | fileName | Path or URL of the CSV file. |
[in] | readHeaders | true if the CSV file contains headers as first row, false otherwise (default true ). |
[in] | delimiter | Delimiter character (default ','). |
[in] | linesChunkSize | bunch of lines to read, use -1 to read all |
[in] | colTypes | Allows users to manually specify column types. Accepts an unordered map with keys being column names, values being type specifiers ('O' for boolean, 'D' for double, 'L' for Long64_t, 'T' for std::string) |
Definition at line 452 of file RCsvDS.cxx.
|
delete |
|
final |
Destructor.
Definition at line 494 of file RCsvDS.cxx.
|
finalprotectedvirtual |
Reimplemented from ROOT::RDF::RDataSource.
Definition at line 98 of file RCsvDS.cxx.
|
private |
Definition at line 369 of file RCsvDS.cxx.
|
private |
Definition at line 165 of file RCsvDS.cxx.
Definition at line 185 of file RCsvDS.cxx.
|
finalvirtual |
Convenience method called after concluding an event-loop.
See Initialize for more details.
Reimplemented from ROOT::RDF::RDataSource.
Definition at line 499 of file RCsvDS.cxx.
|
private |
Definition at line 463 of file RCsvDS.cxx.
|
private |
Definition at line 228 of file RCsvDS.cxx.
|
finalvirtual |
Returns a reference to the collection of the dataset's column names.
Implements ROOT::RDF::RDataSource.
Definition at line 507 of file RCsvDS.cxx.
|
finalvirtual |
If the other GetColumnReaders overload returns an empty vector, this overload will be called instead.
[in] | slot | The data processing slot that needs to be considered |
[in] | name | The name of the column for which a column reader needs to be returned |
[in] | tid | A type_info At least one of the two must return a non-empty/non-null value. |
Reimplemented from ROOT::RDF::RDataSource.
Definition at line 663 of file RCsvDS.cxx.
|
finalprivatevirtual |
type-erased vector of pointers to pointers to column values - one per slot
Implements ROOT::RDF::RDataSource.
Definition at line 246 of file RCsvDS.cxx.
Return ranges of entries to distribute to tasks.
They are required to be contiguous intervals with no entries skipped. Supposing a dataset with nEntries, the intervals must start at 0 and end at nEntries, e.g. [0-5],[5-10] for 10 entries. This function will be invoked repeatedly by RDataFrame as it needs additional entries to process. The same entry range should not be returned more than once. Returning an empty collection of ranges signals to RDataFrame that the processing can stop.
Implements ROOT::RDF::RDataSource.
Definition at line 512 of file RCsvDS.cxx.
|
finalvirtual |
Return a string representation of the datasource type.
The returned string will be used by ROOT::RDF::SaveGraph() to represent the datasource in the visualization of the computation graph. Concrete datasources can override the default implementation.
Reimplemented from ROOT::RDF::RDataSource.
Definition at line 639 of file RCsvDS.cxx.
|
inlinefinalvirtual |
Returns the number of files from which the dataset is constructed.
Reimplemented from ROOT::RDF::RDataSource.
Definition at line 133 of file RCsvDS.hxx.
|
private |
Definition at line 570 of file RCsvDS.cxx.
|
finalvirtual |
Type of a column as a string, e.g.
GetTypeName("x") == "double"
. Required for jitting e.g. df.Filter("x>0")
.
[in] | colName | The name of the column |
Implements ROOT::RDF::RDataSource.
Definition at line 581 of file RCsvDS.cxx.
|
finalvirtual |
Checks if the dataset has a certain column.
[in] | colName | The name of the column |
Implements ROOT::RDF::RDataSource.
Definition at line 586 of file RCsvDS.cxx.
|
private |
Definition at line 270 of file RCsvDS.cxx.
Definition at line 303 of file RCsvDS.cxx.
|
private |
Definition at line 324 of file RCsvDS.cxx.
|
private |
Definition at line 335 of file RCsvDS.cxx.
|
private |
Definition at line 114 of file RCsvDS.cxx.
|
private |
Definition at line 159 of file RCsvDS.cxx.
Advance the "cursors" returned by GetColumnReaders to the selected entry for a particular slot.
[in] | slot | The data processing slot that needs to be considered |
[in] | entry | The entry which needs to be pointed to by the reader pointers Slots are adopted to accommodate parallel data processing. Different workers will loop over different ranges and will be labelled by different "slot" values. Returns true if the entry has to be processed, false otherwise. |
Implements ROOT::RDF::RDataSource.
Definition at line 591 of file RCsvDS.cxx.
|
finalvirtual |
Inform RDataSource of the number of processing slots (i.e.
worker threads) used by the associated RDataFrame. Slots numbers are used to simplify parallel execution: RDataFrame guarantees that different threads will always pass different slot values when calling methods concurrently.
Reimplemented from ROOT::RDF::RDataSource.
Definition at line 622 of file RCsvDS.cxx.
|
private |
Definition at line 251 of file RCsvDS.cxx.
|
private |
Definition at line 100 of file RCsvDS.hxx.
|
private |
Definition at line 93 of file RCsvDS.hxx.
|
private |
Definition at line 91 of file RCsvDS.hxx.
|
private |
Definition at line 90 of file RCsvDS.hxx.
|
private |
Definition at line 92 of file RCsvDS.hxx.
|
private |
Definition at line 86 of file RCsvDS.hxx.
|
private |
Definition at line 83 of file RCsvDS.hxx.
|
private |
Definition at line 82 of file RCsvDS.hxx.
|
private |
Definition at line 95 of file RCsvDS.hxx.
|
private |
Definition at line 87 of file RCsvDS.hxx.
|
staticprivate |
Definition at line 76 of file RCsvDS.hxx.
Definition at line 79 of file RCsvDS.hxx.
Definition at line 79 of file RCsvDS.hxx.
Definition at line 79 of file RCsvDS.hxx.
Definition at line 79 of file RCsvDS.hxx.
Definition at line 79 of file RCsvDS.hxx.
Definition at line 79 of file RCsvDS.hxx.
|
private |
Definition at line 89 of file RCsvDS.hxx.
|
private |
Definition at line 84 of file RCsvDS.hxx.
|
private |
Definition at line 96 of file RCsvDS.hxx.
|
private |
Definition at line 85 of file RCsvDS.hxx.
|
private |
Definition at line 81 of file RCsvDS.hxx.
|
private |
Definition at line 88 of file RCsvDS.hxx.
|
private |
Definition at line 94 of file RCsvDS.hxx.
|
private |
Definition at line 97 of file RCsvDS.hxx.