RSqliteDS is an RDF data source implementation for SQL result sets from sqlite3 files.

The RSqliteDS is able to feed an RDataFrame with data from a SQlite SELECT query. One can use it like

auto rdf = ROOT::RDF::FromSqlite("/path/to/file.sqlite", "select name from table");
auto h = rdf.Define("lName", "name.length()").Histo1D("lName");

The data source has to provide column types for all the columns. Determining column types in SQlite is tricky as it is dynamically typed and in principle each row can have different column types. The following heuristics is used:

If a table column is queried as is ("SELECT colname FROM table"), the default/declared column type is taken.
For expressions ("SELECT 1+1 FROM table"), the type of the first row of the result set determines the column type. That can result in a column to be of thought of type NULL where subsequent rows actually have meaningful values. The provided SELECT query can be used to avoid such ambiguities.

Definition at line 61 of file RSqliteDS.hxx.

Classes
struct	Value_t
	Used to hold a single "cell" of the SELECT query's result table. Can be changed to std::variant once available. More...

Public Member Functions
	RSqliteDS (const RSqliteDS &)=delete
	RSqliteDS (const std::string &fileName, const std::string &query)
	Build the dataframe.
	RSqliteDS (RSqliteDS &&)=delete
	~RSqliteDS () final
	Frees the sqlite resources and closes the file.
virtual void	Finalize ()
	Convenience method called after concluding an event-loop.
virtual void	FinalizeSlot (unsigned int)
	Convenience method called at the end of the data processing associated to a slot.
const std::vector< std::string > &	GetColumnNames () const final
	Returns the SELECT queries names.
template<typename T>
std::vector< T ** >	GetColumnReaders (std::string_view columnName)
	Called at most once per column by RDF.
std::unique_ptr< ROOT::Detail::RDF::RColumnReaderBase >	GetColumnReaders (unsigned int slot, std::string_view colName, const std::type_info &tid) final
	If the other GetColumnReaders overload returns an empty vector, this overload will be called instead.
std::vector< std::pair< ULong64_t, ULong64_t > >	GetEntryRanges () final
	Returns a range of size 1 as long as more rows are available in the SQL result set.
std::string	GetLabel () final
	Return a string representation of the datasource type.
virtual std::size_t	GetNFiles () const
	Returns the number of files from which the dataset is constructed.
std::string	GetTypeName (std::string_view colName) const final
	Returns the C++ type for a given column name, implemented as a linear search through all the columns.
bool	HasColumn (std::string_view colName) const final
	A linear search through the columns for the given name.
void	Initialize () final
	Resets the SQlite query engine at the beginning of the event loop.
virtual void	InitSlot (unsigned int, ULong64_t)
	Convenience method called at the start of the data processing associated to a slot.
RSqliteDS &	operator= (const RSqliteDS &)=delete
RSqliteDS &	operator= (RSqliteDS &&)=delete
bool	SetEntry (unsigned int slot, ULong64_t entry) final
	Stores the result of the current active sqlite query row as a C++ value.
virtual void	SetGlobalEntryRange (std::pair< ULong64_t, ULong64_t > entryRange)
	Restrict processing to a [begin, end) range of entries.
void	SetNSlots (unsigned int nSlots) final
	Almost a no-op, many slots can in fact reduce the performance due to thread synchronization.

Protected Types
using	Record_t = std::vector<void *>

Protected Member Functions
virtual std::string	AsString ()
virtual std::unique_ptr< ROOT::Detail::RDF::RColumnReaderBase >	CreateColumnReader (unsigned int slot, std::string_view col, const std::type_info &tid, TTreeReader *)
	Creates a column reader for the requested column.
virtual ROOT::RDF::RSampleInfo	CreateSampleInfo (unsigned int, const std::unordered_map< std::string, ROOT::RDF::Experimental::RSample * > &) const
virtual std::string	DescribeDataset ()
virtual const std::vector< std::string > &	GetColumnNamesNoDuplicates () const
Record_t	GetColumnReadersImpl (std::string_view name, const std::type_info &) final
	Activates the given column's result value.
virtual const std::vector< std::string > &	GetTopLevelFieldNames () const
virtual std::string	GetTypeNameWithOpts (std::string_view colName, bool) const
virtual void	InitializeWithOpts (const std::set< std::string > &)
virtual void	ProcessMT (ROOT::Detail::RDF::RLoopManager &)
virtual void	RunFinalChecks (bool) const

Protected Attributes
std::optional< std::pair< ULong64_t, ULong64_t > >	fGlobalEntryRange {}
unsigned int	fNSlots {}

Private Types
enum class	ETypes { kInteger , kReal , kText , kBlob , kNull }
	All the types known to SQlite. Changes require changing fgTypeNames, too. More...

Private Member Functions
void	SqliteError (int errcode)
	Helper function to throw an exception if there is a fatal sqlite error, e.g. an I/O error.

Private Attributes
std::vector< std::string >	fColumnNames
std::vector< ETypes >	fColumnTypes
std::unique_ptr< Internal::RSqliteDSDataSet >	fDataSet
ULong64_t	fNRow
std::vector< Value_t >	fValues
	The data source is inherently single-threaded and returns only one row at a time. This vector holds the results.

Static Private Attributes
static constexpr char const *	fgTypeNames []
	Corresponds to the types defined in ETypes.

#include <ROOT/RSqliteDS.hxx>

Inheritance diagram for ROOT::RDF::RSqliteDS:

Member Typedef Documentation

◆ Record_t

using ROOT::RDF::RDataSource::Record_t = std::vector<void *>

protectedinherited

Definition at line 142 of file RDataSource.hxx.

Member Enumeration Documentation

◆ ETypes

enum class ROOT::RDF::RSqliteDS::ETypes

strongprivate

All the types known to SQlite. Changes require changing fgTypeNames, too.

Enumerator
kInteger
kReal
kText
kBlob
kNull

Definition at line 65 of file RSqliteDS.hxx.

Constructor & Destructor Documentation

◆ RSqliteDS() [1/3]

ROOT::RDF::RSqliteDS::RSqliteDS	(	const std::string &	fileName,
		const std::string &	query )

Build the dataframe.

Parameters

[in]	fileName	The path to an sqlite3 file, will be opened read-only
[in]	query	A valid sqlite3 SELECT query

The constructor opens the sqlite file, prepares the query engine and determines the column names and types.

Definition at line 352 of file RSqliteDS.cxx.

◆ RSqliteDS() [2/3]

ROOT::RDF::RSqliteDS::RSqliteDS ( const RSqliteDS & )

delete

◆ RSqliteDS() [3/3]

ROOT::RDF::RSqliteDS::RSqliteDS ( RSqliteDS && )

delete

◆ ~RSqliteDS()

ROOT::RDF::RSqliteDS::~RSqliteDS ( )

final

Frees the sqlite resources and closes the file.

Definition at line 438 of file RSqliteDS.cxx.

Member Function Documentation

◆ AsString()

virtual std::string ROOT::RDF::RDataSource::AsString ( )

inlineprotectedvirtualinherited

Reimplemented in ROOT::Internal::RDF::RRootDS, ROOT::Internal::RDF::RTTreeDS, ROOT::Internal::RDF::RVecDS< ColumnTypes >, ROOT::RDF::RCsvDS, ROOT::RDF::RLazyDS< ColumnTypes >, and ROOT::RDF::RTrivialDS.

Definition at line 145 of file RDataSource.hxx.

◆ CreateColumnReader()

virtual std::unique_ptr< ROOT::Detail::RDF::RColumnReaderBase > ROOT::RDF::RDataSource::CreateColumnReader	(	unsigned int	slot,
		std::string_view	col,
		const std::type_info &	tid,
		TTreeReader *	)

inlineprotectedvirtualinherited

Creates a column reader for the requested column.

In the general case, this is just a redirect to the right GetColumnReaders overload. The signature notably also has a TTreeReader * parameter. This is currently necessary to still allow the TTree-based MT scheduling via TTreeProcessorMT. We use the TTreeProcessorMT::Process method to launch the same kernel across all threads. In each thread task, TTreeProcessorMT creates a thread-local instance of a TTreeReader which is going to read the range of events assigned to that task. That TTreeReader instance is what is passed to this method whenever a column reader needs to be created in a thread task. In the future this method might be removed by either allowing to request a handle to the thread-local TTreeReader instance programmatically from the TTreeProcessorMT, or refactoring the TTreeProcessorMT scheduling into RTTreeDS altogether.

Reimplemented in ROOT::Internal::RDF::RTTreeDS.

Definition at line 195 of file RDataSource.hxx.

◆ CreateSampleInfo()

ROOT::RDF::RSampleInfo ROOT::RDF::RDataSource::CreateSampleInfo	(	unsigned int	,
		const std::unordered_map< std::string, ROOT::RDF::Experimental::RSample * > &	) const

protectedvirtualinherited

Reimplemented in ROOT::Internal::RDF::RTTreeDS, and ROOT::RDF::RNTupleDS.

Definition at line 11 of file RDataSource.cxx.

◆ DescribeDataset()

virtual std::string ROOT::RDF::RDataSource::DescribeDataset ( )

inlineprotectedvirtualinherited

Reimplemented in ROOT::Internal::RDF::RTTreeDS.

Definition at line 165 of file RDataSource.hxx.

◆ Finalize()

virtual void ROOT::RDF::RDataSource::Finalize ( )

inlinevirtualinherited

Convenience method called after concluding an event-loop.

See Initialize for more details.

Reimplemented in ROOT::Internal::RDF::RTTreeDS, ROOT::RDF::RCsvDS, and ROOT::RDF::RNTupleDS.

Definition at line 317 of file RDataSource.hxx.

◆ FinalizeSlot()

virtual void ROOT::RDF::RDataSource::FinalizeSlot ( unsigned int )

inlinevirtualinherited

Convenience method called at the end of the data processing associated to a slot.

Parameters

[in] slot The data processing slot wihch needs to be finalized This method might be called multiple times per thread per event-loop.

Reimplemented in ROOT::Internal::RDF::RRootDS, and ROOT::RDF::RNTupleDS.

Definition at line 311 of file RDataSource.hxx.

◆ GetColumnNames()

const std::vector< std::string > & ROOT::RDF::RSqliteDS::GetColumnNames ( ) const

finalvirtual

Returns the SELECT queries names.

The column names have been cached in the constructor. For expressions, the column name is the string of the expression unless the query defines a column name with as like in "SELECT 1 + 1 as mycolumn FROM table"

Implements ROOT::RDF::RDataSource.

Definition at line 451 of file RSqliteDS.cxx.

◆ GetColumnNamesNoDuplicates()

virtual const std::vector< std::string > & ROOT::RDF::RDataSource::GetColumnNamesNoDuplicates ( ) const

inlineprotectedvirtualinherited

Reimplemented in ROOT::Internal::RDF::RTTreeDS.

Definition at line 159 of file RDataSource.hxx.

◆ GetColumnReaders() [1/2]

template<typename T>

std::vector< T ** > ROOT::RDF::RDataSource::GetColumnReaders ( std::string_view columnName )

inlineinherited

Called at most once per column by RDF.

Return vector of pointers to pointers to column values - one per slot.

Template Parameters

T	The type of the data stored in the column

Parameters

[in] columnName The name of the column

These pointers are veritable cursors: it's a responsibility of the RDataSource implementation that they point to the "right" memory region.

Definition at line 248 of file RDataSource.hxx.

◆ GetColumnReaders() [2/2]

std::unique_ptr< ROOT::Detail::RDF::RColumnReaderBase > ROOT::RDF::RSqliteDS::GetColumnReaders	(	unsigned int	,
		std::string_view	,
		const std::type_info &	)

finalvirtual

If the other GetColumnReaders overload returns an empty vector, this overload will be called instead.

Parameters

[in]	slot	The data processing slot that needs to be considered
[in]	name	The name of the column for which a column reader needs to be returned
[in]	tid	A type_info At least one of the two must return a non-empty/non-null value.

Reimplemented from ROOT::RDF::RDataSource.

Definition at line 596 of file RSqliteDS.cxx.

◆ GetColumnReadersImpl()

RDataSource::Record_t ROOT::RDF::RSqliteDS::GetColumnReadersImpl	(	std::string_view	name,
		const std::type_info &	)

finalprotectedvirtual

Activates the given column's result value.

Implements ROOT::RDF::RDataSource.

Definition at line 458 of file RSqliteDS.cxx.

◆ GetEntryRanges()

std::vector< std::pair< ULong64_t, ULong64_t > > ROOT::RDF::RSqliteDS::GetEntryRanges ( )

finalvirtual

Returns a range of size 1 as long as more rows are available in the SQL result set.

This inherently serialized the RDF independent of the number of slots.

Implements ROOT::RDF::RDataSource.

Definition at line 467 of file RSqliteDS.cxx.

◆ GetLabel()

std::string ROOT::RDF::RSqliteDS::GetLabel ( )

finalvirtual

Return a string representation of the datasource type.

The returned string will be used by ROOT::RDF::SaveGraph() to represent the datasource in the visualization of the computation graph. Concrete datasources can override the default implementation.

Reimplemented from ROOT::RDF::RDataSource.

Definition at line 515 of file RSqliteDS.cxx.

◆ GetNFiles()

virtual std::size_t ROOT::RDF::RDataSource::GetNFiles ( ) const

inlinevirtualinherited

Returns the number of files from which the dataset is constructed.

Reimplemented in ROOT::Internal::RDF::RRootDS, ROOT::Internal::RDF::RTTreeDS, ROOT::RDF::RCsvDS, and ROOT::RDF::RNTupleDS.

Definition at line 222 of file RDataSource.hxx.

◆ GetTopLevelFieldNames()

virtual const std::vector< std::string > & ROOT::RDF::RDataSource::GetTopLevelFieldNames ( ) const

inlineprotectedvirtualinherited

Reimplemented in ROOT::Internal::RDF::RTTreeDS, and ROOT::RDF::RNTupleDS.

Definition at line 155 of file RDataSource.hxx.

◆ GetTypeName()

std::string ROOT::RDF::RSqliteDS::GetTypeName ( std::string_view colName ) const

finalvirtual

Returns the C++ type for a given column name, implemented as a linear search through all the columns.

Implements ROOT::RDF::RDataSource.

Definition at line 486 of file RSqliteDS.cxx.

◆ GetTypeNameWithOpts()

virtual std::string ROOT::RDF::RDataSource::GetTypeNameWithOpts	(	std::string_view	colName,
		bool	) const

inlineprotectedvirtualinherited

Reimplemented in ROOT::Internal::RDF::RTTreeDS.

Definition at line 152 of file RDataSource.hxx.

◆ HasColumn()

bool ROOT::RDF::RSqliteDS::HasColumn ( std::string_view colName ) const

finalvirtual

A linear search through the columns for the given name.

Implements ROOT::RDF::RDataSource.

Definition at line 500 of file RSqliteDS.cxx.

◆ Initialize()

void ROOT::RDF::RSqliteDS::Initialize ( )

finalvirtual

Resets the SQlite query engine at the beginning of the event loop.

Reimplemented from ROOT::RDF::RDataSource.

Definition at line 507 of file RSqliteDS.cxx.

◆ InitializeWithOpts()

virtual void ROOT::RDF::RDataSource::InitializeWithOpts ( const std::set< std::string > & )

inlineprotectedvirtualinherited

Reimplemented in ROOT::Internal::RDF::RTTreeDS.

Definition at line 162 of file RDataSource.hxx.

◆ InitSlot()

virtual void ROOT::RDF::RDataSource::InitSlot	(	unsigned int	,
		ULong64_t	)

inlinevirtualinherited

Convenience method called at the start of the data processing associated to a slot.

Parameters

[in]	slot	The data processing slot wihch needs to be initialized
[in]	firstEntry	The first entry of the range that the task will process. This method might be called multiple times per thread per event-loop.

Reimplemented in ROOT::Internal::RDF::RRootDS, ROOT::RDF::RArrowDS, and ROOT::RDF::RNTupleDS.

Definition at line 304 of file RDataSource.hxx.

◆ operator=() [1/2]

RSqliteDS & ROOT::RDF::RSqliteDS::operator= ( const RSqliteDS & )

delete

◆ operator=() [2/2]

RSqliteDS & ROOT::RDF::RSqliteDS::operator= ( RSqliteDS && )

delete

◆ ProcessMT()

void ROOT::RDF::RDataSource::ProcessMT ( ROOT::Detail::RDF::RLoopManager & lm )

protectedvirtualinherited

Reimplemented in ROOT::Internal::RDF::RTTreeDS.

Definition at line 17 of file RDataSource.cxx.

◆ RunFinalChecks()

virtual void ROOT::RDF::RDataSource::RunFinalChecks ( bool ) const

inlineprotectedvirtualinherited

Reimplemented in ROOT::Internal::RDF::RTTreeDS.

Definition at line 174 of file RDataSource.hxx.

◆ SetEntry()

bool ROOT::RDF::RSqliteDS::SetEntry	(	unsigned int	slot,
		ULong64_t	entry )

finalvirtual

Stores the result of the current active sqlite query row as a C++ value.

Implements ROOT::RDF::RDataSource.

Definition at line 532 of file RSqliteDS.cxx.

◆ SetGlobalEntryRange()

virtual void ROOT::RDF::RDataSource::SetGlobalEntryRange ( std::pair< ULong64_t, ULong64_t > entryRange )

inlinevirtualinherited

Restrict processing to a [begin, end) range of entries.

Parameters

entryRange The range of entries to process.

Definition at line 327 of file RDataSource.hxx.

◆ SetNSlots()

void ROOT::RDF::RSqliteDS::SetNSlots ( unsigned int nSlots )

finalvirtual

Almost a no-op, many slots can in fact reduce the performance due to thread synchronization.

Reimplemented from ROOT::RDF::RDataSource.

Definition at line 569 of file RSqliteDS.cxx.

◆ SqliteError()

void ROOT::RDF::RSqliteDS::SqliteError ( int errcode )

private

Helper function to throw an exception if there is a fatal sqlite error, e.g. an I/O error.

Definition at line 580 of file RSqliteDS.cxx.

Member Data Documentation

◆ fColumnNames

std::vector<std::string> ROOT::RDF::RSqliteDS::fColumnNames

private

Definition at line 92 of file RSqliteDS.hxx.

◆ fColumnTypes

std::vector<ETypes> ROOT::RDF::RSqliteDS::fColumnTypes

private

Definition at line 93 of file RSqliteDS.hxx.

◆ fDataSet

std::unique_ptr<Internal::RSqliteDSDataSet> ROOT::RDF::RSqliteDS::fDataSet

private

Definition at line 90 of file RSqliteDS.hxx.

◆ fGlobalEntryRange

std::optional<std::pair<ULong64_t, ULong64_t> > ROOT::RDF::RDataSource::fGlobalEntryRange {}

protectedinherited

Definition at line 149 of file RDataSource.hxx.

◆ fgTypeNames

char const * ROOT::RDF::RSqliteDS::fgTypeNames

staticconstexprprivate

Initial value:

                                        = {
   "Long64_t",
   "double",
   "std::string",
   "std::vector<unsigned char>",
   "void *"
}

Corresponds to the types defined in ETypes.

Definition at line 99 of file RSqliteDS.hxx.

◆ fNRow

ULong64_t ROOT::RDF::RSqliteDS::fNRow

private

Definition at line 91 of file RSqliteDS.hxx.

◆ fNSlots

unsigned int ROOT::RDF::RDataSource::fNSlots {}

protectedinherited

Definition at line 147 of file RDataSource.hxx.

◆ fValues

std::vector<Value_t> ROOT::RDF::RSqliteDS::fValues

private

The data source is inherently single-threaded and returns only one row at a time. This vector holds the results.

Definition at line 95 of file RSqliteDS.hxx.

The documentation for this class was generated from the following files:

tree/dataframe/inc/ROOT/RSqliteDS.hxx
tree/dataframe/src/RSqliteDS.cxx

Classes

Public Member Functions

Protected Types

Protected Member Functions

Protected Attributes

Private Types

Private Member Functions

Private Attributes

Static Private Attributes

Member Typedef Documentation

◆ Record_t

Member Enumeration Documentation

◆ ETypes

Constructor & Destructor Documentation

◆ RSqliteDS() [1/3]

◆ RSqliteDS() [2/3]

◆ RSqliteDS() [3/3]

◆ ~RSqliteDS()

Member Function Documentation

◆ AsString()

◆ CreateColumnReader()

◆ CreateSampleInfo()

◆ DescribeDataset()

◆ Finalize()

◆ FinalizeSlot()

◆ GetColumnNames()

◆ GetColumnNamesNoDuplicates()

◆ GetColumnReaders() [1/2]

◆ GetColumnReaders() [2/2]

◆ GetColumnReadersImpl()

◆ GetEntryRanges()

◆ GetLabel()

◆ GetNFiles()

◆ GetTopLevelFieldNames()

◆ GetTypeName()

◆ GetTypeNameWithOpts()

◆ HasColumn()

◆ Initialize()

◆ InitializeWithOpts()

◆ InitSlot()

◆ operator=() [1/2]

◆ operator=() [2/2]

◆ ProcessMT()

◆ RunFinalChecks()

◆ SetEntry()

◆ SetGlobalEntryRange()

◆ SetNSlots()

◆ SqliteError()

Member Data Documentation

◆ fColumnNames

◆ fColumnTypes

◆ fDataSet

◆ fGlobalEntryRange

◆ fgTypeNames

◆ fNRow

◆ fNSlots

◆ fValues