Abstract interface to read data from an ntuple.

The page source is initialized with the columns of interest. Alias columns from projected fields are mapped to the corresponding physical columns. Pages from the columns of interest can then be mapped into memory. The page source also gives access to the ntuple's meta-data.

Definition at line 398 of file RPageStorage.hxx.

Classes
class	RActivePhysicalColumns
	Keeps track of the requested physical column IDs. More...

struct	RCounters
	Default I/O performance counters that get registered in fMetrics. More...

struct	REntryRange
	Used in SetEntryRange / GetEntryRange. More...

class	RExclDescriptorGuard
	An RAII wrapper used for the writable access to RPageSource::fDescriptor. See GetSharedDescriptorGuard(). More...

class	RSharedDescriptorGuard
	An RAII wrapper used for the read-only access to RPageSource::fDescriptor. See GetExclDescriptorGuard(). More...

Public Member Functions
	RPageSource (const RPageSource &)=delete

	RPageSource (RPageSource &&)=delete

	RPageSource (std::string_view ntupleName, const RNTupleReadOptions &fOptions)

	~RPageSource () override

ColumnHandle_t	AddColumn (DescriptorId_t fieldId, const RColumn &column) override
	Register a new column.

void	Attach ()
	Open the physical storage container for the tree.

virtual std::unique_ptr< RPageSource >	Clone () const =0
	Open the same storage multiple time, e.g. for reading in multiple threads.

void	DropColumn (ColumnHandle_t columnHandle) override
	Unregisters a column.

ColumnId_t	GetColumnId (ColumnHandle_t columnHandle)

REntryRange	GetEntryRange () const

NTupleSize_t	GetNElements (ColumnHandle_t columnHandle)

NTupleSize_t	GetNEntries ()

const RNTupleReadOptions &	GetReadOptions () const

const RSharedDescriptorGuard	GetSharedDescriptorGuard () const
	Takes the read lock for the descriptor.

EPageStorageType	GetType () final
	Whether the concrete implementation is a sink or a source.

virtual std::vector< std::unique_ptr< RCluster > >	LoadClusters (std::span< RCluster::RKey > clusterKeys)=0
	Populates all the pages of the given cluster ids and columns; it is possible that some columns do not contain any pages.

virtual void	LoadSealedPage (DescriptorId_t physicalColumnId, RClusterIndex clusterIndex, RSealedPage &sealedPage)=0
	Read the packed and compressed bytes of a page into the memory buffer provided by selaedPage.

RPageSource &	operator= (const RPageSource &)=delete

RPageSource &	operator= (RPageSource &&)=delete

virtual RPage	PopulatePage (ColumnHandle_t columnHandle, NTupleSize_t globalIndex)=0
	Allocates and fills a page that contains the index-th element.

virtual RPage	PopulatePage (ColumnHandle_t columnHandle, RClusterIndex clusterIndex)=0
	Another version of PopulatePage that allows to specify cluster-relative indexes.

void	SetEntryRange (const REntryRange &range)
	Promise to only read from the given entry range.

RPage	UnsealPage (const RSealedPage &sealedPage, const RColumnElementBase &element, DescriptorId_t physicalColumnId)
	Helper for unstreaming a page.

void	UnzipCluster (RCluster *cluster)
	Parallel decompression and unpacking of the pages in the given cluster.

Public Member Functions inherited from ROOT::Experimental::Internal::RPageStorage
	RPageStorage (const RPageStorage &other)=delete

	RPageStorage (RPageStorage &&other)=default

	RPageStorage (std::string_view name)

virtual	~RPageStorage ()

virtual Detail::RNTupleMetrics &	GetMetrics ()
	Returns the default metrics object.

const std::string &	GetNTupleName () const
	Returns the NTuple name.

RPageStorage &	operator= (const RPageStorage &other)=delete

RPageStorage &	operator= (RPageStorage &&other)=default

virtual void	ReleasePage (RPage &page)=0
	Every page store needs to be able to free pages it handed out.

void	SetTaskScheduler (RTaskScheduler *taskScheduler)

Static Public Member Functions
static std::unique_ptr< RPageSource >	Create (std::string_view ntupleName, std::string_view location, const RNTupleReadOptions &options=RNTupleReadOptions())
	Guess the concrete derived page source from the file name (location)

Protected Member Functions
virtual RNTupleDescriptor	AttachImpl ()=0

void	EnableDefaultMetrics (const std::string &prefix)
	Enables the default set of metrics provided by RPageSource.

RExclDescriptorGuard	GetExclDescriptorGuard ()
	Note that the underlying lock is not recursive. See GetSharedDescriptorGuard() for further information.

void	PrepareLoadCluster (const RCluster::RKey &clusterKey, ROnDiskPageMap &pageZeroMap, std::function< void(DescriptorId_t, NTupleSize_t, const RClusterDescriptor::RPageRange::RPageInfo &)> perPageFunc)
	Prepare a page range read for the column set in `clusterKey`.

virtual void	UnzipClusterImpl (RCluster *)

Protected Member Functions inherited from ROOT::Experimental::Internal::RPageStorage
void	WaitForAllTasks ()

Protected Attributes
RActivePhysicalColumns	fActivePhysicalColumns
	The active columns are implicitly defined by the model fields or views.

std::unique_ptr< RCounters >	fCounters

std::unique_ptr< RNTupleDecompressor >	fDecompressor
	Helper to unzip pages and header/footer; comprises a 16MB (kMAXZIPBUF) unzip buffer.

RNTupleReadOptions	fOptions

Protected Attributes inherited from ROOT::Experimental::Internal::RPageStorage
Detail::RNTupleMetrics	fMetrics

std::string	fNTupleName

RTaskScheduler *	fTaskScheduler = nullptr

Private Attributes
RNTupleDescriptor	fDescriptor

std::shared_mutex	fDescriptorLock

REntryRange	fEntryRange
	Used by the cluster pool to prevent reading beyond the given range.

Additional Inherited Members
Public Types inherited from ROOT::Experimental::Internal::RPageStorage
using	ColumnHandle_t = RColumnHandle
	The column handle identifies a column with the current open page storage.

using	SealedPageSequence_t = std::deque< RSealedPage >

#include <ROOT/RPageStorage.hxx>

Inheritance diagram for ROOT::Experimental::Internal::RPageSource:

[legend]

Constructor & Destructor Documentation

◆ RPageSource() [1/3]

ROOT::Experimental::Internal::RPageSource::RPageSource	(	std::string_view	ntupleName,
		const RNTupleReadOptions &	fOptions
	)

Definition at line 93 of file RPageStorage.cxx.

◆ RPageSource() [2/3]

ROOT::Experimental::Internal::RPageSource::RPageSource ( const RPageSource & )

delete

◆ RPageSource() [3/3]

ROOT::Experimental::Internal::RPageSource::RPageSource ( RPageSource && )

delete

◆ ~RPageSource()

ROOT::Experimental::Internal::RPageSource::~RPageSource ( )

override

Definition at line 98 of file RPageStorage.cxx.

Member Function Documentation

◆ AddColumn()

ROOT::Experimental::Internal::RPageStorage::ColumnHandle_t ROOT::Experimental::Internal::RPageSource::AddColumn	(	DescriptorId_t	fieldId,
		const RColumn &	column
	)

overridevirtual

Register a new column.

When reading, the column must exist in the ntuple on disk corresponding to the meta-data. When writing, every column can only be attached once.

Implements ROOT::Experimental::Internal::RPageStorage.

Reimplemented in ROOT::Experimental::Internal::RPageSourceFriends.

Definition at line 121 of file RPageStorage.cxx.

◆ Attach()

void ROOT::Experimental::Internal::RPageSource::Attach ( )

inline

Open the physical storage container for the tree.

Definition at line 557 of file RPageStorage.hxx.

◆ AttachImpl()

virtual RNTupleDescriptor ROOT::Experimental::Internal::RPageSource::AttachImpl ( )

protectedpure virtual

Implemented in ROOT::Experimental::Internal::RPageSourceFriends, ROOT::Experimental::Internal::RPageSourceDaos, and ROOT::Experimental::Internal::RPageSourceFile.

◆ Clone()

virtual std::unique_ptr< RPageSource > ROOT::Experimental::Internal::RPageSource::Clone ( ) const

pure virtual

Open the same storage multiple time, e.g. for reading in multiple threads.

Implemented in ROOT::Experimental::Internal::RPageSourceFriends, ROOT::Experimental::Internal::RPageSourceDaos, and ROOT::Experimental::Internal::RPageSourceFile.

◆ Create()

std::unique_ptr< ROOT::Experimental::Internal::RPageSource > ROOT::Experimental::Internal::RPageSource::Create	(	std::string_view	ntupleName,
		std::string_view	location,
		const RNTupleReadOptions &	options = `RNTupleReadOptions()`
	)

static

Guess the concrete derived page source from the file name (location)

Definition at line 101 of file RPageStorage.cxx.

◆ DropColumn()

void ROOT::Experimental::Internal::RPageSource::DropColumn ( ColumnHandle_t columnHandle )

overridevirtual

Unregisters a column.

A page source decreases the reference counter for the corresponding active column. For a page sink, dropping columns is currently a no-op.

Implements ROOT::Experimental::Internal::RPageStorage.

Reimplemented in ROOT::Experimental::Internal::RPageSourceFriends.

Definition at line 130 of file RPageStorage.cxx.

◆ EnableDefaultMetrics()

void ROOT::Experimental::Internal::RPageSource::EnableDefaultMetrics ( const std::string & prefix )

protected

Enables the default set of metrics provided by RPageSource.

prefix will be used as the prefix for the counters registered in the internal RNTupleMetrics object. A subclass using the default set of metrics is responsible for updating the counters appropriately, e.g. fCounters->fNRead.Inc() Alternatively, a subclass might provide its own RNTupleMetrics object by overriding the GetMetrics() member function.

Definition at line 188 of file RPageStorage.cxx.

◆ GetColumnId()

ROOT::Experimental::ColumnId_t ROOT::Experimental::Internal::RPageSource::GetColumnId ( ColumnHandle_t columnHandle )

Definition at line 153 of file RPageStorage.cxx.

◆ GetEntryRange()

REntryRange ROOT::Experimental::Internal::RPageSource::GetEntryRange ( ) const

inline

Definition at line 565 of file RPageStorage.hxx.

◆ GetExclDescriptorGuard()

RExclDescriptorGuard ROOT::Experimental::Internal::RPageSource::GetExclDescriptorGuard ( )

inlineprotected

Note that the underlying lock is not recursive. See GetSharedDescriptorGuard() for further information.

Definition at line 523 of file RPageStorage.hxx.

◆ GetNElements()

ROOT::Experimental::NTupleSize_t ROOT::Experimental::Internal::RPageSource::GetNElements ( ColumnHandle_t columnHandle )

Definition at line 148 of file RPageStorage.cxx.

◆ GetNEntries()

ROOT::Experimental::NTupleSize_t ROOT::Experimental::Internal::RPageSource::GetNEntries ( )

Definition at line 143 of file RPageStorage.cxx.

◆ GetReadOptions()

const RNTupleReadOptions & ROOT::Experimental::Internal::RPageSource::GetReadOptions ( ) const

inline

Definition at line 539 of file RPageStorage.hxx.

◆ GetSharedDescriptorGuard()

const RSharedDescriptorGuard ROOT::Experimental::Internal::RPageSource::GetSharedDescriptorGuard ( ) const

inline

Takes the read lock for the descriptor.

Multiple threads can take the lock concurrently. The underlying std::shared_mutex, however, is neither read nor write recursive: within one thread, only one lock (shared or exclusive) must be acquired at the same time. This requires special care in sections protected by GetSharedDescriptorGuard() and GetExclDescriptorGuard() especially to avoid that the locks are acquired indirectly (e.g. by a call to GetNEntries()). As a general guideline, no other method of the page source should be called (directly or indirectly) in a guarded section.

Definition at line 548 of file RPageStorage.hxx.

◆ GetType()

EPageStorageType ROOT::Experimental::Internal::RPageSource::GetType ( )

inlinefinalvirtual

Whether the concrete implementation is a sink or a source.

Implements ROOT::Experimental::Internal::RPageStorage.

Definition at line 538 of file RPageStorage.hxx.

◆ LoadClusters()

virtual std::vector< std::unique_ptr< RCluster > > ROOT::Experimental::Internal::RPageSource::LoadClusters ( std::span< RCluster::RKey > clusterKeys )

pure virtual

Populates all the pages of the given cluster ids and columns; it is possible that some columns do not contain any pages.

The page source may load more columns than the minimal necessary set from columns. To indicate which columns have been loaded, LoadClusters() must mark them with SetColumnAvailable(). That includes the ones from the columns that don't have pages; otherwise subsequent requests for the cluster would assume an incomplete cluster and trigger loading again. LoadClusters() is typically called from the I/O thread of a cluster pool, i.e. the method runs concurrently to other methods of the page source.

Implemented in ROOT::Experimental::Internal::RPageSourceFriends, ROOT::Experimental::Internal::RPageSourceDaos, and ROOT::Experimental::Internal::RPageSourceFile.

◆ LoadSealedPage()

virtual void ROOT::Experimental::Internal::RPageSource::LoadSealedPage	(	DescriptorId_t	physicalColumnId,
		RClusterIndex	clusterIndex,
		RSealedPage &	sealedPage
	)

pure virtual

Read the packed and compressed bytes of a page into the memory buffer provided by selaedPage.

The sealed page can be used subsequently in a call to RPageSink::CommitSealedPage. The fSize and fNElements member of the sealedPage parameters are always set. If sealedPage.fBuffer is nullptr, no data will be copied but the returned size information can be used by the caller to allocate a large enough buffer and call LoadSealedPage again.

Implemented in ROOT::Experimental::Internal::RPageSourceFriends, ROOT::Experimental::Internal::RPageSourceDaos, and ROOT::Experimental::Internal::RPageSourceFile.

◆ operator=() [1/2]

RPageSource & ROOT::Experimental::Internal::RPageSource::operator= ( const RPageSource & )

delete

◆ operator=() [2/2]

RPageSource & ROOT::Experimental::Internal::RPageSource::operator= ( RPageSource && )

delete

◆ PopulatePage() [1/2]

virtual RPage ROOT::Experimental::Internal::RPageSource::PopulatePage	(	ColumnHandle_t	columnHandle,
		NTupleSize_t	globalIndex
	)

pure virtual

Allocates and fills a page that contains the index-th element.

Implemented in ROOT::Experimental::Internal::RPageSourceFriends, ROOT::Experimental::Internal::RPageSourceDaos, and ROOT::Experimental::Internal::RPageSourceFile.

◆ PopulatePage() [2/2]

virtual RPage ROOT::Experimental::Internal::RPageSource::PopulatePage	(	ColumnHandle_t	columnHandle,
		RClusterIndex	clusterIndex
	)

pure virtual

Another version of PopulatePage that allows to specify cluster-relative indexes.

Implemented in ROOT::Experimental::Internal::RPageSourceFriends, ROOT::Experimental::Internal::RPageSourceDaos, and ROOT::Experimental::Internal::RPageSourceFile.

◆ PrepareLoadCluster()

void ROOT::Experimental::Internal::RPageSource::PrepareLoadCluster	(	const RCluster::RKey &	clusterKey,
		ROnDiskPageMap &	pageZeroMap,
		std::function< void(DescriptorId_t, NTupleSize_t, const RClusterDescriptor::RPageRange::RPageInfo &)>	perPageFunc
	)

protected

Prepare a page range read for the column set in clusterKey.

Specifically, pages referencing the kTypePageZero locator are filled in pageZeroMap; otherwise, perPageFunc is called for each page. This is commonly used as part of LoadClusters() in derived classes.

Definition at line 165 of file RPageStorage.cxx.

◆ SetEntryRange()

void ROOT::Experimental::Internal::RPageSource::SetEntryRange ( const REntryRange & range )

Promise to only read from the given entry range.

If set, prevents the cluster pool from reading-ahead beyond the given range. The range needs to be within [0, GetNEntries()).

Definition at line 135 of file RPageStorage.cxx.

◆ UnsealPage()

ROOT::Experimental::Internal::RPage ROOT::Experimental::Internal::RPageSource::UnsealPage	(	const RSealedPage &	sealedPage,
		const RColumnElementBase &	element,
		DescriptorId_t	physicalColumnId
	)

Helper for unstreaming a page.

This is commonly used in derived, concrete page sources. The implementation currently always makes a memory copy, even if the sealed page is uncompressed and in the final memory layout. The optimization of directly mapping pages is left to the concrete page source implementations. Usage of this method requires construction of fDecompressor. Memory is allocated via RPageAllocatorHeap; use RPageAllocatorHeap::DeletePage() to deallocate returned pages.

Definition at line 283 of file RPageStorage.cxx.

◆ UnzipCluster()

void ROOT::Experimental::Internal::RPageSource::UnzipCluster ( RCluster * cluster )

Parallel decompression and unpacking of the pages in the given cluster.

The unzipped pages are supposed to be preloaded in a page pool attached to the source. The method is triggered by the cluster pool's unzip thread. It is an optional optimization, the method can safely do nothing. In particular, the actual implementation will only run if a task scheduler is set. In practice, a task scheduler is set if implicit multi-threading is turned on.

Definition at line 159 of file RPageStorage.cxx.

◆ UnzipClusterImpl()

virtual void ROOT::Experimental::Internal::RPageSource::UnzipClusterImpl ( RCluster * )

inlineprotectedvirtual

Reimplemented in ROOT::Experimental::Internal::RPageSourceDaos, and ROOT::Experimental::Internal::RPageSourceFile.

Definition at line 504 of file RPageStorage.hxx.

Member Data Documentation

◆ fActivePhysicalColumns

RActivePhysicalColumns ROOT::Experimental::Internal::RPageSource::fActivePhysicalColumns

protected

The active columns are implicitly defined by the model fields or views.

Definition at line 495 of file RPageStorage.hxx.

◆ fCounters

std::unique_ptr<RCounters> ROOT::Experimental::Internal::RPageSource::fCounters

protected

Definition at line 491 of file RPageStorage.hxx.

◆ fDecompressor

std::unique_ptr<RNTupleDecompressor> ROOT::Experimental::Internal::RPageSource::fDecompressor

protected

Helper to unzip pages and header/footer; comprises a 16MB (kMAXZIPBUF) unzip buffer.

Not all page sources need a decompressor (e.g. virtual ones for chains and friends don't), thus we leave it up to the derived class whether or not the decompressor gets constructed.

Definition at line 500 of file RPageStorage.hxx.

◆ fDescriptor

RNTupleDescriptor ROOT::Experimental::Internal::RPageSource::fDescriptor

private

Definition at line 452 of file RPageStorage.hxx.

◆ fDescriptorLock

std::shared_mutex ROOT::Experimental::Internal::RPageSource::fDescriptorLock

mutableprivate

Definition at line 453 of file RPageStorage.hxx.

◆ fEntryRange

REntryRange ROOT::Experimental::Internal::RPageSource::fEntryRange

private

Used by the cluster pool to prevent reading beyond the given range.

Definition at line 454 of file RPageStorage.hxx.

◆ fOptions

RNTupleReadOptions ROOT::Experimental::Internal::RPageSource::fOptions

protected

Definition at line 493 of file RPageStorage.hxx.

Libraries for ROOT::Experimental::Internal::RPageSource:

[legend]

The documentation for this class was generated from the following files:

tree/ntuple/v7/inc/ROOT/RPageStorage.hxx
tree/ntuple/v7/src/RPageStorage.cxx

Classes

Public Member Functions

Static Public Member Functions

Protected Member Functions

Protected Attributes

Private Attributes

Additional Inherited Members

Constructor & Destructor Documentation

◆ RPageSource() [1/3]

◆ RPageSource() [2/3]

◆ RPageSource() [3/3]

◆ ~RPageSource()

Member Function Documentation

◆ AddColumn()

◆ Attach()

◆ AttachImpl()

◆ Clone()

◆ Create()

◆ DropColumn()

◆ EnableDefaultMetrics()

◆ GetColumnId()

◆ GetEntryRange()

◆ GetExclDescriptorGuard()

◆ GetNElements()

◆ GetNEntries()

◆ GetReadOptions()

◆ GetSharedDescriptorGuard()

◆ GetType()

◆ LoadClusters()

◆ LoadSealedPage()

◆ operator=() [1/2]

◆ operator=() [2/2]

◆ PopulatePage() [1/2]

◆ PopulatePage() [2/2]

◆ PrepareLoadCluster()

◆ SetEntryRange()

◆ UnsealPage()

◆ UnzipCluster()

◆ UnzipClusterImpl()

Member Data Documentation

◆ fActivePhysicalColumns

◆ fCounters

◆ fDecompressor

◆ fDescriptor

◆ fDescriptorLock

◆ fEntryRange

◆ fOptions