Logo ROOT  
Reference Guide
Loading...
Searching...
No Matches
RPageSource Class Referenceabstract

Abstract interface to read data from an ntuple.

The page source is initialized with the columns of interest. Alias columns from projected fields are mapped to the corresponding physical columns. Pages from the columns of interest can then be mapped into memory. The page source also gives access to the ntuple's metadata.

Definition at line 569 of file RPageStorage.hxx.

Classes

class  RActivePhysicalColumns
 Keeps track of the requested physical column IDs and their in-memory target type via a column element identifier. More...
struct  RClusterInfo
 Summarizes cluster-level information that are necessary to load a certain page. More...
struct  RCounters
 Default I/O performance counters that get registered in fMetrics. More...
struct  REntryRange
 Used in SetEntryRange / GetEntryRange. More...
class  RExclDescriptorGuard
 An RAII wrapper used for the writable access to RPageSource::fDescriptor. See GetSharedDescriptorGuard(). More...
class  RSharedDescriptorGuard
 An RAII wrapper used for the read-only access to RPageSource::fDescriptor. See `GetExclDescriptorGuard()``. More...

Public Member Functions

 RPageSource (const RPageSource &)=delete
 RPageSource (RPageSource &&)=delete
 RPageSource (std::string_view ntupleName, const ROOT::RNTupleReadOptions &fOptions)
 ~RPageSource () override
ColumnHandle_t AddColumn (ROOT::DescriptorId_t fieldId, ROOT::Internal::RColumn &column) override
 Register a new column.
void Attach (ROOT::Internal::RNTupleSerializer::EDescriptorDeserializeMode mode=ROOT::Internal::RNTupleSerializer::EDescriptorDeserializeMode::kForReading)
 Open the physical storage container and deserialize header and footer.
std::unique_ptr< RPageSourceClone () const
 Open the same storage multiple time, e.g.
void DropColumn (ColumnHandle_t columnHandle) override
 Unregisters a column.
REntryRange GetEntryRange () const
ROOT::NTupleSize_t GetNElements (ColumnHandle_t columnHandle)
ROOT::NTupleSize_t GetNEntries ()
const std::unordered_set< ROOT::DescriptorId_t > & GetPinnedClusters () const
const ROOT::RNTupleReadOptionsGetReadOptions () const
const RSharedDescriptorGuard GetSharedDescriptorGuard () const
 Takes the read lock for the descriptor.
EPageStorageType GetType () final
 Whether the concrete implementation is a sink or a source.
virtual ROOT::Internal::RPageRef LoadPage (ColumnHandle_t columnHandle, RNTupleLocalIndex localIndex)
 Another version of LoadPage that allows to specify cluster-relative indexes.
virtual ROOT::Internal::RPageRef LoadPage (ColumnHandle_t columnHandle, ROOT::NTupleSize_t globalIndex)
 Allocates and fills a page that contains the index-th element.
virtual void LoadSealedPage (ROOT::DescriptorId_t physicalColumnId, RNTupleLocalIndex localIndex, RSealedPage &sealedPage)=0
 Read the packed and compressed bytes of a page into the memory buffer provided by sealedPage.
virtual void LoadStreamerInfo ()=0
 Forces the loading of ROOT StreamerInfo from the underlying file.
void LoadStructure ()
 Loads header and footer without decompressing or deserializing them.
virtual std::unique_ptr< RPageSourceOpenWithDifferentAnchor (const ROOT::Internal::RNTupleLink &anchorLink, const ROOT::RNTupleReadOptions &options={})=0
 Creates a new PageSource using the same underlying file as this but referring to a different RNTuple, described by anchorLink.
RPageSourceoperator= (const RPageSource &)=delete
RPageSourceoperator= (RPageSource &&)=delete
void PinCluster (ROOT::DescriptorId_t clusterId)
 Instructs the cluster pool and page pool to consider the given cluster as active (should stay cached).
void RegisterStreamerInfos ()
 Builds the streamer info records from the descriptor's extra type info section.
void SetEntryRange (const REntryRange &range)
 Promise to only read from the given entry range.
void UnpinCluster (ROOT::DescriptorId_t clusterId)
 Allows the given cluster to be evicted from the cluster pool and page pool.
RResult< ROOT::Internal::RPageUnsealPage (const RSealedPage &sealedPage, const ROOT::Internal::RColumnElementBase &element)
void UnzipCluster (ROOT::Internal::RCluster *cluster)
 Populates all the pages of the given cluster ids and columns; it is possible that some columns do not contain any pages.

Static Public Member Functions

static std::unique_ptr< RPageSourceCreate (std::string_view ntupleName, std::string_view location, const ROOT::RNTupleReadOptions &options=ROOT::RNTupleReadOptions())
 Guess the concrete derived page source from the file name (location).
static RResult< ROOT::Internal::RPageUnsealPage (const RSealedPage &sealedPage, const ROOT::Internal::RColumnElementBase &element, ROOT::Internal::RPageAllocator &pageAlloc)
 Helper for unstreaming a page.

Protected Member Functions

virtual ROOT::RNTupleDescriptor AttachImpl (ROOT::Internal::RNTupleSerializer::EDescriptorDeserializeMode mode)=0
 LoadStructureImpl() has been called before AttachImpl() is called
virtual std::unique_ptr< RPageSourceCloneImpl () const=0
 Returns a new, unattached page source for the same data set.
void EnableDefaultMetrics (const std::string &prefix)
 Enables the default set of metrics provided by RPageSource.
RExclDescriptorGuard GetExclDescriptorGuard ()
 Note that the underlying lock is not recursive. See GetSharedDescriptorGuard() for further information.
virtual ROOT::Internal::RPageRef LoadPageImpl (ColumnHandle_t columnHandle, const RClusterInfo &clusterInfo, ROOT::NTupleSize_t idxInCluster)=0
virtual void LoadStructureImpl ()=0
void PrepareLoadCluster (const ROOT::Internal::RCluster::RKey &clusterKey, ROOT::Internal::ROnDiskPageMap &pageZeroMap, std::function< void(ROOT::DescriptorId_t, ROOT::NTupleSize_t, const ROOT::RClusterDescriptor::RPageInfo &)> perPageFunc)
 Prepare a page range read for the column set in clusterKey.
virtual void UnzipClusterImpl (ROOT::Internal::RCluster *cluster)

Protected Attributes

RActivePhysicalColumns fActivePhysicalColumns
 The active columns are implicitly defined by the model fields or views.
ROOT::Internal::RClusterPool fClusterPool
 The cluster pool asynchronously preloads the next few clusters.
std::unique_ptr< RCountersfCounters
ROOT::RNTupleReadOptions fOptions
ROOT::Internal::RPagePool fPagePool
 Pages that are unzipped with IMT are staged into the page pool.
std::unordered_set< ROOT::DescriptorId_tfPinnedClusters
 Pinned clusters and their $2 * (cluster bunch size) - 1$ successors will not be evicted from the cluster pool.

Private Member Functions

void UpdateLastUsedCluster (ROOT::DescriptorId_t clusterId)
 Does nothing if fLastUsedCluster == clusterId.

Private Attributes

ROOT::RNTupleDescriptor fDescriptor
std::shared_mutex fDescriptorLock
REntryRange fEntryRange
 Used by the cluster pool to prevent reading beyond the given range.
bool fHasStreamerInfosRegistered = false
 Set to true when RegisterStreamerInfos() is called.
bool fHasStructure = false
 Set to true once LoadStructure() is called.
bool fIsAttached = false
 Set to true once Attach() is called.
ROOT::DescriptorId_t fLastUsedCluster = ROOT::kInvalidDescriptorId
 Remembers the last cluster id from which a page was requested.
std::map< ROOT::NTupleSize_t, ROOT::DescriptorId_tfPreloadedClusters
 Clusters from where pages got preloaded in UnzipClusterImpl(), ordered by first entry number of the clusters.

#include <ROOT/RPageStorage.hxx>

Inheritance diagram for RPageSource:
ROOT::Internal::RPageStorage ROOT::Experimental::Internal::RPageSourceDaos ROOT::Internal::RPageSourceFile

Constructor & Destructor Documentation

◆ RPageSource() [1/3]

ROOT::Internal::RPageSource::RPageSource ( std::string_view ntupleName,
const ROOT::RNTupleReadOptions & fOptions )

Definition at line 166 of file RPageStorage.cxx.

◆ RPageSource() [2/3]

◆ RPageSource() [3/3]

◆ ~RPageSource()

Definition at line 174 of file RPageStorage.cxx.

Member Function Documentation

◆ AddColumn()

Register a new column.

When reading, the column must exist in the ntuple on disk corresponding to the metadata. When writing, every column can only be attached once.

Implements ROOT::Internal::RPageStorage.

Definition at line 197 of file RPageStorage.cxx.

◆ Attach()

◆ AttachImpl()

LoadStructureImpl() has been called before AttachImpl() is called

◆ Clone()

Open the same storage multiple time, e.g.

for reading in multiple threads. If the source is already attached, the clone will be attached, too. The clone will use, however, it's own connection to the underlying storage (e.g., file descriptor, XRootD handle, etc.)

Definition at line 235 of file RPageStorage.cxx.

◆ CloneImpl()

virtual std::unique_ptr< RPageSource > ROOT::Internal::RPageSource::CloneImpl ( ) const
protectedpure virtual

Returns a new, unattached page source for the same data set.

◆ Create()

std::unique_ptr< ROOT::Internal::RPageSource > ROOT::Internal::RPageSource::Create ( std::string_view ntupleName,
std::string_view location,
const ROOT::RNTupleReadOptions & options = ROOT::RNTupleReadOptions() )
static

Guess the concrete derived page source from the file name (location).

Definition at line 177 of file RPageStorage.cxx.

◆ DropColumn()

void ROOT::Internal::RPageSource::DropColumn ( ColumnHandle_t columnHandle)
overridevirtual

Unregisters a column.

A page source decreases the reference counter for the corresponding active column. For a page sink, dropping columns is currently a no-op.

Implements ROOT::Internal::RPageStorage.

Definition at line 207 of file RPageStorage.cxx.

◆ EnableDefaultMetrics()

void ROOT::Internal::RPageSource::EnableDefaultMetrics ( const std::string & prefix)
protected

Enables the default set of metrics provided by RPageSource.

prefix will be used as the prefix for the counters registered in the internal RNTupleMetrics object. A subclass using the default set of metrics is responsible for updating the counters appropriately, e.g. fCounters->fNRead.Inc() Alternatively, a subclass might provide its own RNTupleMetrics object by overriding the GetMetrics() member function.

Definition at line 471 of file RPageStorage.cxx.

◆ GetEntryRange()

Definition at line 805 of file RPageStorage.hxx.

◆ GetExclDescriptorGuard()

Note that the underlying lock is not recursive. See GetSharedDescriptorGuard() for further information.

Definition at line 750 of file RPageStorage.hxx.

◆ GetNElements()

◆ GetNEntries()

◆ GetPinnedClusters()

const std::unordered_set< ROOT::DescriptorId_t > & ROOT::Internal::RPageSource::GetPinnedClusters ( ) const
inline

Definition at line 843 of file RPageStorage.hxx.

◆ GetReadOptions()

Definition at line 775 of file RPageStorage.hxx.

◆ GetSharedDescriptorGuard()

Takes the read lock for the descriptor.

Multiple threads can take the lock concurrently. The underlying std::shared_mutex, however, is neither read nor write recursive: within one thread, only one lock (shared or exclusive) must be acquired at the same time. This requires special care in sections protected by GetSharedDescriptorGuard() and GetExclDescriptorGuard() especially to avoid that the locks are acquired indirectly. As a general guideline, no other method of the page source should be called (directly or indirectly) in a guarded section.

Definition at line 783 of file RPageStorage.hxx.

◆ GetType()

EPageStorageType ROOT::Internal::RPageSource::GetType ( )
inlinefinalvirtual

Whether the concrete implementation is a sink or a source.

Implements ROOT::Internal::RPageStorage.

Definition at line 774 of file RPageStorage.hxx.

◆ LoadPage() [1/2]

ROOT::Internal::RPageRef ROOT::Internal::RPageSource::LoadPage ( ColumnHandle_t columnHandle,
RNTupleLocalIndex localIndex )
virtual

Another version of LoadPage that allows to specify cluster-relative indexes.

Returns a default-constructed RPage for suppressed columns.

Definition at line 435 of file RPageStorage.cxx.

◆ LoadPage() [2/2]

Allocates and fills a page that contains the index-th element.

The default implementation searches the page and calls LoadPageImpl(). Returns a default-constructed RPage for suppressed columns.

Definition at line 396 of file RPageStorage.cxx.

◆ LoadPageImpl()

virtual ROOT::Internal::RPageRef ROOT::Internal::RPageSource::LoadPageImpl ( ColumnHandle_t columnHandle,
const RClusterInfo & clusterInfo,
ROOT::NTupleSize_t idxInCluster )
protectedpure virtual

◆ LoadSealedPage()

virtual void ROOT::Internal::RPageSource::LoadSealedPage ( ROOT::DescriptorId_t physicalColumnId,
RNTupleLocalIndex localIndex,
RSealedPage & sealedPage )
pure virtual

Read the packed and compressed bytes of a page into the memory buffer provided by sealedPage.

The sealed page can be used subsequently in a call to RPageSink::CommitSealedPage. The fSize and fNElements member of the sealedPage parameters are always set. If sealedPage.fBuffer is nullptr, no data will be copied but the returned size information can be used by the caller to allocate a large enough buffer and call LoadSealedPage again.

◆ LoadStreamerInfo()

virtual void ROOT::Internal::RPageSource::LoadStreamerInfo ( )
pure virtual

Forces the loading of ROOT StreamerInfo from the underlying file.

This currently only has an effect for TFile-backed sources.

◆ LoadStructure()

Loads header and footer without decompressing or deserializing them.

This can be used to asynchronously open a file in the background. The method is idempotent and it is called as a first step in Attach(). Pages sources may or may not make use of splitting loading and processing metadata. Therefore, LoadStructure() may do nothing and defer loading the metadata to Attach().

Definition at line 220 of file RPageStorage.cxx.

◆ LoadStructureImpl()

virtual void ROOT::Internal::RPageSource::LoadStructureImpl ( )
protectedpure virtual

◆ OpenWithDifferentAnchor()

virtual std::unique_ptr< RPageSource > ROOT::Internal::RPageSource::OpenWithDifferentAnchor ( const ROOT::Internal::RNTupleLink & anchorLink,
const ROOT::RNTupleReadOptions & options = {} )
pure virtual

Creates a new PageSource using the same underlying file as this but referring to a different RNTuple, described by anchorLink.

◆ operator=() [1/2]

RPageSource & ROOT::Internal::RPageSource::operator= ( const RPageSource & )
delete

◆ operator=() [2/2]

RPageSource & ROOT::Internal::RPageSource::operator= ( RPageSource && )
delete

◆ PinCluster()

Instructs the cluster pool and page pool to consider the given cluster as active (should stay cached).

Definition at line 840 of file RPageStorage.hxx.

◆ PrepareLoadCluster()

Prepare a page range read for the column set in clusterKey.

Specifically, pages referencing the kTypePageZero locator are filled in pageZeroMap; otherwise, perPageFunc is called for each page. This is commonly used as part of LoadClusters() in derived classes.

Definition at line 334 of file RPageStorage.cxx.

◆ RegisterStreamerInfos()

Builds the streamer info records from the descriptor's extra type info section.

This is necessary when connecting streamer fields so that emulated classes can be read.

Definition at line 608 of file RPageStorage.cxx.

◆ SetEntryRange()

Promise to only read from the given entry range.

If set, prevents the cluster pool from reading-ahead beyond the given range. The range needs to be within [0, GetNEntries()).

Definition at line 212 of file RPageStorage.cxx.

◆ UnpinCluster()

Allows the given cluster to be evicted from the cluster pool and page pool.

Definition at line 842 of file RPageStorage.hxx.

◆ UnsealPage() [1/2]

◆ UnsealPage() [2/2]

Helper for unstreaming a page.

This is commonly used in derived, concrete page sources. The implementation currently always makes a memory copy, even if the sealed page is uncompressed and in the final memory layout. The optimization of directly mapping pages is left to the concrete page source implementations.

Definition at line 569 of file RPageStorage.cxx.

◆ UnzipCluster()

Populates all the pages of the given cluster ids and columns; it is possible that some columns do not contain any pages.

The page source may load more columns than the minimal necessary set from columns. To indicate which columns have been loaded, LoadClusters()`` must mark them with SetColumnAvailable(). That includes the ones from the columns that don't have pages; otherwise subsequent requests for the cluster would assume an incomplete cluster and trigger loading again. LoadClusters()` is typically called from the I/O thread of a cluster pool, i.e. the method runs concurrently to other methods of the page source. */ virtual std::vector<std::unique_ptr<ROOT::Internal::RCluster>> LoadClusters(std::span<ROOT::Internal::RCluster::RKey> clusterKeys) = 0;

/** Parallel decompression and unpacking of the pages in the given cluster. The unzipped pages are supposed to be preloaded in a page pool attached to the source. The method is triggered by the cluster pool's unzip thread. It is an optional optimization, the method can safely do nothing. In particular, the actual implementation will only run if a task scheduler is set. In practice, a task scheduler is set if implicit multi-threading is turned on.

Definition at line 256 of file RPageStorage.cxx.

◆ UnzipClusterImpl()

Definition at line 262 of file RPageStorage.cxx.

◆ UpdateLastUsedCluster()

Does nothing if fLastUsedCluster == clusterId.

Otherwise, updated fLastUsedCluster and evict unused paged from the page pool of all previous clusters. Must not be called when the descriptor guard is taken.

Definition at line 361 of file RPageStorage.cxx.

Member Data Documentation

◆ fActivePhysicalColumns

The active columns are implicitly defined by the model fields or views.

Definition at line 710 of file RPageStorage.hxx.

◆ fClusterPool

The cluster pool asynchronously preloads the next few clusters.

Note that derived classes should call fClusterPool.StopBackgroundThread() in their destructor so that the I/O background thread does not call methods from the destructed derived class.

Definition at line 718 of file RPageStorage.hxx.

◆ fCounters

Definition at line 706 of file RPageStorage.hxx.

◆ fDescriptor

◆ fDescriptorLock

std::shared_mutex ROOT::Internal::RPageSource::fDescriptorLock
mutableprivate

Definition at line 625 of file RPageStorage.hxx.

◆ fEntryRange

Used by the cluster pool to prevent reading beyond the given range.

Definition at line 626 of file RPageStorage.hxx.

◆ fHasStreamerInfosRegistered

Set to true when RegisterStreamerInfos() is called.

Definition at line 629 of file RPageStorage.hxx.

◆ fHasStructure

Set to true once LoadStructure() is called.

Definition at line 627 of file RPageStorage.hxx.

◆ fIsAttached

Set to true once Attach() is called.

Definition at line 628 of file RPageStorage.hxx.

◆ fLastUsedCluster

Remembers the last cluster id from which a page was requested.

Definition at line 632 of file RPageStorage.hxx.

◆ fOptions

◆ fPagePool

Pages that are unzipped with IMT are staged into the page pool.

Definition at line 720 of file RPageStorage.hxx.

◆ fPinnedClusters

Pinned clusters and their $2 * (cluster bunch size) - 1$ successors will not be evicted from the cluster pool.

Pages of pinned clusters won't be evicted from the page pool.

Definition at line 713 of file RPageStorage.hxx.

◆ fPreloadedClusters

Clusters from where pages got preloaded in UnzipClusterImpl(), ordered by first entry number of the clusters.

If the last used cluster changes in LoadPage(), all unused pages from previous clusters are evicted from the page pool. Pinned clusters won't be evicted.

Definition at line 636 of file RPageStorage.hxx.


The documentation for this class was generated from the following files: