Logo ROOT  
Reference Guide
 
All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Properties Friends Macros Modules Pages
Loading...
Searching...
No Matches
ROOT::Experimental::Internal::RPagePersistentSink Class Referenceabstract

Base class for a sink with a physical storage backend.

Definition at line 441 of file RPageStorage.hxx.

Classes

struct  RCounters
 Default I/O performance counters that get registered in fMetrics. More...
 
struct  RFeatures
 Set of optional features supported by the persistent sink. More...
 

Public Member Functions

 RPagePersistentSink (const RPagePersistentSink &)=delete
 
 RPagePersistentSink (RPagePersistentSink &&)=default
 
 RPagePersistentSink (std::string_view ntupleName, const ROOT::RNTupleWriteOptions &options)
 
 ~RPagePersistentSink () override
 
ColumnHandle_t AddColumn (ROOT::DescriptorId_t fieldId, ROOT::Internal::RColumn &column) final
 Register a new column.
 
void CommitClusterGroup () final
 Write out the page locations (page list envelope) for all the committed clusters since the last call of CommitClusterGroup (or the beginning of writing).
 
void CommitDatasetImpl () final
 
void CommitPage (ColumnHandle_t columnHandle, const ROOT::Internal::RPage &page) final
 Write a page to the storage. The column must have been added before.
 
void CommitSealedPage (ROOT::DescriptorId_t physicalColumnId, const RPageStorage::RSealedPage &sealedPage) final
 Write a preprocessed page to storage. The column must have been added before.
 
void CommitSealedPageV (std::span< RPageStorage::RSealedPageGroup > ranges) final
 Write a vector of preprocessed pages to storage. The corresponding columns must have been added before.
 
void CommitStagedClusters (std::span< RStagedCluster > clusters) final
 Commit staged clusters, logically appending them to the ntuple descriptor.
 
void CommitSuppressedColumn (ColumnHandle_t columnHandle) final
 Commits a suppressed column for the current cluster.
 
const ROOT::RNTupleDescriptorGetDescriptor () const final
 Return the RNTupleDescriptor being constructed.
 
ROOT::NTupleSize_t GetNEntries () const final
 
std::unique_ptr< RNTupleModelInitFromDescriptor (const ROOT::RNTupleDescriptor &descriptor, bool copyClusters)
 Initialize sink based on an existing descriptor and fill into the descriptor builder, optionally copying over the descriptor's clusters to this sink's descriptor.
 
void InitImpl (RNTupleModel &model) final
 Updates the descriptor and calls InitImpl() that handles the backend-specific details (file, DAOS, etc.)
 
RPagePersistentSinkoperator= (const RPagePersistentSink &)=delete
 
RPagePersistentSinkoperator= (RPagePersistentSink &&)=default
 
RStagedCluster StageCluster (ROOT::NTupleSize_t nNewEntries) final
 Stage the current cluster and create a new one for the following data.
 
void UpdateExtraTypeInfo (const ROOT::RExtraTypeInfoDescriptor &extraTypeInfo) final
 Adds an extra type information record to schema.
 
void UpdateSchema (const ROOT::Internal::RNTupleModelChangeset &changeset, ROOT::NTupleSize_t firstEntry) final
 Incorporate incremental changes to the model into the ntuple descriptor.
 
- Public Member Functions inherited from ROOT::Experimental::Internal::RPageSink
 RPageSink (const RPageSink &)=delete
 
 RPageSink (RPageSink &&)=default
 
 RPageSink (std::string_view ntupleName, const ROOT::RNTupleWriteOptions &options)
 
 ~RPageSink () override
 
virtual std::uint64_t CommitCluster (ROOT::NTupleSize_t nNewEntries)
 Finalize the current cluster and create a new one for the following data.
 
void CommitDataset ()
 Run the registered callbacks and finalize the current cluster and the entrire data set.
 
void DropColumn (ColumnHandle_t) final
 Unregisters a column.
 
virtual RSinkGuard GetSinkGuard ()
 
EPageStorageType GetType () final
 Whether the concrete implementation is a sink or a source.
 
const ROOT::RNTupleWriteOptionsGetWriteOptions () const
 Returns the sink's write options.
 
void Init (RNTupleModel &model)
 Physically creates the storage container to hold the ntuple (e.g., a keys a TFile or an S3 bucket) Init() associates column handles to the columns referenced by the model.
 
bool IsInitialized () const
 
RPageSinkoperator= (const RPageSink &)=delete
 
RPageSinkoperator= (RPageSink &&)=default
 
void RegisterOnCommitDatasetCallback (Callback_t callback)
 The registered callback is executed at the beginning of CommitDataset();.
 
virtual ROOT::Internal::RPage ReservePage (ColumnHandle_t columnHandle, std::size_t nElements)
 Get a new, empty page for the given column that can be filled with up to nElements; nElements must be larger than zero.
 
- Public Member Functions inherited from ROOT::Experimental::Internal::RPageStorage
 RPageStorage (const RPageStorage &other)=delete
 
 RPageStorage (RPageStorage &&other)=default
 
 RPageStorage (std::string_view name)
 
virtual ~RPageStorage ()
 
ROOT::DescriptorId_t GetColumnId (ColumnHandle_t columnHandle) const
 
virtual Detail::RNTupleMetricsGetMetrics ()
 Returns the default metrics object.
 
const std::string & GetNTupleName () const
 Returns the NTuple name.
 
RPageStorageoperator= (const RPageStorage &other)=delete
 
RPageStorageoperator= (RPageStorage &&other)=default
 
void SetTaskScheduler (RTaskScheduler *taskScheduler)
 

Static Public Member Functions

static std::unique_ptr< RPageSinkCreate (std::string_view ntupleName, std::string_view location, const ROOT::RNTupleWriteOptions &options=ROOT::RNTupleWriteOptions())
 Guess the concrete derived page source from the location.
 
- Static Public Member Functions inherited from ROOT::Experimental::Internal::RPageSink
static RSealedPage SealPage (const RSealPageConfig &config)
 Seal a page using the provided info.
 

Protected Member Functions

virtual RNTupleLocator CommitClusterGroupImpl (unsigned char *serializedPageList, std::uint32_t length)=0
 Returns the locator of the page list envelope of the given buffer that contains the serialized page list.
 
virtual void CommitDatasetImpl (unsigned char *serializedFooter, std::uint32_t length)=0
 
virtual RNTupleLocator CommitPageImpl (ColumnHandle_t columnHandle, const ROOT::Internal::RPage &page)=0
 
virtual RNTupleLocator CommitSealedPageImpl (ROOT::DescriptorId_t physicalColumnId, const RPageStorage::RSealedPage &sealedPage)=0
 
virtual std::vector< RNTupleLocatorCommitSealedPageVImpl (std::span< RPageStorage::RSealedPageGroup > ranges, const std::vector< bool > &mask)
 Vector commit of preprocessed pages.
 
void EnableDefaultMetrics (const std::string &prefix)
 Enables the default set of metrics provided by RPageSink.
 
virtual void InitImpl (unsigned char *serializedHeader, std::uint32_t length)=0
 
virtual std::uint64_t StageClusterImpl ()=0
 Returns the number of bytes written to storage (excluding metadata)
 
- Protected Member Functions inherited from ROOT::Experimental::Internal::RPageSink
RSealedPage SealPage (const ROOT::Internal::RPage &page, const ROOT::Internal::RColumnElementBase &element)
 Helper for streaming a page.
 
- Protected Member Functions inherited from ROOT::Experimental::Internal::RPageStorage
void WaitForAllTasks ()
 

Protected Attributes

std::unique_ptr< RCountersfCounters
 
ROOT::Internal::RNTupleDescriptorBuilder fDescriptorBuilder
 
RFeatures fFeatures
 
- Protected Attributes inherited from ROOT::Experimental::Internal::RPageSink
bool fIsInitialized = false
 Flag if sink was initialized.
 
std::unique_ptr< ROOT::RNTupleWriteOptionsfOptions
 
- Protected Attributes inherited from ROOT::Experimental::Internal::RPageStorage
Detail::RNTupleMetrics fMetrics
 
std::string fNTupleName
 
std::unique_ptr< ROOT::Internal::RPageAllocatorfPageAllocator
 For the time being, we will use the heap allocator for all sources and sinks. This may change in the future.
 
RTaskSchedulerfTaskScheduler = nullptr
 

Private Attributes

std::uint64_t fNextClusterInGroup = 0
 Remembers the starting cluster id for the next cluster group.
 
std::vector< ROOT::RClusterDescriptor::RColumnRangefOpenColumnRanges
 Keeps track of the number of elements in the currently open cluster. Indexed by column id.
 
std::vector< ROOT::RClusterDescriptor::RPageRangefOpenPageRanges
 Keeps track of the written pages in the currently open cluster. Indexed by column id.
 
ROOT::NTupleSize_t fPrevClusterNEntries = 0
 Used to calculate the number of entries in the current cluster.
 
RNTupleSerializer::RContext fSerializationContext
 Used to map the IDs of the descriptor to the physical IDs issued during header/footer serialization.
 
RNTupleSerializer::StreamerInfoMap_t fStreamerInfos
 Union of the streamer info records that are sent from streamer fields to the sink before committing the dataset.
 

Additional Inherited Members

- Public Types inherited from ROOT::Experimental::Internal::RPageSink
using Callback_t = std::function<void(RPageSink &)>
 
- Public Types inherited from ROOT::Experimental::Internal::RPageStorage
using ColumnHandle_t = RColumnHandle
 The column handle identifies a column with the current open page storage.
 
using SealedPageSequence_t = std::deque<RSealedPage>
 
- Static Public Attributes inherited from ROOT::Experimental::Internal::RPageStorage
static constexpr std::size_t kNBytesPageChecksum = sizeof(std::uint64_t)
 The page checksum is a 64bit xxhash3.
 

#include <ROOT/RPageStorage.hxx>

Inheritance diagram for ROOT::Experimental::Internal::RPagePersistentSink:
[legend]

Constructor & Destructor Documentation

◆ RPagePersistentSink() [1/3]

ROOT::Experimental::Internal::RPagePersistentSink::RPagePersistentSink ( std::string_view ntupleName,
const ROOT::RNTupleWriteOptions & options )

Definition at line 784 of file RPageStorage.cxx.

◆ RPagePersistentSink() [2/3]

ROOT::Experimental::Internal::RPagePersistentSink::RPagePersistentSink ( const RPagePersistentSink & )
delete

◆ RPagePersistentSink() [3/3]

ROOT::Experimental::Internal::RPagePersistentSink::RPagePersistentSink ( RPagePersistentSink && )
default

◆ ~RPagePersistentSink()

ROOT::Experimental::Internal::RPagePersistentSink::~RPagePersistentSink ( )
override

Definition at line 790 of file RPageStorage.cxx.

Member Function Documentation

◆ AddColumn()

ROOT::Experimental::Internal::RPageStorage::ColumnHandle_t ROOT::Experimental::Internal::RPagePersistentSink::AddColumn ( ROOT::DescriptorId_t fieldId,
ROOT::Internal::RColumn & column )
finalvirtual

Register a new column.

When reading, the column must exist in the ntuple on disk corresponding to the metadata. When writing, every column can only be attached once.

Implements ROOT::Experimental::Internal::RPageStorage.

Definition at line 793 of file RPageStorage.cxx.

◆ CommitClusterGroup()

void ROOT::Experimental::Internal::RPagePersistentSink::CommitClusterGroup ( )
finalvirtual

Write out the page locations (page list envelope) for all the committed clusters since the last call of CommitClusterGroup (or the beginning of writing).

Implements ROOT::Experimental::Internal::RPageSink.

Definition at line 1194 of file RPageStorage.cxx.

◆ CommitClusterGroupImpl()

virtual RNTupleLocator ROOT::Experimental::Internal::RPagePersistentSink::CommitClusterGroupImpl ( unsigned char * serializedPageList,
std::uint32_t length )
protectedpure virtual

Returns the locator of the page list envelope of the given buffer that contains the serialized page list.

Typically, the implementation takes care of compressing and writing the provided buffer.

Implemented in ROOT::Experimental::Internal::RPageSinkDaos, and ROOT::Experimental::Internal::RPageSinkFile.

◆ CommitDatasetImpl() [1/2]

void ROOT::Experimental::Internal::RPagePersistentSink::CommitDatasetImpl ( )
finalvirtual

◆ CommitDatasetImpl() [2/2]

virtual void ROOT::Experimental::Internal::RPagePersistentSink::CommitDatasetImpl ( unsigned char * serializedFooter,
std::uint32_t length )
protectedpure virtual

◆ CommitPage()

void ROOT::Experimental::Internal::RPagePersistentSink::CommitPage ( ColumnHandle_t columnHandle,
const ROOT::Internal::RPage & page )
finalvirtual

Write a page to the storage. The column must have been added before.

Implements ROOT::Experimental::Internal::RPageSink.

Definition at line 1017 of file RPageStorage.cxx.

◆ CommitPageImpl()

virtual RNTupleLocator ROOT::Experimental::Internal::RPagePersistentSink::CommitPageImpl ( ColumnHandle_t columnHandle,
const ROOT::Internal::RPage & page )
protectedpure virtual

◆ CommitSealedPage()

void ROOT::Experimental::Internal::RPagePersistentSink::CommitSealedPage ( ROOT::DescriptorId_t physicalColumnId,
const RPageStorage::RSealedPage & sealedPage )
finalvirtual

Write a preprocessed page to storage. The column must have been added before.

Implements ROOT::Experimental::Internal::RPageSink.

Definition at line 1029 of file RPageStorage.cxx.

◆ CommitSealedPageImpl()

virtual RNTupleLocator ROOT::Experimental::Internal::RPagePersistentSink::CommitSealedPageImpl ( ROOT::DescriptorId_t physicalColumnId,
const RPageStorage::RSealedPage & sealedPage )
protectedpure virtual

◆ CommitSealedPageV()

void ROOT::Experimental::Internal::RPagePersistentSink::CommitSealedPageV ( std::span< RPageStorage::RSealedPageGroup > ranges)
finalvirtual

Write a vector of preprocessed pages to storage. The corresponding columns must have been added before.

Implements ROOT::Experimental::Internal::RPageSink.

Definition at line 1057 of file RPageStorage.cxx.

◆ CommitSealedPageVImpl()

std::vector< ROOT::RNTupleLocator > ROOT::Experimental::Internal::RPagePersistentSink::CommitSealedPageVImpl ( std::span< RPageStorage::RSealedPageGroup > ranges,
const std::vector< bool > & mask )
protectedvirtual

Vector commit of preprocessed pages.

The ranges array specifies a range of sealed pages to be committed for each column. The returned vector contains, in order, the RNTupleLocator for each page on each range in ranges, i.e. the first N entries refer to the N pages in ranges[0], followed by M entries that refer to the M pages in ranges[1], etc. The mask allows to skip writing out certain pages. The vector has the size of all the pages. For every false value in the mask, the corresponding locator is skipped (missing) in the output vector. The default is to call CommitSealedPageImpl for each page; derived classes may provide an optimized implementation though.

Reimplemented in ROOT::Experimental::Internal::RPageSinkDaos, and ROOT::Experimental::Internal::RPageSinkFile.

Definition at line 1041 of file RPageStorage.cxx.

◆ CommitStagedClusters()

void ROOT::Experimental::Internal::RPagePersistentSink::CommitStagedClusters ( std::span< RStagedCluster > clusters)
finalvirtual

Commit staged clusters, logically appending them to the ntuple descriptor.

Implements ROOT::Experimental::Internal::RPageSink.

Definition at line 1157 of file RPageStorage.cxx.

◆ CommitSuppressedColumn()

void ROOT::Experimental::Internal::RPagePersistentSink::CommitSuppressedColumn ( ColumnHandle_t columnHandle)
finalvirtual

Commits a suppressed column for the current cluster.

Can be called anytime before CommitCluster(). For any given column and cluster, there must be no calls to both CommitSuppressedColumn() and page commits.

Implements ROOT::Experimental::Internal::RPageSink.

Definition at line 1012 of file RPageStorage.cxx.

◆ Create()

std::unique_ptr< ROOT::Experimental::Internal::RPageSink > ROOT::Experimental::Internal::RPagePersistentSink::Create ( std::string_view ntupleName,
std::string_view location,
const ROOT::RNTupleWriteOptions & options = ROOT::RNTupleWriteOptions() )
static

Guess the concrete derived page source from the location.

Definition at line 763 of file RPageStorage.cxx.

◆ EnableDefaultMetrics()

void ROOT::Experimental::Internal::RPagePersistentSink::EnableDefaultMetrics ( const std::string & prefix)
protected

Enables the default set of metrics provided by RPageSink.

prefix will be used as the prefix for the counters registered in the internal RNTupleMetrics object. This set of counters can be extended by a subclass by calling fMetrics.MakeCounter<...>().

A subclass using the default set of metrics is always responsible for updating the counters appropriately, e.g. fCounters->fNPageCommited.Inc()

Definition at line 1267 of file RPageStorage.cxx.

◆ GetDescriptor()

const ROOT::RNTupleDescriptor & ROOT::Experimental::Internal::RPagePersistentSink::GetDescriptor ( ) const
inlinefinalvirtual

Return the RNTupleDescriptor being constructed.

Implements ROOT::Experimental::Internal::RPageSink.

Definition at line 524 of file RPageStorage.hxx.

◆ GetNEntries()

ROOT::NTupleSize_t ROOT::Experimental::Internal::RPagePersistentSink::GetNEntries ( ) const
inlinefinalvirtual

Implements ROOT::Experimental::Internal::RPageSink.

Definition at line 526 of file RPageStorage.hxx.

◆ InitFromDescriptor()

std::unique_ptr< ROOT::RNTupleModel > ROOT::Experimental::Internal::RPagePersistentSink::InitFromDescriptor ( const ROOT::RNTupleDescriptor & descriptor,
bool copyClusters )

Initialize sink based on an existing descriptor and fill into the descriptor builder, optionally copying over the descriptor's clusters to this sink's descriptor.

Returns
The model created from the new sink's descriptor. This model should be kept alive for at least as long as the sink.

Definition at line 942 of file RPageStorage.cxx.

◆ InitImpl() [1/2]

void ROOT::Experimental::Internal::RPagePersistentSink::InitImpl ( ROOT::RNTupleModel & model)
finalvirtual

Updates the descriptor and calls InitImpl() that handles the backend-specific details (file, DAOS, etc.)

Implements ROOT::Experimental::Internal::RPageSink.

Reimplemented in ROOT::Experimental::Internal::RPageSinkDaos, and ROOT::Experimental::Internal::RPageSinkFile.

Definition at line 913 of file RPageStorage.cxx.

◆ InitImpl() [2/2]

virtual void ROOT::Experimental::Internal::RPagePersistentSink::InitImpl ( unsigned char * serializedHeader,
std::uint32_t length )
protectedpure virtual

◆ operator=() [1/2]

RPagePersistentSink & ROOT::Experimental::Internal::RPagePersistentSink::operator= ( const RPagePersistentSink & )
delete

◆ operator=() [2/2]

RPagePersistentSink & ROOT::Experimental::Internal::RPagePersistentSink::operator= ( RPagePersistentSink && )
default

◆ StageCluster()

ROOT::Experimental::Internal::RPageSink::RStagedCluster ROOT::Experimental::Internal::RPagePersistentSink::StageCluster ( ROOT::NTupleSize_t nNewEntries)
finalvirtual

Stage the current cluster and create a new one for the following data.

Returns the object that must be passed to CommitStagedClusters to logically append the staged cluster to the ntuple descriptor.

Implements ROOT::Experimental::Internal::RPageSink.

Definition at line 1128 of file RPageStorage.cxx.

◆ StageClusterImpl()

virtual std::uint64_t ROOT::Experimental::Internal::RPagePersistentSink::StageClusterImpl ( )
protectedpure virtual

Returns the number of bytes written to storage (excluding metadata)

Implemented in ROOT::Experimental::Internal::RPageSinkDaos, and ROOT::Experimental::Internal::RPageSinkFile.

◆ UpdateExtraTypeInfo()

void ROOT::Experimental::Internal::RPagePersistentSink::UpdateExtraTypeInfo ( const ROOT::RExtraTypeInfoDescriptor & extraTypeInfo)
finalvirtual

Adds an extra type information record to schema.

The extra type information will be written to the extension header. The information in the record will be merged with the existing information, e.g. duplicate streamer info records will be removed. This method is called by the "on commit dataset" callback registered by specific fields (e.g., streamer field) and during merging.

Implements ROOT::Experimental::Internal::RPageSink.

Definition at line 904 of file RPageStorage.cxx.

◆ UpdateSchema()

void ROOT::Experimental::Internal::RPagePersistentSink::UpdateSchema ( const ROOT::Internal::RNTupleModelChangeset & changeset,
ROOT::NTupleSize_t firstEntry )
finalvirtual

Incorporate incremental changes to the model into the ntuple descriptor.

This happens, e.g. if new fields were added after the initial call to RPageSink::Init(RNTupleModel &). firstEntry specifies the global index for the first stored element in the added columns.

Implements ROOT::Experimental::Internal::RPageSink.

Definition at line 814 of file RPageStorage.cxx.

Member Data Documentation

◆ fCounters

std::unique_ptr<RCounters> ROOT::Experimental::Internal::RPagePersistentSink::fCounters
protected

Definition at line 477 of file RPageStorage.hxx.

◆ fDescriptorBuilder

ROOT::Internal::RNTupleDescriptorBuilder ROOT::Experimental::Internal::RPagePersistentSink::fDescriptorBuilder
protected

Definition at line 465 of file RPageStorage.hxx.

◆ fFeatures

RFeatures ROOT::Experimental::Internal::RPagePersistentSink::fFeatures
protected

Definition at line 464 of file RPageStorage.hxx.

◆ fNextClusterInGroup

std::uint64_t ROOT::Experimental::Internal::RPagePersistentSink::fNextClusterInGroup = 0
private

Remembers the starting cluster id for the next cluster group.

Definition at line 447 of file RPageStorage.hxx.

◆ fOpenColumnRanges

std::vector<ROOT::RClusterDescriptor::RColumnRange> ROOT::Experimental::Internal::RPagePersistentSink::fOpenColumnRanges
private

Keeps track of the number of elements in the currently open cluster. Indexed by column id.

Definition at line 451 of file RPageStorage.hxx.

◆ fOpenPageRanges

std::vector<ROOT::RClusterDescriptor::RPageRange> ROOT::Experimental::Internal::RPagePersistentSink::fOpenPageRanges
private

Keeps track of the written pages in the currently open cluster. Indexed by column id.

Definition at line 453 of file RPageStorage.hxx.

◆ fPrevClusterNEntries

ROOT::NTupleSize_t ROOT::Experimental::Internal::RPagePersistentSink::fPrevClusterNEntries = 0
private

Used to calculate the number of entries in the current cluster.

Definition at line 449 of file RPageStorage.hxx.

◆ fSerializationContext

RNTupleSerializer::RContext ROOT::Experimental::Internal::RPagePersistentSink::fSerializationContext
private

Used to map the IDs of the descriptor to the physical IDs issued during header/footer serialization.

Definition at line 444 of file RPageStorage.hxx.

◆ fStreamerInfos

RNTupleSerializer::StreamerInfoMap_t ROOT::Experimental::Internal::RPagePersistentSink::fStreamerInfos
private

Union of the streamer info records that are sent from streamer fields to the sink before committing the dataset.

Definition at line 456 of file RPageStorage.hxx.

Libraries for ROOT::Experimental::Internal::RPagePersistentSink:

The documentation for this class was generated from the following files: