Wrapper sink that coalesces cluster column page writes.
Definition at line 39 of file RPageSinkBuf.hxx.
Classes | |
| class | RColumnBuf |
| A buffered column. More... | |
| struct | RCounters |
| I/O performance counters that get registered in fMetrics. More... | |
Public Types | |
| using | Callback_t = std::function<void(RPageSink &)> |
| using | ColumnHandle_t = RColumnHandle |
| The column handle identifies a column with the current open page storage. | |
| using | SealedPageSequence_t = std::deque<RSealedPage> |
Public Member Functions | |
| RPageSinkBuf (const RPageSinkBuf &)=delete | |
| RPageSinkBuf (RPageSinkBuf &&)=delete | |
| RPageSinkBuf (std::unique_ptr< RPageSink > inner) | |
| ~RPageSinkBuf () override | |
| ColumnHandle_t | AddColumn (ROOT::DescriptorId_t fieldId, RColumn &column) final |
| Register a new column. | |
| std::unique_ptr< RPageSink > | CloneAsHidden (std::string_view name, const RNTupleWriteOptions &opts) const final |
| Creates a new sink with the same underlying storage as this but writing to a different RNTuple named name. | |
| void | CommitAttributeSet (std::string_view attrSetName, const RNTupleLink &attrAnchorInfo) final |
| Adds the given anchor information (name + locator) into the main RNTuple's descriptor as an attribute set linked to it with the given name. | |
| std::uint64_t | CommitCluster (ROOT::NTupleSize_t nNewEntries) final |
| Finalize the current cluster and create a new one for the following data. | |
| void | CommitClusterGroup () final |
| Write out the page locations (page list envelope) for all the committed clusters since the last call of CommitClusterGroup (or the beginning of writing). | |
| RNTupleLink | CommitDataset () |
| Run the registered callbacks and finalize the current cluster and the entrire data set. | |
| RNTupleLink | CommitDatasetImpl () final |
| void | CommitPage (ColumnHandle_t columnHandle, const RPage &page) final |
| Write a page to the storage. The column must have been added before. | |
| void | CommitSealedPage (ROOT::DescriptorId_t physicalColumnId, const RSealedPage &sealedPage) final |
| Write a preprocessed page to storage. The column must have been added before. | |
| void | CommitSealedPageV (std::span< RPageStorage::RSealedPageGroup > ranges) final |
| Write a vector of preprocessed pages to storage. The corresponding columns must have been added before. | |
| void | CommitStagedClusters (std::span< RStagedCluster > clusters) final |
| Commit staged clusters, logically appending them to the ntuple descriptor. | |
| void | CommitSuppressedColumn (ColumnHandle_t columnHandle) final |
| Commits a suppressed column for the current cluster. | |
| void | DropColumn (ColumnHandle_t) final |
| Unregisters a column. | |
| ROOT::DescriptorId_t | GetColumnId (ColumnHandle_t columnHandle) const |
| const ROOT::RNTupleDescriptor & | GetDescriptor () const final |
| Return the RNTupleDescriptor being constructed. | |
| virtual ROOT::Experimental::Detail::RNTupleMetrics & | GetMetrics () |
| Returns the default metrics object. | |
| ROOT::NTupleSize_t | GetNEntries () const final |
| const std::string & | GetNTupleName () const |
| Returns the NTuple name. | |
| virtual RSinkGuard | GetSinkGuard () |
| EPageStorageType | GetType () final |
| Whether the concrete implementation is a sink or a source. | |
| const ROOT::RNTupleWriteOptions & | GetWriteOptions () const |
| Returns the sink's write options. | |
| void | Init (RNTupleModel &model) |
| Physically creates the storage container to hold the ntuple (e.g., a keys a TFile or an S3 bucket) Init() associates column handles to the columns referenced by the model. | |
| void | InitImpl (ROOT::RNTupleModel &model) final |
| bool | IsInitialized () const |
| RPageSinkBuf & | operator= (const RPageSinkBuf &)=delete |
| RPageSinkBuf & | operator= (RPageSinkBuf &&)=delete |
| void | RegisterOnCommitDatasetCallback (Callback_t callback) |
| The registered callback is executed at the beginning of CommitDataset();. | |
| RPage | ReservePage (ColumnHandle_t columnHandle, std::size_t nElements) final |
| Get a new, empty page for the given column that can be filled with up to nElements; nElements must be larger than zero. | |
| void | SetTaskScheduler (RTaskScheduler *taskScheduler) |
| RStagedCluster | StageCluster (ROOT::NTupleSize_t nNewEntries) final |
| Stage the current cluster and create a new one for the following data. | |
| void | UpdateExtraTypeInfo (const ROOT::RExtraTypeInfoDescriptor &extraTypeInfo) final |
| Adds an extra type information record to schema. | |
| void | UpdateSchema (const RNTupleModelChangeset &changeset, ROOT::NTupleSize_t firstEntry) final |
| Incorporate incremental changes to the model into the ntuple descriptor. | |
Static Public Member Functions | |
| static RSealedPage | SealPage (const RSealPageConfig &config) |
| Seal a page using the provided info. | |
Static Public Attributes | |
| static constexpr std::size_t | kNBytesPageChecksum = sizeof(std::uint64_t) |
| The page checksum is a 64bit xxhash3. | |
Protected Member Functions | |
| RSealedPage | SealPage (const ROOT::Internal::RPage &page, const ROOT::Internal::RColumnElementBase &element) |
| Helper for streaming a page. | |
| void | WaitForAllTasks () |
Protected Attributes | |
| bool | fIsInitialized = false |
| Flag if sink was initialized. | |
| ROOT::Experimental::Detail::RNTupleMetrics | fMetrics |
| std::string | fNTupleName |
| std::unique_ptr< ROOT::RNTupleWriteOptions > | fOptions |
| std::unique_ptr< ROOT::Internal::RPageAllocator > | fPageAllocator |
| For the time being, we will use the heap allocator for all sources and sinks. This may change in the future. | |
| RTaskScheduler * | fTaskScheduler = nullptr |
Private Member Functions | |
| void | ConnectFields (const std::vector< ROOT::RFieldBase * > &fields, ROOT::NTupleSize_t firstEntry) |
| void | FlushClusterImpl (std::function< void(void)> FlushClusterFn) |
Private Attributes | |
| std::vector< RColumnBuf > | fBufferedColumns |
| Vector of buffered column pages. Indexed by column id. | |
| std::atomic< std::size_t > | fBufferedUncompressed = 0 |
| The sum of uncompressed bytes in buffered pages. Used to heuristically reduce the memory usage. | |
| std::unique_ptr< RCounters > | fCounters |
| std::unique_ptr< ROOT::RNTupleModel > | fInnerModel |
| The buffered page sink maintains a copy of the RNTupleModel for the inner sink. | |
| std::unique_ptr< RPageSink > | fInnerSink |
| The inner sink, responsible for actually performing I/O. | |
| ROOT::DescriptorId_t | fNColumns = 0 |
| ROOT::DescriptorId_t | fNFields = 0 |
| std::vector< Callback_t > | fOnDatasetCommitCallbacks |
| std::vector< unsigned char > | fSealPageBuffer |
| Used as destination buffer in the simple SealPage overload. | |
| std::vector< ColumnHandle_t > | fSuppressedColumns |
| Columns committed as suppressed are stored and passed to the inner sink at cluster commit. | |
| RWritePageMemoryManager | fWritePageMemoryManager |
| Used in ReservePage to maintain the page buffer budget. | |
#include <ROOT/RPageSinkBuf.hxx>
|
inherited |
Definition at line 258 of file RPageStorage.hxx.
|
inherited |
The column handle identifies a column with the current open page storage.
Definition at line 180 of file RPageStorage.hxx.
|
inherited |
Definition at line 130 of file RPageStorage.hxx.
|
explicit |
Definition at line 40 of file RPageSinkBuf.cxx.
|
delete |
|
delete |
|
override |
Definition at line 56 of file RPageSinkBuf.cxx.
|
finalvirtual |
Register a new column.
When reading, the column must exist in the ntuple on disk corresponding to the metadata. When writing, every column can only be attached once.
Implements ROOT::Internal::RPageStorage.
Definition at line 65 of file RPageSinkBuf.cxx.
|
finalvirtual |
Creates a new sink with the same underlying storage as this but writing to a different RNTuple named name.
Only one of the two sinks can safely write at the same time. The RNTuple written by this cloned sink will be stored in a hidden key (this is a convenient assumption we make now since this method is only used to create attribute RNTuples).
Implements ROOT::Internal::RPageSink.
Definition at line 325 of file RPageSinkBuf.cxx.
|
finalvirtual |
Adds the given anchor information (name + locator) into the main RNTuple's descriptor as an attribute set linked to it with the given name.
The attribute set must have already been written to storage via RNTupleAttrSetWriter::Commit(). Note that, by RNTuple specs, this is only legal to call on a non-attribute RNTuple's sink.
Implements ROOT::Internal::RPageSink.
Definition at line 330 of file RPageSinkBuf.cxx.
|
finalvirtual |
Finalize the current cluster and create a new one for the following data.
Returns the number of bytes written to storage (excluding metadata).
Reimplemented from ROOT::Internal::RPageSink.
Definition at line 284 of file RPageSinkBuf.cxx.
|
finalvirtual |
Write out the page locations (page list envelope) for all the committed clusters since the last call of CommitClusterGroup (or the beginning of writing).
Implements ROOT::Internal::RPageSink.
Definition at line 305 of file RPageSinkBuf.cxx.
|
inherited |
Run the registered callbacks and finalize the current cluster and the entrire data set.
Definition at line 774 of file RPageStorage.cxx.
|
finalvirtual |
Implements ROOT::Internal::RPageSink.
Definition at line 312 of file RPageSinkBuf.cxx.
|
finalvirtual |
Write a page to the storage. The column must have been added before.
Implements ROOT::Internal::RPageSink.
Definition at line 149 of file RPageSinkBuf.cxx.
|
finalvirtual |
Write a preprocessed page to storage. The column must have been added before.
Implements ROOT::Internal::RPageSink.
Definition at line 240 of file RPageSinkBuf.cxx.
|
finalvirtual |
Write a vector of preprocessed pages to storage. The corresponding columns must have been added before.
Implements ROOT::Internal::RPageSink.
Definition at line 246 of file RPageSinkBuf.cxx.
|
finalvirtual |
Commit staged clusters, logically appending them to the ntuple descriptor.
Implements ROOT::Internal::RPageSink.
Definition at line 298 of file RPageSinkBuf.cxx.
|
finalvirtual |
Commits a suppressed column for the current cluster.
Can be called anytime before CommitCluster(). For any given column and cluster, there must be no calls to both CommitSuppressedColumn() and page commits.
Implements ROOT::Internal::RPageSink.
Definition at line 144 of file RPageSinkBuf.cxx.
|
private |
Definition at line 70 of file RPageSinkBuf.cxx.
|
inlinefinalvirtualinherited |
Unregisters a column.
A page source decreases the reference counter for the corresponding active column. For a page sink, dropping columns is currently a no-op.
Implements ROOT::Internal::RPageStorage.
Definition at line 307 of file RPageStorage.hxx.
|
private |
Definition at line 255 of file RPageSinkBuf.cxx.
|
inlineinherited |
Definition at line 188 of file RPageStorage.hxx.
|
finalvirtual |
Return the RNTupleDescriptor being constructed.
Implements ROOT::Internal::RPageSink.
Definition at line 88 of file RPageSinkBuf.cxx.
|
inlinevirtualinherited |
Returns the default metrics object.
Subclasses might alternatively provide their own metrics object by overriding this.
Definition at line 192 of file RPageStorage.hxx.
|
inlinefinalvirtual |
Implements ROOT::Internal::RPageSink.
Definition at line 137 of file RPageSinkBuf.hxx.
|
inlineinherited |
Returns the NTuple name.
Definition at line 195 of file RPageStorage.hxx.
|
inlinevirtualinherited |
Definition at line 433 of file RPageStorage.hxx.
|
inlinefinalvirtualinherited |
Whether the concrete implementation is a sink or a source.
Implements ROOT::Internal::RPageStorage.
Definition at line 303 of file RPageStorage.hxx.
|
inlineinherited |
Returns the sink's write options.
Definition at line 305 of file RPageStorage.hxx.
|
inlineinherited |
Physically creates the storage container to hold the ntuple (e.g., a keys a TFile or an S3 bucket) Init() associates column handles to the columns referenced by the model.
Definition at line 318 of file RPageStorage.hxx.
|
finalvirtual |
Implements ROOT::Internal::RPageSink.
Definition at line 93 of file RPageSinkBuf.cxx.
|
inlineinherited |
Definition at line 309 of file RPageStorage.hxx.
|
delete |
|
delete |
|
inlineinherited |
The registered callback is executed at the beginning of CommitDataset();.
Definition at line 402 of file RPageStorage.hxx.
|
finalvirtual |
Get a new, empty page for the given column that can be filled with up to nElements; nElements must be larger than zero.
Reimplemented from ROOT::Internal::RPageSink.
Definition at line 319 of file RPageSinkBuf.cxx.
|
protectedinherited |
Helper for streaming a page.
This is commonly used in derived, concrete page sinks. Note that if compressionSetting is 0 (uncompressed) and the page is mappable and not checksummed, the returned sealed page will point directly to the input page buffer. Otherwise, the sealed page references fSealPageBuffer. Thus, the buffer pointed to by the RSealedPage should never be freed.
Definition at line 757 of file RPageStorage.cxx.
|
staticinherited |
Seal a page using the provided info.
Definition at line 719 of file RPageStorage.cxx.
|
inlineinherited |
Definition at line 197 of file RPageStorage.hxx.
|
finalvirtual |
Stage the current cluster and create a new one for the following data.
Returns the object that must be passed to CommitStagedClusters to logically append the staged cluster to the ntuple descriptor.
Implements ROOT::Internal::RPageSink.
Definition at line 291 of file RPageSinkBuf.cxx.
|
finalvirtual |
Adds an extra type information record to schema.
The extra type information will be written to the extension header. The information in the record will be merged with the existing information, e.g. duplicate streamer info records will be removed. This method is called by the "on commit dataset" callback registered by specific fields (e.g., streamer field) and during merging.
Implements ROOT::Internal::RPageSink.
Definition at line 137 of file RPageSinkBuf.cxx.
|
finalvirtual |
Incorporate incremental changes to the model into the ntuple descriptor.
This happens, e.g. if new fields were added after the initial call to RPageSink::Init(RNTupleModel &). firstEntry specifies the global index for the first stored element in the added columns.
Implements ROOT::Internal::RPageSink.
Definition at line 101 of file RPageSinkBuf.cxx.
|
inlineprotectedinherited |
Definition at line 153 of file RPageStorage.hxx.
|
private |
Vector of buffered column pages. Indexed by column id.
Definition at line 116 of file RPageSinkBuf.hxx.
|
private |
The sum of uncompressed bytes in buffered pages. Used to heuristically reduce the memory usage.
Definition at line 114 of file RPageSinkBuf.hxx.
|
private |
Definition at line 107 of file RPageSinkBuf.hxx.
|
private |
The buffered page sink maintains a copy of the RNTupleModel for the inner sink.
For the unbuffered case, the RNTupleModel is instead managed by a RNTupleWriter.
Definition at line 112 of file RPageSinkBuf.hxx.
|
private |
The inner sink, responsible for actually performing I/O.
Definition at line 109 of file RPageSinkBuf.hxx.
|
protectedinherited |
Flag if sink was initialized.
Definition at line 279 of file RPageStorage.hxx.
|
protectedinherited |
Definition at line 146 of file RPageStorage.hxx.
|
private |
Definition at line 120 of file RPageSinkBuf.hxx.
|
private |
Definition at line 119 of file RPageSinkBuf.hxx.
|
protectedinherited |
Definition at line 151 of file RPageStorage.hxx.
|
privateinherited |
Definition at line 288 of file RPageStorage.hxx.
|
protectedinherited |
Definition at line 276 of file RPageStorage.hxx.
|
protectedinherited |
For the time being, we will use the heap allocator for all sources and sinks. This may change in the future.
Definition at line 149 of file RPageStorage.hxx.
|
privateinherited |
Used as destination buffer in the simple SealPage overload.
Definition at line 289 of file RPageStorage.hxx.
|
private |
Columns committed as suppressed are stored and passed to the inner sink at cluster commit.
Definition at line 118 of file RPageSinkBuf.hxx.
|
protectedinherited |
Definition at line 152 of file RPageStorage.hxx.
|
privateinherited |
Used in ReservePage to maintain the page buffer budget.
Definition at line 292 of file RPageStorage.hxx.
|
staticconstexprinherited |
The page checksum is a 64bit xxhash3.
Definition at line 73 of file RPageStorage.hxx.