Storage provider that write ntuple pages into a file.

The written file can be either in ROOT format or in RNTuple bare format.

Definition at line 55 of file RPageStorageFile.hxx.

Classes
struct	CommitBatch

Public Member Functions
	RPageSinkFile (const RPageSinkFile &)=delete

	RPageSinkFile (RPageSinkFile &&)=default

	RPageSinkFile (std::string_view ntupleName, ROOT::Experimental::RFile &file, std::string_view ntupleDir, const ROOT::RNTupleWriteOptions &options)

	RPageSinkFile (std::string_view ntupleName, std::string_view path, const ROOT::RNTupleWriteOptions &options)

	RPageSinkFile (std::string_view ntupleName, TDirectory &fileOrDirectory, const ROOT::RNTupleWriteOptions &options)

	~RPageSinkFile () override

std::unique_ptr< RPageSink >	CloneWithDifferentName (std::string_view name, const ROOT::RNTupleWriteOptions &opts) const override
	Creates a new sink that uses the same underlying file/directory but writes to a different RNTuple with the given `name`.

RPageSinkFile &	operator= (const RPageSinkFile &)=delete

RPageSinkFile &	operator= (RPageSinkFile &&)=default

void	UpdateSchema (const ROOT::Internal::RNTupleModelChangeset &changeset, ROOT::NTupleSize_t firstEntry) final
	Incorporate incremental changes to the model into the ntuple descriptor.

Public Member Functions inherited from ROOT::Internal::RPagePersistentSink
	RPagePersistentSink (const RPagePersistentSink &)=delete

	RPagePersistentSink (RPagePersistentSink &&)=default

	RPagePersistentSink (std::string_view ntupleName, const ROOT::RNTupleWriteOptions &options)

	~RPagePersistentSink () override

ColumnHandle_t	AddColumn (ROOT::DescriptorId_t fieldId, ROOT::Internal::RColumn &column) final
	Register a new column.

void	CommitClusterGroup () final
	Write out the page locations (page list envelope) for all the committed clusters since the last call of CommitClusterGroup (or the beginning of writing).

void	CommitPage (ColumnHandle_t columnHandle, const ROOT::Internal::RPage &page) final
	Write a page to the storage. The column must have been added before.

void	CommitSealedPage (ROOT::DescriptorId_t physicalColumnId, const RPageStorage::RSealedPage &sealedPage) final
	Write a preprocessed page to storage. The column must have been added before.

void	CommitSealedPageV (std::span< RPageStorage::RSealedPageGroup > ranges) final
	Write a vector of preprocessed pages to storage. The corresponding columns must have been added before.

void	CommitStagedClusters (std::span< RStagedCluster > clusters) final
	Commit staged clusters, logically appending them to the ntuple descriptor.

void	CommitSuppressedColumn (ColumnHandle_t columnHandle) final
	Commits a suppressed column for the current cluster.

const ROOT::RNTupleDescriptor &	GetDescriptor () const final
	Return the RNTupleDescriptor being constructed.

ROOT::NTupleSize_t	GetNEntries () const final

std::unique_ptr< RNTupleModel >	InitFromDescriptor (const ROOT::RNTupleDescriptor &descriptor, bool copyClusters)
	Initialize sink based on an existing descriptor and fill into the descriptor builder, optionally copying over the descriptor's clusters to this sink's descriptor.

RPagePersistentSink &	operator= (const RPagePersistentSink &)=delete

RPagePersistentSink &	operator= (RPagePersistentSink &&)=default

RStagedCluster	StageCluster (ROOT::NTupleSize_t nNewEntries) final
	Stage the current cluster and create a new one for the following data.

void	UpdateExtraTypeInfo (const ROOT::RExtraTypeInfoDescriptor &extraTypeInfo) final
	Adds an extra type information record to schema.

Public Member Functions inherited from ROOT::Internal::RPageSink
	RPageSink (const RPageSink &)=delete

	RPageSink (RPageSink &&)=default

	RPageSink (std::string_view ntupleName, const ROOT::RNTupleWriteOptions &options)

	~RPageSink () override

virtual std::uint64_t	CommitCluster (ROOT::NTupleSize_t nNewEntries)
	Finalize the current cluster and create a new one for the following data.

void	CommitDataset ()
	Run the registered callbacks and finalize the current cluster and the entrire data set.

void	DropColumn (ColumnHandle_t) final
	Unregisters a column.

virtual RSinkGuard	GetSinkGuard ()

EPageStorageType	GetType () final
	Whether the concrete implementation is a sink or a source.

const ROOT::RNTupleWriteOptions &	GetWriteOptions () const
	Returns the sink's write options.

void	Init (RNTupleModel &model)
	Physically creates the storage container to hold the ntuple (e.g., a keys a TFile or an S3 bucket) Init() associates column handles to the columns referenced by the model.

bool	IsInitialized () const

RPageSink &	operator= (const RPageSink &)=delete

RPageSink &	operator= (RPageSink &&)=default

void	RegisterOnCommitDatasetCallback (Callback_t callback)
	The registered callback is executed at the beginning of CommitDataset();.

virtual ROOT::Internal::RPage	ReservePage (ColumnHandle_t columnHandle, std::size_t nElements)
	Get a new, empty page for the given column that can be filled with up to nElements; nElements must be larger than zero.

Public Member Functions inherited from ROOT::Internal::RPageStorage
	RPageStorage (const RPageStorage &other)=delete

	RPageStorage (RPageStorage &&other)=default

	RPageStorage (std::string_view name)

virtual	~RPageStorage ()

ROOT::DescriptorId_t	GetColumnId (ColumnHandle_t columnHandle) const

virtual ROOT::Experimental::Detail::RNTupleMetrics &	GetMetrics ()
	Returns the default metrics object.

const std::string &	GetNTupleName () const
	Returns the NTuple name.

RPageStorage &	operator= (const RPageStorage &other)=delete

RPageStorage &	operator= (RPageStorage &&other)=default

void	SetTaskScheduler (RTaskScheduler *taskScheduler)

Protected Member Functions
RNTupleLocator	CommitClusterGroupImpl (unsigned char *serializedPageList, std::uint32_t length) final
	Returns the locator of the page list envelope of the given buffer that contains the serialized page list.

void	CommitDatasetImpl () final

void	CommitDatasetImpl (unsigned char *serializedFooter, std::uint32_t length) final

virtual void	CommitDatasetImpl (unsigned char *serializedFooter, std::uint32_t length)=0

RNTupleLocator	CommitPageImpl (ColumnHandle_t columnHandle, const RPage &page) override

RNTupleLocator	CommitSealedPageImpl (ROOT::DescriptorId_t physicalColumnId, const RPageStorage::RSealedPage &sealedPage) final

std::vector< RNTupleLocator >	CommitSealedPageVImpl (std::span< RPageStorage::RSealedPageGroup > ranges, const std::vector< bool > &mask) final
	Vector commit of preprocessed pages.

void	InitImpl (RNTupleModel &model) final
	Updates the descriptor and calls InitImpl() that handles the backend-specific details (file, DAOS, etc.)

void	InitImpl (unsigned char *serializedHeader, std::uint32_t length) final

virtual void	InitImpl (unsigned char *serializedHeader, std::uint32_t length)=0

std::uint64_t	StageClusterImpl () final
	Returns the number of bytes written to storage (excluding metadata)

Protected Member Functions inherited from ROOT::Internal::RPagePersistentSink
void	EnableDefaultMetrics (const std::string &prefix)
	Enables the default set of metrics provided by RPageSink.

Protected Member Functions inherited from ROOT::Internal::RPageSink
RSealedPage	SealPage (const ROOT::Internal::RPage &page, const ROOT::Internal::RColumnElementBase &element)
	Helper for streaming a page.

Protected Member Functions inherited from ROOT::Internal::RPageStorage
void	WaitForAllTasks ()

Private Member Functions
	RPageSinkFile (std::string_view ntupleName, const ROOT::RNTupleWriteOptions &options)

	RPageSinkFile (std::unique_ptr< ROOT::Internal::RNTupleFileWriter > writer, const ROOT::RNTupleWriteOptions &options)

void	CommitBatchOfPages (CommitBatch &batch, std::vector< RNTupleLocator > &locators)
	Subroutine of CommitSealedPageVImpl, used to perform a vector write of the (multi-)range of pages contained in `batch`.

RNTupleLocator	WriteSealedPage (const RPageStorage::RSealedPage &sealedPage, std::size_t bytesPacked)
	We pass bytesPacked so that TFile::ls() reports a reasonable value for the compression ratio of the corresponding key.

Private Attributes
ROOT::Internal::RNTupleSerializer::StreamerInfoMap_t	fInfosOfClassFields
	On UpdateSchema(), the new class fields register the corresponding streamer info here so that the streamer info records in the file can be properly updated on dataset commit.

std::uint64_t	fNBytesCurrentCluster = 0
	Number of bytes committed to storage in the current cluster.

std::unique_ptr< ROOT::Internal::RNTupleFileWriter >	fWriter

Additional Inherited Members
Public Types inherited from ROOT::Internal::RPageSink
using	Callback_t = std::function<void(RPageSink &)>

Public Types inherited from ROOT::Internal::RPageStorage
using	ColumnHandle_t = RColumnHandle
	The column handle identifies a column with the current open page storage.

using	SealedPageSequence_t = std::deque<RSealedPage>

Static Public Member Functions inherited from ROOT::Internal::RPagePersistentSink
static std::unique_ptr< RPageSink >	Create (std::string_view ntupleName, std::string_view location, const ROOT::RNTupleWriteOptions &options=ROOT::RNTupleWriteOptions())
	Guess the concrete derived page source from the location.

Static Public Member Functions inherited from ROOT::Internal::RPageSink
static RSealedPage	SealPage (const RSealPageConfig &config)
	Seal a page using the provided info.

Static Public Attributes inherited from ROOT::Internal::RPageStorage
static constexpr std::size_t	kNBytesPageChecksum = sizeof(std::uint64_t)
	The page checksum is a 64bit xxhash3.

Protected Attributes inherited from ROOT::Internal::RPagePersistentSink
std::unique_ptr< RCounters >	fCounters

ROOT::Internal::RNTupleDescriptorBuilder	fDescriptorBuilder

RFeatures	fFeatures

Protected Attributes inherited from ROOT::Internal::RPageSink
bool	fIsInitialized = false
	Flag if sink was initialized.

std::unique_ptr< ROOT::RNTupleWriteOptions >	fOptions

Protected Attributes inherited from ROOT::Internal::RPageStorage
ROOT::Experimental::Detail::RNTupleMetrics	fMetrics

std::string	fNTupleName

std::unique_ptr< ROOT::Internal::RPageAllocator >	fPageAllocator
	For the time being, we will use the heap allocator for all sources and sinks. This may change in the future.

RTaskScheduler *	fTaskScheduler = nullptr

#include <ROOT/RPageStorageFile.hxx>

Inheritance diagram for ROOT::Internal::RPageSinkFile:

[legend]

Constructor & Destructor Documentation

◆ RPageSinkFile() [1/7]

ROOT::Internal::RPageSinkFile::RPageSinkFile	(	std::string_view	ntupleName,
		const ROOT::RNTupleWriteOptions &	options )

private

Definition at line 47 of file RPageStorageFile.cxx.

◆ RPageSinkFile() [2/7]

ROOT::Internal::RPageSinkFile::RPageSinkFile	(	std::unique_ptr< ROOT::Internal::RNTupleFileWriter >	writer,
		const ROOT::RNTupleWriteOptions &	options )

private

Definition at line 75 of file RPageStorageFile.cxx.

◆ RPageSinkFile() [3/7]

ROOT::Internal::RPageSinkFile::RPageSinkFile	(	std::string_view	ntupleName,
		std::string_view	path,
		const ROOT::RNTupleWriteOptions &	options )

Definition at line 54 of file RPageStorageFile.cxx.

◆ RPageSinkFile() [4/7]

ROOT::Internal::RPageSinkFile::RPageSinkFile	(	std::string_view	ntupleName,
		TDirectory &	fileOrDirectory,
		const ROOT::RNTupleWriteOptions &	options )

Definition at line 61 of file RPageStorageFile.cxx.

◆ RPageSinkFile() [5/7]

ROOT::Internal::RPageSinkFile::RPageSinkFile	(	std::string_view	ntupleName,
		ROOT::Experimental::RFile &	file,
		std::string_view	ntupleDir,
		const ROOT::RNTupleWriteOptions &	options )

Definition at line 68 of file RPageStorageFile.cxx.

◆ RPageSinkFile() [6/7]

ROOT::Internal::RPageSinkFile::RPageSinkFile ( const RPageSinkFile & )

delete

◆ RPageSinkFile() [7/7]

ROOT::Internal::RPageSinkFile::RPageSinkFile ( RPageSinkFile && )

default

◆ ~RPageSinkFile()

ROOT::Internal::RPageSinkFile::~RPageSinkFile ( )

override

Definition at line 82 of file RPageStorageFile.cxx.

Member Function Documentation

◆ CloneWithDifferentName()

std::unique_ptr< ROOT::Internal::RPageSink > ROOT::Internal::RPageSinkFile::CloneWithDifferentName	(	std::string_view	name,
		const ROOT::RNTupleWriteOptions &	opts ) const

overridevirtual

Creates a new sink that uses the same underlying file/directory but writes to a different RNTuple with the given name.

Implements ROOT::Internal::RPageSink.

Definition at line 302 of file RPageStorageFile.cxx.

◆ CommitBatchOfPages()

void ROOT::Internal::RPageSinkFile::CommitBatchOfPages	(	CommitBatch &	batch,
		std::vector< RNTupleLocator > &	locators )

private

Subroutine of CommitSealedPageVImpl, used to perform a vector write of the (multi-)range of pages contained in batch.

The locators for the written pages are appended to locators. This procedure also updates some internal metrics of the page sink, hence it's not const. batch gets reset to size 0 after the writing is done (but its begin and end are not updated).

Definition at line 163 of file RPageStorageFile.cxx.

◆ CommitClusterGroupImpl()

ROOT::RNTupleLocator ROOT::Internal::RPageSinkFile::CommitClusterGroupImpl	(	unsigned char *	serializedPageList,
		std::uint32_t	length )

finalprotectedvirtual

Returns the locator of the page list envelope of the given buffer that contains the serialized page list.

Typically, the implementation takes care of compressing and writing the provided buffer.

Implements ROOT::Internal::RPagePersistentSink.

Definition at line 268 of file RPageStorageFile.cxx.

◆ CommitDatasetImpl() [1/3]

void ROOT::Internal::RPagePersistentSink::CommitDatasetImpl ( )

finalprotectedvirtual

Reimplemented from ROOT::Internal::RPagePersistentSink.

Definition at line 545 of file RPageStorage.cxx.

◆ CommitDatasetImpl() [2/3]

void ROOT::Internal::RPageSinkFile::CommitDatasetImpl	(	unsigned char *	serializedFooter,
		std::uint32_t	length )

finalprotectedvirtual

Implements ROOT::Internal::RPagePersistentSink.

Definition at line 280 of file RPageStorageFile.cxx.

◆ CommitDatasetImpl() [3/3]

virtual void ROOT::Internal::RPagePersistentSink::CommitDatasetImpl	(	unsigned char *	serializedFooter,
		std::uint32_t	length )

protectedvirtual

Implements ROOT::Internal::RPagePersistentSink.

◆ CommitPageImpl()

ROOT::RNTupleLocator ROOT::Internal::RPageSinkFile::CommitPageImpl	(	ColumnHandle_t	columnHandle,
		const RPage &	page )

overrideprotectedvirtual

Implements ROOT::Internal::RPagePersistentSink.

Definition at line 142 of file RPageStorageFile.cxx.

◆ CommitSealedPageImpl()

ROOT::RNTupleLocator ROOT::Internal::RPageSinkFile::CommitSealedPageImpl	(	ROOT::DescriptorId_t	physicalColumnId,
		const RPageStorage::RSealedPage &	sealedPage )

finalprotectedvirtual

Implements ROOT::Internal::RPagePersistentSink.

Definition at line 155 of file RPageStorageFile.cxx.

◆ CommitSealedPageVImpl()

std::vector< ROOT::RNTupleLocator > ROOT::Internal::RPageSinkFile::CommitSealedPageVImpl	(	std::span< RPageStorage::RSealedPageGroup >	ranges,
		const std::vector< bool > &	mask )

finalprotectedvirtual

Vector commit of preprocessed pages.

The ranges array specifies a range of sealed pages to be committed for each column. The returned vector contains, in order, the RNTupleLocator for each page on each range in ranges, i.e. the first N entries refer to the N pages in ranges[0], followed by M entries that refer to the M pages in ranges[1], etc. The mask allows to skip writing out certain pages. The vector has the size of all the pages. For every false value in the mask, the corresponding locator is skipped (missing) in the output vector. The default is to call CommitSealedPageImpl for each page; derived classes may provide an optimized implementation though.

Reimplemented from ROOT::Internal::RPagePersistentSink.

Definition at line 190 of file RPageStorageFile.cxx.

◆ InitImpl() [1/3]

void ROOT::Internal::RPagePersistentSink::InitImpl ( ROOT::RNTupleModel & model )

finalprotectedvirtual

Updates the descriptor and calls InitImpl() that handles the backend-specific details (file, DAOS, etc.)

Reimplemented from ROOT::Internal::RPagePersistentSink.

Definition at line 527 of file RPageStorage.cxx.

◆ InitImpl() [2/3]

void ROOT::Internal::RPageSinkFile::InitImpl	(	unsigned char *	serializedHeader,
		std::uint32_t	length )

finalprotectedvirtual

Implements ROOT::Internal::RPagePersistentSink.

Definition at line 84 of file RPageStorageFile.cxx.

◆ InitImpl() [3/3]

virtual void ROOT::Internal::RPagePersistentSink::InitImpl	(	unsigned char *	serializedHeader,
		std::uint32_t	length )

protectedvirtual

Implements ROOT::Internal::RPagePersistentSink.

◆ operator=() [1/2]

RPageSinkFile & ROOT::Internal::RPageSinkFile::operator= ( const RPageSinkFile & )

delete

◆ operator=() [2/2]

RPageSinkFile & ROOT::Internal::RPageSinkFile::operator= ( RPageSinkFile && )

default

◆ StageClusterImpl()

std::uint64_t ROOT::Internal::RPageSinkFile::StageClusterImpl ( )

finalprotectedvirtual

Returns the number of bytes written to storage (excluding metadata)

Implements ROOT::Internal::RPagePersistentSink.

Definition at line 260 of file RPageStorageFile.cxx.

◆ UpdateSchema()

void ROOT::Internal::RPageSinkFile::UpdateSchema	(	const ROOT::Internal::RNTupleModelChangeset &	changeset,
		ROOT::NTupleSize_t	firstEntry )

finalvirtual

Incorporate incremental changes to the model into the ntuple descriptor.

This happens, e.g. if new fields were added after the initial call to RPageSink::Init(RNTupleModel &). firstEntry specifies the global index for the first stored element in the added columns.

Reimplemented from ROOT::Internal::RPagePersistentSink.

Definition at line 92 of file RPageStorageFile.cxx.

◆ WriteSealedPage()

ROOT::RNTupleLocator ROOT::Internal::RPageSinkFile::WriteSealedPage	(	const RPageStorage::RSealedPage &	sealedPage,
		std::size_t	bytesPacked )

inlineprivate

We pass bytesPacked so that TFile::ls() reports a reasonable value for the compression ratio of the corresponding key.

It is not strictly necessary to write and read the sealed page.

Definition at line 124 of file RPageStorageFile.cxx.

Member Data Documentation

◆ fInfosOfClassFields

ROOT::Internal::RNTupleSerializer::StreamerInfoMap_t ROOT::Internal::RPageSinkFile::fInfosOfClassFields

private

On UpdateSchema(), the new class fields register the corresponding streamer info here so that the streamer info records in the file can be properly updated on dataset commit.

Definition at line 73 of file RPageStorageFile.hxx.

◆ fNBytesCurrentCluster

std::uint64_t ROOT::Internal::RPageSinkFile::fNBytesCurrentCluster = 0

private

Number of bytes committed to storage in the current cluster.

Definition at line 70 of file RPageStorageFile.hxx.

◆ fWriter

std::unique_ptr<ROOT::Internal::RNTupleFileWriter> ROOT::Internal::RPageSinkFile::fWriter

private

Definition at line 68 of file RPageStorageFile.hxx.

Libraries for ROOT::Internal::RPageSinkFile:

[legend]

The documentation for this class was generated from the following files:

tree/ntuple/inc/ROOT/RPageStorageFile.hxx
tree/ntuple/src/RPageStorageFile.cxx

Classes

Public Member Functions

Protected Member Functions

Private Member Functions

Private Attributes

Additional Inherited Members

Constructor & Destructor Documentation

◆ RPageSinkFile() [1/7]

◆ RPageSinkFile() [2/7]

◆ RPageSinkFile() [3/7]

◆ RPageSinkFile() [4/7]

◆ RPageSinkFile() [5/7]

◆ RPageSinkFile() [6/7]

◆ RPageSinkFile() [7/7]

◆ ~RPageSinkFile()

Member Function Documentation

◆ CloneWithDifferentName()

◆ CommitBatchOfPages()

◆ CommitClusterGroupImpl()

◆ CommitDatasetImpl() [1/3]

◆ CommitDatasetImpl() [2/3]

◆ CommitDatasetImpl() [3/3]

◆ CommitPageImpl()

◆ CommitSealedPageImpl()

◆ CommitSealedPageVImpl()

◆ InitImpl() [1/3]

◆ InitImpl() [2/3]

◆ InitImpl() [3/3]

◆ operator=() [1/2]

◆ operator=() [2/2]

◆ StageClusterImpl()

◆ UpdateSchema()

◆ WriteSealedPage()

Member Data Documentation

◆ fInfosOfClassFields

◆ fNBytesCurrentCluster

◆ fWriter