Storage provider that writes ntuple pages to into a DAOS container.

Currently, an object is allocated for ntuple metadata (anchor/header/footer). Objects can correspond to pages or clusters of pages depending on the RNTuple-DAOS mapping strategy.

Definition at line 135 of file RPageStorageDaos.hxx.

Public Member Functions
	RPageSinkDaos (std::string_view ntupleName, std::string_view uri, const RNTupleWriteOptions &options)

	~RPageSinkDaos () override

void	ReleasePage (RPage &page) final
	Every page store needs to be able to free pages it handed out.

RPage	ReservePage (ColumnHandle_t columnHandle, std::size_t nElements) final
	Get a new, empty page for the given column that can be filled with up to nElements.

Public Member Functions inherited from ROOT::Experimental::Detail::RPageSink
	RPageSink (const RPageSink &)=delete

	RPageSink (RPageSink &&)=default

	RPageSink (std::string_view ntupleName, const RNTupleWriteOptions &options)

	~RPageSink () override

ColumnHandle_t	AddColumn (DescriptorId_t fieldId, const RColumn &column) final
	Register a new column.

std::uint64_t	CommitCluster (NTupleSize_t nEntries)
	Finalize the current cluster and create a new one for the following data.

void	CommitClusterGroup ()
	Write out the page locations (page list envelope) for all the committed clusters since the last call of CommitClusterGroup (or the beginning of writing).

void	CommitDataset ()
	Finalize the current cluster and the entrire data set.

void	CommitPage (ColumnHandle_t columnHandle, const RPage &page)
	Write a page to the storage. The column must have been added before.

void	CommitSealedPage (DescriptorId_t columnId, const RPageStorage::RSealedPage &sealedPage)
	Write a preprocessed page to storage. The column must have been added before.

void	CommitSealedPageV (std::span< RPageStorage::RSealedPageGroup > ranges)
	Write a vector of preprocessed pages to storage. The corresponding columns must have been added before.

void	Create (RNTupleModel &model)
	Physically creates the storage container to hold the ntuple (e.g., a keys a TFile or an S3 bucket) To do so, Create() calls CreateImpl() after updating the descriptor.

void	DropColumn (ColumnHandle_t) final
	Unregisters a column.

RNTupleMetrics &	GetMetrics () override
	Returns the default metrics object. Subclasses might alternatively provide their own metrics object by overriding this.

EPageStorageType	GetType () final
	Whether the concrete implementation is a sink or a source.

const RNTupleWriteOptions &	GetWriteOptions () const
	Returns the sink's write options.

RPageSink &	operator= (const RPageSink &)=delete

RPageSink &	operator= (RPageSink &&)=default

Public Member Functions inherited from ROOT::Experimental::Detail::RPageStorage
	RPageStorage (const RPageStorage &other)=delete

	RPageStorage (RPageStorage &&other)=default

	RPageStorage (std::string_view name)

virtual	~RPageStorage ()

const std::string &	GetNTupleName () const
	Returns the NTuple name.

RPageStorage &	operator= (const RPageStorage &other)=delete

RPageStorage &	operator= (RPageStorage &&other)=default

void	SetTaskScheduler (RTaskScheduler *taskScheduler)

Protected Member Functions
RNTupleLocator	CommitClusterGroupImpl (unsigned char *serializedPageList, std::uint32_t length) final
	Returns the locator of the page list envelope of the given buffer that contains the serialized page list.

std::uint64_t	CommitClusterImpl (NTupleSize_t nEntries) final
	Returns the number of bytes written to storage (excluding metadata)

void	CommitDatasetImpl (unsigned char *serializedFooter, std::uint32_t length) final

RNTupleLocator	CommitPageImpl (ColumnHandle_t columnHandle, const RPage &page) final

RNTupleLocator	CommitSealedPageImpl (DescriptorId_t columnId, const RPageStorage::RSealedPage &sealedPage) final

std::vector< RNTupleLocator >	CommitSealedPageVImpl (std::span< RPageStorage::RSealedPageGroup > ranges) final
	Vector commit of preprocessed pages.

void	CreateImpl (const RNTupleModel &model, unsigned char *serializedHeader, std::uint32_t length) final

void	WriteNTupleAnchor ()

void	WriteNTupleFooter (const void *data, size_t nbytes, size_t lenFooter)

void	WriteNTupleHeader (const void *data, size_t nbytes, size_t lenHeader)

Protected Member Functions inherited from ROOT::Experimental::Detail::RPageSink
void	EnableDefaultMetrics (const std::string &prefix)
	Enables the default set of metrics provided by RPageSink.

RSealedPage	SealPage (const RPage &page, const RColumnElementBase &element, int compressionSetting)
	Helper for streaming a page.

Private Attributes
std::atomic< std::uint64_t >	fClusterGroupId {0}
	Cluster group counter for the next committed cluster pagelist; incremented in `CommitClusterGroupImpl()`

std::unique_ptr< RDaosContainer >	fDaosContainer
	Underlying DAOS container.

std::uint64_t	fNBytesCurrentCluster {0}
	Tracks the number of bytes committed to the current cluster.

RDaosNTupleAnchor	fNTupleAnchor

ntuple_index_t	fNTupleIndex {0}

std::unique_ptr< RPageAllocatorHeap >	fPageAllocator

std::atomic< std::uint64_t >	fPageId {0}
	Page identifier for the next committed page; it is automatically incremented in `CommitSealedPageImpl()`

std::string	fURI
	A URI to a DAOS pool of the form 'daos://pool-label/container-label'.

Additional Inherited Members
Public Types inherited from ROOT::Experimental::Detail::RPageStorage
using	ColumnHandle_t = RColumnHandle
	The column handle identifies a column with the current open page storage.

using	SealedPageSequence_t = std::deque< RSealedPage >

Static Public Member Functions inherited from ROOT::Experimental::Detail::RPageSink
static std::unique_ptr< RPageSink >	Create (std::string_view ntupleName, std::string_view location, const RNTupleWriteOptions &options=RNTupleWriteOptions())
	Guess the concrete derived page source from the file name (location)

Static Protected Member Functions inherited from ROOT::Experimental::Detail::RPageSink
static RSealedPage	SealPage (const RPage &page, const RColumnElementBase &element, int compressionSetting, void *buf)
	Seal a page using the provided buffer.

Protected Attributes inherited from ROOT::Experimental::Detail::RPageSink
std::unique_ptr< RNTupleCompressor >	fCompressor
	Helper to zip pages and header/footer; includes a 16MB (kMAXZIPBUF) zip buffer.

std::unique_ptr< RCounters >	fCounters

RNTupleDescriptorBuilder	fDescriptorBuilder

RNTupleMetrics	fMetrics

std::uint64_t	fNextClusterInGroup = 0
	Remembers the starting cluster id for the next cluster group.

std::vector< RClusterDescriptor::RColumnRange >	fOpenColumnRanges
	Keeps track of the number of elements in the currently open cluster. Indexed by column id.

std::vector< RClusterDescriptor::RPageRange >	fOpenPageRanges
	Keeps track of the written pages in the currently open cluster. Indexed by column id.

std::unique_ptr< RNTupleWriteOptions >	fOptions

NTupleSize_t	fPrevClusterNEntries = 0
	Used to calculate the number of entries in the current cluster.

Protected Attributes inherited from ROOT::Experimental::Detail::RPageStorage
std::string	fNTupleName

RTaskScheduler *	fTaskScheduler = nullptr

#include <ROOT/RPageStorageDaos.hxx>

Inheritance diagram for ROOT::Experimental::Detail::RPageSinkDaos:

[legend]

Constructor & Destructor Documentation

◆ RPageSinkDaos()

ROOT::Experimental::Detail::RPageSinkDaos::RPageSinkDaos	(	std::string_view	ntupleName,
		std::string_view	uri,
		const RNTupleWriteOptions &	options
	)

Definition at line 212 of file RPageStorageDaos.cxx.

◆ ~RPageSinkDaos()

ROOT::Experimental::Detail::RPageSinkDaos::~RPageSinkDaos ( )

overridedefault

Member Function Documentation

◆ CommitClusterGroupImpl()

ROOT::Experimental::RNTupleLocator ROOT::Experimental::Detail::RPageSinkDaos::CommitClusterGroupImpl	(	unsigned char *	serializedPageList,
		std::uint32_t	length
	)

finalprotectedvirtual

Returns the locator of the page list envelope of the given buffer that contains the serialized page list.

Typically, the implementation takes care of compressing and writing the provided buffer.

Implements ROOT::Experimental::Detail::RPageSink.

Definition at line 340 of file RPageStorageDaos.cxx.

◆ CommitClusterImpl()

std::uint64_t ROOT::Experimental::Detail::RPageSinkDaos::CommitClusterImpl ( NTupleSize_t nEntries )

finalprotectedvirtual

Returns the number of bytes written to storage (excluding metadata)

Implements ROOT::Experimental::Detail::RPageSink.

Definition at line 334 of file RPageStorageDaos.cxx.

◆ CommitDatasetImpl()

void ROOT::Experimental::Detail::RPageSinkDaos::CommitDatasetImpl	(	unsigned char *	serializedFooter,
		std::uint32_t	length
	)

finalprotectedvirtual

Implements ROOT::Experimental::Detail::RPageSink.

Definition at line 359 of file RPageStorageDaos.cxx.

◆ CommitPageImpl()

ROOT::Experimental::RNTupleLocator ROOT::Experimental::Detail::RPageSinkDaos::CommitPageImpl	(	ColumnHandle_t	columnHandle,
		const RPage &	page
	)

finalprotectedvirtual

Implements ROOT::Experimental::Detail::RPageSink.

Definition at line 249 of file RPageStorageDaos.cxx.

◆ CommitSealedPageImpl()

ROOT::Experimental::RNTupleLocator ROOT::Experimental::Detail::RPageSinkDaos::CommitSealedPageImpl	(	DescriptorId_t	columnId,
		const RPageStorage::RSealedPage &	sealedPage
	)

finalprotectedvirtual

Implements ROOT::Experimental::Detail::RPageSink.

Definition at line 263 of file RPageStorageDaos.cxx.

◆ CommitSealedPageVImpl()

std::vector< ROOT::Experimental::RNTupleLocator > ROOT::Experimental::Detail::RPageSinkDaos::CommitSealedPageVImpl ( std::span< RPageStorage::RSealedPageGroup > ranges )

finalprotectedvirtual

Vector commit of preprocessed pages.

The ranges array specifies a range of sealed pages to be committed for each column. The returned vector contains, in order, the RNTupleLocator for each page on each range in ranges, i.e. the first N entries refer to the N pages in ranges[0], followed by M entries that refer to the M pages in ranges[1], etc. The default is to call CommitSealedPageImpl for each page; derived classes may provide an optimized implementation though.

Reimplemented from ROOT::Experimental::Detail::RPageSink.

Definition at line 285 of file RPageStorageDaos.cxx.

◆ CreateImpl()

void ROOT::Experimental::Detail::RPageSinkDaos::CreateImpl	(	const RNTupleModel &	model,
		unsigned char *	serializedHeader,
		std::uint32_t	length
	)

finalprotectedvirtual

Implements ROOT::Experimental::Detail::RPageSink.

Definition at line 224 of file RPageStorageDaos.cxx.

◆ ReleasePage()

void ROOT::Experimental::Detail::RPageSinkDaos::ReleasePage ( RPage & page )

finalvirtual

Every page store needs to be able to free pages it handed out.

But Sinks and sources have different means of allocating pages.

Implements ROOT::Experimental::Detail::RPageStorage.

Definition at line 405 of file RPageStorageDaos.cxx.

◆ ReservePage()

ROOT::Experimental::Detail::RPage ROOT::Experimental::Detail::RPageSinkDaos::ReservePage	(	ColumnHandle_t	columnHandle,
		std::size_t	nElements
	)

finalvirtual

Get a new, empty page for the given column that can be filled with up to nElements.

If nElements is zero, the page sink picks an appropriate size.

Implements ROOT::Experimental::Detail::RPageSink.

Definition at line 397 of file RPageStorageDaos.cxx.

◆ WriteNTupleAnchor()

void ROOT::Experimental::Detail::RPageSinkDaos::WriteNTupleAnchor ( )

protected

Definition at line 386 of file RPageStorageDaos.cxx.

◆ WriteNTupleFooter()

void ROOT::Experimental::Detail::RPageSinkDaos::WriteNTupleFooter	(	const void *	data,
		size_t	nbytes,
		size_t	lenFooter
	)

protected

Definition at line 377 of file RPageStorageDaos.cxx.

◆ WriteNTupleHeader()

void ROOT::Experimental::Detail::RPageSinkDaos::WriteNTupleHeader	(	const void *	data,
		size_t	nbytes,
		size_t	lenHeader
	)

protected

Definition at line 368 of file RPageStorageDaos.cxx.

Member Data Documentation

◆ fClusterGroupId

std::atomic<std::uint64_t> ROOT::Experimental::Detail::RPageSinkDaos::fClusterGroupId {0}

private

Cluster group counter for the next committed cluster pagelist; incremented in CommitClusterGroupImpl()

Definition at line 147 of file RPageStorageDaos.hxx.

◆ fDaosContainer

std::unique_ptr<RDaosContainer> ROOT::Experimental::Detail::RPageSinkDaos::fDaosContainer

private

Underlying DAOS container.

An internal std::shared_ptr keep the pool connection alive. ISO C++ ensures the correct destruction order, i.e., ~RDaosContainer is invoked first (which calls daos_cont_close(); the destructor for the std::shared_ptr<RDaosPool> is invoked after (which calls daos_pool_disconect()).

Definition at line 143 of file RPageStorageDaos.hxx.

◆ fNBytesCurrentCluster

std::uint64_t ROOT::Experimental::Detail::RPageSinkDaos::fNBytesCurrentCluster {0}

private

Tracks the number of bytes committed to the current cluster.

Definition at line 151 of file RPageStorageDaos.hxx.

◆ fNTupleAnchor

RDaosNTupleAnchor ROOT::Experimental::Detail::RPageSinkDaos::fNTupleAnchor

private

Definition at line 153 of file RPageStorageDaos.hxx.

◆ fNTupleIndex

ntuple_index_t ROOT::Experimental::Detail::RPageSinkDaos::fNTupleIndex {0}

private

Definition at line 154 of file RPageStorageDaos.hxx.

◆ fPageAllocator

std::unique_ptr<RPageAllocatorHeap> ROOT::Experimental::Detail::RPageSinkDaos::fPageAllocator

private

Definition at line 137 of file RPageStorageDaos.hxx.

◆ fPageId

std::atomic<std::uint64_t> ROOT::Experimental::Detail::RPageSinkDaos::fPageId {0}

private

Page identifier for the next committed page; it is automatically incremented in CommitSealedPageImpl()

Definition at line 145 of file RPageStorageDaos.hxx.

◆ fURI

std::string ROOT::Experimental::Detail::RPageSinkDaos::fURI

private

A URI to a DAOS pool of the form 'daos://pool-label/container-label'.

Definition at line 149 of file RPageStorageDaos.hxx.

tree/ntuple/v7/inc/ROOT/RPageStorageDaos.hxx

tree/ntuple/v7/src/RPageStorageDaos.cxx

Public Member Functions

Protected Member Functions

Private Attributes

Additional Inherited Members

Constructor & Destructor Documentation

◆ RPageSinkDaos()

◆ ~RPageSinkDaos()

Member Function Documentation

◆ CommitClusterGroupImpl()

◆ CommitClusterImpl()

◆ CommitDatasetImpl()

◆ CommitPageImpl()

◆ CommitSealedPageImpl()

◆ CommitSealedPageVImpl()

◆ CreateImpl()

◆ ReleasePage()

◆ ReservePage()

◆ WriteNTupleAnchor()

◆ WriteNTupleFooter()

◆ WriteNTupleHeader()

Member Data Documentation

◆ fClusterGroupId

◆ fDaosContainer

◆ fNBytesCurrentCluster

◆ fNTupleAnchor

◆ fNTupleIndex

◆ fPageAllocator

◆ fPageId

◆ fURI