Feed ROOT data directly into models for machine learning training.
RDataLoader streams ROOT data into machine learning frameworks as batches ready for training. It reads any TTree or RNTuple, from local or remote files or from an RDataFrame, giving you access to the full ROOT ecosystem for filtering, defining new variables and applying selections; it delivers batches of your dataset for NumPy, PyTorch, TensorFlow and JAX through a simple iteration interface.
RDataLoader is part of ROOT.Experimental.ML and is currently experimental. The API may change between ROOT releases, and feedback is much appreciated!
The loader reads the file in the background while you train, shuffles the entries every epoch and hands them over as tensors. The rest of this page goes through each step in detail.
A one-page quick reference covering the API.
This walkthrough trains a small PyTorch classifier to separate a signal from a background sample, stored in two files. Each step points to the section of this page that explains it further.
Selections, new variables and labels are defined with RDataFrame, before the data reaches the loader. Here, each sample gets a label column: 1 for signal, 0 for background.
Filters run before any data is copied internally. Selecting events and columns in RDataFrame is therefore the cheapest way to reduce how much data the loader processes.Both samples are combined into one stream of batches. columns lists the features, target the label (see Columns, targets and weights):
Each for loop over a loader is one epoch. The entry indices are re-shuffled at the start of every epoch, so that the batches are built in a different order at each epoch:
float32, which is what most models expect. For a loss that needs integer class indices, like torch.nn.CrossEntropyLoss, convert the target with y.long().squeeze(1) (this casts the Pytorch tensor to a 64-bit integer and removes a dimension of size 1 at index 1).The training and validation datasets can be written to disk exactly as they were loaded, e.g. to share them or to train other models on the same split (see Saving datasets):
The tutorials show complete examples, with NumPy, PyTorch, TensorFlow and XGBoost.
RDataLoader takes the dataset to read as its first argument. All the other arguments are keyword-only.
| Input | Reads |
|---|---|
("events", "data.root") | the TTree or RNTuple events from one file |
("events", ["data_1.root", "data_2.root"]), ("events", "data_*.root") | events from several files, as one dataset |
ROOT.RDataFrame(...), with any Filter and Define | the entries and columns of the dataframe |
[("signal", "sig.root"), ("background", "bkg.root")], [df1, df2] | several datasets, one after the other in the same stream of batches |
A (dataset_name, files) tuple is opened like ROOT.RDataFrame(dataset_name, files). The files can be given as strings or pathlib.Path objects, and can be remote:
Pass an RDataFrame instead as soon as you need to prepare the data, e.g. to apply selections or to define the target column:
columns selects the feature columns to load, and target names the column (or list of columns) to predict. When iterating, the targets are returned separately as y, so you don't need to split them manually:
If columns is not given, all the columns of the dataset are used as features, except for the target and weights columns, you can inspect the loader to check yourself:
Without a target, each batch is a single array holding all the columns.
If your dataset has a weight column, pass its name to weights. It is returned as a third value w, with the shape (batch_size, 1):
weights can only be used together with a target.Columns holding a variable number of values per entry, such as ROOT::RVec or std::vector, are loaded as a fixed number of features. Give their maximum size in max_vec_sizes: shorter vectors are padded with vec_padding (0 by default) and longer ones are truncated.
max_vec_sizes.nJets above) so that the model can tell real values from padding.Any transformation of the features can be done with RDataFrame before loading, e.g. to normalize them:
train_test_split() splits the dataset into two loaders, each holding a fraction of the entries:
The validation entries are rounded down: with 1001 entries and test_size=0.2, 200 entries are used for validation and 801 for training. Without shuffling, the last entries are used for validation.
set_seed is given. Fix the seed whenever you split, so that the validation entries are the same every time you run, e.g. when resuming a training or evaluating a model later.To also keep a test dataset that is never seen during the training, there are two options. In both examples below, 60% of the events are used for training, 20% for validation and 20% for the test.
With a filter. Select the test events with a Filter, e.g. on the event number, and give them their own loader:
The test events are then always the same, whatever the seed, and nothing is written to disk.
By splitting twice. Split off the test dataset first, save the rest with save(), and split the saved dataset again into training and validation:
The events are then selected at random, and no column is needed to select them.
train_test_split() on a loader returned by train_test_split() returns the same two datasets again, whatever the test_size.drop_remainder=False, otherwise save() drops the entries of the last incomplete batch (see Saving datasets). The test dataset can be saved the same way, to evaluate the model later or with other tools. Vector columns are saved padded to their maximum size, so give the same max_vec_sizes to the second loader.Training, validation and test can also simply be read from different files.
The batches can be returned in four formats.
| Method | Batches |
|---|---|
as_numpy() | numpy.ndarray |
as_torch(device=None) | torch.Tensor, optionally moved to a device such as "cuda" |
as_tensorflow() | a tf.data.Dataset of tf.Tensor |
as_jax(device=None) | jax.Array, optionally moved to a device such as "gpu" |
All the values are float32. With a single target or weights column, y and w have the shape (batch_size, 1).
as_tensorflow() returns a tf.data.Dataset, which can be passed to Keras directly:
drop_remainder=False, the last batch is completed with rows of zeros. Keep the default drop_remainder=True, or use a weights column so that the added rows get a weight of zero.batch_size to the number of entries and drop_remainder=False, then take it with X, y = next(iter(train.as_numpy())). See the ml_dataloader_XGBoost.py tutorial.Transform each batch with Python's map:
For a TensorFlow dataset, use Dataset.map:
When the number of entries is not a multiple of batch_size, the remaining entries are dropped by default. Pass drop_remainder=False to get them in a smaller last batch. num_batches and last_batch_no_of_rows tell how many batches an epoch has and how many rows the last one holds; they are available once the loader is split or iterated.
By default, the loader reads the data lazily: it loads a few clusters of entries (groups of consecutive entries stored together on disk) at a time in a background thread while you train, so that the memory usage does not depend on the size of the dataset. With load_eager=True, the whole dataset is loaded into memory at the start, and data is served from memory at every epoch.
| Lazy (default) | Eager (load_eager=True) | |
|---|---|---|
| Memory | about batch_size × batches_in_memory entries | the whole dataset |
| First epoch | starts right away | starts when the dataset is loaded |
| Next epochs | read subsets of the files again | served from memory |
| Shuffling | within a buffer of clusters | across the whole dataset |
| Resampling | not available | available |
The memory used in lazy loading is controlled with batches_in_memory, the approximate number of batches held in memory at a time (10 by default):
batches_in_memory ↑**: larger shuffle buffer, better randomization, higher memory usagebatches_in_memory ↓**: lower memory usage, less randomizationThe entries are shuffled by default, and shuffled differently at every epoch. The seed makes runs reproducible:
With the default set_seed=0, a random seed is drawn at every run.
To make reading efficient, lazy loading follows the layout of the data on disk: at every epoch, it reads the clusters of entries in a random order, and shuffles the entries of a few clusters together, in a buffer of about batches_in_memory batches. Eager loading shuffles the entries across the whole dataset.
batches_in_memory in lazy loading so that each buffer mixes more clusters, or use eager loading.Pass shuffle=False to read the entries in their order on disk, e.g. for the evaluation of a model.
When one class has many more entries than the other, the loader can rebalance them at every epoch, by drawing fewer entries of the majority class (undersampling) or more entries of the minority class (oversampling). Give the two classes as two datasets:
The dataset with the fewest entries is taken as the minority class. With sampling_type="undersampling", replacement=True lets the majority entries be drawn more than once.
load_eager=True), and needs exactly two input datasets.save() writes a dataset to disk exactly as it would be loaded for training: after the filters, the split, the shuffling and the padding of the vector columns. The output is a TTree by default, or an RNTuple:
The column names are kept, and vector columns are written with their fixed, padded size, so a saved file can be loaded back by a new RDataLoader with the same columns and target:
save() follows the batch_size and drop_remainder of the loader: create it with drop_remainder=False to write all the entries. Saving is also a good way to run an expensive preparation of the data once, and train many times on the result.columns instead of loading all the columns of a dataset.Filters and Defines of the RDataFrame run again at every epoch. If they are expensive, use eager loading, or save the prepared dataset with save() (or Snapshot) and train on the saved file.batches_in_memory trades memory for shuffling quality.RDataLoader does not support ImplicitMT yet, and raises a RuntimeError if it is enabled. You can still use multi-threading in the rest of your script, e.g. to prepare and save the data with Snapshot, as long as you call ROOT.DisableImplicitMT() before creating the loader and its inputs.| Tutorial | Shows |
|---|---|
| ml_dataloader_NumPy.py | Loading batches as NumPy arrays. |
| ml_dataloader_PyTorch.py | A basic PyTorch training loop. |
| ml_dataloader_TensorFlow.py | A basic TensorFlow training loop. |
| ml_dataloader_filters_vectors.py | Filtered datasets and vector columns. |
| ml_dataloader_resampling.py | Balancing an imbalanced dataset with oversampling and undersampling. |
| ml_dataloader_Higgs_Classification.py | Normalizing the features and training a classifier on training, validation and test files. |
| ml_dataloader_XGBoost.py | Training an XGBoost classifier from remote files, with class-balancing weights. |
The full API is described below.
Classes | |
| class | ROOT._pythonization._ml_dataloader.FormattedLoader |
| Iterable that converts each batch to the requested format. More... | |
| class | ROOT._pythonization._ml_dataloader.RDataLoader |
| Entry point for ML batch loading from a ROOT RDataFrame. More... | |
Functions | |
| None | ROOT._pythonization._ml_dataloader.RDataLoader.__init__ (self, ROOT.RDF.RNode|Spec|Iterable[ROOT.RDF.RNode|Spec] inputs, *int batch_size=64, int batches_in_memory=10, list[str]|None columns=None, dict[str, int]|None max_vec_sizes=None, float vec_padding=0.0, str|list[str]|None target=None, str weights="", bool shuffle=True, bool drop_remainder=True, int set_seed=0, bool load_eager=False, str sampling_type="", float sampling_ratio=1.0, bool replacement=False) |
| Args: inputs: RDataFrame or (dataset_name, files) tuple, or a list of them, to load from. | |
| FormattedLoader | ROOT._pythonization._ml_dataloader.RDataLoader.as_jax (self, str|Any device=None) |
| Return an iterable that yields batches as JAX arrays. | |
| FormattedLoader | ROOT._pythonization._ml_dataloader.RDataLoader.as_numpy (self) |
| Return an iterable that yields batches as NumPy arrays. | |
| tf.data.Dataset | ROOT._pythonization._ml_dataloader.RDataLoader.as_tensorflow (self) |
| Return a tf.data.Dataset over batches as TensorFlow tensors. | |
| FormattedLoader | ROOT._pythonization._ml_dataloader.RDataLoader.as_torch (self, str|torch.device|None device=None) |
| Return an iterable that yields batches as PyTorch tensors. | |
| list[str] | ROOT._pythonization._ml_dataloader.RDataLoader.columns (self) |
| All column names as they appear in each batch tensor. | |
| list[str] | ROOT._pythonization._ml_dataloader.RDataLoader.feature_columns (self) |
| Feature column names (columns minus target and weights). | |
| int | ROOT._pythonization._ml_dataloader.RDataLoader.last_batch_no_of_rows (self) |
| Number of rows in the last (remainder) batch, 0 if no remainder. | |
| int | ROOT._pythonization._ml_dataloader.RDataLoader.num_batches (self) |
| Total number of batches in this split for one epoch. | |
| None | ROOT._pythonization._ml_dataloader.RDataLoader.save (self, str dataset_name, str|os.PathLike filename, *str output_format="ttree") |
| Write this split to disk as it would be loaded for training. | |
| list[str] | ROOT._pythonization._ml_dataloader.RDataLoader.target_columns (self) |
| Target column names. | |
| Tuple[RDataLoader, RDataLoader] | ROOT._pythonization._ml_dataloader.RDataLoader.train_test_split (self, float test_size=0.2) |
| Partition the dataset into training and validation splits. | |
| str | ROOT._pythonization._ml_dataloader.RDataLoader.weights_column (self) |
| Weights column name, or empty string if not set. | |
| None ROOT._pythonization._ml_dataloader.RDataLoader.__init__ | ( | self, | |
| ROOT.RDF.RNode | Spec | Iterable[ROOT.RDF.RNode | Spec] | inputs, | ||
| *int | batch_size = 64, | ||
| int | batches_in_memory = 10, | ||
| list[str] | None | columns = None, | ||
| dict[str, int] | None | max_vec_sizes = None, | ||
| float | vec_padding = 0.0, | ||
| str | list[str] | None | target = None, | ||
| str | weights = "", | ||
| bool | shuffle = True, | ||
| bool | drop_remainder = True, | ||
| int | set_seed = 0, | ||
| bool | load_eager = False, | ||
| str | sampling_type = "", | ||
| float | sampling_ratio = 1.0, | ||
| bool | replacement = False ) |
Args: inputs: RDataFrame or (dataset_name, files) tuple, or a list of them, to load from.
A tuple is used to construct a single RDataFrame to read dataset_name from one or more files. batch_size: Number of entries per batch. batches_in_memory: Approximate number of batches held in the shuffle buffer at any time. Larger values improve shuffle quality across cluster boundaries at the cost of higher memory usage. Acts as a soft cap: the buffer may temporarily exceed this. Defaults to 10. columns: Names of feature columns to load. If not given, all columns are used. max_vec_sizes: Maximum size per vector column. Required for RVec columns. vec_padding: Padding value for vectors shorter than their max size. Defaults to 0. target: Name or list of names of target column(s). weights: Column to use for event weighting. Requires a target. shuffle: Whether to shuffle data across cluster boundaries every epoch. Defaults to True. drop_remainder: Drop the last batch if smaller than batch_size. Defaults to True. set_seed: Seed for the random number generator. 0 means a random seed is drawn from the system. Defaults to 0. load_eager: If True, load the full dataset into memory before training. If False (default), load lazily in chunks. sampling_type: Resampling strategy: "undersampling" or "oversampling". Requires load_eager=True and exactly two input dataframes. sampling_ratio: Ratio of minority to majority entries in the resampled dataset. Requires load_eager=True and sampling_type set. replacement: Whether undersampling is with replacement. Requires load_eager=True and sampling_type="undersampling".
Definition at line 626 of file _ml_dataloader.py.
| FormattedLoader ROOT._pythonization._ml_dataloader.RDataLoader.as_jax | ( | self, | |
| str | Any | device = None ) |
Return an iterable that yields batches as JAX arrays.
Args: device: If given, the returned arrays are moved to the specified device. Can be a string (e.g. "cpu", "gpu", "tpu") or any of JAX's device objects.
Definition at line 843 of file _ml_dataloader.py.
| FormattedLoader ROOT._pythonization._ml_dataloader.RDataLoader.as_numpy | ( | self | ) |
Return an iterable that yields batches as NumPy arrays.
Definition at line 780 of file _ml_dataloader.py.
| tf.data.Dataset ROOT._pythonization._ml_dataloader.RDataLoader.as_tensorflow | ( | self | ) |
Return a tf.data.Dataset over batches as TensorFlow tensors.
Definition at line 805 of file _ml_dataloader.py.
| FormattedLoader ROOT._pythonization._ml_dataloader.RDataLoader.as_torch | ( | self, | |
| str | torch.device | None | device = None ) |
Return an iterable that yields batches as PyTorch tensors.
Args: device: If given, the returned tensors are moved to the specified device.
Definition at line 788 of file _ml_dataloader.py.
| list[str] ROOT._pythonization._ml_dataloader.RDataLoader.columns | ( | self | ) |
All column names as they appear in each batch tensor.
Definition at line 862 of file _ml_dataloader.py.
| list[str] ROOT._pythonization._ml_dataloader.RDataLoader.feature_columns | ( | self | ) |
Feature column names (columns minus target and weights).
Definition at line 876 of file _ml_dataloader.py.
| int ROOT._pythonization._ml_dataloader.RDataLoader.last_batch_no_of_rows | ( | self | ) |
Number of rows in the last (remainder) batch, 0 if no remainder.
Definition at line 921 of file _ml_dataloader.py.
| int ROOT._pythonization._ml_dataloader.RDataLoader.num_batches | ( | self | ) |
Total number of batches in this split for one epoch.
Definition at line 906 of file _ml_dataloader.py.
| None ROOT._pythonization._ml_dataloader.RDataLoader.save | ( | self, | |
| str | dataset_name, | ||
| str | os.PathLike | filename, | ||
| *str | output_format = "ttree" ) |
Write this split to disk as it would be loaded for training.
This function preserves the input column names when writing the output dataset. Input columns of collection types keep the same type, but their values will be fixed-sized and padded according to the configuration during loading (e.g. via max_vec_sizes and vec_padding).
Args: dataset_name: Name of the output TTree (or RNTuple). filename: Output file path. output_format: "ttree" (default) or "rntuple".
Definition at line 760 of file _ml_dataloader.py.
| list[str] ROOT._pythonization._ml_dataloader.RDataLoader.target_columns | ( | self | ) |
Target column names.
Definition at line 886 of file _ml_dataloader.py.
| Tuple[RDataLoader, RDataLoader] ROOT._pythonization._ml_dataloader.RDataLoader.train_test_split | ( | self, | |
| float | test_size = 0.2 ) |
Partition the dataset into training and validation splits.
Returns two RDataLoader instances that share the same underlying C++ backend and can each be iterated independently.
Definition at line 744 of file _ml_dataloader.py.
| str ROOT._pythonization._ml_dataloader.RDataLoader.weights_column | ( | self | ) |
Weights column name, or empty string if not set.
Definition at line 896 of file _ml_dataloader.py.