Definition at line 85 of file _ml_dataloader.py.
Public Member Functions | |
| None | __init__ (self, ROOT.RDF.RNode|Spec|Iterable[ROOT.RDF.RNode|Spec] inputs, *int batch_size=0, int batches_in_memory=1, list[str]|None columns=None, dict[str, int]|None max_vec_sizes=None, int vec_padding=0, str|list[str]|None target=None, str weights="", float test_size=0.0, bool shuffle=True, bool drop_remainder=True, int set_seed=0, bool load_eager=False, str sampling_type="", float sampling_ratio=1.0, bool replacement=False) |
| Wrapper around the C++ DataLoader. | |
| Activate (self) | |
| Initialize the generator to be used for a loop, this spawns the loading thread. | |
| ActivateTrainingEpoch (self) | |
| Activate the training epoch of the generator. | |
| ActivateValidationEpoch (self) | |
| Activate the validation epoch of the generator. | |
| Any | ConvertBatchToJAX (self, Any batch, device=None) |
| Convert the batch into a JAX array. | |
| np.ndarray | ConvertBatchToNumpy (self, batch) |
| Convert the batch into a NumPy array. | |
| torch.Tensor | ConvertBatchToPyTorch (self, Any batch, device=None) |
| Convert the batch into a PyTorch tensor. | |
| Any | ConvertBatchToTF (self, Any batch) |
| Convert the batch into a TensorFlow tensor. | |
| CreateTrainBatches (self) | |
| Create the first training batches from the first cluster. | |
| CreateValidationBatches (self) | |
| Create the first validation batches from the first cluster. | |
| DeActivate (self) | |
| Deactivate the generator. | |
| DeActivateTrainingEpoch (self) | |
| Deactivate the training epoch of the generator. | |
| DeActivateValidationEpoch (self) | |
| Deactivate the validation epoch of the generator. | |
| GetSample (self) | |
| Return a sample of data that has the same size and types as the actual result. | |
| Any | GetTrainBatch (self) |
| Return the next training batch of data from the given RDataFrame. | |
| Any | GetValidationBatch (self) |
| Return the next training batch of data from the given RDataFrame. | |
| isActive (self) | |
| isTrainingActive (self) | |
| isValidationActive (self) | |
| int | num_training_batches (self) |
| int | num_validation_batches (self) |
| None | Save (self, str dataset_name, str filename, bool is_training, str output_format) |
| int | train_remainder_rows (self) |
| int | val_remainder_rows (self) |
Protected Member Functions | |
| np.ndarray | _get_raw_array (self, batch) |
| list[int] | _resolve_columns (self, ROOT.RDF.RNode x_rdf, list[str]|None columns, list[str] target_columns, str weights_column, dict[str, int]|None max_vec_sizes=None) |
| Resolve the columns to load based on the given RDataFrame and columns. | |
| np.ndarray|Tuple[np.ndarray, np.ndarray]|Tuple[np.ndarray, np.ndarray, np.ndarray] | _split_target_and_weights (self, np.ndarray data) |
| None ROOT._pythonization._ml_dataloader._RDataLoader.__init__ | ( | self, | |
| ROOT.RDF.RNode | Spec | Iterable[ROOT.RDF.RNode | Spec] | inputs, | ||
| *int | batch_size = 0, | ||
| int | batches_in_memory = 1, | ||
| list[str] | None | columns = None, | ||
| dict[str, int] | None | max_vec_sizes = None, | ||
| int | vec_padding = 0, | ||
| str | list[str] | None | target = None, | ||
| str | weights = "", | ||
| float | test_size = 0.0, | ||
| bool | shuffle = True, | ||
| bool | drop_remainder = True, | ||
| int | set_seed = 0, | ||
| bool | load_eager = False, | ||
| str | sampling_type = "", | ||
| float | sampling_ratio = 1.0, | ||
| bool | replacement = False ) |
Wrapper around the C++ DataLoader.
Args: inputs (ROOT.RDF.RNode | Spec | Iterable[ROOT.RDF.RNode | Spec]): RDataFrame or (dataset_name, files) tuple, or a list of them, to load from. A tuple is used to construct a single RDataFrame to read dataset_name from one or more files. batch_size (int): Number of entries per batch returned by the generator. batches_in_memory (int): Approximate number of batches that should be kept in memory at the same time. Higher value results in faster loading, but also higher memory usage. Defaults to 1. columns (list[str] | None): Names of feature columns to load. If not given, all columns are used. max_vec_sizes (dict[str, int] | None): Mapping from vector column name to the maximum size of the vector. Required when using vector based columns. vec_padding (int): Value used to pad vectors with if the vector is smaller than the given max vector length. Defaults to 0. target (str | list[str] | None): Name or list of names of target column(s). weights (str): Column used to weight events. Can only be used when a target is given. test_size (float): The ratio of batches being kept for validation. Value has to be between 0 and 1. Defaults to 0.0. shuffle (bool): Batches consist of random events and are shuffled every epoch. Defaults to True. drop_remainder (bool): Drop the remainder of data that is too small to compose full batch. Defaults to True. set_seed (int): For reproducibility: Set the seed for the random number generator used to split the dataset into training and validation as well as shuffing of the entries. Defaults to 0 which means that the seed is set to the random device. load_eager (bool): If True, load the full dataset(s) into memory. If False, load data lazily in clusters. Defaults to False. sampling_type (str): Describes the mode of sampling from the minority and majority dataframes. Supported values are "undersampling" and "oversampling". Requires load_eager=True. Defaults to "". For 'undersampling' and 'oversampling' it requires a list of exactly two dataframes as input, where the dataframe with the most entries is the majority dataframe and the dataframe with the fewest entries is the minority dataframe. sampling_ratio (float): Ratio of minority and majority entries in the resampled dataset. Requires load_eager=True and sampling_type="undersampling" or "oversampling". Defaults to 1.0. replacement (bool): Whether the sampling is with (True) or without (False) replacement. Requires load_eager=True and sampling_type="undersampling". Defaults to False.
Definition at line 164 of file _ml_dataloader.py.
|
protected |
Definition at line 395 of file _ml_dataloader.py.
|
protected |
Resolve the columns to load based on the given RDataFrame and columns.
Args: x_rdf (RNode): RDataFrame or RNode object. columns (list[str], optional): Feature columns to load. Defaults to all columns in the RDataFrame, except for target_columns and weights_column. target_columns (list[str]): Name(s) of target column(s). weights_column (str): Name of the weight column, or "" for none. max_vec_sizes (dict[str, int], optional): Mapping from vector column name to the maximum size of the vector. Required when using vector based columns.
Returns: list[int]: List of max vector sizes
Definition at line 86 of file _ml_dataloader.py.
|
protected |
Definition at line 405 of file _ml_dataloader.py.
| ROOT._pythonization._ml_dataloader._RDataLoader.Activate | ( | self | ) |
Initialize the generator to be used for a loop, this spawns the loading thread.
Definition at line 305 of file _ml_dataloader.py.
| ROOT._pythonization._ml_dataloader._RDataLoader.ActivateTrainingEpoch | ( | self | ) |
Activate the training epoch of the generator.
Definition at line 313 of file _ml_dataloader.py.
| ROOT._pythonization._ml_dataloader._RDataLoader.ActivateValidationEpoch | ( | self | ) |
Activate the validation epoch of the generator.
Definition at line 318 of file _ml_dataloader.py.
| Any ROOT._pythonization._ml_dataloader._RDataLoader.ConvertBatchToJAX | ( | self, | |
| Any | batch, | ||
| device = None ) |
Convert the batch into a JAX array.
Args: batch: Batch returned from the DataLoader
Returns: jax.Array: converted batch
Definition at line 492 of file _ml_dataloader.py.
| np.ndarray ROOT._pythonization._ml_dataloader._RDataLoader.ConvertBatchToNumpy | ( | self, | |
| batch ) |
Convert the batch into a NumPy array.
Args: batch: Batch returned from the DataLoader
Returns: np.ndarray: converted batch
Definition at line 429 of file _ml_dataloader.py.
| torch.Tensor ROOT._pythonization._ml_dataloader._RDataLoader.ConvertBatchToPyTorch | ( | self, | |
| Any | batch, | ||
| device = None ) |
Convert the batch into a PyTorch tensor.
Args: batch: Batch returned from the DataLoader
Returns: torch.Tensor: converted batch
Definition at line 440 of file _ml_dataloader.py.
| Any ROOT._pythonization._ml_dataloader._RDataLoader.ConvertBatchToTF | ( | self, | |
| Any | batch ) |
Convert the batch into a TensorFlow tensor.
Args: batch: Batch returned from the DataLoader
Returns: tensorflow.Tensor: converted batch
Definition at line 458 of file _ml_dataloader.py.
| ROOT._pythonization._ml_dataloader._RDataLoader.CreateTrainBatches | ( | self | ) |
Create the first training batches from the first cluster.
Definition at line 331 of file _ml_dataloader.py.
| ROOT._pythonization._ml_dataloader._RDataLoader.CreateValidationBatches | ( | self | ) |
Create the first validation batches from the first cluster.
Definition at line 335 of file _ml_dataloader.py.
| ROOT._pythonization._ml_dataloader._RDataLoader.DeActivate | ( | self | ) |
Deactivate the generator.
Definition at line 309 of file _ml_dataloader.py.
| ROOT._pythonization._ml_dataloader._RDataLoader.DeActivateTrainingEpoch | ( | self | ) |
Deactivate the training epoch of the generator.
Definition at line 323 of file _ml_dataloader.py.
| ROOT._pythonization._ml_dataloader._RDataLoader.DeActivateValidationEpoch | ( | self | ) |
Deactivate the validation epoch of the generator.
Definition at line 327 of file _ml_dataloader.py.
| ROOT._pythonization._ml_dataloader._RDataLoader.GetSample | ( | self | ) |
Return a sample of data that has the same size and types as the actual result.
This sample can be used to define the shape and size of the output
Returns: np.ndarray: data sample
Definition at line 355 of file _ml_dataloader.py.
| Any ROOT._pythonization._ml_dataloader._RDataLoader.GetTrainBatch | ( | self | ) |
Return the next training batch of data from the given RDataFrame.
Returns: (np.ndarray): Batch of data of size.
Definition at line 517 of file _ml_dataloader.py.
| Any ROOT._pythonization._ml_dataloader._RDataLoader.GetValidationBatch | ( | self | ) |
Return the next training batch of data from the given RDataFrame.
Returns: (np.ndarray): Batch of data of size.
Definition at line 527 of file _ml_dataloader.py.
| ROOT._pythonization._ml_dataloader._RDataLoader.isActive | ( | self | ) |
Definition at line 296 of file _ml_dataloader.py.
| ROOT._pythonization._ml_dataloader._RDataLoader.isTrainingActive | ( | self | ) |
Definition at line 299 of file _ml_dataloader.py.
| ROOT._pythonization._ml_dataloader._RDataLoader.isValidationActive | ( | self | ) |
Definition at line 302 of file _ml_dataloader.py.
| int ROOT._pythonization._ml_dataloader._RDataLoader.num_training_batches | ( | self | ) |
Definition at line 340 of file _ml_dataloader.py.
| int ROOT._pythonization._ml_dataloader._RDataLoader.num_validation_batches | ( | self | ) |
Definition at line 344 of file _ml_dataloader.py.
| None ROOT._pythonization._ml_dataloader._RDataLoader.Save | ( | self, | |
| str | dataset_name, | ||
| str | filename, | ||
| bool | is_training, | ||
| str | output_format ) |
Definition at line 537 of file _ml_dataloader.py.
| int ROOT._pythonization._ml_dataloader._RDataLoader.train_remainder_rows | ( | self | ) |
Definition at line 348 of file _ml_dataloader.py.
| int ROOT._pythonization._ml_dataloader._RDataLoader.val_remainder_rows | ( | self | ) |
Definition at line 352 of file _ml_dataloader.py.
| ROOT._pythonization._ml_dataloader._RDataLoader.all_columns |
Definition at line 124 of file _ml_dataloader.py.
| ROOT._pythonization._ml_dataloader._RDataLoader.batch_size |
Definition at line 262 of file _ml_dataloader.py.
| ROOT._pythonization._ml_dataloader._RDataLoader.DeActivate |
Definition at line 293 of file _ml_dataloader.py.
| ROOT._pythonization._ml_dataloader._RDataLoader.engine |
Definition at line 276 of file _ml_dataloader.py.
| ROOT._pythonization._ml_dataloader._RDataLoader.feature_columns |
Definition at line 160 of file _ml_dataloader.py.
| ROOT._pythonization._ml_dataloader._RDataLoader.given_columns |
Definition at line 123 of file _ml_dataloader.py.
| ROOT._pythonization._ml_dataloader._RDataLoader.noded_rdfs |
Definition at line 246 of file _ml_dataloader.py.
| ROOT._pythonization._ml_dataloader._RDataLoader.num_columns |
Definition at line 261 of file _ml_dataloader.py.
| ROOT._pythonization._ml_dataloader._RDataLoader.target_columns |
Definition at line 249 of file _ml_dataloader.py.
| ROOT._pythonization._ml_dataloader._RDataLoader.target_given |
Definition at line 251 of file _ml_dataloader.py.
| ROOT._pythonization._ml_dataloader._RDataLoader.target_slice |
Definition at line 159 of file _ml_dataloader.py.
| ROOT._pythonization._ml_dataloader._RDataLoader.train_slice |
Definition at line 158 of file _ml_dataloader.py.
| ROOT._pythonization._ml_dataloader._RDataLoader.weights_column |
Definition at line 250 of file _ml_dataloader.py.
| ROOT._pythonization._ml_dataloader._RDataLoader.weights_given |
Definition at line 252 of file _ml_dataloader.py.
| ROOT._pythonization._ml_dataloader._RDataLoader.weights_index |
Definition at line 266 of file _ml_dataloader.py.