25#include <nlohmann/json.hpp>
1234 void Exec(unsigned int slot)
1236 fPerThreadResults[slot]++;
1239 // Called at the end of the event loop.
1242 *fFinalResult = std::accumulate(fPerThreadResults.begin(), fPerThreadResults.end(), 0);
1245 // Called by RDataFrame to retrieve the name of this action.
1246 std::string GetActionName() const { return "MyCounter"; }
1250 ROOT::RDataFrame df(10);
1251 ROOT::RDF::RResultPtr<int> resultPtr = df.Book<>(MyCounter{df.GetNSlots()}, {});
1252 // The GetValue call triggers the event loop
1253 std::cout << "Number of processed entries: " << resultPtr.GetValue() << std::endl;
1257See the Book() method for more information and [this tutorial](https://root.cern/doc/master/df018__customActions_8C.html)
1258for a more complete example.
1260#### Injecting arbitrary code in the event loop with Foreach() and ForeachSlot()
1262Foreach() takes a callable (lambda expression, free function, functor...) and a list of columns and
1263executes the callable on the values of those columns for each event that passes all upstream selections.
1264It can be used to perform actions that are not already available in the interface. For example, the following snippet
1265evaluates the root mean square of column "x":
1267// Single-thread evaluation of RMS of column "x" using Foreach
1270df.Foreach([&sumSq, &n](double x) { ++n; sumSq += x*x; }, {"x"});
1271std::cout << "rms of x: " << std::sqrt(sumSq / n) << std::endl;
1273In multi-thread runs, users are responsible for the thread-safety of the expression passed to Foreach():
1274thread will execute the expression concurrently.
1275The code above would need to employ some resource protection mechanism to ensure non-concurrent writing of `rms`; but
1276this is probably too much head-scratch for such a simple operation.
1278ForeachSlot() can help in this situation. It is an alternative version of Foreach() for which the function takes an
1279additional "processing slot" parameter besides the columns it should be applied to. RDataFrame
1280guarantees that ForeachSlot() will invoke the user expression with different `slot` parameters for different concurrent
1281executions (see [Special helper columns: rdfentry_ and rdfslot_](\ref helper-cols) for more information on the slot parameter).
1282We can take advantage of ForeachSlot() to evaluate a thread-safe root mean square of column "x":
1284// Thread-safe evaluation of RMS of column "x" using ForeachSlot
1285ROOT::EnableImplicitMT();
1286const unsigned int nSlots = df.GetNSlots();
1287std::vector<double> sumSqs(nSlots, 0.);
1288std::vector<unsigned int> ns(nSlots, 0);
1290df.ForeachSlot([&sumSqs, &ns](unsigned int slot, double x) { sumSqs[slot] += x*x; ns[slot] += 1; }, {"x"});
1291double sumSq = std::accumulate(sumSqs.begin(), sumSqs.end(), 0.); // sum all squares
1292unsigned int n = std::accumulate(ns.begin(), ns.end(), 0); // sum all counts
1293std::cout << "rms of x: " << std::sqrt(sumSq / n) << std::endl;
1295Notice how we created one `double` variable for each processing slot and later merged their results via `std::accumulate`.
1299### Dataset joins with friend trees
1301Vertically concatenating multiple trees that have the same columns (creating a logical dataset with the same columns and
1302more rows) is trivial in RDataFrame: just pass the tree name and a list of file names to RDataFrame's constructor, or create a TChain
1303out of the desired trees and pass that to RDataFrame.
1305Horizontal concatenations of trees or chains (creating a logical dataset with the same number of rows and the union of the
1306columns of multiple trees) leverages TTree's "friend" mechanism.
1308Simple joins of trees that do not have the same number of rows are also possible with indexed friend trees (see below).
1310To use friend trees in RDataFrame, set up trees with the appropriate relationships and then instantiate an RDataFrame
1316main.AddFriend(&friend, "myFriend");
1319auto df2 = df.Filter("myFriend.MyCol == 42");
1322The same applies for TChains. Columns coming from the friend trees can be referred to by their full name, like in the example above,
1323or the friend tree name can be omitted in case the column name is not ambiguous (e.g. "MyCol" could be used instead of
1324"myFriend.MyCol" in the example above if there is no column "MyCol" in the main tree).
1326\note A common source of confusion is that trees that are written out from a multi-thread Snapshot() call will have their
1327 entries (block-wise) shuffled with respect to the original tree. Such trees cannot be used as friends of the original
1328 one: rows will be mismatched.
1330Indexed friend trees provide a way to perform simple joins of multiple trees over a common column.
1331When a certain entry in the main tree (or chain) is loaded, the friend trees (or chains) will then load an entry where the
1332"index" columns have a value identical to the one in the main one. For example, in Python:
1338# If a friend tree has an index on `commonColumn`, when the main tree loads
1339# a given row, it also loads the row of the friend tree that has the same
1340# value of `commonColumn`
1341aux_tree.BuildIndex("commonColumn")
1343mainTree.AddFriend(aux_tree)
1345df = ROOT.RDataFrame(mainTree)
1348RDataFrame supports indexed friend TTrees from ROOT v6.24 in single-thread mode and from v6.28/02 in multi-thread mode.
1350\anchor other-file-formats
1351### Reading data formats other than ROOT trees
1352RDataFrame can be interfaced with RDataSources. The ROOT::RDF::RDataSource interface defines an API that RDataFrame can use to read arbitrary columnar data formats.
1354RDataFrame calls into concrete RDataSource implementations to retrieve information about the data, retrieve (thread-local) readers or "cursors" for selected columns
1355and to advance the readers to the desired data entry.
1356Some predefined RDataSources are natively provided by ROOT such as the ROOT::RDF::RCsvDS which allows to read comma separated files:
1358auto tdf = ROOT::RDF::FromCSV("MuRun2010B.csv");
1359auto filteredEvents =
1360 tdf.Filter("Q1 * Q2 == -1")
1361 .Define("m", "sqrt(pow(E1 + E2, 2) - (pow(px1 + px2, 2) + pow(py1 + py2, 2) + pow(pz1 + pz2, 2)))");
1362auto h = filteredEvents.Histo1D("m");
1366See also FromNumpy (Python-only), FromRNTuple(), FromArrow(), FromSqlite().
1369### Computation graphs (storing and reusing sets of transformations)
1371As we saw, transformed dataframes can be stored as variables and reused multiple times to create modified versions of the dataset. This implicitly defines a **computation graph** in which
1372several paths of filtering/creation of columns are executed simultaneously, and finally aggregated results are produced.
1374RDataFrame detects when several actions use the same filter or the same defined column, and **only evaluates each
1375filter or defined column once per event**, regardless of how many times that result is used down the computation graph.
1376Objects read from each column are **built once and never copied**, for maximum efficiency.
1377When "upstream" filters are not passed, subsequent filters, temporary column expressions and actions are not evaluated,
1378so it might be advisable to put the strictest filters first in the graph.
1380\anchor representgraph
1381### Visualizing the computation graph
1382It is possible to print the computation graph from any node to obtain a [DOT (graphviz)](https://en.wikipedia.org/wiki/DOT_(graph_description_language)) representation either on the standard output
1385Invoking the function ROOT::RDF::SaveGraph() on any node that is not the head node, the computation graph of the branch
1386the node belongs to is printed. By using the head node, the entire computation graph is printed.
1388Following there is an example of usage:
1390// First, a sample computational graph is built
1391ROOT::RDataFrame df("tree", "f.root");
1393auto df2 = df.Define("x", []() { return 1; })
1394 .Filter("col0 % 1 == col0")
1395 .Filter([](int b1) { return b1 <2; }, {"cut1"})
1396 .Define("y", []() { return 1; });
1398auto count = df2.Count();
1400// Prints the graph to the rd1.dot file in the current directory
1401ROOT::RDF::SaveGraph(df, "./mydot.dot");
1402// Prints the graph to standard output
1403ROOT::RDF::SaveGraph(df);
1406The generated graph can be rendered using one of the graphviz filters, e.g. `dot`. For instance, the image below can be generated with the following command:
1408$ dot -Tpng computation_graph.dot -ocomputation_graph.png
1411\image html RDF_Graph2.png
1414### Activating RDataFrame execution logs
1416RDataFrame has experimental support for verbose logging of the event loop runtimes and other interesting related information. It is activated as follows:
1418#include <ROOT/RLogger.hxx>
1420// this increases RDF's verbosity level as long as the `verbosity` variable is in scope
1421auto verbosity = ROOT::Experimental::RLogScopedVerbosity(ROOT::Detail::RDF::RDFLogChannel(), ROOT::Experimental::ELogLevel::kInfo);
1428verbosity = ROOT.Experimental.RLogScopedVerbosity(ROOT.Detail.RDF.RDFLogChannel(), ROOT.Experimental.ELogLevel.kInfo)
1431More information (e.g. start and end of each multi-thread task) is printed using `ELogLevel.kDebug` and even more
1432(e.g. a full dump of the generated code that RDataFrame just-in-time-compiles) using `ELogLevel.kDebug+10`.
1434\anchor rdf-from-spec
1435### Creating an RDataFrame from a dataset specification file
1437RDataFrame can be created using a dataset specification JSON file:
1442df = ROOT.RDF.Experimental.FromSpec("spec.json")
1445The input dataset specification JSON file needs to be provided by the user and it describes all necessary samples and
1446their associated metadata information. The main required key is the "samples" (at least one sample is needed) and the
1447required sub-keys for each sample are "trees" and "files". Additionally, one can specify a metadata dictionary for each
1448sample in the "metadata" key.
1450A simple example for the formatting of the specification in the JSON file is the following:
1456 "trees": ["tree1", "tree2"],
1457 "files": ["file1.root", "file2.root"],
1461 "sample_category" = "data"
1465 "trees": ["tree3", "tree4"],
1466 "files": ["file3.root", "file4.root"],
1470 "sample_category" = "MC_background"
1477The metadata information from the specification file can be then accessed using the DefinePerSample function.
1478For example, to access luminosity information (stored as a double):
1481df.DefinePerSample("lumi", 'rdfsampleinfo_.GetD("lumi")')
1484or sample_category information (stored as a string):
1487df.DefinePerSample("sample_category", 'rdfsampleinfo_.GetS("sample_category")')
1490or directly the filename:
1493df.DefinePerSample("name", "rdfsampleinfo_.GetSampleName()")
1496An example implementation of the "FromSpec" method is available in tutorial: df106_HiggstoFourLeptons.py, which also
1497provides a corresponding exemplary JSON file for the dataset specification.
1500### Adding a progress bar
1502A progress bar showing the processed event statistics can be added to any RDataFrame program.
1503The event statistics include elapsed time, currently processed file, currently processed events, the rate of event processing
1504and an estimated remaining time (per file being processed). It is recorded and printed in the terminal every m events and every
1505n seconds (by default m = 1000 and n = 1). The ProgressBar can be also added when the multithread (MT) mode is enabled.
1507ProgressBar is added after creating the dataframe object (df):
1509ROOT::RDataFrame df("tree", "file.root");
1510ROOT::RDF::Experimental::AddProgressBar(df);
1513Alternatively, RDataFrame can be cast to an RNode first, giving the user more flexibility
1514For example, it can be called at any computational node, such as Filter or Define, not only the head node,
1515with no change to the ProgressBar function itself (please see the [Efficient analysis in Python](#python)
1516section for appropriate usage in Python):
1518ROOT::RDataFrame df("tree", "file.root");
1519auto df_1 = ROOT::RDF::RNode(df.Filter("x>1"));
1520ROOT::RDF::Experimental::AddProgressBar(df_1);
1522Examples of implemented progress bars can be seen by running [Higgs to Four Lepton tutorial](https://root.cern/doc/master/df106__HiggsToFourLeptons_8py_source.html) and [Dimuon tutorial](https://root.cern/doc/master/df102__NanoAODDimuonAnalysis_8C.html).
1542 : RInterface(std::make_shared<
RDFDetail::RLoopManager>(nullptr, defaultColumns))
1545 auto msg =
"Invalid TDirectory!";
1546 throw std::runtime_error(msg);
1548 const std::string treeNameInt(treeName);
1549 auto tree =
static_cast<TTree *
>(dirPtr->
Get(treeNameInt.c_str()));
1551 auto msg =
"Tree \"" + treeNameInt +
"\" cannot be found!";
1552 throw std::runtime_error(msg);
1554 GetProxiedPtr()->SetTree(std::shared_ptr<TTree>(tree, [](
TTree *) {}));
1570RDataFrame::RDataFrame(std::string_view treeName, std::string_view fileNameGlob,
const ColumnNames_t &defaultColumns)
1571 : RInterface(
ROOT::Detail::RDF::CreateLMFromFile(treeName, fileNameGlob, defaultColumns))
1575RDataFrame::RDataFrame(std::string_view treeName, std::string_view fileNameGlob,
const ColumnNames_t &defaultColumns)
1576 : RInterface(
ROOT::Detail::RDF::CreateLMFromTTree(treeName, fileNameGlob, defaultColumns))
1594 const ColumnNames_t &defaultColumns)
1595 : RInterface(
ROOT::Detail::RDF::CreateLMFromFile(datasetName, fileNameGlobs, defaultColumns))
1601 : RInterface(
ROOT::Detail::RDF::CreateLMFromTTree(datasetName, fileNameGlobs, defaultColumns))
1670namespace Experimental {
1704 const nlohmann::ordered_json fullData = nlohmann::ordered_json::parse(std::ifstream(jsonFile));
1705 if (!fullData.contains(
"samples") || fullData[
"samples"].empty()) {
1706 throw std::runtime_error(
1707 R
"(The input specification does not contain any samples. Please provide the samples in the specification like:
1711 "trees": ["tree1", "tree2"],
1712 "files": ["file1.root", "file2.root"],
1713 "metadata": {"lumi": 1.0, }
1716 "trees": ["tree3", "tree4"],
1717 "files": ["file3.root", "file4.root"],
1718 "metadata": {"lumi": 0.5, }
1726 for (
const auto &keyValue : fullData[
"samples"].items()) {
1727 const std::string &sampleName = keyValue.key();
1728 const auto &sample = keyValue.value();
1731 if (!sample.contains(
"trees")) {
1732 throw std::runtime_error(
"A list of tree names must be provided for sample " + sampleName +
".");
1734 std::vector<std::string> trees = sample[
"trees"];
1735 if (!sample.contains(
"files")) {
1736 throw std::runtime_error(
"A list of files must be provided for sample " + sampleName +
".");
1738 std::vector<std::string> files = sample[
"files"];
1739 if (!sample.contains(
"metadata")) {
1743 for (
const auto &metadata : sample[
"metadata"].items()) {
1744 const auto &val = metadata.value();
1745 if (val.is_string())
1746 m.Add(metadata.key(), val.get<std::string>());
1747 else if (val.is_number_integer())
1748 m.Add(metadata.key(), val.get<
int>());
1749 else if (val.is_number_float())
1750 m.Add(metadata.key(), val.get<
double>());
1752 throw std::logic_error(
"The metadata keys can only be of type [string|int|double].");
1757 if (fullData.contains(
"friends")) {
1758 for (
const auto &friends : fullData[
"friends"].items()) {
1759 std::string alias = friends.key();
1760 std::vector<std::string> trees = friends.value()[
"trees"];
1761 std::vector<std::string> files = friends.value()[
"files"];
1762 if (files.size() != trees.size() && trees.size() > 1)
1763 throw std::runtime_error(
"Mismatch between trees and files in a friend.");
1768 if (fullData.contains(
"range")) {
1769 std::vector<int> range = fullData[
"range"];
1771 if (range.size() == 1)
1773 else if (range.size() == 2)
1798 throw std::runtime_error(
"Cannot print information about this RDataFrame, "
1799 "it was not properly created. It must be discarded.");
1801 auto *
tree = lm->GetTree();
1802 auto defCols = lm->GetDefaultColumnNames();
1804 std::ostringstream ret;
1806 ret <<
"A data frame built on top of the " <<
tree->GetName() <<
" dataset.";
1807 if (!defCols.empty()) {
1808 if (defCols.size() == 1)
1809 ret <<
"\nDefault column: " << defCols[0];
1811 ret <<
"\nDefault columns:\n";
1812 for (
auto &&col : defCols) {
1813 ret <<
" - " << col <<
"\n";
1818 ret <<
"A data frame associated to the data source \"" << cling::printValue(ds) <<
"\"";
1820 ret <<
"An empty data frame that will create " << lm->GetNEmptyEntries() <<
" entries\n";
unsigned long long ULong64_t
The head node of a RDF computation graph.
The dataset specification for RDataFrame.
RDatasetSpec & WithGlobalFriends(const std::string &treeName, const std::string &fileNameGlob, const std::string &alias="")
Add friend tree to RDatasetSpec object.
RDatasetSpec & AddSample(RSample sample)
Add sample (RSample class object) to the RDatasetSpec object.
RDatasetSpec & WithGlobalRange(const RDatasetSpec::REntryRange &entryRange={})
Create an RDatasetSpec object for a given range of entries.
Class representing a sample which is a grouping of trees and their fileglobs, and,...
RDataSource * fDataSource
Non-owning pointer to a data-source object. Null if no data-source. RLoopManager has ownership of the...
RDFDetail::RLoopManager * GetLoopManager() const
ROOT's RDataFrame offers a modern, high-level interface for analysis of data stored in TTree ,...
RDataFrame(std::string_view treeName, std::string_view filenameglob, const ColumnNames_t &defaultColumns={})
Build the dataframe.
ROOT::RDF::ColumnNames_t ColumnNames_t
Describe directory structure in memory.
virtual TObject * Get(const char *namecycle)
Return pointer to object identified by namecycle.
A TTree represents a columnar dataset.
ROOT::RDataFrame FromSpec(const std::string &jsonFile)
Factory method to create an RDataFrame from a JSON specification file.
std::vector< std::string > ColumnNames_t
tbb::task_arena is an alias of tbb::interface7::task_arena, which doesn't allow to forward declare tb...
std::shared_ptr< const ColumnNames_t > ColumnNamesPtr_t