Logo ROOT   6.12/07
Reference Guide
TTreeCache.cxx
Go to the documentation of this file.
1 // @(#)root/tree:$Id$
2 // Author: Rene Brun 04/06/2006
3 
4 /*************************************************************************
5  * Copyright (C) 1995-2000, Rene Brun and Fons Rademakers. *
6  * All rights reserved. *
7  * *
8  * For the licensing terms see $ROOTSYS/LICENSE. *
9  * For the list of contributors see $ROOTSYS/README/CREDITS. *
10  *************************************************************************/
11 
12 /** \class TTreeCache
13 \ingroup tree
14 
15 A specialized TFileCacheRead object for a TTree.
16 
17 This class acts as a file cache, registering automatically the baskets from
18 the branches being processed (TTree::Draw or TTree::Process and TSelectors)
19 when in the learning phase. The learning phase is by default 100 entries.
20 It can be changed via TTreeCache::SetLearnEntries.
21 
22 This cache speeds-up considerably the performance, in particular
23 when the Tree is accessed remotely via a high latency network.
24 
25 The default cache size (10 Mbytes) may be changed via the function
26 TTree::SetCacheSize
27 
28 Only the baskets for the requested entry range are put in the cache
29 
30 For each Tree being processed a TTreeCache object is created.
31 This object is automatically deleted when the Tree is deleted or
32 when the file is deleted.
33 
34 - Special case of a TChain
35  Once the training is done on the first Tree, the list of branches
36  in the cache is kept for the following files.
37 
38 - Special case of a TEventlist
39  if the Tree or TChain has a TEventlist, only the buffers
40  referenced by the list are put in the cache.
41 
42 The learning period is started or restarted when:
43  - TTree automatically creates a cache. This feature can be
44  controlled with an env. variable or the TTreeCache.Size option.
45  - TTree::SetCacheSize is called with a non-zero size and a cache
46  did not previously exist
47  - TTreeCache::StartLearningPhase is called.
48  - TTreeCache::SetEntryRange is called
49  * and the learning is not yet finished
50  * and has not been set to manual
51  * and the new minimun entry is different.
52 
53 The learning period is stopped (and prefetching is actually started) when:
54  - TTreeCache::StopLearningPhase is called.
55  - An entry outside the 'learning' range is requested
56  The 'learning range is from fEntryMin (default to 0) to
57  fEntryMin + fgLearnEntries (default to 100).
58  - A 'cached' TChain switches over to a new file.
59 
60 ## WHY DO WE NEED the TreeCache when doing data analysis?
61 
62 When writing a TTree, the branch buffers are kept in memory.
63 A typical branch buffersize (before compression) is typically 32 KBytes.
64 After compression, the zipped buffer may be just a few Kbytes.
65 The branch buffers cannot be much larger in case of Trees with several
66 hundred or thousand branches.
67 
68 When writing, this does not generate a performance problem because branch
69 buffers are always written sequentially and the OS is in general clever enough
70 to flush the data to the output file when a few MBytes of data have to be written.
71 When reading at the contrary, one may hit a performance problem when reading
72 across a network (LAN or WAN) and the network latency is high.
73 For example in a WAN with 10ms latency, reading 1000 buffers of 10 KBytes each
74 with no cache will imply 10s penalty where a local read of the 10 MBytes would
75 take about 1 second.
76 
77 The TreeCache will try to prefetch all the buffers for the selected branches
78 such that instead of transferring 1000 buffers of 10 Kbytes, it will be able
79 to transfer one single large buffer of 10 Mbytes in one single transaction.
80 Not only the TreeCache minimizes the number of transfers, but in addition
81 it can sort the blocks to be read in increasing order such that the file
82 is read sequentially.
83 
84 Systems like xrootd, dCache or httpd take advantage of the TreeCache in
85 reading ahead as much data as they can and return to the application
86 the maximum data specified in the cache and have the next chunk of data ready
87 when the next request comes.
88 
89 ## HOW TO USE the TreeCache
90 
91 A few use cases are discussed below. A cache may be created with automatic sizing
92 when a TTree is used:
93 
94 Caches are created and automatically sized for TTrees when TTreeCache.Size or
95 the environment variable ROOT_TTREECACHE_SIZE is set to a sizing factor.
96 
97 But there are many possible configurations where manual control may be wanted.
98 In some applications you know a priori the list of branches to read. In other
99 applications the analysis loop calls several layers of user functions where it
100 is impossible to predict a priori which branches will be used. This
101 is probably the most frequent case. In this case ROOT I/O will flag used
102 branches automatically when a branch buffer is read during the learning phase.
103 The TreeCache interface provides functions to instruct the cache about the used
104 branches if they are known a priori. In the examples below, portions of analysis
105 code are shown. The few statements involving the TreeCache are marked with `//<<<`
106 
107 ### 1. with TTree::Draw
108 
109 the TreeCache is automatically used by TTree::Draw. The function knows
110 which branches are used in the query and it puts automatically these branches
111 in the cache. The entry range is also known automatically.
112 
113 ### 2. with TTree::Process and TSelectors
114 
115 You must enable the cache and tell the system which branches to cache
116 and also specify the entry range. It is important to specify the entry range
117 in case you process only a subset of the events, otherwise you run the risk
118 to store in the cache entries that you do not need.
119 
120 #### example 2a
121 ~~~ {.cpp}
122  TTree *T = (TTree*)f->Get("mytree");
123  Long64_t nentries = T->GetEntries();
124  Int_t cachesize = 10000000; //10 MBytes
125  T->SetCacheSize(cachesize); //<<<
126  T->AddBranchToCache("*",kTRUE); //<<< add all branches to the cache
127  T->Process('myselector.C+");
128  //in the TSelector::Process function we read all branches
129  T->GetEntry(i);
130  ... here you process your entry
131 ~~~
132 #### example 2b
133 
134 in the Process function we read a subset of the branches.
135 Only the branches used in the first entry will be put in the cache
136 ~~~ {.cpp}
137  TTree *T = (TTree*)f->Get("mytree");
138  //we want to process only the 200 first entries
139  Long64_t nentries=200;
140  int efirst= 0;
141  int elast = efirst+nentries;
142  Int_t cachesize = 10000000; //10 MBytes
143  TTreeCache::SetLearnEntries(1); //<<< we can take the decision after 1 entry
144  T->SetCacheSize(cachesize); //<<<
145  T->SetCacheEntryRange(efirst,elast); //<<<
146  T->Process('myselector.C+","",nentries,efirst);
147  // in the TSelector::Process we read only 2 branches
148  TBranch *b1 = T->GetBranch("branch1");
149  b1->GetEntry(i);
150  if (somecondition) return;
151  TBranch *b2 = T->GetBranch("branch2");
152  b2->GetEntry(i);
153  ... here you process your entry
154 ~~~
155 ### 3. with your own event loop
156 
157 #### example 3a
158 
159 in your analysis loop, you always use 2 branches. You want to prefetch
160 the branch buffers for these 2 branches only.
161 ~~~ {.cpp}
162  TTree *T = (TTree*)f->Get("mytree");
163  TBranch *b1 = T->GetBranch("branch1");
164  TBranch *b2 = T->GetBranch("branch2");
165  Long64_t nentries = T->GetEntries();
166  Int_t cachesize = 10000000; //10 MBytes
167  T->SetCacheSize(cachesize); //<<<
168  T->AddBranchToCache(b1,kTRUE); //<<<add branch1 and branch2 to the cache
169  T->AddBranchToCache(b2,kTRUE); //<<<
170  T->StopCacheLearningPhase(); //<<<
171  for (Long64_t i=0;i<nentries;i++) {
172  T->LoadTree(i); //<<< important call when calling TBranch::GetEntry after
173  b1->GetEntry(i);
174  if (some condition not met) continue;
175  b2->GetEntry(i);
176  if (some condition not met) continue;
177  //here we read the full event only in some rare cases.
178  //there is no point in caching the other branches as it might be
179  //more economical to read only the branch buffers really used.
180  T->GetEntry(i);
181  .. process the rare but interesting cases.
182  ... here you process your entry
183  }
184 ~~~
185 #### example 3b
186 
187 in your analysis loop, you always use 2 branches in the main loop.
188 you also call some analysis functions where a few more branches will be read.
189 but you do not know a priori which ones. There is no point in prefetching
190 branches that will be used very rarely.
191 ~~~ {.cpp}
192  TTree *T = (TTree*)f->Get("mytree");
193  Long64_t nentries = T->GetEntries();
194  Int_t cachesize = 10000000; //10 MBytes
195  T->SetCacheSize(cachesize); //<<<
196  T->SetCacheLearnEntries(5); //<<< we can take the decision after 5 entries
197  TBranch *b1 = T->GetBranch("branch1");
198  TBranch *b2 = T->GetBranch("branch2");
199  for (Long64_t i=0;i<nentries;i++) {
200  T->LoadTree(i);
201  b1->GetEntry(i);
202  if (some condition not met) continue;
203  b2->GetEntry(i);
204  //at this point we may call a user function where a few more branches
205  //will be read conditionally. These branches will be put in the cache
206  //if they have been used in the first 10 entries
207  if (some condition not met) continue;
208  //here we read the full event only in some rare cases.
209  //there is no point in caching the other branches as it might be
210  //more economical to read only the branch buffers really used.
211  T->GetEntry(i);
212  .. process the rare but interesting cases.
213  ... here you process your entry
214  }
215 ~~~
216 ## SPECIAL CASES WHERE TreeCache should not be activated
217 
218 When reading only a small fraction of all entries such that not all branch
219 buffers are read, it might be faster to run without a cache.
220 
221 ## HOW TO VERIFY That the TreeCache has been used and check its performance
222 
223 Once your analysis loop has terminated, you can access/print the number
224 of effective system reads for a given file with a code like
225 (where TFile* f is a pointer to your file)
226 ~~~ {.cpp}
227  printf("Reading %lld bytes in %d transactions\n",f->GetBytesRead(), f->GetReadCalls());
228 ~~~
229 */
230 
231 #include "TSystem.h"
232 #include "TEnv.h"
233 #include "TTreeCache.h"
234 #include "TChain.h"
235 #include "TList.h"
236 #include "TBranch.h"
237 #include "TEventList.h"
238 #include "TObjString.h"
239 #include "TRegexp.h"
240 #include "TLeaf.h"
241 #include "TFriendElement.h"
242 #include "TFile.h"
243 #include <limits.h>
244 
246 
248 
249 ////////////////////////////////////////////////////////////////////////////////
250 /// Default Constructor.
251 
253  fEntryMin(0),
254  fEntryMax(1),
255  fEntryCurrent(-1),
256  fEntryNext(-1),
257  fNbranches(0),
258  fNReadOk(0),
259  fNReadMiss(0),
260  fNReadPref(0),
261  fBranches(0),
262  fBrNames(0),
263  fTree(0),
264  fIsLearning(kTRUE),
265  fIsManual(kFALSE),
266  fFirstBuffer(kTRUE),
267  fOneTime(kFALSE),
268  fReverseRead(0),
269  fFillTimes(0),
270  fFirstTime(kTRUE),
271  fFirstEntry(-1),
272  fReadDirectionSet(kFALSE),
273  fEnabled(kTRUE),
274  fPrefillType(GetConfiguredPrefillType()),
275  fAutoCreated(kFALSE)
276 {
277 }
278 
279 ////////////////////////////////////////////////////////////////////////////////
280 /// Constructor.
281 
282 TTreeCache::TTreeCache(TTree *tree, Int_t buffersize) : TFileCacheRead(tree->GetCurrentFile(),buffersize,tree),
283  fEntryMin(0),
284  fEntryMax(tree->GetEntriesFast()),
285  fEntryCurrent(-1),
286  fEntryNext(0),
287  fNbranches(0),
288  fNReadOk(0),
289  fNReadMiss(0),
290  fNReadPref(0),
291  fBranches(0),
292  fBrNames(new TList),
293  fTree(tree),
295  fIsManual(kFALSE),
297  fOneTime(kFALSE),
298  fReverseRead(0),
299  fFillTimes(0),
300  fFirstTime(kTRUE),
301  fFirstEntry(-1),
303  fEnabled(kTRUE),
306 {
308  Int_t nleaves = tree->GetListOfLeaves()->GetEntries();
309  fBranches = new TObjArray(nleaves);
310 }
311 
312 ////////////////////////////////////////////////////////////////////////////////
313 /// Destructor. (in general called by the TFile destructor)
314 
316 {
317  // Informe the TFile that we have been deleted (in case
318  // we are deleted explicitly by legacy user code).
319  if (fFile) fFile->SetCacheRead(0, fTree);
320 
321  delete fBranches;
322  if (fBrNames) {fBrNames->Delete(); delete fBrNames; fBrNames=0;}
323 }
324 
325 ////////////////////////////////////////////////////////////////////////////////
326 /// Add a branch to the list of branches to be stored in the cache
327 /// this function is called by TBranch::GetBasket
328 /// Returns:
329 /// - 0 branch added or already included
330 /// - -1 on error
331 
332 Int_t TTreeCache::AddBranch(TBranch *b, Bool_t subbranches /*= kFALSE*/)
333 {
334  if (!fIsLearning) {
335  return -1;
336  }
337 
338  // Reject branch that are not from the cached tree.
339  if (!b || fTree->GetTree() != b->GetTree()) return -1;
340 
341  // Is this the first addition of a branch (and we are learning and we are in
342  // the expected TTree), then prefill the cache. (We expect that in future
343  // release the Prefill-ing will be the default so we test for that inside the
344  // LearnPrefill call).
345  if (fNbranches == 0 && fEntryMin >= 0 && b->GetReadEntry() == fEntryMin) LearnPrefill();
346 
347  //Is branch already in the cache?
348  Bool_t isNew = kTRUE;
349  for (int i=0;i<fNbranches;i++) {
350  if (fBranches->UncheckedAt(i) == b) {isNew = kFALSE; break;}
351  }
352  if (isNew) {
353  fTree = b->GetTree();
354  fBranches->AddAtAndExpand(b, fNbranches);
355  fBrNames->Add(new TObjString(b->GetName()));
356  fNbranches++;
357  if (gDebug > 0) printf("Entry: %lld, registering branch: %s\n",b->GetTree()->GetReadEntry(),b->GetName());
358  }
359 
360  // process subbranches
361  Int_t res = 0;
362  if (subbranches) {
363  TObjArray *lb = b->GetListOfBranches();
364  Int_t nb = lb->GetEntriesFast();
365  for (Int_t j = 0; j < nb; j++) {
366  TBranch* branch = (TBranch*) lb->UncheckedAt(j);
367  if (!branch) continue;
368  if (AddBranch(branch, subbranches)<0) {
369  res = -1;
370  }
371  }
372  }
373  return res;
374 }
375 
376 ////////////////////////////////////////////////////////////////////////////////
377 /// Add a branch to the list of branches to be stored in the cache
378 /// this is to be used by user (thats why we pass the name of the branch).
379 /// It works in exactly the same way as TTree::SetBranchStatus so you
380 /// probably want to look over there for details about the use of bname
381 /// with regular expressions.
382 /// The branches are taken with respect to the Owner of this TTreeCache
383 /// (i.e. the original Tree)
384 /// NB: if bname="*" all branches are put in the cache and the learning phase stopped
385 /// Returns:
386 /// - 0 branch added or already included
387 /// - -1 on error
388 
389 Int_t TTreeCache::AddBranch(const char *bname, Bool_t subbranches /*= kFALSE*/)
390 {
391  TBranch *branch, *bcount;
392  TLeaf *leaf, *leafcount;
393 
394  Int_t i;
395  Int_t nleaves = (fTree->GetListOfLeaves())->GetEntriesFast();
396  TRegexp re(bname,kTRUE);
397  Int_t nb = 0;
398  Int_t res = 0;
399 
400  // first pass, loop on all branches
401  // for leafcount branches activate/deactivate in function of status
402  Bool_t all = kFALSE;
403  if (!strcmp(bname,"*")) all = kTRUE;
404  for (i=0;i<nleaves;i++) {
405  leaf = (TLeaf*)(fTree->GetListOfLeaves())->UncheckedAt(i);
406  branch = (TBranch*)leaf->GetBranch();
407  TString s = branch->GetName();
408  if (!all) { //Regexp gives wrong result for [] in name
409  TString longname;
410  longname.Form("%s.%s",fTree->GetName(),branch->GetName());
411  if (strcmp(bname,branch->GetName())
412  && longname != bname
413  && s.Index(re) == kNPOS) continue;
414  }
415  nb++;
416  if (AddBranch(branch, subbranches)<0) {
417  res = -1;
418  }
419  leafcount = leaf->GetLeafCount();
420  if (leafcount && !all) {
421  bcount = leafcount->GetBranch();
422  if (AddBranch(bcount, subbranches)<0) {
423  res = -1;
424  }
425  }
426  }
427  if (nb==0 && strchr(bname,'*')==0) {
428  branch = fTree->GetBranch(bname);
429  if (branch) {
430  if (AddBranch(branch, subbranches)<0) {
431  res = -1;
432  }
433  ++nb;
434  }
435  }
436 
437  //search in list of friends
438  UInt_t foundInFriend = 0;
439  if (fTree->GetListOfFriends()) {
440  TIter nextf(fTree->GetListOfFriends());
441  TFriendElement *fe;
442  TString name;
443  while ((fe = (TFriendElement*)nextf())) {
444  TTree *t = fe->GetTree();
445  if (t==0) continue;
446 
447  // If the alias is present replace it with the real name.
448  char *subbranch = (char*)strstr(bname,fe->GetName());
449  if (subbranch!=bname) subbranch = 0;
450  if (subbranch) {
451  subbranch += strlen(fe->GetName());
452  if ( *subbranch != '.' ) subbranch = 0;
453  else subbranch ++;
454  }
455  if (subbranch) {
456  name.Form("%s.%s",t->GetName(),subbranch);
457  if (AddBranch(name, subbranches)<0) {
458  res = -1;
459  }
460  ++foundInFriend;
461  }
462  }
463  }
464  if (!nb && !foundInFriend) {
465  if (gDebug > 0) printf("AddBranch: unknown branch -> %s \n", bname);
466  Error("AddBranch", "unknown branch -> %s", bname);
467  return -1;
468  }
469  //if all branches are selected stop the learning phase
470  if (*bname == '*') {
471  fEntryNext = -1; // We are likely to have change the set of branches, so for the [re-]reading of the cluster.
473  }
474  return res;
475 }
476 
477 ////////////////////////////////////////////////////////////////////////////////
478 /// Remove a branch to the list of branches to be stored in the cache
479 /// this function is called by TBranch::GetBasket.
480 /// Returns:
481 /// - 0 branch dropped or not in cache
482 /// - -1 on error
483 
484 Int_t TTreeCache::DropBranch(TBranch *b, Bool_t subbranches /*= kFALSE*/)
485 {
486  if (!fIsLearning) {
487  return -1;
488  }
489 
490  // Reject branch that are not from the cached tree.
491  if (!b || fTree->GetTree() != b->GetTree()) return -1;
492 
493  //Is branch already in the cache?
494  if (fBranches->Remove(b)) {
495  --fNbranches;
496  if (gDebug > 0) printf("Entry: %lld, un-registering branch: %s\n",b->GetTree()->GetReadEntry(),b->GetName());
497  }
498  delete fBrNames->Remove(fBrNames->FindObject(b->GetName()));
499 
500  // process subbranches
501  Int_t res = 0;
502  if (subbranches) {
503  TObjArray *lb = b->GetListOfBranches();
504  Int_t nb = lb->GetEntriesFast();
505  for (Int_t j = 0; j < nb; j++) {
506  TBranch* branch = (TBranch*) lb->UncheckedAt(j);
507  if (!branch) continue;
508  if (DropBranch(branch, subbranches)<0) {
509  res = -1;
510  }
511  }
512  }
513  return res;
514 }
515 
516 ////////////////////////////////////////////////////////////////////////////////
517 /// Remove a branch to the list of branches to be stored in the cache
518 /// this is to be used by user (thats why we pass the name of the branch).
519 /// It works in exactly the same way as TTree::SetBranchStatus so you
520 /// probably want to look over there for details about the use of bname
521 /// with regular expressions.
522 /// The branches are taken with respect to the Owner of this TTreeCache
523 /// (i.e. the original Tree)
524 /// NB: if bname="*" all branches are put in the cache and the learning phase stopped
525 /// Returns:
526 /// - 0 branch dropped or not in cache
527 /// - -1 on error
528 
529 Int_t TTreeCache::DropBranch(const char *bname, Bool_t subbranches /*= kFALSE*/)
530 {
531  TBranch *branch, *bcount;
532  TLeaf *leaf, *leafcount;
533 
534  Int_t i;
535  Int_t nleaves = (fTree->GetListOfLeaves())->GetEntriesFast();
536  TRegexp re(bname,kTRUE);
537  Int_t nb = 0;
538  Int_t res = 0;
539 
540  // first pass, loop on all branches
541  // for leafcount branches activate/deactivate in function of status
542  Bool_t all = kFALSE;
543  if (!strcmp(bname,"*")) all = kTRUE;
544  for (i=0;i<nleaves;i++) {
545  leaf = (TLeaf*)(fTree->GetListOfLeaves())->UncheckedAt(i);
546  branch = (TBranch*)leaf->GetBranch();
547  TString s = branch->GetName();
548  if (!all) { //Regexp gives wrong result for [] in name
549  TString longname;
550  longname.Form("%s.%s",fTree->GetName(),branch->GetName());
551  if (strcmp(bname,branch->GetName())
552  && longname != bname
553  && s.Index(re) == kNPOS) continue;
554  }
555  nb++;
556  if (DropBranch(branch, subbranches)<0) {
557  res = -1;
558  }
559  leafcount = leaf->GetLeafCount();
560  if (leafcount && !all) {
561  bcount = leafcount->GetBranch();
562  if (DropBranch(bcount, subbranches)<0) {
563  res = -1;
564  }
565  }
566  }
567  if (nb==0 && strchr(bname,'*')==0) {
568  branch = fTree->GetBranch(bname);
569  if (branch) {
570  if (DropBranch(branch, subbranches)<0) {
571  res = -1;
572  }
573  ++nb;
574  }
575  }
576 
577  //search in list of friends
578  UInt_t foundInFriend = 0;
579  if (fTree->GetListOfFriends()) {
580  TIter nextf(fTree->GetListOfFriends());
581  TFriendElement *fe;
582  TString name;
583  while ((fe = (TFriendElement*)nextf())) {
584  TTree *t = fe->GetTree();
585  if (t==0) continue;
586 
587  // If the alias is present replace it with the real name.
588  char *subbranch = (char*)strstr(bname,fe->GetName());
589  if (subbranch!=bname) subbranch = 0;
590  if (subbranch) {
591  subbranch += strlen(fe->GetName());
592  if ( *subbranch != '.' ) subbranch = 0;
593  else subbranch ++;
594  }
595  if (subbranch) {
596  name.Form("%s.%s",t->GetName(),subbranch);
597  if (DropBranch(name, subbranches)<0) {
598  res = -1;
599  }
600  ++foundInFriend;
601  }
602  }
603  }
604  if (!nb && !foundInFriend) {
605  if (gDebug > 0) printf("DropBranch: unknown branch -> %s \n", bname);
606  Error("DropBranch", "unknown branch -> %s", bname);
607  return -1;
608  }
609  //if all branches are selected stop the learning phase
610  if (*bname == '*') {
611  fEntryNext = -1; // We are likely to have change the set of branches, so for the [re-]reading of the cluster.
612  }
613  return res;
614 }
615 
616 ////////////////////////////////////////////////////////////////////////////////
617 /// Fill the cache buffer with the branches in the cache.
618 
620 {
621  if (fNbranches <= 0) return kFALSE;
623  Long64_t entry = tree->GetReadEntry();
624  Long64_t fEntryCurrentMax = 0;
625 
626  if (fEnablePrefetching) { // Prefetching mode
627  if (fIsLearning) { // Learning mode
628  if (fEntryNext >= 0 && entry >= fEntryNext) {
629  // entry is outside the learn range, need to stop the learning
630  // phase. Doing so may trigger a recursive call to FillBuffer in
631  // the process of filling both prefetching buffers
633  fIsManual = kFALSE;
634  }
635  }
636  if (fIsLearning) { // Learning mode
637  entry = 0;
638  }
639  if (fFirstTime) {
640  //try to detect if it is normal or reverse read
641  fFirstEntry = entry;
642  }
643  else {
644  if (fFirstEntry == entry) return kFALSE;
645  // Set the read direction
646  if (!fReadDirectionSet) {
647  if (entry < fFirstEntry) {
650  }
651  else if (entry > fFirstEntry) {
654  }
655  }
656 
657  if (fReverseRead) {
658  // Reverse reading with prefetching
659  if (fEntryCurrent >0 && entry < fEntryNext) {
660  // We can prefetch the next buffer
661  if (entry >= fEntryCurrent) {
662  entry = fEntryCurrent - tree->GetAutoFlush() * fFillTimes;
663  }
664  if (entry < 0) entry = 0;
665  }
666  else if (fEntryCurrent >= 0) {
667  // We are still reading from the oldest buffer, no need to prefetch a new one
668  return kFALSE;
669  }
670  if (entry < 0) return kFALSE;
672  }
673  else {
674  // Normal reading with prefetching
675  if (fEnablePrefetching) {
676  if (entry < 0 && fEntryNext > 0) {
677  entry = fEntryCurrent;
678  } else if (entry >= fEntryCurrent) {
679  if (entry < fEntryNext) {
680  entry = fEntryNext;
681  }
682  }
683  else {
684  // We are still reading from the oldest buffer,
685  // no need to prefetch a new one
686  return kFALSE;
687  }
689  }
690  }
691  }
692  }
693 
694  // If the entry is in the range we previously prefetched, there is
695  // no point in retrying. Note that this will also return false
696  // during the training phase (fEntryNext is then set intentional to
697  // the end of the training phase).
698  if (fEntryCurrent <= entry && entry < fEntryNext) return kFALSE;
699 
700  // Triggered by the user, not the learning phase
701  if (entry == -1) entry = 0;
702 
703  fEntryCurrentMax = fEntryCurrent;
704  TTree::TClusterIterator clusterIter = tree->GetClusterIterator(entry);
705 
706  auto entryCurrent = clusterIter();
707  auto entryNext = clusterIter.GetNextEntry();
708 
709  if (entryNext < fEntryMin || fEntryMax < entryCurrent) {
710  // There is no overlap betweent the cluster we found [entryCurrent, entryNext[
711  // and the authorized range [fEntryMin, fEntryMax]
712  // so we have nothing to do
713  return kFALSE;
714  }
715 
716  fEntryCurrent = entryCurrent;
717  fEntryNext = entryNext;
718 
719 
720  auto firstClusterEnd = fEntryNext;
721 
723  if (fEntryMax <= 0) fEntryMax = tree->GetEntries();
725 
726  if ( fEnablePrefetching ) {
727  if ( entry == fEntryMax ) {
728  // We are at the end, no need to do anything else
729  return kFALSE;
730  }
731  }
732 
733  // Check if owner has a TEventList set. If yes we optimize for this
734  // Special case reading only the baskets containing entries in the
735  // list.
736  TEventList *elist = fTree->GetEventList();
737  Long64_t chainOffset = 0;
738  if (elist) {
739  if (fTree->IsA() ==TChain::Class()) {
740  TChain *chain = (TChain*)fTree;
741  Int_t t = chain->GetTreeNumber();
742  chainOffset = chain->GetTreeOffset()[t];
743  }
744  }
745 
746  //clear cache buffer
747  Int_t fNtotCurrentBuf = 0;
748  if (fEnablePrefetching){ //prefetching mode
749  if (fFirstBuffer) {
751  fNtotCurrentBuf = fNtot;
752  }
753  else {
755  fNtotCurrentBuf = fBNtot;
756  }
757  }
758  else {
760  fNtotCurrentBuf = fNtot;
761  }
762 
763  //store baskets
764  Int_t clusterIterations = 0;
765  Long64_t minEntry = fEntryCurrent;
766  Int_t prevNtot;
767  Int_t minBasket = 0; // We will use this to avoid re-checking the first baskets in the 2nd (or more) run in the while loop.
768  Long64_t maxReadEntry = minEntry; // If we are stopped before the end of the 2nd pass, this marker will where we need to start next time.
769  do {
770  prevNtot = fNtotCurrentBuf;
771  Int_t nextMinBasket = INT_MAX;
772  UInt_t pass = 0;
773  while (pass < 2) {
774  // The first pass we add one basket per branches.
775  // then in the second pass we add the other baskets of the cluster.
776  // This is to support the case where the cache is too small to hold a full cluster.
777  ++pass;
778  for (Int_t i=0;i<fNbranches;i++) {
780  if (b->GetDirectory()==0) continue;
781  if (b->GetDirectory()->GetFile() != fFile) continue;
782  Int_t nb = b->GetMaxBaskets();
783  Int_t *lbaskets = b->GetBasketBytes();
784  Long64_t *entries = b->GetBasketEntry();
785  if (!lbaskets || !entries) continue;
786  //we have found the branch. We now register all its baskets
787  //from the requested offset to the basket below fEntrymax
788  Int_t blistsize = b->GetListOfBaskets()->GetSize();
789  Int_t j=minBasket; // We need this out of the loop so we can find out how far we went.
790  Bool_t firstBasketSeen = kFALSE;
791  for (;j<nb;j++) {
792  // This basket has already been read, skip it
793  if (j<blistsize && b->GetListOfBaskets()->UncheckedAt(j)) continue;
794 
795  Long64_t pos = b->GetBasketSeek(j);
796  Int_t len = lbaskets[j];
797  if (pos <= 0 || len <= 0) continue;
798  if (len > fBufferSizeMin) {
799  // Do not cache a basket if it is bigger than the cache size!
800  continue;
801  }
802  //important: do not try to read fEntryNext, otherwise you jump to the next autoflush
803  if (entries[j] >= fEntryNext) break; // break out of the for each branch loop.
804  if (entries[j] < minEntry && (j<nb-1 && entries[j+1] <= minEntry)) continue;
805  if (elist) {
806  Long64_t emax = fEntryMax;
807  if (j<nb-1) emax = entries[j+1]-1;
808  if (!elist->ContainsRange(entries[j]+chainOffset,emax+chainOffset)) continue;
809  }
810  if (pass==2 && !firstBasketSeen) {
811  // Okay, this has already been requested in the first pass.
812  firstBasketSeen = kTRUE;
813  continue;
814  }
815  fNReadPref++;
816 
817  if ( (fNtotCurrentBuf+len) > fBufferSizeMin ) {
818  // Humm ... we are going to go over the requested size.
819  if (clusterIterations > 0) {
820  // We already have a full cluster and now we would go over the requested
821  // size, let's stop caching (and make sure we start next time from the
822  // end of the previous cluster).
823  if (gDebug > 5) {
824  Info("FillBuffer","Breaking early because %d is greater than %d at cluster iteration %d will restart at %lld",(fNtotCurrentBuf+len), fBufferSizeMin, clusterIterations,minEntry);
825  }
826  fEntryNext = minEntry;
827  break;
828  } else {
829  if (pass == 1) {
830  if ( (fNtotCurrentBuf+len) > 4*fBufferSizeMin ) {
831  // Okay, so we have not even made one pass and we already have
832  // accumulated request for more than twice the memory size ...
833  // So stop for now, and will restart at the same point, hoping
834  // that the basket will still be in memory and not asked again ..
835  fEntryNext = maxReadEntry;
836  if (gDebug > 5) {
837  Info("FillBuffer","Breaking early because %d is greater than 2*%d at cluster iteration %d pass %d will restart at %lld",(fNtotCurrentBuf+len), fBufferSizeMin, clusterIterations,pass,fEntryNext);
838  }
839  break;
840  }
841  } else {
842  // We have made one pass through the branches and thus already
843  // requested one basket per branch, let's stop prefetching
844  // now.
845  if ( (fNtotCurrentBuf+len) > 2*fBufferSizeMin ) {
846  fEntryNext = maxReadEntry;
847  if (gDebug > 5) {
848  Info("FillBuffer","Breaking early because %d is greater than 2*%d at cluster iteration %d pass %d will restart at %lld",(fNtotCurrentBuf+len), fBufferSizeMin, clusterIterations,pass,fEntryNext);
849  }
850  break;
851  }
852  }
853  }
854  }
855  if (fEnablePrefetching){
856  if (fFirstBuffer) {
857  TFileCacheRead::Prefetch(pos,len);
858  fNtotCurrentBuf = fNtot;
859  }
860  else {
862  fNtotCurrentBuf = fBNtot;
863  }
864  }
865  else {
866  TFileCacheRead::Prefetch(pos,len);
867  fNtotCurrentBuf = fNtot;
868  }
869  if ( ( j < (nb-1) ) && entries[j+1] > maxReadEntry ) {
870  maxReadEntry = entries[j+1];
871  }
872  if (fNtotCurrentBuf > 4*fBufferSizeMin) {
873  // Humm something wrong happened.
874  Warning("FillBuffer","There is more data in this cluster (starting at entry %lld to %lld, current=%lld) than usual ... with %d %.3f%% of the branches we already have %d bytes (instead of %d)",
875  fEntryCurrent,fEntryNext, entries[j], i, (100.0*i) / ((float)fNbranches), fNtotCurrentBuf,fBufferSizeMin);
876 
877  }
878  if (pass==1) {
879  // In the first pass, we record one basket per branch and move on to the next branch.
880  break;
881  }
882  }
883 
884  if (j < nextMinBasket) nextMinBasket = j;
885  if (gDebug > 0) printf("Entry: %lld, registering baskets branch %s, fEntryNext=%lld, fNseek=%d, fNtotCurrentBuf=%d\n",minEntry,((TBranch*)fBranches->UncheckedAt(i))->GetName(),fEntryNext,fNseek,fNtotCurrentBuf);
886  }
887  } // loop for the 2 passes.
888  clusterIterations++;
889 
890  minEntry = clusterIter.Next();
891  if (fIsLearning) {
892  fFillTimes++;
893  }
894 
895  // Continue as long as we still make progress (prevNtot < fNtotCurrentBuf), that the next entry range to be looked at,
896  // which start at 'minEntry', is not past the end of the requested range (minEntry < fEntryMax)
897  // and we guess that we not going to go over the requested amount of memory by asking for another set
898  // of entries (fBufferSizeMin > ((Long64_t)fNtotCurrentBuf*(clusterIterations+1))/clusterIterations).
899  // fNtotCurrentBuf / clusterIterations is the average size we are accumulated so far at each loop.
900  // and thus (fNtotCurrentBuf / clusterIterations) * (clusterIterations+1) is a good guess at what the next total size
901  // would be if we run the loop one more time. fNtotCurrentBuf and clusterIterations are Int_t but can sometimes
902  // be 'large' (i.e. 30Mb * 300 intervals) and can overflow the numerical limit of Int_t (i.e. become
903  // artificially negative). To avoid this issue we promote fNtotCurrentBuf to a long long (64 bits rather than 32 bits)
904  if (!((fBufferSizeMin > ((Long64_t)fNtotCurrentBuf*(clusterIterations+1))/clusterIterations) && (prevNtot < fNtotCurrentBuf) && (minEntry < fEntryMax)))
905  break;
906 
907  //for the reverse reading case
908  if (!fIsLearning && fReverseRead){
909  if (clusterIterations >= fFillTimes)
910  break;
911  if (minEntry >= fEntryCurrentMax && fEntryCurrentMax >0)
912  break;
913  }
914  minBasket = nextMinBasket;
915  fEntryNext = clusterIter.GetNextEntry();
917  } while (kTRUE);
918 
919  if (fEntryCurrent > entry || entry >= fEntryNext) {
920  // Something went very wrong and even-though we searched for the baskets
921  // holding 'entry' we somehow ended up with a range of entries that does
922  // validate. So we must have been unable to find or fit the needed basket.
923  // And thus even-though, we know the corresponding baskets wont be in the cache,
924  // Let's make it official that 'entry' is within the range of this TTreeCache ('s search.)
925 
926  // Without this, the next read will be flagged as 'out-of-range' and then we start at
927  // the exact same point as this FillBuffer execution resulting in both the requested
928  // entry still not being part of the cache **and** the beginning of the cluster being
929  // read **again**.
930 
931  fEntryNext = firstClusterEnd;
932  }
933 
934  if (fEnablePrefetching) {
935  if (fIsLearning) {
937  }
938  if (!fIsLearning && fFirstTime){
939  // First time we add autoFlush entries , after fFillTimes * autoFlush
940  // only in reverse prefetching mode
941  fFirstTime = kFALSE;
942  }
943  }
945  return kTRUE;
946 }
947 
948 ////////////////////////////////////////////////////////////////////////////////
949 /// Return the desired prefill type from the environment or resource variable
950 /// - 0 - No prefill
951 /// - 1 - All branches
952 
954 {
955  const char *stcp;
956  Int_t s = 0;
957 
958  if (!(stcp = gSystem->Getenv("ROOT_TTREECACHE_PREFILL")) || !*stcp) {
959  s = gEnv->GetValue("TTreeCache.Prefill", 1);
960  } else {
961  s = TString(stcp).Atoi();
962  }
963 
964  return static_cast<TTreeCache::EPrefillType>(s);
965 }
966 
967 ////////////////////////////////////////////////////////////////////////////////
968 /// Give the total efficiency of the cache... defined as the ratio
969 /// of blocks found in the cache vs. the number of blocks prefetched
970 /// ( it could be more than 1 if we read the same block from the cache more
971 /// than once )
972 ///
973 /// Note: This should eb used at the end of the processing or we will
974 /// get incomplete stats
975 
977 {
978  if ( !fNReadPref )
979  return 0;
980 
981  return ((Double_t)fNReadOk / (Double_t)fNReadPref);
982 }
983 
984 ////////////////////////////////////////////////////////////////////////////////
985 /// This will indicate a sort of relative efficiency... a ratio of the
986 /// reads found in the cache to the number of reads so far
987 
989 {
990  if ( !fNReadOk && !fNReadMiss )
991  return 0;
992 
993  return ((Double_t)fNReadOk / (Double_t)(fNReadOk + fNReadMiss));
994 }
995 
996 ////////////////////////////////////////////////////////////////////////////////
997 /// Static function returning the number of entries used to train the cache
998 /// see SetLearnEntries
999 
1001 {
1002  return fgLearnEntries;
1003 }
1004 
1005 ////////////////////////////////////////////////////////////////////////////////
1006 /// Print cache statistics. Like:
1007 ///
1008 /// ~~~ {.cpp}
1009 /// ******TreeCache statistics for file: cms2.root ******
1010 /// Number of branches in the cache ...: 1093
1011 /// Cache Efficiency ..................: 0.997372
1012 /// Cache Efficiency Rel...............: 1.000000
1013 /// Learn entries......................: 100
1014 /// Reading............................: 72761843 bytes in 7 transactions
1015 /// Readahead..........................: 256000 bytes with overhead = 0 bytes
1016 /// Average transaction................: 10394.549000 Kbytes
1017 /// Number of blocks in current cache..: 210, total size: 6280352
1018 /// ~~~
1019 ///
1020 /// - if option = "a" the list of blocks in the cache is printed
1021 /// see also class TTreePerfStats.
1022 /// - if option contains 'cachedbranches', the list of branches being
1023 /// cached is printed.
1024 
1025 void TTreeCache::Print(Option_t *option) const
1026 {
1027  TString opt = option;
1028  opt.ToLower();
1029  printf("******TreeCache statistics for tree: %s in file: %s ******\n",fTree ? fTree->GetName() : "no tree set",fFile ? fFile->GetName() : "no file set");
1030  if (fNbranches <= 0) return;
1031  printf("Number of branches in the cache ...: %d\n",fNbranches);
1032  printf("Cache Efficiency ..................: %f\n",GetEfficiency());
1033  printf("Cache Efficiency Rel...............: %f\n",GetEfficiencyRel());
1034  printf("Learn entries......................: %d\n",TTreeCache::GetLearnEntries());
1035  if ( opt.Contains("cachedbranches") ) {
1036  opt.ReplaceAll("cachedbranches","");
1037  printf("Cached branches....................:\n");
1038  const TObjArray *cachedBranches = this->GetCachedBranches();
1039  Int_t nbranches = cachedBranches->GetEntriesFast();
1040  for (Int_t i = 0; i < nbranches; ++i) {
1041  TBranch* branch = (TBranch*) cachedBranches->UncheckedAt(i);
1042  printf("Branch name........................: %s\n",branch->GetName());
1043  }
1044  }
1045  TFileCacheRead::Print(opt);
1046 }
1047 
1048 ////////////////////////////////////////////////////////////////////////////////
1049 /// Old method ReadBuffer before the addition of the prefetch mechanism.
1050 
1052  //Is request already in the cache?
1053  if (TFileCacheRead::ReadBuffer(buf,pos,len) == 1){
1054  fNReadOk++;
1055  return 1;
1056  }
1057 
1058  //not found in cache. Do we need to fill the cache?
1059  Bool_t bufferFilled = FillBuffer();
1060  if (bufferFilled) {
1061  Int_t res = TFileCacheRead::ReadBuffer(buf,pos,len);
1062 
1063  if (res == 1)
1064  fNReadOk++;
1065  else if (res == 0)
1066  fNReadMiss++;
1067 
1068  return res;
1069  }
1070  fNReadMiss++;
1071 
1072  return 0;
1073 }
1074 
1075 ////////////////////////////////////////////////////////////////////////////////
1076 /// Used to read a chunk from a block previously fetched. It will call FillBuffer
1077 /// even if the cache lookup succeeds, because it will try to prefetch the next block
1078 /// as soon as we start reading from the current block.
1079 
1081  if (TFileCacheRead::ReadBuffer(buf, pos, len) == 1){
1082  //call FillBuffer to prefetch next block if necessary
1083  //(if we are currently reading from the last block available)
1084  FillBuffer();
1085  fNReadOk++;
1086  return 1;
1087  }
1088 
1089  //keep on prefetching until request is satisfied
1090  // try to prefetch a couple of times and if request is still not satisfied then
1091  // fall back to normal reading without prefetching for the current request
1092  Int_t counter = 0;
1093  while (1) {
1094  if(TFileCacheRead::ReadBuffer(buf, pos, len)) {
1095  break;
1096  }
1097  FillBuffer();
1098  fNReadMiss++;
1099  counter++;
1100  if (counter>1) {
1101  return 0;
1102  }
1103  }
1104 
1105  fNReadOk++;
1106  return 1;
1107 }
1108 
1109 ////////////////////////////////////////////////////////////////////////////////
1110 /// Read buffer at position pos if the request is in the list of
1111 /// prefetched blocks read from fBuffer.
1112 /// Otherwise try to fill the cache from the list of selected branches,
1113 /// and recheck if pos is now in the list.
1114 /// Returns:
1115 /// - -1 in case of read failure,
1116 /// - 0 in case not in cache,
1117 /// - 1 in case read from cache.
1118 /// This function overloads TFileCacheRead::ReadBuffer.
1119 
1121 {
1122  if (!fEnabled) return 0;
1123 
1124  if (fEnablePrefetching)
1125  return TTreeCache::ReadBufferPrefetch(buf, pos, len);
1126  else
1127  return TTreeCache::ReadBufferNormal(buf, pos, len);
1128 }
1129 
1130 ////////////////////////////////////////////////////////////////////////////////
1131 /// This will simply clear the cache
1132 
1134 {
1136 
1137  if (fEnablePrefetching) {
1138  fFirstTime = kTRUE;
1140  }
1141 }
1142 
1143 ////////////////////////////////////////////////////////////////////////////////
1144 /// Change the underlying buffer size of the cache.
1145 /// If the change of size means some cache content is lost, or if the buffer
1146 /// is now larger, setup for a cache refill the next time there is a read
1147 /// Returns:
1148 /// - 0 if the buffer content is still available
1149 /// - 1 if some or all of the buffer content has been made unavailable
1150 /// - -1 on error
1151 
1153 {
1154  Int_t prevsize = GetBufferSize();
1155  Int_t res = TFileCacheRead::SetBufferSize(buffersize);
1156  if (res < 0) {
1157  return res;
1158  }
1159 
1160  if (res == 0 && buffersize <= prevsize) {
1161  return res;
1162  }
1163 
1164  // if content was removed from the buffer, or the buffer was enlarged then
1165  // empty the prefetch lists and prime to fill the cache again
1166 
1168  if (fEnablePrefetching) {
1170  }
1171 
1172  fEntryCurrent = -1;
1173  if (!fIsLearning) {
1174  fEntryNext = -1;
1175  }
1176 
1177  return 1;
1178 }
1179 
1180 ////////////////////////////////////////////////////////////////////////////////
1181 /// Set the minimum and maximum entry number to be processed
1182 /// this information helps to optimize the number of baskets to read
1183 /// when prefetching the branch buffers.
1184 
1186 {
1187  // This is called by TTreePlayer::Process in an automatic way...
1188  // don't restart it if the user has specified the branches.
1189  Bool_t needLearningStart = (fEntryMin != emin) && fIsLearning && !fIsManual;
1190 
1191  fEntryMin = emin;
1192  fEntryMax = emax;
1194  if (gDebug > 0)
1195  Info("SetEntryRange", "fEntryMin=%lld, fEntryMax=%lld, fEntryNext=%lld",
1197 
1198  if (needLearningStart) {
1199  // Restart learning
1201  }
1202 }
1203 
1204 ////////////////////////////////////////////////////////////////////////////////
1205 /// Overload to make sure that the object specific
1206 
1208 {
1209  // The infinite recursion is 'broken' by the fact that
1210  // TFile::SetCacheRead remove the entry from fCacheReadMap _before_
1211  // calling SetFile (and also by setting fFile to zero before the calling).
1212  if (fFile) {
1213  TFile *prevFile = fFile;
1214  fFile = 0;
1215  prevFile->SetCacheRead(0, fTree, action);
1216  }
1217  TFileCacheRead::SetFile(file, action);
1218 }
1219 
1220 ////////////////////////////////////////////////////////////////////////////////
1221 /// Static function to set the number of entries to be used in learning mode
1222 /// The default value for n is 10. n must be >= 1
1223 
1225 {
1226  if (n < 1) n = 1;
1227  fgLearnEntries = n;
1228 }
1229 
1230 ////////////////////////////////////////////////////////////////////////////////
1231 /// Set whether the learning period is started with a prefilling of the
1232 /// cache and which type of prefilling is used.
1233 /// The two value currently supported are:
1234 /// - TTreeCache::kNoPrefill disable the prefilling
1235 /// - TTreeCache::kAllBranches fill the cache with baskets from all branches.
1236 /// The default prefilling behavior can be controlled by setting
1237 /// TTreeCache.Prefill or the environment variable ROOT_TTREECACHE_PREFILL.
1238 
1240 {
1241  fPrefillType = type;
1242 }
1243 
1244 ////////////////////////////////////////////////////////////////////////////////
1245 /// The name should be enough to explain the method.
1246 /// The only additional comments is that the cache is cleaned before
1247 /// the new learning phase.
1248 
1250 {
1251  fIsLearning = kTRUE;
1252  fIsManual = kFALSE;
1253  fNbranches = 0;
1254  if (fBrNames) fBrNames->Delete();
1256  fEntryCurrent = -1;
1257 }
1258 
1259 ////////////////////////////////////////////////////////////////////////////////
1260 /// This is the counterpart of StartLearningPhase() and can be used to stop
1261 /// the learning phase. It's useful when the user knows exactly what branches
1262 /// they are going to use.
1263 /// For the moment it's just a call to FillBuffer() since that method
1264 /// will create the buffer lists from the specified branches.
1265 
1267 {
1268  if (fIsLearning) {
1269  // This will force FillBuffer to read the buffers.
1270  fEntryNext = -1;
1271  fIsLearning = kFALSE;
1272  }
1273  fIsManual = kTRUE;
1274 
1275  //fill the buffers only once during learning
1276  if (fEnablePrefetching && !fOneTime) {
1277  fIsLearning = kTRUE;
1278  FillBuffer();
1279  fOneTime = kTRUE;
1280  }
1281 }
1282 
1283 ////////////////////////////////////////////////////////////////////////////////
1284 /// Update pointer to current Tree and recompute pointers to the branches in the cache.
1285 
1287 {
1288 
1289  fTree = tree;
1290 
1291  fEntryMin = 0;
1292  fEntryMax = fTree->GetEntries();
1293 
1294  fEntryCurrent = -1;
1295 
1296  if (fBrNames->GetEntries() == 0 && fIsLearning) {
1297  // We still need to learn.
1299  } else {
1300  // We learnt from a previous file.
1301  fIsLearning = kFALSE;
1302  fEntryNext = -1;
1303  }
1304  fNbranches = 0;
1305 
1306  TIter next(fBrNames);
1307  TObjString *os;
1308  while ((os = (TObjString*)next())) {
1309  TBranch *b = fTree->GetBranch(os->GetName());
1310  if (!b) {
1311  continue;
1312  }
1313  fBranches->AddAt(b, fNbranches);
1314  fNbranches++;
1315  }
1316 }
1317 
1318 ////////////////////////////////////////////////////////////////////////////////
1319 /// Perform an initial prefetch, attempting to read as much of the learning
1320 /// phase baskets for all branches at once
1321 
1323 {
1324  // This is meant for the learning phase
1325  if (!fIsLearning) return;
1326 
1327  // This should be called before reading entries, otherwise we'll
1328  // always exit here, since TBranch adds itself before reading
1329  if (fNbranches > 0) return;
1330 
1331  // Is the LearnPrefill enabled (using an Int_t here to allow for future
1332  // extension to alternative Prefilling).
1333  if (fPrefillType == kNoPrefill) return;
1334 
1335  // Force only the learn entries to be cached by temporarily setting min/max
1336  // to the learning phase entry range
1337  // But save all the old values, so we can restore everything to how it was
1338  Long64_t eminOld = fEntryMin;
1339  Long64_t emaxOld = fEntryMax;
1340  Long64_t ecurrentOld = fEntryCurrent;
1341  Long64_t enextOld = fEntryNext;
1342 
1345 
1346  // Add all branches to be cached. This also sets fIsManual, stops learning,
1347  // and makes fEntryNext = -1 (which forces a cache fill, which is good)
1348  AddBranch("*");
1349  fIsManual = kFALSE; // AddBranch sets fIsManual, so we reset it
1350 
1351  // Now, fill the buffer with the learning phase entry range
1352  FillBuffer();
1353 
1354  // Leave everything the way we found it
1355  fIsLearning = kTRUE;
1356  DropBranch("*"); // This doesn't work unless we're already learning
1357 
1358  // Restore entry values
1359  fEntryMin = eminOld;
1360  fEntryMax = emaxOld;
1361  fEntryCurrent = ecurrentOld;
1362  fEntryNext = enextOld;
1363 }
virtual const char * GetName() const
Returns name of object.
Definition: TNamed.h:47
Int_t fNtot
Total size of prefetched blocks.
A TLeaf describes individual elements of a TBranch See TBranch structure in TTree.
Definition: TLeaf.h:32
Long64_t fEntryMax
! last entry in the cache
Definition: TTreeCache.h:37
An array of TObjects.
Definition: TObjArray.h:37
Long64_t * GetBasketEntry() const
Definition: TBranch.h:165
Long64_t GetNextEntry()
Definition: TTree.h:271
virtual void Delete(Option_t *option="")
Remove all objects from the list AND delete all heap based objects.
Definition: TList.cxx:467
Int_t fNReadOk
Number of blocks read and found in the cache.
Definition: TTreeCache.h:41
virtual void Info(const char *method, const char *msgfmt,...) const
Issue info message.
Definition: TObject.cxx:854
TFile * fFile
Pointer to file.
long long Long64_t
Definition: RtypesCore.h:69
const TObjArray * GetCachedBranches() const
Definition: TTreeCache.h:76
TObjArray * GetListOfBaskets()
Definition: TBranch.h:193
Collectable string class.
Definition: TObjString.h:28
TObjArray * fBranches
! List of branches to be stored in the cache
Definition: TTreeCache.h:44
A cache when reading files over the network.
const char Option_t
Definition: RtypesCore.h:62
Bool_t fOneTime
! used in the learning phase
Definition: TTreeCache.h:50
virtual void SetCacheRead(TFileCacheRead *cache, TObject *tree=0, ECacheAction action=kDisconnect)
Set a pointer to the read cache.
Definition: TFile.cxx:2242
const Ssiz_t kNPOS
Definition: RtypesCore.h:111
TString & ReplaceAll(const TString &s1, const TString &s2)
Definition: TString.h:638
virtual void SetLearnPrefill(EPrefillType type=kNoPrefill)
Set whether the learning period is started with a prefilling of the cache and which type of prefillin...
A specialized TFileCacheRead object for a TTree.
Definition: TTreeCache.h:30
A ROOT file is a suite of consecutive data records (TKey instances) with a well defined format...
Definition: TFile.h:46
virtual Int_t GetEntries() const
Definition: TCollection.h:177
virtual TList * GetListOfFriends() const
Definition: TTree.h:409
virtual Long64_t GetAutoFlush() const
Definition: TTree.h:366
Regular expression class.
Definition: TRegexp.h:31
TDirectory * GetDirectory() const
Definition: TBranch.h:173
virtual TObject * Remove(TObject *obj)
Remove object from array.
Definition: TObjArray.cxx:703
Ssiz_t Index(const char *pat, Ssiz_t i=0, ECaseCompare cmp=kExact) const
Definition: TString.h:585
Long64_t fEntryMin
! first entry in the cache
Definition: TTreeCache.h:36
Basic string class.
Definition: TString.h:125
void ToLower()
Change string to lower-case.
Definition: TString.cxx:1099
virtual void Prefetch(Long64_t pos, Int_t len)
Add block of length len at position pos in the list of blocks to be prefetched.
int Int_t
Definition: RtypesCore.h:41
bool Bool_t
Definition: RtypesCore.h:59
virtual void StopLearningPhase()
This is the counterpart of StartLearningPhase() and can be used to stop the learning phase...
virtual void LearnPrefill()
Perform an initial prefetch, attempting to read as much of the learning phase baskets for all branche...
void StartLearningPhase()
The name should be enough to explain the method.
virtual Int_t AddBranch(TBranch *b, Bool_t subgbranches=kFALSE)
Add a branch to the list of branches to be stored in the cache this function is called by TBranch::Ge...
Definition: TTreeCache.cxx:332
Long64_t * GetTreeOffset() const
Definition: TChain.h:117
Bool_t fAutoCreated
! true if cache was automatically created
Definition: TTreeCache.h:59
virtual TObject * FindObject(const char *name) const
Delete a TObjLink object.
Definition: TList.cxx:574
virtual Int_t ReadBufferNormal(char *buf, Long64_t pos, Int_t len)
Old method ReadBuffer before the addition of the prefetch mechanism.
Helper class to iterate over cluster of baskets.
Definition: TTree.h:234
Int_t * GetBasketBytes() const
Definition: TBranch.h:164
virtual void SecondPrefetch(Long64_t, Int_t)
virtual void Print(Option_t *option="") const
Print cache statistics.
void Class()
Definition: Class.C:29
virtual Long64_t GetReadEntry() const
Definition: TTree.h:426
Bool_t fIsTransferred
True when fBuffer contains something valid.
virtual Bool_t ContainsRange(Long64_t entrymin, Long64_t entrymax)
Return TRUE if list contains entries from entrymin to entrymax included.
Definition: TEventList.cxx:171
virtual const char * Getenv(const char *env)
Get environment variable.
Definition: TSystem.cxx:1638
virtual TClusterIterator GetClusterIterator(Long64_t firstentry)
Return an iterator over the cluster of baskets starting at firstentry.
Definition: TTree.cxx:5161
virtual Int_t ReadBuffer(char *buf, Long64_t pos, Int_t len)
Read buffer at position pos if the request is in the list of prefetched blocks read from fBuffer...
virtual void ResetCache()
This will simply clear the cache.
Bool_t fEnabled
! cache enabled for cached reading
Definition: TTreeCache.h:56
TObjArray * GetListOfBranches()
Definition: TBranch.h:194
virtual TFile * GetFile() const
Definition: TDirectory.h:147
Double_t GetEfficiencyRel() const
This will indicate a sort of relative efficiency...
Definition: TTreeCache.cxx:988
virtual TTree * GetTree() const
Definition: TTree.h:434
A doubly linked list.
Definition: TList.h:44
virtual TBranch * GetBranch(const char *name)
Return pointer to the branch with the given name in this tree or its friends.
Definition: TTree.cxx:4985
TTree * fTree
! pointer to the current Tree
Definition: TTreeCache.h:46
const char * GetName() const
Returns name of object.
Definition: TObjString.h:39
EPrefillType GetConfiguredPrefillType() const
Return the desired prefill type from the environment or resource variable.
Definition: TTreeCache.cxx:953
virtual void SetEntryRange(Long64_t emin, Long64_t emax)
Set the minimum and maximum entry number to be processed this information helps to optimize the numbe...
virtual void AddAtAndExpand(TObject *obj, Int_t idx)
Add object at position idx.
Definition: TObjArray.cxx:234
virtual Int_t DropBranch(TBranch *b, Bool_t subbranches=kFALSE)
Remove a branch to the list of branches to be stored in the cache this function is called by TBranch:...
Definition: TTreeCache.cxx:484
R__EXTERN TSystem * gSystem
Definition: TSystem.h:540
TTree * GetTree() const
Definition: TTreeCache.h:84
virtual TObject * Remove(TObject *obj)
Remove object from the list.
Definition: TList.cxx:818
virtual Int_t ReadBufferPrefetch(char *buf, Long64_t pos, Int_t len)
Used to read a chunk from a block previously fetched.
virtual ~TTreeCache()
Destructor. (in general called by the TFile destructor)
Definition: TTreeCache.cxx:315
void Form(const char *fmt,...)
Formats a string using a printf style format descriptor.
Definition: TString.cxx:2343
unsigned int UInt_t
Definition: RtypesCore.h:42
Int_t GetEntriesFast() const
Definition: TObjArray.h:64
virtual void Error(const char *method, const char *msgfmt,...) const
Issue error message.
Definition: TObject.cxx:880
virtual TLeaf * GetLeafCount() const
Definition: TLeaf.h:72
virtual void SetFile(TFile *file, TFile::ECacheAction action=TFile::kDisconnect)
Set the file using this cache and reset the current blocks (if any).
Int_t GetMaxBaskets() const
Definition: TBranch.h:196
A TEventList object is a list of selected events (entries) in a TTree.
Definition: TEventList.h:31
Bool_t fReadDirectionSet
! read direction established
Definition: TTreeCache.h:55
Int_t fNReadMiss
Number of blocks read and not found in the cache.
Definition: TTreeCache.h:42
static void SetLearnEntries(Int_t n=10)
Static function to set the number of entries to be used in learning mode The default value for n is 1...
virtual Int_t SetBufferSize(Int_t buffersize)
Sets the buffer size.
virtual Bool_t FillBuffer()
Fill the cache buffer with the branches in the cache.
Definition: TTreeCache.cxx:619
virtual void AddAt(TObject *obj, Int_t idx)
Add object at position ids.
Definition: TObjArray.cxx:253
Int_t fNReadPref
Number of blocks that were prefetched.
Definition: TTreeCache.h:43
virtual Long64_t GetBasketSeek(Int_t basket) const
Return address of basket in the file.
Definition: TBranch.cxx:1244
const Bool_t kFALSE
Definition: RtypesCore.h:88
Bool_t fReverseRead
! reading in reverse mode
Definition: TTreeCache.h:51
Bool_t fFirstTime
! save the fact that we processes the first entry
Definition: TTreeCache.h:53
Long64_t fEntryCurrent
! current lowest entry number in the cache
Definition: TTreeCache.h:38
Int_t fBufferSizeMin
Original size of fBuffer.
Long64_t Next()
Move on to the next cluster and return the starting entry of this next cluster.
Definition: TTree.cxx:602
TObject * UncheckedAt(Int_t i) const
Definition: TObjArray.h:89
TTreeCache()
Default Constructor.
Definition: TTreeCache.cxx:252
virtual void Print(Option_t *option="") const
Print cache statistics.
#define ClassImp(name)
Definition: Rtypes.h:359
double Double_t
Definition: RtypesCore.h:55
Bool_t fIsLearning
! true if cache is in learning mode
Definition: TTreeCache.h:47
virtual Int_t GetTreeNumber() const
Definition: TChain.h:116
int type
Definition: TGX11.cxx:120
R__EXTERN TEnv * gEnv
Definition: TEnv.h:171
virtual Int_t GetBufferSize() const
Bool_t Contains(const char *pat, ECaseCompare cmp=kExact) const
Definition: TString.h:570
virtual void SetFile(TFile *file, TFile::ECacheAction action=TFile::kDisconnect)
Overload to make sure that the object specific.
static constexpr double s
EPrefillType fPrefillType
Whether a pre-filling is enabled (and if applicable which type)
Definition: TTreeCache.h:57
Int_t fFillTimes
! how many times we can fill the current buffer
Definition: TTreeCache.h:52
virtual Int_t SetBufferSize(Int_t buffersize)
Change the underlying buffer size of the cache.
virtual Long64_t GetEntries() const
Definition: TTree.h:382
Int_t fNbranches
! Number of branches in the cache
Definition: TTreeCache.h:40
Bool_t fEnablePrefetching
reading by prefetching asynchronously
virtual void Add(TObject *obj)
Definition: TList.h:87
Long64_t fFirstEntry
! save the value of the first entry
Definition: TTreeCache.h:54
Definition: file.py:1
A TFriendElement TF describes a TTree object TF in a file.
A chain is a collection of files containing TTree objects.
Definition: TChain.h:33
TList * fBrNames
! list of branch names in the cache
Definition: TTreeCache.h:45
Long64_t GetReadEntry() const
Definition: TBranch.h:185
you should not use this method at all Int_t Int_t Double_t Double_t Double_t Int_t Double_t Double_t Double_t Double_t b
Definition: TRolke.cxx:630
Bool_t fFirstBuffer
! true if first buffer is used for prefetching
Definition: TTreeCache.h:49
static Int_t fgLearnEntries
number of entries used for learning mode
Definition: TTreeCache.h:58
TTree * GetTree() const
Definition: TBranch.h:200
Int_t GetEntries() const
Return the number of objects in array (i.e.
Definition: TObjArray.cxx:522
R__EXTERN Int_t gDebug
Definition: Rtypes.h:86
Int_t Atoi() const
Return integer value of string.
Definition: TString.cxx:1975
virtual Int_t ReadBuffer(char *buf, Long64_t pos, Int_t len)
Read buffer at position pos.
Definition: tree.py:1
A TTree object has a header with a name and a title.
Definition: TTree.h:70
TEventList * GetEventList() const
Definition: TTree.h:392
TBranch * GetBranch() const
Definition: TLeaf.h:71
Bool_t fIsManual
! true if cache is StopLearningPhase was used
Definition: TTreeCache.h:48
virtual const char * GetName() const
Returns name of object.
Definition: TObject.cxx:357
virtual Int_t GetSize() const
Definition: TCollection.h:180
A TTree is a list of TBranches.
Definition: TBranch.h:59
Long64_t fEntryNext
! next entry number where cache must be filled
Definition: TTreeCache.h:39
virtual Int_t GetValue(const char *name, Int_t dflt) const
Returns the integer value for a resource.
Definition: TEnv.cxx:491
virtual void UpdateBranches(TTree *tree)
Update pointer to current Tree and recompute pointers to the branches in the cache.
const Bool_t kTRUE
Definition: RtypesCore.h:87
const Int_t n
Definition: legend1.C:16
Double_t GetEfficiency() const
Give the total efficiency of the cache...
Definition: TTreeCache.cxx:976
char name[80]
Definition: TGX11.cxx:109
static Int_t GetLearnEntries()
Static function returning the number of entries used to train the cache see SetLearnEntries.
virtual void Warning(const char *method, const char *msgfmt,...) const
Issue warning message.
Definition: TObject.cxx:866
Int_t fNseek
Number of blocks to be prefetched.
virtual TObjArray * GetListOfLeaves()
Definition: TTree.h:408
ECacheAction
TTreeCache flushing semantics.
Definition: TFile.h:63