Pilot Blade Reconstruction

Link to the git repository

Data Reco

Runs in December

Global runs

Runs 286520

Fill 5575: https://cmswbm.web.cern.ch/cmswbm/cmsdb/servlet/FillReport?FILL=5575
Around 8 hours long.
Golden run with everything at the best timing: "new configuration for Physics, moving to globaldelay25 of 18 ns (for BmO a compromise) and additionally shifting the clock for BmI a further 6 ns (to total 24 ns). This last change is done in portcard 24. Run 286520 started at 2121 UTC. Dead time looks acceptable (~1%)."

RECO Dataset:  /ExpressPhysicsPA/tvami-PilotBlade_pA_data_RECO_Filtered_FEVT_Express-2016HI_Runs286520_CMSSW8023_v1-01d89551f6d39ecac8c03bedefbd04e0/USER
Creation time: 2016-12-09 06:08:01, Dataset size: 391.1GB, Number of blocks: 7, Number of events: 2985781

Runs 286519

http://cmsonline.cern.ch/cms-elog/963173

Dataset: /ExpressPhysicsPA/PARun2016D-Express-v1/FEVT

0ns LS 21-40
4ns LS 61-80
8ns LS 101-120
12ns  LS 141-160
16ns  LS 121-140
20ns LS 81-100
24ns LS 41-60

Runs 286518

"From the DQM plots, I presume you have started run 286518 with BmI at 167 and BmO at 166?

Close. We shifted the BmI TPLL delay so that both can be at 166."

Runs 286517

Run 286517

LS 1-20: Physics
LS 22-50: WBC 168
LS 51-70: WBC 169
LS 71-90: WBC 165
LS 91-110: WBC 166
LS 111-130: WBC 167
LS 131-? : Physics

With the usual caveats that the endpoints of the LS pieces are a bit fuzzy.

miniDAQ

Runs 286497-286504

LHC Fill 5573  Pb-p 6500 Z.GeV
CMS trigger rates up to 95 kHz
Added request of 1kHz random triggers


MiniDAQ WBC scan 5-10 minutes per point.  Tier 0 transfer on for all runs.

Run    WBC
286497 168  run started during ramping, stable from ~30mins
286498 168  0 OOS
286499 167  0 OOS
286500 169  0 OOS
286501 166  0 OOS
286502 165  0 OOS, BXid errors from pix FED 1240 ?
286503 164  0 OOS
286504 165  0 OOS, repeated due to BXid errors above

Runs in November

Runs 285975

LHC Fill: 5546
100_200ns_224Pb_288p_162_162_54_8inj

CMS Run 285975
Stable beams declared 25 minutes in the run around 18:35.
Pilot ROCs WBC=167 in the beginning.
118 - 166 : WBC 165
169 - 220 : WBC 166
221 - 271 : WBC 167
274 - 325 : WBC 168
327 - 365 : WBC 169

Runs 285505

InjectionScheme 100_200ns_288p_224Pb_162_162_54_8inj

Runs 285369

Timing         LS
Run number: 285369 Orbit counter: 189375069 Lumi-section number: 723 0ns start
Run number: 285369 Orbit counter: 193569514 Lumi-section number: 739 0ns stop
Run number: 285369 Orbit counter: 193748361 Lumi-section number: 740 2ns start
Run number: 285369 Orbit counter: 196693794 Lumi-section number: 751 2ns stop
Run number: 285369 Orbit counter: 196977870 Lumi-section number: 752 4ns start
Run number: 285369 Orbit counter: 201462757 Lumi-section number: 769 4ns stop
Run number: 285369 Orbit counter: 201718362 Lumi-section number: 770 6ns start
Run number: 285369 Orbit counter: 206679014 Lumi-section number: 789 6ns stop
Run number: 285369 Orbit counter: 206907818 Lumi-section number: 790 8ns start
Run number: 285369 Orbit counter: 211879067 Lumi-section number: 809 8ns stop
Run number: 285369 Orbit counter: 212166777 Lumi-section number: 810 10ns start
Run number: 285369 Orbit counter: 217225898 Lumi-section number: 829 10ns stop
Run number: 285369 Orbit counter: 217328179 Lumi-section number: 830 12ns start
Run number: 285369 Orbit counter: 222566286 Lumi-section number: 850 12ns stop
Run number: 285369 Orbit counter: 222617518 Lumi-section number: 850 14ns start
Run number: 285369 Orbit counter: 227777975 Lumi-section number: 869 14ns stop
Run number: 285369 Orbit counter: 227879944 Lumi-section number: 870 16ns start
Run number: 285369 Orbit counter: 233048169 Lumi-section number: 889 16ns stop
Run number: 285369 Orbit counter: 233124069 Lumi-section number: 890 18ns start
Run number: 285369 Orbit counter: 235601098 Lumi-section number: 899 18ns stop
Run number: 285369 Orbit counter: 235677589 Lumi-section number: 900 20ns start
Run number: 285369 Orbit counter: 238208074 Lumi-section number: 909 20ns stop
Run number: 285369 Orbit counter: 238336308 Lumi-section number: 910 22ns start
Run number: 285369 Orbit counter: 240762130 Lumi-section number: 919 22ns stop
Run number: 285369 Orbit counter: 240889799 Lumi-section number: 919 24ns start
Run number: 285369 Orbit counter: 243600803 Lumi-section number: 930 24ns stop

RECO Dataset: /ExpressPhysicsPA/tvami-PilotBlade_pA_data_RECO_Filtered_FEVT_Expres-2016HI_Runs285369_CMSSW8023_v1-01d89551f6d39ecac8c03bedefbd04e0/USER
Creation time: 2016-11-22 20:21:31, Dataset size: 523.9GB, Number of blocks: 6, Number of events: 3291817

Runs 285244

fill 5510

Runs 285216-285218

fill 5507 (21h)

Runs 285030-285047

RAW ZeroBias dataset: /ZeroBias/PARun2016B-v1/RAW

Runs 285090-285091

LHC Fill 5505 Single_20p_20Pb_10_10_9_1non_coll Stable Beams declared L1A rate ~13.5k Hz

RAW Dataset: /ZeroBias/PARun2016B-v1/RAW

RECO Dataset: /ZeroBias/tvami-PilotBlade_pA_data_RECO_Filtered_RAW_ZeroBias-2016H_Runs285090-285091_CMSSW8020_v2-795711265f3d8929b82f139d10d39249/USER
Dataset size: 3.8GB, Number of blocks: 2, Number of events: 115475

Plots: https://tvami.web.cern.ch/tvami/projects/PilotBlade/Runs285090-285091_pA_ZeroBias

Runs 284992-285009

RAW ZeroBias dataset: /ZeroBias/PARun2016B-v1/RAW
fileNames = cms.untracked.vstring(
'root://cms-xrd-global.cern.ch//store/hidata/PARun2016B/ZeroBias/RAW/v1/000/284/992/00000/C6F93894-7FA6-E611-B12E-02163E0138F1.root'
'root://cms-xrd-global.cern.ch//store/hidata/PARun2016B/ZeroBias/RAW/v1/000/285/007/00000/323F689A-84A6-E611-ADAA-FA163EA2F1C6.root'
),

Runs 284958

RAW ZeroBias dataset: /ZeroBias/PARun2016B-v1/RAW
fileNames = cms.untracked.vstring(
'root://cms-xrd-global.cern.ch//store/hidata/PARun2016B/ZeroBias/RAW/v1/000/284/958/00000/569080FB-09A6-E611-BF25-FA163E41ABC5.root',
'root://cms-xrd-global.cern.ch//store/hidata/PARun2016B/ZeroBias/RAW/v1/000/284/958/00000/68226679-1BA6-E611-A0C6-02163E013611.root',
),

miniDAQ

Runs 286310-286322

LHC fill:5563 - VdM scan

Run 286310
FED FW v11.10 - L1A rate ~70k Hz
Lots of OOS published in LPM controller page.
ENE burst from ch15 and 16 in the error dumps.

Error fifo was not filled with ENE in the previous fw releases (till 10.12 - I think) 
With FW v11.10 that is fixed!

Run 286312
FED FW v11.5 - L1A rate ~15k Hz + 55k Hz (randoms)
No OOS published in LPM controller page.
No ENEs in the error dumps. (Known to be broken)

Run 286313
FED FW v11.10 - L1A rate ~15k Hz + 55k Hz (randoms)
OOS rate ~1000 per minute
ENE burst from ch15 and 16 in the error dumps.

Run 286315
FED FW v11.10 - L1A rate ~70k Hz 
OOS and ENE as usual.

One of the main source of ENEs seems to be TBM event number 0, which is not a valid event number. 
Take a look at the dump copied at the end of the e-log.(*)

I wonder if FED FW 11.5 (and backwards till 10.12) was checking for ENEs?!?!

Decided to check auto_mask _valid functionality. 
This should be able to mask a tbm channel in the FED after the channel has thrown programmable number of OOSs.
While checking it became clear that OOS counters for FED tbm channels reset when they receive a reysnc from TCDS. 
This is not a desired, since by this way masking (on the fly) can't work. 
If the threshold for the OOS counter(s)  set to 0, than channels are masked.
So, the FW logic works, but resetting the counters at each resync should be changed.
Discussed it with L. Charles. 

Wrote a new detconfig/58 w/ 5 modules. Kept ch15 and 16 out.

Run 286321
LHC dumped the beam and CMS is running cosmics
HV OFF + LV ON
w/ 5 modules 
FED FW v11.10 - L1A rate ~600 Hz + 550 Hz (randoms)  
This time ch19 and 20 are having ENEs in the error dumps.
OOS rate is high. 

Wrote a new detconfig/58 w/ 5 modules. Kept ch19 and 20 out.
Run 286322
w/ 4 moudules + same LHC and trigger conditions as previous run
No OOS, no ENE burst. BmI modules seem to be happy.
Increased randoms. BmI modules are still happy.

Runs 286053-286059

LHC Fill: 5553
100_200ns_540Pb_684_405_351_251_20inj
L1A rate ~85k Hz
Run 286053 - WBC 167
Run 286055 - WBC 168
Run 286056 - WBC 168
Run 286057 - WBC 166
Run 286058 - WBC 169
Run 286059 - WBC 165

Runs 285752

I changed the FED fw back to v11.3 and started a miniDAQ run w/ data transfer on. Run 285752.

Runs in October

Global runs

Runs 284035-284038 and 284077-284078

Date: 2016-10-26 and 2016-10-27
284077-284078 was in LHC Fill 5456 - Single_nominals_10b_5_0_5_MD1814 - started at ~12:00

284077 WBC: 169
284078 WBC: 169

284035 WBC: 170
284036 WBC: 167
284037 WBC: 168
284038 WBC: 166

Dataset: /ZeroBias/Run2016H-v1/RAW

RECO dataset: /ZeroBias/tvami-PilotBlade_pp_data_RECO_Filtered_RAW_ZeroBias-2016H_Runs284035-284038_CMSSW8020_v1-795711265f3d8929b82f139d10d39249/USER
Dataset size: 251.7GB, Number of blocks: 7, Number of events: 1003064

and

RECO dataset: /ZeroBias/tvami-PilotBlade_pp_data_RECO_Filtered_RAW_ZeroBias-2016H_Runs284077-284078_CMSSW8020_v1-795711265f3d8929b82f139d10d39249/USER
Dataset size: 70.0GB, Number of blocks: 5, Number of events: 2195243

Plots: https://tvami.web.cern.ch/tvami/projects/PilotBlade/Runs284035-284038and284077-284078/

Runs 283834-283835

#Event: 133134

Run 283834: WBC=168
Run 283835 
WBC LSStart LSEnd
167 1       13
169 15      27
166 28      40
170 41      53
165 54      68
168 69      end

RAW Dataset:

 /ZeroBias/Run2016H-v1/RAW

RAW RECO Dataset:

/ZeroBias/tvami-PilotBlade_pp_data_RECO_Filtered_RAW_ZeroBias-2016H_Runs283834-283835_CMSSW8020_v1-795711265f3d8929b82f139d10d39249/USER
Dataset size: 22.8GB, Number of blocks: 2, Number of events: 133134

Ntuple Dataset:

Tier2 Budapest:
/tvami/PB/October/ZeroBias/PilotBlade_data_Ntuplizer_pp_Runs283834-283835_CMSSW8020_ZeroBias_Fidu3_v2//161027_180958/0000

Plots: https://tvami.web.cern.ch/tvami/projects/PilotBlade/Runs283834-283835_ZeroBias/

Run 282663

One hour of WBC “0” collisions during the 600x600 run

Run 282649-282650

WBC Key  BmI BmO Run     LS   comment
 0  2443 166 167 282649  all  pause/resume at LS 191 Physics_p74ns_p49ns_6
-1  2569 165 166 282650  5-22 Physics_p99ns_p74ns_6 - start LS 05 - end 22
+1  2565 167 168 282650 24-40 Physics_p49ns_p24ns_6 - start LS 24 - end 40
+2  2563 168 169 282650 43-56 Physics_p24ns_m1ns_6  - start LS 43 - end 56
-2  2571 164 165 282650 58-74 Physics_p124ns_p99ns_6 - start LS 58 - end 74

RAW Dataset:

/ZeroBias/Run2016H-v1/RAW

miniDAQ

Runs 284533-284541

Run in miniDAQ with the latest dac settings, Key 3195; WBC 169 and ch37 out. Run 284533. Duration ~10 min.

Upload the latest FED fw v10.13. 
Run in miniDAQ with WBC 169 and ch37 out. Run 284539. Duration ~10 min.

New Key 3197 with WBC 167.
Run in miniDAQ with WBC 167 and ch37 out. Run 284540. Duration ~10 min.

New Key 3198 w/ detconfig 52. Ch 37 in.
Run in miniDAQ with WBC 167 and ch37 in. Run 284541. Duration ~10 min. Data Transfer ON.

Run 284012

Tier 0 Transfer on for 6 minutes
Changed timing on port card based on Yuta's analysis
Added 1 BX of delay to BmI in port card TPLL
Should make relative alignment of BmI and BmO (finally)
WBC is 170 for this run, which should match the Bunch structure.
Still not clear if we have Bc0 delay in the LPM correct for miniDAQ
No OOS in 284012.  Error FIFO looks like only PKAM sequences.

Run 284007-284011

Run 283974

With FED FW 10.12 in miniDAQ with 100kHz random triggers and cyclic re-sync generators run 283974 lasted > 5 hours, no stuck in BUSY or sync loss. 3 OOS, some timeouts ending run during LHC ramping

Run 283868

RUN 283868
- fw v10.11 
- removed channels with high error rate
- switch to default detconfig 56 with 2 BmI + 2BmO modules
- ran for a rather long time ~3:45h before being stopped manually
- observation: trigger rate seen by pilot goes up with time (lumi dependent?)
- OOS rate ~4Hz (54000 OOS / 3h 45min) but not sure if there is also a time dependency

Run 283862-283867

RUN 283862
- fw v10.8_sc
- Key: 3083
- all Pilot Blade modules included 
- saw a very high rate of OOS ~30Hz, most offending