Date: Thu, 17 Jul 2003 09:41:16 -0700 (PDT) From: John Zweizig To: Keith Riles Cc: Stan Whitcomb , "kats@ligo.mit.edu" , Peter Shawhan Subject: Tabulation of bad data sections Hi Keith, It doesn't seem that I will be able to put together a final list of questionable time periods before I leave today, but I have made considerable progress and I thought that I could a lest give you an idea of where things are. I made one significant discovery, that plotting the maximum values of the DataQual Band limited RMS trends gives a much more "black or white" view of the data and is therefore makes decisions on cutting the data much easier. The are now maximum plots for all the wide rms bands in http://london.ligo-la.caltech.edu/gds/monitor_repots/DataQual/60daysummary/l1-max_bands.pdf http://blue.ligo-wa.caltech.edu/gds/monitor_repots/DataQual/60daysummary/h1-max_bands.pdf http://blue.ligo-wa.caltech.edu/gds/monitor_repots/DataQual/60daysummary/h2-max_bands.pdf Since the fft used to calculate the band rms are repeated every 4 seconds, the maximum plots show the rms in the worst 4 second interval. As you can see from the above links, there are many striking feature in these trends. It is not too surprising that short feature (ring-downs losses of lock, PD saturation) show up more clearly in this plot. I was surprised to see, however, that the ring-up of some optics modes seem clearer in the maximum plot than in the average band (see February 16 at 0230 UTC for example). The danger of course is that in many cases a gravitational wave could show up as a similar spike. To prevent GW effects from showing up in the veto list, the list of times generated by thresholding the maximum plots must be scanned for appropriateness. At present I am doing this by hand, which takes a lot of time. After much contemplation and discussion on this matter, there are two types of segments that I think should be vetoed based on BLRMS. 1) Data in which the BLRMS levels are sufficiently far out of the normal operating range that there is substantial doubt that the IFO is operating in a normal, well understood (or at least reproducible) way. 2) Data in which the rate of change of the general noise level is large enough that it would be difficult to track in the analysis. Note that I don't count the regularly occurring violin modes in this group because it is possible to subtract or notch these away. Furthermore, any veto based on the sensitive bands of AS_Q should show anomalous behavior over a period of at least several minutes to distance it from GW effects. Event vetos will take over for shorter periods of time. I assume that because of the magnitude of the cut thresholds in the 10-100Hz band (>10000 times greater than the noise levels in the other bands) that these cuts can be made with impunity. I assume that it is worthwhile to reject data segments in which there are large effects in the low frequency band even if there is no obvious increase in the RMS of other bands based on my unwillingness to believe that such large effects can really be isolated to the low frequency band. FWIW, there are 159 minutes of L1 data with max>150 in the 10-100Hz band (my current working cut). This corresponds to ~0.5% of the locked L1 data. You can see lists of times where the maximum RMS was greater than my cuts in http://www.ligo.caltech.edu/~jzweizig/S2_Data_Quality/H1-annotated-vetos http://www.ligo.caltech.edu/~jzweizig/S2_Data_Quality/H2-annotated-vetos http://www.ligo.caltech.edu/~jzweizig/S2_Data_Quality/L1-annotated-vetos each line in these files has a GPS time, a UTC time, the number of successive minutes which surpass the cut, the band number (0=10-100Hz, 1=100-200Hz, 2=200-400Hz, 3=400-1kHz, 4=1k-7kHz), a segment ID and a code indicating other known information, i.e. 1 = Data at start of locked segment D = Daq error L = Data at end of locked segment P = Peak-Peak variation flagged by Gaby V = AS_Q PD saturated (only L1) The flagged L1 data seem to be mostly accounted for by known effects. I believe that a reasonably short list of other times will account for everything else (this is in preparation, I will send it from Ireland, Email access permitting). There isn't as much tabulated about saturations/failures in the LHO IFOs, but the lists of times over threshold aren't as long either, so these should be possble to summarize without too much effort also. I have all the DataQual trends on my laptop so I should be able to finish this up relatively quickly. Best regards, John