S2 Segment Data Quality Repository

For each detector, there is an ASCII file ( S2H1_seginfo_work.txt , S2H2_seginfo_work.txt , S2L1_seginfo_work.txt , S2A1_seginfo_work.txt ) which details any information learned about each segment which indicates that it is bad or potentially bad data. These files are maintained by hand. As information is learned and added to these files, they are periodically copied to different names incorporating a specific version number, and the information is also distilled and presented in a few other files:

• S2*v*_seginfo.html :     a copy of the seginfo file with some HTML mark-up.
• S2*v*_segs.html :     a more compact, fixed-format list, consisting of:
1. Segment index (hyperlinked to the seginfo file)
2. Start time
3. End time
4. Duration
5. Bitmask value representing bad-flags which apply to this time interval
6. List of bad-flag names, if any, which apply to this time interval
Note that there may be more than one line for a given segment index, if the set of applicable bad-flags changes during the segment. See, for example, segment 1 in S2H1v01_segs.html . The user is expected to decide which bad-flags should be heeded, omit those time intervals from the list, and stitch back together any contiguous intervals which were not omitted. For example, if you don't care that the first H1 segment was partially outside of the official S2 run, then you can analyze the whole segment (729271373-729274548).
• S2*v*_segs.txt :     like S2*v*_segs.html, but without the hyperlinks.

The goal of this effort is to determine what data is suitable for searching for gravitational waves, and to provide this information in a machine-readable form. The success of this effort will depend on individuals examining the data, identifying effects which might cause the data to be bad, and then evaluating which effects are serious enough to disqualify data. Contributions are eagerly awaited; Keith Riles (kriles@umich.edu) has agreed to collect this information and maintain the segment info files. (Notes for maintainers.)

Ongoing and Suggested Investigations  (New volunteers welcome!)

Meetings concerning data quality repository:

May 15 teleconference: Agenda and summary

Version 01 files -- May 6, 2003

• Resolved discrepancies between conlog segments and SegGener 'IFOLocked' segments. See seginfo files for details.
• Found a few segments which extend beyond the official endpoints of the S2 run, and marked these with the bad-flag OUTSIDE_S2.
• Checked the IFO-SV_DAQ channel in the raw conlog files, and added a note in the seginfo file for each segment which ended with a DAQ "Not OK" condition. Sometimes this led to missing frame data; marked these times with the bad-flag MISSING_RAW.
• Made lists of all EPICS control setting differences from one segment to the next ( S2H1controls.txt , S2H2controls.txt , S2L1controls.txt ), but these have been only partially digested. Following up on a discovery by Betsy Bland, tracked down anomalies in the on/off state of certain filters in the MICH servo loop; these have been marked with the bad-flag MICH_FILT, though it will take further investigation to determine whether or not the data is really bad. Also found a handful of segments with anomalous values of DARM_GAIN, but these should be OK as long as the calibration team takes the gain change into account.

Double IFO:
Segment lists (ASCII): S2H1H2_segs.txt , S2H1L1_segs.txt , S2H2L1_segs.txt

Triple IFO:
Segment list (ASCII): S2H1H2L1_segs.txt

Notes on coincidence segment lists

Version 02 files -- June 19, 2003

• More MICH filter troubles (H2 only, H1 and L1 done in version 1)
• DAQ dropouts (all three IFO's - JohnZ)
• ASQ saturations (all three IFO's - JohnZ)
• Inferred AS Photodiode saturations (L1 only - JohnZ)
• Large peak-to-peak swings in AS_Q near end of lock (L1 only - Gaby)
• Annotations & notes pertaining to controls changes identified in v01 by PeterSh (L1 only - Gaby)

--> Email summary of findings - Only red-flagged segments are actually flagged in version 2 segments below (L1 only - Gaby, Rana)
• Notes on elog entries and email summary (L1 only - Rana)
• Notes on controls changes (H1 only - Betsy/Rana)
• Note 1, note 2 on controls changes and email summary (H2 only - Betsy/Rana)
• Segment-indexed links to L1 trend plots produced by Gaby
• Graphical summaries of segment quality

• Some artifacts in H1 and H2 corresponding to those marked "L1 only" above
• Cuts on the DataQual monitor glitch rates and band-limited RMS
• Cuts on calibration reliability
• Suspected L1 end-station timing problem that affected ~2 days of data

Time / Frequency plots of burst candidate output for S2 playground  data segments (all three IFO's - Laura)

Single IFO:
Segment lists (html): S2H1v02_segs.html , S2H2v02_segs.html , S2L1v02_segs.html
Segment lists (ASCII): S2H1v02_segs.txt , S2H2v02_segs.txt , S2L1v02_segs.txt
Full segment info (html): S2H1v02_seginfo.html , S2H2v02_seginfo.html , S2L1v02_seginfo.html
Full segment info (ASCII): S2H1v02_seginfo.txt , S2H2v02_seginfo.txt , S2L1v02_seginfo.txt

Graphical summaries of segment data quality

Double IFO:
Segment lists (ASCII): S2H1H2_segs.txt , S2H1L1_segs.txt , S2H2L1_segs.txt

Triple IFO:
Segment list (ASCII): S2H1H2L1_segs.txt

Notes on coincidence segment lists

Version 03 files -- August 29, 2003

• Gaby's IFO performance and power minute trends with histograms now available for H1 and L1
• Invalid timing (see note from Brian)
• Missing RDS (see note from PeterSh)
• Unavailable or unreliable calibration (H1, L1)

(see Gaby's H1 note, Gaby's L1 note, and PatrickS's SenseMon page)
• ASQ outliers (H1, L1)

(see Gaby's H1 note and document, Gaby's L1 note and document,
concerning safety of  ASQ outliers correlated with auxiliary channel outliers, see exchange between PeterSh and Gaby, and Keith's followup note)
• ASQ Band-Limited RMS Outliers (H1)

(see JohnZ's note on H1/H2/L1 raw outlier info and his followup recommendations and tables for H1)
• PeterSh has created segment manipulation utilities and posted to LIGOtools(segwizard is especially useful)
• PeterSh has also written a script to produce a lookup table from GPS time to science mode in the three IFO's.
• MikeA has created and posted trends of magnitude and phase of the three calibration lines for each IFO.

• There were no working magnetometers at the LLO end stations during S2

(see note from Doug)
• There was a sample-swapping error affecting some H1 auxiliary channels

(see note from Daniel, elog list of affected channels, memo from Rolf Bork & Dennis Coyne),
note from Keith on RDS channels, and note from JohnZ on the DMT handling of affected channels)
• The ASQ_OUTLIER_CORRELATED data quality flag should be used with care (see ASQ outlier notes above)
• The ASQ_UPPERBAND_OUTLIER data quality flag has intervals as short as 1 minute (see JohnZ's notes above)
• The CALIB_LINE_NO_RDS_V03 data quality flag may be superseded in a later release

Graphical summaries of segment data quality

List of data quality flags with summaries of meanings

Double IFO:
Segment lists (ASCII): S2H1H2_segs.txt , S2H1L1_segs.txt , S2H2L1_segs.txt

Triple IFO:
Segment list (ASCII): S2H1H2L1_segs.txt

Notes on coincidence segment lists

Version 04 files -- May 7, 2004

• Gaby's and PeterSh's updated compilations (H1, H2, L1) of intervals of missing or low-strength calibration lines

(please note that the old flag NO_CALIB_LINE has been superseded by NO_CALIB)
• Addition of Allegro detector segments (filenames, day index, GPS times) (Peter Zhang)
• Flags (used by burst group) for H1 intervals with >5 minutes of "growly" AS_Q in the 200-400 Hz band (John Zweizig)

Graphical summaries of segment data quality

List of data quality flags with summaries of meanings

Double IFO:
Segment lists (ASCII): S2H1H2_segs.txt , S2H1L1_segs.txt , S2H2L1_segs.txt , S2L1A1_segs.txt

Triple IFO:
Segment list (ASCII): S2H1H2L1_segs.txt

Notes on coincidence segment lists

Version 05 files -- June 10, 2004

• JohnZ's refined list of intervals with missing calibration lines
• See also Xavier's list of times (H1,  H2,  L1) and plots (H1,  H2,  L1) when excitation were missing or apparently weak

Graphical summaries of segment data quality

List of data quality flags with summaries of meanings

Double IFO:
Segment lists (ASCII): S2H1H2_segs.txt , S2H1L1_segs.txt , S2H2L1_segs.txt , S2L1A1_segs.txt

Triple IFO:
Segment list (ASCII): S2H1H2L1_segs.txt

Notes on coincidence segment lists

Version 06 files -- November 26, 2004

• Elevated acoustic noise (John Zweizig)
• Airplanes (John Zweizig, Katherine Rawlins)
• Allegro flags for conditions / vetoes (Martin McHugh)
• Three azimuthal orientations (degrees): (252_XARM, 297_NULL, 252_YARM)
• Re-injection of current into transducer: Before and After
• No calibration info
• Phase of reference oscillator under adjustment

Graphical summaries of segment data quality

List of data quality flags with summaries of meanings

Double IFO:
Segment lists (ASCII): S2H1H2_segs.txt , S2H1L1_segs.txt , S2H2L1_segs.txt , S2L1A1_segs.txt

Triple IFO:
Segment list (ASCII): S2H1H2L1_segs.txt

Notes on coincidence segment lists