Migrating S5 offline DQ flags to S6 online DMT monitors or cron scripts


DQ Flags already Created Online

During the S5 run there were a handful of data quality flags set automatically by DMT monitors (written by John Zweizig):

S5 Flag name
(click for details)
ASC_Overflow
ASI_CORR_OVERFLOW
H1_Not_Locked
H2_Not_Locked
Injection
LSC_OVERFLOW
PD_Overflow
SEVERE_LSC_OVERFLOW
Wind_Over_30MPH


Candidate Flags for Migration

As we look ahead to the S6 run, we should migrate many more flags that are now determined from offline studies to online DMT monitors (or to cron jobs looking at second/minute trends that are monitored by scimons). The table below lists candidates for such migration (click on links to see background information on flaginfo page), a summary of the method used to determine the flag now, the person or persons who have compiled the flags for S5, and possible ways to do it faster for S6.

(A complete list of S5 offline DQ flags (defined to date) with documentation links can be found at http://gallatin.physics.lsa.umich.edu/~keithr/S5DQ/flaginfo.html.)

S5 Flag name
(click for details)
Method used Investigator(s) Candidate migration path(s)
CALIB_BAD_COEFFS_60 Determined as part of calibration release from large excursions in alpha, beta Myungkee Sung Online DMT monitor, using StrainWatch class
or using cron job looking at h(t) generation
CALIB_DROPOUT_1SAMPLE
CALIB_DROPOUT_1SEC CALIB_DROPOUT_AWG_STUCK CALIB_GLITCH_ZG
Scan of Zglitch triggers John Zweizig Online DMT monitor
CHECKSUM_MISMATCH Parsing of output of automated (but unreliable) program Ben Johnson Cron job with scimon monitoring - easy restart needed
DARM_09_11_dHz_LOWTHRESH
DARM_09_11_dHz_MEDTHRESH
DARM_09_11_dHz_HIGHTHRESH
DARM_11_13_dHz_LOWTHRESH
DARM_11_13_dHz_MEDTHRESH
DARM_11_13_dHz_HIGHTHRESH
DARM_18_24_dHz_LOWTHRESH
DARM_18_24_dHz_MEDTHRESH
DARM_18_24_dHz_HIGHTHRESH
DARM_50_70_dHz_LOWTHRESH
DARM_50_70_dHz_MEDTHRESH
DARM_50_70_dHz_HIGHTHRESH
Customized offline DMT run with threshold on minute trends Justin Garofoli Online DMT monitor
DEWAR_GLITCH Inspection of glitches reported by DewarMon Emelie Harstad Online DMT monitor
INVALID_DARMERR Offline scan of RDS frames John Zweizig Cron job with scimon monitoring - easy restart needed
LIGHTDIP_02_PERCENT LIGHTDIP_03_PERCENT
LIGHTDIP_04_PERCENT
LIGHTDIP_06_PERCENT
LIGHTDIP_07_PERCENT
LIGHTDIP_08_PERCENT
LIGHTDIP_10_PERCENT
LIGHTDIP_12_PERCENT
LIGHTDIP_15_PERCENT
LIGHTDIP_20_PERCENT
Offline scan of second trends Peter Shawhan Online DMT monitor
MASTER_OVERFLOW_ASC MASTER_OVERFLOW_LSC
MASTER_OVERFLOW_SUS_MC2
MASTER_OVERFLOW_SUS_RM
Offline scan of second trends Peter Shawhan Online DMT monitor or cron job
MISSING_RAW
MISSING_RDS_LEVEL_1
MISSING_RDS_LEVEL_3
MISSING_RDS_LEVEL_4
Automated scripts Greg Mendell Cron job with scimon monitoring - easy restart needed
MISSING_RDS_C02_LX
MISSING_RDS_C03_L1
MISSING_RDS_C03_L2
Detection of h(t) gaps Xavi Siemens,
Shourov Chatterji,
Greg Mendell,
Drew Keppel
Cron job with scimon monitoring - easy restart needed
NO_CALIB_LINE Determined as part of calibration release Myungkee Sung Online DMT monitor
OUT_OF_LOCK
PRE_LOCKLOSS_10_SEC
PRE_LOCKLOSS_30_SEC
PRE_LOCKLOSS_60_SEC
PRE_LOCKLOSS_120_SEC
Offline scan of second trends Peter Shawhan Online DMT monitor or cron job
PHOTODIODE_GLITCH
PHOTODIODE_OFF
Study of PDnMon triggers John Zweizig Online DMT monitor
POWMAG Study of KleineWelle magnetometer glitches Laura Cadonati Online DMT monitor
SEIS_DARMERR_5_7HZ Study of glitch rates and inspiral range Masahiro Ito Online DMT monitor
SEISMIC_EY_99PCTL_3_10HZ Study of minute trends Masahiro Ito Online DMT monitor
SEISMIC_X_30_100_mHz_LOWTHRESH
SEISMIC_X_30_100_mHz_MEDTHRESH
SEISMIC_X_30_100_mHz_HIGHTHRESH
SEISMIC_Y_30_100_mHz_LOWTHRESH
SEISMIC_Y_30_100_mHz_MEDTHRESH
SEISMIC_Y_30_100_mHz_HIGHTHRESH
SEISMIC_Y_30_100_mHz_LOWTHRESH
SEISMIC_Y_30_100_mHz_MEDTHRESH
SEISMIC_Y_30_100_mHz_HIGHTHRESH
Study of minute trends Alistair Hayden Online DMT monitor
SIDECOIL_ETMX
SIDECOIL_ETMY
Study of second trends Laura Cadonati Online DMT monitor
SIDECOIL_ETMX_RMS_6HZ
SIDECOIL_ETMY_RMS_6HZ
Offline DMT scan of band-limited RMS John Zweizig Online DMT monitor
TCS_GLITCH_LOUD Study of KleineWelle triggers Shantanu Desai Online DMT monitor
TIDAL_SERVO_DESATURATION
TIDAL_SERVO_PRESATURATION
Study of KleineWelle triggers Shantanu Desai Online DMT monitor or cron job
TRAIN_LIKELY Offline scan of seismic minute trends Jake Slutsky Online DMT monitor


Consolidating Programs

Many of these tasks can be combined together into DMT monitors or cron scripts. Here is one possible grouping:

S5 Flag name
(click for details)
Candidate migration path(s)
CALIB_BAD_COEFFS_60
MISSING_RDS_C02_LX
MISSING_RDS_C03_L1
MISSING_RDS_C03_L2
Online DMT monitor, using StrainWatch class
or using cron job looking at h(t) generation or both
CALIB_DROPOUT_1SAMPLE
CALIB_DROPOUT_1SEC CALIB_DROPOUT_AWG_STUCK CALIB_GLITCH_ZG
Online DMT monitor (enhanced ZGlitch?)
CHECKSUM_MISMATCH
INVALID_DARMERR
MISSING_RAW
MISSING_RDS_LEVEL_1
MISSING_RDS_LEVEL_3
MISSING_RDS_LEVEL_4
Cron job with scimon monitoring - easy restart needed
DARM_09_11_dHz_LOWTHRESH
DARM_09_11_dHz_MEDTHRESH
DARM_09_11_dHz_HIGHTHRESH
DARM_11_13_dHz_LOWTHRESH
DARM_11_13_dHz_MEDTHRESH
DARM_11_13_dHz_HIGHTHRESH
DARM_18_24_dHz_LOWTHRESH
DARM_18_24_dHz_MEDTHRESH
DARM_18_24_dHz_HIGHTHRESH
DARM_50_70_dHz_LOWTHRESH
DARM_50_70_dHz_MEDTHRESH
DARM_50_70_dHz_HIGHTHRESH
Online DMT monitor
DEWAR_GLITCH Online DMT monitor (enhanced DewarMon?)
LIGHTDIP_02_PERCENT LIGHTDIP_03_PERCENT
LIGHTDIP_04_PERCENT
LIGHTDIP_06_PERCENT
LIGHTDIP_07_PERCENT
LIGHTDIP_08_PERCENT
LIGHTDIP_10_PERCENT
LIGHTDIP_12_PERCENT
LIGHTDIP_15_PERCENT
LIGHTDIP_20_PERCENT
OUT_OF_LOCK
PRE_LOCKLOSS_10_SEC
PRE_LOCKLOSS_30_SEC
PRE_LOCKLOSS_60_SEC
PRE_LOCKLOSS_120_SEC
Online DMT monitor (enhanced LockLoss?)
MASTER_OVERFLOW_ASC MASTER_OVERFLOW_LSC
MASTER_OVERFLOW_SUS_MC2
MASTER_OVERFLOW_SUS_RM
Online DMT monitor (enhanced SegGener?) or cron job
NO_CALIB_LINE Online DMT monitor (enhanced ZGlitch?)
PHOTODIODE_GLITCH
PHOTODIODE_OFF
Online DMT monitor (enhanced PDnMon?)
POWMAG Online DMT monitor
SEIS_DARMERR_5_7HZ
SEISMIC_EY_99PCTL_3_10HZ
SEISMIC_X_30_100_mHz_LOWTHRESH
SEISMIC_X_30_100_mHz_MEDTHRESH
SEISMIC_X_30_100_mHz_HIGHTHRESH
SEISMIC_Y_30_100_mHz_LOWTHRESH
SEISMIC_Y_30_100_mHz_MEDTHRESH
SEISMIC_Y_30_100_mHz_HIGHTHRESH
SEISMIC_Y_30_100_mHz_LOWTHRESH
SEISMIC_Y_30_100_mHz_MEDTHRESH
SEISMIC_Y_30_100_mHz_HIGHTHRESH
TRAIN_LIKELY
Online DMT monitor (enhanced SEIS_BLRMS?)
SIDECOIL_ETMX
SIDECOIL_ETMY
SIDECOIL_ETMX_RMS_6HZ
SIDECOIL_ETMY_RMS_6HZ
Online DMT monitor (enhanced RMSBANDS?, enhanced SuspensionMon?)
TCS_GLITCH_LOUD Online DMT monitor (enhanced ZGlitch?)
TIDAL_SERVO_DESATURATION
TIDAL_SERVO_PRESATURATION
Online DMT monitor or cron job


General Glitchiness

In addition, there are several S5 flags defined by general "glitchiness" of DARM_ERR, based on qualitative offline assessment:

Flag name
(click for details)
Method used Investigator(s)
AUTOBURT_GLITCHES Examining Fourier transforms of KleineWelle trigger rates Erik Katsavounidis
BADRANGE_GLITCHINESS
EARTHQUAKE_GLITCHINESS
ELEVATED_GLITCHINESS
HURRICANE_GLITCHINESS
SEVERE_GLITCHINESS
SPOB_GLITCHINESS
Examining inspiral range fluctuations Gaby Gonzalez

Although it would be difficult to mimic setting these criteria faithfully in an online DMT monitor, it would be desirable to flag periods of glitchiness already recognized in the control via measures of non-stationarity, such as the BurstMon pixel fraction. Hence it probably makes sense to enhance BurstMon to create DQ segments on the fly. The same may be true for the Station monitor.

Similarly, it would make sense to run KleineWelle online (on a relatively small set of channels) and create DQ flags based on high rates of triggers with high significance, especially given how useful KleineWelle triggers have proven in defining many S5 DQ flags described above.


K. Riles - December 7, 2007