From gmendell@ligo-wa.caltech.edu Thu Jan 31 17:36:24 2008
Date: Thu, 31 Jan 2008 14:36:44 -0800
From: Gregory Mendell 
To: Keith Riles , Xavier Siemens 
Cc: LDAS_ADMIN_ALL 
Subject: [Fwd: Re: gaps in h(t) data (fwd)]

Hi Keith,

Below are links to segment lists with the times Level 1 C03 h(t) data
does not exist during Science,Injection mode, including the
16 s, 16 s, and 8 s times at the start and end of Science mode for H1,
H2, and L1 respectively.

The gaps are identical to those I sent to you on 14 Jan 2008 for the Level 2
C03 h(t) data (see that email below).

MISSING H1_RDS_C03_L1:

http://www.ldas-cit.ligo.caltech.edu/ldas_outgoing/GapChecker/S5hoft/archive/MISSING_H1_RDS_C03_L1.Fri_Jan_25_16_34_34_PST_2008.txt

MISSING H2_RDS_C03_L1:

http://www.ldas-cit.ligo.caltech.edu/ldas_outgoing/GapChecker/S5hoft/archive/MISSING_H2_RDS_C03_L1.Thu_Jan_31_11_04_35_PST_2008.txt

MISSING L1_RDS_C03_L1:

http://www.ldas-cit.ligo.caltech.edu/ldas_outgoing/GapChecker/S5hoft/archive/MISSING_L1_RDS_C03_L1.Fri_Jan_25_16_34_23_PST_2008.txt

Note that of the above times, the only ones that are NOT the16 s, 16 s,
and 8 s times at the start and end of Science mode for H1, H2, and L1
respectively are these, again the same as for the Level 2 C03 h(t):

FOR H1_RDS_C03_L1:

823201104 823201168 64
823209520 823209584 64
823261712 823261776 64
851144096 851144112 16
854319712 854319728 16
869323312 869323472 160

FOR H2_RDS_C03_L1:

823225552 823225616 64
823261712 823261776 64
840047851 840047883 32
851144096 851144112 16


FOR L1_RDS_C03_L1:

838066816 838066840 24
863181976 863182008 32
864054039 864054048 9

Also, I am not currently aware of any bad Level 1 h(t) frames that need to
be marked as bad with data quality flags.

I will also email I hope later today a final set of gaps in the S5 raw and
RDS data to add on from the last set I sent on May, 30, 2007. These gaps are
all from times when CDS did not write frames, and so are not Science mode
times.  However, I will send them for completeness.

Regards,
Greg

-- 
Gregory Mendell | Ph: 509-372-8300 ext. 189 | Fax: 509-372-8137
LIGO Hanford Observatory, PO Box 159, MS S9-02, Richland WA 99352
http://www.ligo.caltech.edu | http://www.ligo-wa.caltech.edu

-------- Original Message --------
Subject: Re: gaps in h(t) data (fwd)
Date: Mon, 14 Jan 2008 11:48:38 -0800
From: Gregory Mendell 
To: Keith Riles ,  Xavier Siemens

References:

<4786A5C6.6040804@ligo-wa.caltech.edu>
<47882C15.3060908@ligo-wa.caltech.edu>

Hi Keith,

Below are links to segment lists with the times Level 2 C03 h(t) data
does not exist during Science,Injection mode, including the
16 s, 16 s, and 8 s times at the start and end of Science mode for H1,
H2, and L1 respectively.

MISSING H1 RDS C03_L2:

http://www.ldas-cit.ligo.caltech.edu/ldas_outgoing/GapChecker/S5hoft/archive
/MISSING_H1_RDS_C03_L2.Fri_Jan_11_17_36_39_PST_2008.txt

MISSING H2 RDS C03_L2:

http://www.ldas-cit.ligo.caltech.edu/ldas_outgoing/GapChecker/S5hoft/archive
/MISSING_H2_RDS_C03_L2.Fri_Jan_11_17_36_51_PST_2008.txt

MISSING L1 RDS C03 L2:

http://www.ldas-cit.ligo.caltech.edu/ldas_outgoing/GapChecker/S5hoft/archive
/MISSING_L1_RDS_C03_L2.Fri_Jan_11_16_03_26_PST_2008.txt

Note that of the above times, the only ones that are NOT the16 s, 16 s,
and 8 s times at the start and end of Science mode for H1, H2, and L1
respectively are these:

For H1 RDS C03_L2:

823201104 823201168 64
823209520 823209584 64
823261712 823261776 64
851144096 851144112 16
854319712 854319728 16
869323312 869323472 160

For H2 RDS C03_L2:

823225552 823225616 64
823261712 823261776 64
840047851 840047883 32
851144096 851144112 16

For L1 RDS C03_L2:

838066816 838066840 24
863181976 863182008 32
864054039 864054048 9

Finally, this is the time that the bad H1 RDS C03_L2 h(t) given in my
email below exists:

867344496 867344624 128

I will be repeating this for Level 1 C03 h(t) data next.  I am still
working to resolve some transfer problems with this data set, but should
have similar lists (actually it should be identical lists) for these by
the end of this week.

Regards,
Greg

Gregory Mendell wrote:
> Hi,
> 
> Keith,
> 
> I have filled in all the gaps in the Level 2 C03 h(t) data at CIT that
> can be filled in.  My scripts are running now to generate a final list
> of segments missing data of this type during Science,Injection times.  I
> will email these to you by Monday at the latest.
> 
> (There are still gaps to be filled in due to transfer errors in the
> Level 1 C03 h(t), where my previous attempts to fix this failed.  That
> will be next weeks task.)
> 
> Xavi,
> 
> There is one file at LHO and now at CIT that fails FrCheck.  See below
> The publishing scripts, if configured correctly are smart enough to not
> transfer files that fail in this manner, but my fix did not catch this,
> and the file then transferred.
> 
> Let me know if you want to regenerate this file, or if we need a BAD
> flag for the time segment it covers.
> 
> FrCheck -i
> /archive/cluster/strain-L2/LHO/H-H1_RDS_C03_L2-8673/H-H1_RDS_C03_L2-867344
> 496-128.gwf
>  Checking file
> /archive/cluster/strain-L2/LHO/H-H1_RDS_C03_L2-8673/H-H1_RDS_C03_L2-867344
> 496-128.gwf
>  Compression will NOT be checked
> 
>   Sequentiel file reading. Please wait....
>         1 frames read
>  Read Error. Last errors are:
>  *** FrError: in FrEndOfFileRead File CRC Checksum error: read=7ed49963
> computed=3ded754e
> file:/archive/cluster/strain-L2/LHO/H-H1_RDS_C03_L2-8673/H-H1_RDS_C03_L2-8
> 67344496-128.gwf
>  *** FrError: in FrameRead Cheksum Error
> 
>    1 frame(s) in file.
> 
> Regards,
> Greg
> 
> Gregory Mendell wrote:
> > Hi,
> >
> > There are a large number of gaps in the C03 h(t) at CIT from the second
> > half of S5 that exist due to problems transferring the files.  Just
> > before the holidays I ran scripts to try to fill in these gaps.  I need
> > to rerun these scripts to check what gaps remain.  So many other things
> > were broken, that I have not had a chance to do this yet, but I am
> > starting those scripts now.
> >
> > As for small gaps, my scripts run LSCsegFind for
> > --type="Injection,Science" to find Science
> > and Injection times during the run, and then trim the start and end of
> > each segment by 16 s, 16 s, and 8 s for H1, H2, and L1 respectively.
> >
> > Thus, you may wish to forward to the glitch group the above information
> > about trimming the start and end times.  (Note that I found the 16 s, 16
> > s, and 8 s times for H1, H2, and L1 respectively empirically.  By using
> > these, almost all of the small gaps my script found went away.   Xavi
> > might know more precisely how much trimming his code did.)
> >
> > Before the holidays, below are the gaps I found in the
> > [H1|H2|L1_RDS_C03_L2 data at CIT after doing the above trimming
> > procedure.  Please do NOT forward these to the glitch group --- since
> > these are the gaps I tried to fill in before the holidays. Instead WAIT
> > until I rerun my scripts.  If some gaps remain, I'll have to try to fill
> > those in again, until I get down to a final list of gaps that exist
> > because the data does not exist at LHO or LLO.   I have scripts check
> > for this, but it will take another day or so to have a final list.  OK,
> > so here are the gaps I found from before the holidays:
> >
> > MISSING H1_RDS_C03_L2:
> >
> > 823201104 823201168 64
> > 823209520 823209584 64
> > 823261712 823261776 64
> > 851144096 851144112 16
> > 854319712 854319728 16
> > 867344496 867344624 128
> > 869323312 869323472 160
> > 872302367 872302719 352
> > 872395007 872395327 320
> > 873668895 873670090 1195
> >
> > MISSING H2_RDS_C03_L2:
> >
> > 823225552 823225616 64
> > 823261712 823261776 64
> > 840047851 840047883 32
> > 851144096 851144112 16
> > 871993631 871993887 256
> >
> > MISSING L1_RDS_C03_L2:
> >
> > 838066816 838066840 24
> > 863181976 863182008 32
> > 864054039 864054048 9
> > 865355460 865356021 561
> > 865356749 865364750 8001
> > 865365066 865365706 640
> > 865379196 865380552 1356
> > 865380668 865405577 24909
> > 865406738 865407400 662
> > 871711727 871711983 256
> >
> > Note that of the above, those from the 1st part of S5 are probably true
> > gaps, but some from the last part of S5 are due to transfer failures.
> > As I said above, I already ran script to fill in the gaps due to
> > transfer errors before the holidays.  But I have yet checked how
> > successfully those scripts were.  (My scripts just "unstick" things; LDR
> > then has to do the transferring, and some of these could have got
> > "stuck" again.) I let you know in the next few days when I have my final
> > list.  I send the above, just in case is in urgent need to check what
> > was known to be missing before the holidays.
> >
> > Regards,
> > Greg
> >
> > Keith Riles wrote:
> > > Hi Greg,
> > >
> > > As you can see from the message below from Igor,
> > > the burst group is now running over the full S5 data,
> > > and the occasional gaps in h(t) are causing problems.
> > > Can you update your missing-data intervals for the
> > > rest of S5?
> > >
> > > thanks very much,
> > > Keith
> > >
> > > ---------- Forwarded message ----------
> > > Date: Thu, 10 Jan 2008 11:06:46 -0600
> > > From: Igor Yakushin 
> > > To: Laura Cadonati ,
> > >     Xavier Siemens ,
> > >     Scott Koranda ,
> > >     Keith Riles ,
> > >     Burst Analysis Group 
> > > Subject: gaps in h(t) data
> > >
> > > Laura, Xavi, Scott, Keith,
> > >
> > > I am trying to understand why about 400 of my jobs on the second half
> > > of S5
> > > cannot complete due to the gaps in h(t) data. Below is the procedure I
> > > used to
> > > find out exactly what gaps give me trouble.
> > >
> > >
> > > LSCdataFind -s 848908303 -e 877591821 -t H1_RDS_C03_L2 -o H -u file >
> > > H1.out
> > > LSCdataFind -s 848908303 -e 877591821 -t H2_RDS_C03_L2 -o H -u file >
> > > H2.out
> > > LSCdataFind -s 848908303 -e 877591821 -t L1_RDS_C03_L2 -o L -u file >
> > > L1.out
> > > ./segs1.py > gaps.out
> > >
> > > segs1.py substracts the segments covered by h(t) from category 1
> > > segments and
> > > finds gaps in H1 and L1 h(t) at CIT:
> > > **********************************************************************
> > > ***************************
> > > #!/usr/bin/python
> > >
> > > from glue.segments import *
> > > from glue.segmentsUtils import *
> > >
> > > H1=fromfilenames(open("H1.out"))
> > > H1.coalesce()
> > > H2=fromfilenames(open("H2.out"))
> > > H2.coalesce()
> > > L1=fromfilenames(open("L1.out"))
> > > L1.coalesce()
> > > J=fromsegwizard(open("jobs.txt"))
> > > J.coalesce()
> > >
> > > H1mJ=J-H1
> > > H1mJ.coalesce()
> > >
> > > H2mJ=J-H2
> > > H2mJ.coalesce()
> > >
> > > L1mJ=J-L1
> > > L1mJ.coalesce()
> > >
> > > print "="*100
> > > print H1mJ
> > > print map(lambda x:x.duration(),H1mJ)
> > > print H1mJ.duration()
> > >
> > > print "="*100
> > > print H2mJ
> > > print map(lambda x:x.duration(),H2mJ)
> > > print H2mJ.duration()
> > >
> > > print "="*100
> > > print L1mJ
> > > print map(lambda x:x.duration(),L1mJ)
> > > print L1mJ.duration()
> > > **********************************************************************
> > > ***************************
> > > [CIT:igor@ldas-pcdev1 S5_H1H2L1_run47_2]$ cat gaps.out
> > > ======================================================================
> > > ==============================
> > > [segment(867344496, 867344624), segment(869323312, 869323472),
> > > segment(870467408, 870467411), segment(870560004, 870560006),
> > > segment(870568229,
> > > 870568234), segment(870743020, 870743024), segment(870763683,
> > > 870763688),
> > > segment(870905938, 870905943), segment(870939905, 870939908),
> > > segment(870980988,
> > > 870980991), segment(870991911, 870991915), segment(871016633,
> > > 871016636),
> > > segment(871032510, 871032515), segment(871243427, 871243429),
> > > segment(871392589,
> > > 871392591), segment(871597087, 871597091), segment(872103989,
> > > 872103992),
> > > segment(872242591, 872242596), segment(872885762, 872885763),
> > > segment(873119792,
> > > 873119796), segment(873147564, 873147569), segment(873179437,
> > > 873179443),
> > > segment(873222854, 873222855), segment(873670107, 873670110),
> > > segment(873959125,
> > > 873959129), segment(873994932, 873994936), segment(874132554,
> > > 874132559),
> > > segment(874275426, 874275427), segment(874514084, 874514086),
> > > segment(874817608,
> > > 874817611), segment(875022640, 875022646), segment(875041712,
> > > 875041714)]
> > > [128, 160, 3, 2, 5, 4, 5, 5, 3, 3, 4, 3, 5, 2, 2, 4, 3, 5, 1, 4, 5, 6,
> > > 1, 3, 4,
> > > 4, 5, 1, 2, 3, 6, 2]
> > > 393
> > > ======================================================================
> > > ==============================
> > > [segment(870453089, 870453094), segment(870467407, 870467411),
> > > segment(870526849, 870526855), segment(870578140, 870578144),
> > > segment(870716142,
> > > 870716146), segment(870788350, 870788354), segment(870813950,
> > > 870813955),
> > > segment(870824942, 870824947), segment(870891401, 870891406),
> > > segment(870905939,
> > > 870905943), segment(870939903, 870939908), segment(870956297,
> > > 870956302),
> > > segment(870991910, 870991915), segment(871105560, 871105565),
> > > segment(871236627,
> > > 871236631), segment(871243426, 871243429), segment(871289563,
> > > 871289568),
> > > segment(871312983, 871312988), segment(871352659, 871352666),
> > > segment(871356225,
> > > 871356232), segment(871392584, 871392591), segment(871409392,
> > > 871409398),
> > > segment(871414309, 871414315), segment(871441437, 871441442),
> > > segment(871590268,
> > > 871590273), segment(871597086, 871597091), segment(871619290,
> > > 871619294),
> > > segment(871653647, 871653651), segment(871718967, 871718971),
> > > segment(871803499,
> > > 871803503), segment(871920977, 871920980), segment(871933671,
> > > 871933675),
> > > segment(872103987, 872103992), segment(872142125, 872142129),
> > > segment(872167774,
> > > 872167778), segment(872183074, 872183078), segment(872201827,
> > > 872201831),
> > > segment(872299570, 872299573), segment(872304710, 872304715),
> > > segment(872315350,
> > > 872315354), segment(872331574, 872331579), segment(872419749,
> > > 872419754),
> > > segment(872442798, 872442800), segment(872461191, 872461197),
> > > segment(872704665,
> > > 872704671), segment(872731123, 872731127), segment(872816904,
> > > 872816908),
> > > segment(872871888, 872871892), segment(872885759, 872885763),
> > > segment(872894721,
> > > 872894726), segment(872979792, 872979796), segment(872997216,
> > > 872997220),
> > > segment(873039664, 873039669), segment(873222850, 873222855),
> > > segment(873257938,
> > > 873257943), segment(873318424, 873318428), segment(873413473,
> > > 873413477),
> > > segment(873489654, 873489655), segment(873583005, 873583010),
> > > segment(873618095,
> > > 873618099), segment(873631779, 873631783), segment(873713256,
> > > 873713261),
> > > segment(873733469, 873733474), segment(873736107, 873736110),
> > > segment(873745608,
> > > 873745612), segment(873769276, 873769282), segment(873786055,
> > > 873786061),
> > > segment(873806613, 873806616), segment(873814248, 873814253),
> > > segment(873834757,
> > > 873834762), segment(873844502, 873844507), segment(873903825,
> > > 873903831),
> > > segment(874015300, 874015304), segment(874023757, 874023763),
> > > segment(874087548,
> > > 874087553), segment(874189111, 874189116), segment(874272862,
> > > 874272866),
> > > segment(874275422, 874275427), segment(874313427, 874313431),
> > > segment(874360657,
> > > 874360662), segment(874439964, 874439969), segment(874478784,
> > > 874478789),
> > > segment(874501476, 874501480), segment(874514081, 874514086),
> > > segment(874524801,
> > > 874524805), segment(874526299, 874526303), segment(874530194,
> > > 874530199),
> > > segment(874536278, 874536283), segment(874567554, 874567559),
> > > segment(874622903,
> > > 874622907), segment(874650109, 874650114), segment(874687183,
> > > 874687188),
> > > segment(874695969, 874695973), segment(874706762, 874706766),
> > > segment(874802172,
> > > 874802177), segment(874817606, 874817611), segment(874878327,
> > > 874878332),
> > > segment(874901670, 874901674), segment(874959065, 874959072),
> > > segment(874977366,
> > > 874977372), segment(875022639, 875022646), segment(875041707,
> > > 875041714),
> > > segment(875056048, 875056055), segment(875104038, 875104045),
> > > segment(875109321,
> > > 875109328), segment(875120291, 875120296), segment(875154079,
> > > 875154084),
> > > segment(875200553, 875200560), segment(875230303, 875230309),
> > > segment(875269584,
> > > 875269590), segment(875434993, 875434999)]
> > > [5, 4, 6, 4, 4, 4, 5, 5, 5, 4, 5, 5, 5, 5, 4, 3, 5, 5, 7, 7, 7, 6, 6,
> > > 5, 5, 5,
> > > 4, 4, 4, 4, 3, 4, 5, 4, 4, 4, 4, 3, 5, 4, 5, 5, 2, 6, 6, 4, 4, 4, 4,
> > > 5, 4, 4, 5,
> > > 5, 5, 4, 4, 1, 5, 4, 4, 5, 5, 3, 4, 6, 6, 3, 5, 5, 5, 6, 4, 6, 5, 5,
> > > 4, 5, 4, 5,
> > > 5, 5, 4, 5, 4, 4, 5, 5, 5, 4, 5, 5, 4, 4, 5, 5, 5, 4, 7, 6, 7, 7, 7,
> > > 7, 7, 5, 5,
> > > 7, 6, 6, 6]
> > > 534
> > > ======================================================================
> > > ==============================
> > > []
> > > []
> > > 0
> > > **********************************************************************
> > > ***************************
> > >
> > > Most of the gaps are very short and most likely are due to the fact
> > > that h(t)
> > > generation need some offset from Science segment boundary for filters.
> > > However,
> > > this should be taken into account by some DQ flag and apparently when
> > > Laura
> > > generated that segment list at the end of November either this flag
> > > was not
> > > taken into account or it did not exist or was not complete. There is
> > > also a
> > > possibility that the corresponding h(t) were produced by not
> > > transferred by LDR
> > > to CIT. Does anybody know answers to these questions? Should I just
> > > change
> > > category 1 segments and consider the missing h(t) really missing or is
> > > there a
> > > chance that they should be there but were either mistakenly not
> > > generated or not
> > > transferred?
> > >
> > > Thank you,
> > >
> >
> 
>