S3 Planning - General 

Each observatory site will have local run coordinators with authorization to make decisions on non-standard configurations and to be on call to deal with miscellaneous problems. At Hanford, these duties will be shared by Fred Raab and Daniel Sigg. At Livingston, they will be shared by Mike Zucker, Brian O'Reilly and Valera Frolov. The rotation scheme will be locally decided. Stan will announce soon the S3 teleconference schedule.

During E10 H1 had a 75% duty cycle. The inspiral range was disturbingly variable, ranging as high as 1.8 Mpc, but it was discovered that lowering the laser power stabilized the range at its high end. It also lowered glitch rates that were presumably due to a hidden saturation. H2 had a 45% duty cycle during E10, having started pretty ragged. The IFO stabilized as the week wore on, but it's not clear why. A typical inspiral range was 600 kpc. The large, every-few-minutes glitches on H1 have persisted and are not yet understood. The best monitor of them so far has been watching the raw AS_Q spectrum in "movie mode'. The present DataQual glitch rates do not pick them up well, although the DataQual band-limited RMS for that band (around 100 Hz) does show excess at glitch times. RayleighMon picks up about 20% of the glitches. Fred suggested adding narrower bands (40 Hz) up to about 230 Hz to DataQual's lineup. KR suggested adding a 12-sigma glitch rate monitor to DataQual to catch what are anomalously large excursions. John agreed to try modifiying the config file for weekend running. Fred remarked that having the inspiral range available now in common-mode (but pre-science-mode) running was proving convenient and requested that DataQual report in that mode too. John confirmed he will be changing the config file later in the day.
L1 has struggled all week, with most time given over to commissioning. There were few good locks and almost no science mode running. Although the source of the noise that has degraded inspiral ranges by ~10 w.r.t. S2 is not yet known, a number of suspected culprits have been ruled out. A laser power supply problem that added to troubles was replaced. Fred suggested trying to lower laser power in case L1 is subject to the same saturation trouble evident in H1.

Thanks to Caltech and Hanford, six of the seven unassigned expert shifts at LHO have now been filled. The one remaining shift unassigned to any institution is the Sunday day shift (Nov 9) preceding the LSC meeting. A volunteer is most welcome.

Most of the scimon procedure instruction sheets leftover from S2 were updated just before or during the first weekend of E10, with a few new sheets added during the week. Nonetheless, a handful could use some more attention

• Calibration   (Gaby Gonzalez, Mike Landry)

A full-up calibration was done for H1, including verification of the DC global scale that had been propagated from S2. The LSC model was also updated. Similar work has been done for H2, except that the DC scale hasn't yet been checked (but will be soon). Calibration functions will be posted later in the day for H1 and H2. It has been verified that the autocal and SenseMon inspiral ranges agree to within a few percent, confirming that SenseMon is correctly tracking the relevant dynamic input matrix element. The inspiral range for H2 started the week at 800 kpc, but the addition of a dewhitening filter to improve stability lowered the range to 600 kpc for reasons not understood. Fred remarked that there is some skepticism about the DC scale for H2, since at 1-2 kHz, the sensitivity comes within 2-3 of design sensitivity despite low power. Peter Fritschel wondered if violin mode notches were doing any good to reduce excitations. Fred replied that the notches seemed to do nothing and that large excitations are seen even on masses that are not driven. The source of the excitations will be investigated further before S3.
Gaby reported that no serious full calibration has been attempted on L1 yet, but a quick n dirty attempt last weekend gave a range of about 200 kpc and provided Patrick Sutton with functions to use in SenseMon.
 
• Timing (Szabi Marka, Daniel Sigg)

Szabi reported by e-mail from Japan.Briefly, a new atomic clock system has been commissioned at both sites and seems to be working.
 
• Hardware signal injections (Peter Shawhan, Szabi Marka)

Peter and crew have been working mostly on the pulsar and stochastic signal injections during the last week. A number of technical obstacles were overcome with code optimization, monitoring scripts and cron jobs. There remains one problem with simultaneous injection of stochastic signals since the random number seed depends on the GPS start time, requiring (at the moment) that the jobs at the different sites start within the same second. A scheme to make the simultaneity requirement more tolerant will be tried out soon. Workstations in the control rooms (control7 at LHO, control14 at LLO) are dedicated to the injections and should not be used for other work, to avoid overloading. There remains one failure mode in which a test point is cleared, the injections continue, but there is no DAQ recording of the excitation. Mike will check with Dave Barker to see if the state vector can be set up to turn off science mode if the test point disappears. In general, Stan is concerned about an injection system that could prove to be an operational headache and suggested considering an increase in signal amplitude with injections carried out during only 10% of the run. KR wants to try out the full-run injections arrangement for a while, but revisit the issue soon into the run, if failures occur frequently. Peter has put together an information page on the injections. Some of that info will be condensed into a scimon instruction sheet.
 
• Reduced data set generation - channel list (Philip Charlton, Isabel Leonor)

Philip reported that the bugs have been worked out of the RDS generation at the sites; some earlier troubles with knowing promptly when data has been recorded have been fixed. There is now a latency of about half an hour for level 1 RDS generation and an hour to an hour and a half for level 3 (AS_Q only) generation. The transfer of level 3 data from the sites to CIT and then to the Tier 2 centers (UWM, PSU, MIT, AEI) has proven more difficult because of LDR (LIGO Data Replicator) troubles and network firewalls between LLO and CIT. There is also a bottleneck in which RDS transfers out of CIT must await data from both sites. So the slower link holds up the process. Because of startup troubles, level 3 RDS transfers for E10 data are about four days behind, but it is believed that the latency will be a few hours once the transfers reach steady-state. Philip will test this as soon as the E10 data transfers finish.
Isabel reported that the extra weather channels Robert requested have been added to the RDS level 1 list. There are still a few channels slated for the RDS that are not yet present in the LHO & LLO data streams. KR asked her to send him the list of missing channels ; he will try to find out the status/plans for those channels.
 
• Data quality (John Zweizig, Keith Riles)

John has been looking at the 3-day DataQual summaries from LHO. He has been seeing some glitch rates from H1 that appear alarming, both the ambient rates for 4-sigma excursions which should be about 1 Hz, but which are running at several Hz in good times, and some much larger excursions. KR wondered how much of this was due to the apparent laser power saturation problems (see above) seen most of the week and suggested looking at the last 24 hours of data from after the power was tweaked down. For S3 John is also working on a new scheme to keep track of data quality segments in near real-time by running cron jobs to look back over trends produced by a variety of DMT monitors. The "database" will be directories that are filled with statistics derived from those trends. He expects to have the new infrastructure by the start (or soon after the start) of S3. Peter Shawhan mentioned that he will be generating a segment list for E10 shortly.
 
• PEM injections (Robert Schofield)

Robert will be doing the first round of LLO PEM injections on one or more nights of Nov 4-6. He will work with Josh Dalrymple of Syracuse who will later do the second round of LLO injections near the end of S3. Robert will do the first round of LHO PEM injections Nov 9-11 and the second LHO round near the end of S3. In each round of injections at each site, approximately four hours of nominally science-mode data is needed.
 
• Glitches / Burst/Inspiral search group vetoes (Laura Cadonati, Erik Katsavounidis, Peter Saulson, Julien Sylvestre)

Erik and Laura were unable to attend but reported by e-mail that problems with the burst DSO's had prevented looking closely at E10 data yet, but they expect to have some results by next Friday's detchar telecon. Their plan for S3 is to make glitch investigation as close to real time as possible. Julien reported by e-mail that at least qualitatively the E10 data looked like the S2 data from the TFClusters perspective, but with larger trigger rate variations. Peter Saulson reported on a look at DataQual glitch rates for all three interferometers. By and large, E10 rates are running significantly higher than in S2, consistent with John's findings above. Fred mentioned observing the odd behavior that DataQual glitch rates actually go down sometimes when large glitches occur. John believes this is because the very large glitches are distorting the noise spectrum enough to hide the rate of smaller glitches. Fred also mentioned an observation by Stan that there are variations of an order of magnitude in the 60 Hz line in AS_Q. Generally, stability of the line correlates well with stable inspiral range. He wondered whether that variation affected DataQual glitch rates. John believed not because of low-pass filtering and notches used in DataQual.
 
• Stochastic search group coherence studies (Joe Romano, Fred Raab)

Joe reported that Tania Regenbau has been working on a long-term E10 H1/H2 coherence measurement, but had had some LDAS troubles and wasn't quite ready to report. Fred reported on a several-hour H1/H2 measurement he had done with the DTT which had not turned up any significant coherence in the astrophysics band other than in the 60 Hz harmonics. John suggested constructing a figure of merit for the LHO control room based on the H1-H2 coherence. Some discussion ensued on whether that was a worthwhile check, given the tininess of the expected correlation and the little one could likely do about an excess. KR suggested that an alarm on the coherence rising above a certain high threshold would be useful to catch gross screwups in the LVEA leading to coherent acoustic contamination (Fred's example: a phone left on that rings).
 
• Violin modes (S. Klimenko, J. Castiglione)

Sergey reported that the violin modes he has monitored in E10 data have amplitudes that are consistent with thermal most of the time, with most excursions occurring at the start of lock stretches, as expected. Dick wondered if the beam splitter test mass vibration modes are being monitored by LineMon, since an excess has been observed in H2 during E10. Sergey thought they were being monitored, but would double-check. Sergey urged anyone with line-tracking requests to contact him soon.
 
• Bilinear couplings (S. Penn, V. Chickarmane)

Steve's new BicoMon background monitor was running since the second day of E10. Although the results summaries were being written sensibly to the html directory, he ran into the problem that the monitor occasionally falls behind the data (clear from the time stamps in the summary files), at which point the information sent to the DMT viewer gets zeroed out because of an inconsistency check. John suggested using his new DMT synchronization class to avoid the DMT viewer dropouts. Steve has implemented a config file with cross-bicoherence channel choices suggested by Vijay Chickarmane and Nelson Christensen.
 
• Correlated inter-site environmental transients (R. Schofield)

Robert reported that the merged PEM RDS for S3 would be made at LHO after level 1 RDS tapes are shipped from LLO. Isabel will take care of the merging jobs. She mentioned, however, that there is presently an incompatibility between the LDAS-merged frames and the DTT, which Robert will use in his studies. KR will see if this can get sorted out by the start of S3.
 
• Local environmental disturbances (R. Schofield)

From E10 elog entries, Robert reports that liquid nitrogen dewar creaks are clearly causing some lock losses. Other seismic disturbances have been reported too. Since not all elog reports of seismic disturbance state which seismometers have been looked at, it's hard to draw conclusions about sources. Robert urged everyone to look at seismometers from all locations when investigating lock losses. Winds were a problem at the start of E10 at LHO. (See Robert's PEM injections report above for S3 plans.)
 
• Data Access (Peter Shawhan)

Peter summarized the data creation / transfer plans for S3. Raw data will be written at each site to tapes which are copied and shipped to Caltech (~once per week) for creation of level 1 (hundreds of channels) and level 2 (handful of channels) reduced data sets for distribution to Tier 2 centers. Levels 1 and 2 will also be generated and kept locally at the sites. Level 3 (AS_Q only) will be generated at the sites and sent over the network to Caltech and then out to the Tier 2 centers (see Philip's report above). A data access page based on the S2 version has been linked to the S3 web page. Volunteers are welcome to fill in some of the missing information on accessing certain data sets.

DMT Software Status Review:

(Since the telecon had nearly ended by the time the agenda items below came up, KR asked for only urgent reports concerning recent/planned changes to DMT code. Some e-mail reports were also summarized.)

• Status of DMT infrastructure

 
• John Zweizig: Version updates, deadlines, etc.

There are some serious loading issues on certain DMT machines due to a problem in the web viewer interface (web emulator of DMT viewer). Szabi has been looking at the problem this week, but hasn't been able to track it down yet. If it persists through next week, John will probably shut down the web serving to avoid interference with monitoring programs. John expects a handful of  code and config file changes in the next few days. He has set a deadline of Monday night for any code changes to be run during S3. He will be unreachable Wednesday morning through Friday afternoon, but Patrick Sutton has agreed to fill in for him during that time, in particular, to restart all monitors at both sites Thursday night in conjunction with the installation of S3 LDAS databases at the sites. 


• Status of DMT monitors

 
 
• Dave Chin / Tim Bodiya / Keith Riles: LockLoss and ServoMon/SpecMon

No recent changes in LockLoss/ServoMon other than a config file fix to allow new LHO LockLoss trends to be seen by the data viewer. KR exercised SpecMon last weekend at LLO and sent more bug reports and change requests to Tim Bodiya.
 
• Ken Ganezer: seis_blrms

Ken asks that bug reports on seis_blrms be sent to him.
 
• Masahiro Ito: glitchMon

No report.
 
• Sergey Klimenko: LineMonitor and WaveMon (reported by e-mail after the telecon)

Heterodyning is now implemented in the LineMon version that looks at excitation channels. So far the heterodyning method sees larger fluctuations than Sergey's standard algorithm. Sergey attributes the increased variance to the absence of background noise estimation in the heterodyning.
 
• Szabi Marka: IRIG-B, TimeMon, and ShapeMon

No report
 
• Tim Olson: SpectrumArchiver (report by e-mail)

Tim is working on a version to run under the process manager and expects to have it ready for S3.
 
• Steve Penn / Vijay Chickarmane: BicoMon / BiLinMon

See Steve's report above.
 
• Brian Stubbs: SuspensionMon (report by e-mail)

The latest Hanford version of SuspensionMon ran stably, but the Livingston config file had problems. Brian saw some odd behavior in the H1 end mass channels he is trying to understand. Coding work for DMT trends is not quite done. Trigger statistics don't seem to indicate that SuspensionMon's triggers are being received.
 
• Patrick Sutton / Mike Ashley: SenseMonitor / LineAmp and RayleighMonitor

See remarks above on SenseMon. The new code seems to be working beautifully.
 
• Natalia Zotov: PTMon

Natalia was unable to connect to the telecon (20 ports weren't enough?!). So she reported by e-mail . Briefly, she has retuned downsampling and threshold parameters, which seems to have produced more robust results at LLO. She will be looking at LHO data this weekend.
 
• John Zweizig: BitTest,  DataQual, HistCompr, PSLMon, SatMon , and SegGener

No time to report.

A.O.B.

• Next detector characterization telecon: Friday October 31 at 1:30 EDT