S3 Planning - General
 

Earlier in the week Stan circulated instructions for joining weekly S3 telecons at 5:00 EST (2:00 PST)., where general run status will be reviewed and proposed configuration changes discussed. Because L1 's performance is rapidly changing (good news!), there will likely be some continued commissioning during the first week of S3, but unlike in E10, there will be significant time set aside for science mode running. Mike Zucker will make the call each night.

H1 and H2 are both locking readily, with decent duty cycles. H1 has inspiral ranges spanning 1.7 to 2.5 Mpc, where the variation is dominated by fluctuations in somewhat broadband noise centered on 100 Hz (not the infamous 100-150 Hz glitches, however, which remain mysterious). H2 is running with typical inspiral ranges of 800-900 kpc. DC calibrations on both IFO's were completed yesterday. Swept sines will be carried out tonight, with calibration line injection amplitudes to be tuned today. The calibration lines are injected from the operator 4k and 2k workstations via awg scripts (unlike signal injections). Dave Barker has redefined the LHO state vector bit to require the excitation test points for calibration lines and pulsars to be defined for science mode to apply. This safeguard is to ensure that any injections during science mode are properly recorded in the data.
L1 inspiral ranges are now getting above 1 Mpc after fixes within the last several days. Lock stretches of half to one hour have been observed, but have been interrupted by commissioning work. It is thought that several hours can be achieved. The IFO seems relatively stable (not well quantified yet), with low glitch rates. Calibrations have not been seriously addressed yet, but work will begin tonight. Baseline measurements and determination of injection line amplitudes are the main goals in the first night. The "up" script for reaching science-mode-ready state is not quite final. Brian relayed a statement from Mike Zucker that science mode running will be emphasized over commissioning in the coming days, if all goes well.

As was true last week, one shift (Sunday Nov 9 day shift before LSC meeting) remains unassigned to any institution. A volunteer is most welcome.

There have been a number of updates of instruction sheets in the last week, but work still remains. KR will be doing some updating at LHO over the weekend. Stefan raised several issues that had come up in going through the checklist for today's LHO day shift:

• The script for checking that SpectrumArchiver is functioning properly seems to be missing. KR will send Tim Olson a note on it.
• The pages for checking the status of inspiral and burst DSO's running in LDAS don't seem to be working. Albert said he expected things to be up and running in the next day or so.
• There was some discussion about the burst group's plans. In S2 the group produced nice summary web plots in near real-time of DSO results. Erik reported that, except for WaveBurst, the burst group's DSO's are not quite ready to run on real-time streaming data, but will be soon. Once those are running, then the old web infrastructure can be used again. WaveBurst requires coincident streams from both sites. TFClusters and the PowerExcess DSO's will look at single-IFO data. All will rely on the level 3 reduced data set (AS_Q only). Sergey mentioned that he planned to create some infrastructure for displaying WaveBurst results on the web.

• Glitches / Burst/Inspiral search group vetoes (Laura Cadonati, Natalia Zotov)

Laura summarized a study of the E10 glitches reported by glitchmon on the three IFO's. Using Peter Shawhan's segment list, she looked at 117 hours of H1 data (358 AS_Q glitches), 62 hours of H2 data (917 AS_Q glitches), and 2 hours of L1 data (76 AS_Q glitches). In each case triggers from every non-PEM auxiliary channel monitored by glitchMon were evaluated for coincidence with AS_Q glitches. Dead-times, veto success (fraction of auxiliary glitches matched to AS_Q glitch), and veto efficiency (fraction of AS_Q glitches matched to auxiliary channel glitch) were evaluated. Time-lag histogram plots were also produced and displayed on the results page if a statistically significant peak at zero time lag was observed. Summary:
• H1: No single, dramatic correlation, but the following channels showed significant correlations: AS_I, POB_Q, POB_I, REFL_Q, SPOB_MON
• H2: The POB_I channel is quite important here, with 55% veto success and 25% efficiency and only 0.06% deadtime. AS_I is quite similar, with lesser but still significant help coming from AS_DC, AS_AC, POB_Q, REFL_(I/Q/DC), and QP(DX/DY)_DC.
• L1: Four channels are found to be very important: POB_I, AS_I, MICH_CTRL, PRC_CTRL, with MICH_CTRL giving veto success of 100%(!) and efficiency of 59%, PRC_CTRl giving veto succcess of 95% and efficiency of 84% (!). Caveats: only 76 AS_Q glitches were available to be studied, and the IFO is a very different beast now from what it was in E10.

Laura also mentioned that Julien Sylvestre has generated TFClusters triggers for E10, but she did not believe they had been analyzed yet.
Natalia reported that a third of the glitch triggers found by PTMon in H1 data are in the worrisome range of 80-120 Hz. She finds correlated glitches in several other primary IFO channels in most cases.

• Calibration   (Mike Landry, Brian O'Reilly)

See above remarks under IFO status.
 
• Timing (Szabi Marka, Daniel Sigg)

No update from last week's report. (Szabi in Japan, Daniel on shift tonight).
 
• Hardware signal injections (Peter Shawhan, Szabi Marka)

The mechanics of hardware injections are working well now after the work in E10. A problem was discovered this week, however, in which numerical precision errors in LAL (code now fixed by Teviet Creighton) were causing the pulsar injection code to inject line signals at other than at the 10 chosen pulsar values. Peter discovered the problem when looking at FFT's of the excitation channel. The code seems to be working fine now. Sukanta Bose is working on revisions to the stochastic injection code and on the setting of amplitudes. When that code is ready, a request will be made at the weekly S3 telecon for dedicated injection time. It had been hoped to carry out a series of burst/inspiral injections over the last couple of nights, but the IFO's did not cooperate. Setting aside time for those injections will be discussed at this afternoon's S3 telecon.
It is no longer necessary (or desirable!) for operators/scimons to restart stopped injections. A checking script should either verify that injections are running or reveal why they are not (e.g., other injections going on temporarily). Any other outcome should be reported by e-mail to Peter. Two new workstations have been installed in the LHO mass storage room, dedicated to injections into the IFO's and freeing up the control room workstation previously reserved for these.
 
• Reduced data set generation - channel list (Philip Charlton, Isabel Leonor)

Philip reported RDS generation is going well at Hanford, but that the level 2 generation will need to be restarted at LLO because of a channel list update. During the telecon data started flowing from LHO to Caltech at 10 times real-time so that we should be reaching steady-state soon. There was no opportunity this week to measure total latency between data taking and transfer to Tier 2 centers, but that measurement will be available soon and is expected to be an hour or so. (Note added next day: latency from LHO/LLO to CIT stabilized at 1.5 hours. Additional latency from CIT to other sites not yet available.) Raw data tapes will be shipped from the sites to Caltech roughly once per week where level 1 and 2 RDS will be regenerated for storage there and distribution over the network to Tier 2 centers. Level 1 RDS tapes will also be made for each site and shipped to the other, giving a total delay of up to two weeks between data recording at one site and RDS appearance at the other.
Isabel reported that there were a few last-minute changes to the level 1 RDS channel list because some channels had been renamed or had not been made ready in time. The channel list should now be considered final. Daniel has fixed a problem in the DTT that prevented reading of merged, multi-IFO RDS frames.
 
• Data quality (John Zweizig, Keith Riles)

John could not make the telecon, but KR reported that John is testing out a new scheme to keep track of data quality segments in near real-time by running cron jobs to look back over trends produced by a variety of DMT monitors. KR has promised some scripts for viewing and annotating segment information, but has not gotten far yet on the coding.
 
• Stochastic search group coherence studies (Tania Regimbau)

Tania showed a coherence plot of H1/H2 for the entire E10 run, averaged over 30-second intervals (bin size 0.03 Hz). There is strong structure at the 60 Hz line harmonics and at very low frequencies (below ~30 Hz), as expected qualitatively. The flat noise floor is about where it should be for this length of time average. Aside from a handful of other small peaks, the coherence looks very nice, a dramatic improvement over what was seen in the S2 run. Peter Shawhan pointed out that the injected pulsars do not appear to be showing up in the coherence, a reassuring indication that those injections are truly non-intrusive. He suggested that someone from the pulsar group compute the expected coherence from these injections as a diagnostic.
 
• Violin modes (S. Klimenko)

Sergey is monitoring all of the major violin mode harmonics on each IFO. A problem in LineMon that was causing loading of its DMT machine during E10 has been fixed with a refined heterodyne algorithm. Patrick noted, however, that even with the fixes, there is only 10% idle CPU on the LineMon machine (obsidian) at LHO when both IFO's are in science mode. He (or John) may have to move monitors around to ensure sufficient headroom on all machines. Patrick wondered how one would know if LineMon were losing data. Sergey replied that the DMT viewer plot should show a gap, the html summary page would show inconsistent total times, and its machine's spi page would show lost frames. (BrianO mentioned that a problem was seen with seis_blrms in E10 when it could not keep up with the data, in which there would be apparent seismic spikes on the figure of merit screen, leading to scimon confusion.)
Sergey has added a new heterodyne option to LineMon with an anti-aliasing filter to try to reduce the large fluctuations seen in that line tracking option in E10. He wondered when he could try it out at the sites. KR suggested that he bring it up at this afternoon's S3 telecon and imagined it would be a good thing to try in a few days, after things settle down a bit. Rauha wondered whether LineMon should provide audible alarms on missing calibration lines. Sergey said that his standard measure of line strength would do so, but not the heterodyne option recently commissioned. Since both are running, we should be okay. KR remarked that if it is established within the first week of running that the heterodyne determination adds no value to the standard technique, we may want to disable that tracking option, relieving the load on obsidian.
 
• Bilinear couplings (S. Penn, V. Chickarmane)

Steve has a new BicoMon background monitor nearly debugged (hope to finish today) that allows monitoring of multiple bilinear couplings measures in one program. KR was leery about starting up significantly new code after the start of S3, but suggested Steve talk to John and raise the issue at this afternoon's S3 telecon.BicoMon does not (yet) provide a figure of merit display; so changes are not critical to operations, but bugs are a worry.
 
• Correlated inter-site environmental transients (R. Schofield)

No update (Robert on owl shift tonight)
 
• Local environmental disturbances (R. Schofield)

Ditto.
 
• Data Access (Peter Shawhan)

Peter is working on an interface problem that prevents the DTT from reading new RDS frames via LARS.  He mentioned that the fix that allows the DTT to read old merged RDS frames may not appear in LIGOtools for a while.

DMT Software Status Review:

• Status of DMT infrastructure

 
• Patrick Sutton (filling in for John Zweizig): Version updates, deadlines, etc.

All DMT monitors were restarted Thursday night at both sites. All DMT authors are urged to verify that their monitors are working properly! As mentioned above, there is serious loading of two LHO machines (obsidian & gypsum) that may prove troublesome and require moving monitors around. All of the new monitoring code received by last Monday night's deadline has been installed and is running. There seems still to be a problem with the web emulator of the DMT viewer going into loops that drain the stone and delaronde cpu's. This will have to be watched carefully and perhaps that interface turned off until it is fixed.
 
• Szabi Marka (alarms and web interface)

(Szabi in Japan). KR has sent Szabi an e-mail concerning the alarm-silencing version of the DMT alarms page. The one at LHO seems to work fine and presumably should become the default alarm page. KR does not know the corresponding URL for the LLO version of the alarm silencer.


• Status of DMT monitors

 
• Dave Chin / Tim Bodiya / Keith Riles: LockLoss and ServoMon/SpecMon

No recent changes in LockLoss/ServoMon. KR will exercise a new version of the interactive SpecMon program this weekend.

• Ken Ganezer: seis_blrms

Ken had some questions about accessing the running versions of seis_blrms at the sites. He has started looking at the code on the saiph machine at Caltech.
 
• Masahiro Ito: glitchMon

No report.
 
• Sergey Klimenko: LineMonitor and WaveMon

LineMon reported on above under violin modes entry. No recent changes to WaveMon
 
• Szabi Marka: IRIG-B, TimeMon, and ShapeMon

No report
 
• Tim Olson: SpectrumArchiver (report by e-mail)

Tim is waiting for a response from John on how to use a new synchronization class he would like to use for running SpectrumArchiver under the process manager.
 
• Steve Penn / Vijay Chickarmane: BicoMon / BiLinMon

See Steve's report above under bilinear couplings..
 
• Brian Stubbs: SuspensionMon

SuspensionMon is running stably at both sites. Some earlier trigger problems at LHO seem to be fixed. KR urged running guild to verify that the triggers do actually make it into the database. Since alarms are not yet supported by the program, it's important that scimons periodically check either the log file or the triggers to make sure no mirror side motion has been kicked into unstable oscillations. KR asked that Brian evaluate a reasonable checking scheme (once/day) and write up a short scimon instruction sheet describing it. KR wondered if the thresholds need to be raised on the monitors, since we generally want triggers only when the side servos go haywire. One (presumably) should not have received the triggers Brian verified had been generated this week.

• Patrick Sutton / Mike Ashley: SenseMonitor / LineAmp and RayleighMonitor

There were a few minor fixes to SenseMonitor since E10. The calibrated power spectra and inspiral range integrand are now displayed properly in the DMT viewer. Patrick asked that someone define and archive reference curves for these soon to serve as benchmarks. Patrick has also revised RayleighMonitor to provide better defaults for launching in the mode usually used in the control room. There is also new functionality in that an option allows whitening the power spectra before looking for deviations, where the whitening is based on the first N (parameter) spectra observed during initialization. The whitening option is turned off by default.

• Natalia Zotov: PTMon

Natalia has fixed the retriggering problem seen before and now sees reasonable glitch rates. She can see via guild the triggers being written to the LHO database, but cannot see them in the LLO database (this was resolved after the telecon by Peter Shawhan who discovered a stale config file was being used for PTMon at LLO and by Patrick who fixed the problem).
 
• John Zweizig: BitTest,  DataQual, HistCompr, PSLMon, SatMon , and SegGener

No update..

A.O.B.

• Rauha mentioned that he is working on a scheme to notify the operators via a pop-up window on a control room workstation when a gamma ray burst (GRB) has been detected by the worldwide satellite network (typically 0-3 seen per day). He would like operators to hold off an hour after such an event before deliberately taking down an interferometer. (Ray reminded us that during the biggest GRB of the S2 run, L1 was down, and H1 was up only because the operator had decided to wait until the turn of the hour before leaving science mode.) Rauha has not yet worked out a mechanism but wondered if he could one of the DMT machines. KR suggested consulting John before carrying out any experiments on any DMT machines running online monitors. Another method may be to go through the red and river gateway machines at LHO and LLO, respectively. Stan should also be consulted about implementing the scheme once it's ready..

 
• Next detector characterization meeting: Wednesday November 12 at LHO (LSC meeting)