Minutes of E8 DMT software status review ---------------------------------------- (June 10 @ 11:00 a.m. PDT in LHO control room) ------------------------------------------------------------------------------ Introduction - KR This meeting is to review the progress and plans of DMT monitor authors in addressing the feedback from E8 testers and to discuss some general issues concerning DMT infrastructure. The focus of the individual reports should be work carried out within the last few days or work in progress. ------------------------------------------------------------------------------ LockLoss - Dave Chin Dave is working on a revised version with the following improvements: * Storage / retrieval of time histories now plotted in the dmt viewer so that restarts of the monitor don't cause loss of history. A general class is being written so that other DMT monitor authors can benefit in their history storage. * More lock definitions are being kept track of, one for each state vector bit plus a few key combinations. Daniel suggested providing a multi-level history in addition to the current binary 0 or 1 display, where the highest level would correspond to all state vector bits being on. Plotting the state vector value itself is probably not a good choice, however. * Lock states will be trended so that they are visible in the data viewer * Falling out of lock will be alarmed One improvement completed during E8 is the production of a 2-week history file to be used in Szabi's DMT global summary page. ------------------------------------------------------------------------------ ServoMon - Dave Chin Dave is working on a revised version with the following improvements: * Storage / retrieval of time histories now plotted in the dmt viewer so that restarts of the monitor don't cause loss of history, using the same class developed for LockLoss. * Trending of trigger rates to be visible in the data viewer * Setting alarms for triggers ------------------------------------------------------------------------------ glitchMon - Masahiro Ito Masahiro has nearly finished the following improvements: * 24-hour histories of trigger rates in the data viewer * Revised histograms of trigger amplitudes in the dmt viewer * More readable config file with more flexibility Before S1 he plans to implement trending of trigger rates ------------------------------------------------------------------------------ LineMonitor - Sergey Klimenko Sergey will try before S1 to set up triggers & alarms on line strengths that exceed certain thresholds. There was some discussion on avoiding high false alarm rates. Sergey would prefer someone in the control room at each site to define the thresholds and to enable alarms only during official data taking. ------------------------------------------------------------------------------ IRIG-B and TimeMon - Szabi Marka Szabi is working on improved documentation and guidance for the user for both monitors ------------------------------------------------------------------------------ MTLineMon - Adrian Ottewill The monitor will soon become part of the standard lineup started up by the process manager. Adrian is working on converting his present root graphics display of line tracking to a dmt viewer display. He is also working on a summary html file similar to what Sergey has written for his line tracker. He will also implement trending. Mike Landry volunteered to serve as a tester once Adrian is back in Dublin. Adrian also hopes to provide by the time of S1 a GUI-controlled version of MTLineMon for interactive exploration. ------------------------------------------------------------------------------ CorrMon - Adrian Ottewill This monitor has mostly been used for producing files for offline analysis of inter-channel correlations, but could also be used in the control room for exploration or for setting alarms on too-strong correlations. Adrian plans now has trigger capability and will implement alarms. As for MTLineMon, he hopes to provide by the time of S1 a GUI-controlled version for interactive exploration. ------------------------------------------------------------------------------ BicoMon - Steve Penn Steve has fixed some thread and buffer problems that led to instability in testing. Speed is still a severe problem if one runs on one of new blade workstation but displays graphics on a CDS workstation in the control room. Displaying graphics on the same machine the program is running seems to help. Daniel suggested that if Steve verifies he can get good throughput on 16 kHz channels by running and displaying on a blade, then a blade with monitor could be moved into the control room to host interactive monitors like BicoMon and RayleighMonitor. [This test was done, and it was found that 16 KHz channels display was two times slower than real-time. Steve may move to a default update time of 2 seconds instead of 1 second. Steve is working on some known bugs that affect normalization of the bicoherence and the calculation of cross-bispectral densities. Launching of the monitor from the GDS diagnostics menu still does not work. It will not be easy to fix a reported problem with mouse-click response, even when display speed is increased. Steve will look at data taken last night by Robert Schofield in which strong upconversion was deliberately induced to understand why Robert was unable to see the upconversion with BicoMon. Steve hopes it's just a problem with binning. KR suggested providing more guidance to the user on decreasing frequency bin size. ------------------------------------------------------------------------------ MultiVolt - Daniel Sigg The program's trend writing no longer works. Daniel will investigate and fix. There is a problem with a PEM voltage sensor that gives voltage values a factor of 10 too high. The corresponding channels for the 4K are not hooked up. The hardware and software problems should be fixed by S1. Good documentation exists, but the DMT documentation web page isn't finding it at the moment (a problem that other monitor authors have reported). ------------------------------------------------------------------------------ SenseMonitor - Patrick Sutton Patrick is working on a new summary html file. He will work with Sergey to use its line tracking method to keep track of the strength of the injected calibration lines so that on-the-fly adjustments to transfer functions can be made as IFO gains drift. At the moment drifting alignment can lower the arm power and apparent noise, fooling the monitor into claiming improved instead of degraded sensitivity. Patrick will also work with Mike to make sure there is a systematic way to use the most up-to-date calibration files. A long-term goal is a fully automated scheme, but for S1 something simpler should suffice. To the extent that injected calibration lines indicate gain losses that are not constant across the spectrum, Patrick will print a warning message in the summary html file. ------------------------------------------------------------------------------ RayleighMonitor - Patrick Sutton Patrick is working on automated plot file writing. He is also working on a sped-up version of the display using some scrolling graphics code from Daniel, but so far the development version has proved unstable. The old version can keep up with data in real time if the graphics is displayed on the host machine (verified on blades). This is another monitor for which a dedicated DMT workstation in the control room makes sense. ------------------------------------------------------------------------------ PTMon - Natalia Zotov Natalia's brand-new program now runs, but is not (yet) ready for user testing. ------------------------------------------------------------------------------ BitTest - John Zweizig John has addressed all of the requests for improvement from Saturday's test and has also added alarms for serious problems. ------------------------------------------------------------------------------ HistCompr - John Zweizig As noted in the e-log, this program would make a good engine for a background monitor that periodically (e.g., once per hour) stores histograms for a large number of channels. An interactive foreground monitor to retrieve and display such histograms for comparison with each other and with current conditions would be very nice. John and Szabi are considering asking one or both of their SURF students to work on this project. (Editorial comment by KR: this is a great project for an undergraduate; creating a self-contained but very useful tool for use in a grand scientific enterprise like LIGO ought to feel pretty rewarding!) There was also discussion of the display of overflow/underflow bins for histograms in the DMT viewer. Masahiro will look into making the display of those bins the default so that gross outliers are not overlooked in the fine print of the histogram's legend. John will look into problems reported with inappropriate range choices for histogrammed channels and with some MC and REFL_I channels. ------------------------------------------------------------------------------ PSLMon - John Zweizig ZGlitch: John will look into recording glitch rates via DMT viewer plots and setting conservative (~1 Hz) alarm thresholds on rates. Responding to an elog report, John said that eliminating empty time series snippets from the dmt viewer selection list is not at all easy and not necessary, since one wouldn't normally look at a snippet unless a glitch had been reported for that channel. RMSBands: A brief summary html file will be written and lock state checked before recording band-limited RMS for non-PEM channels (so that huge, out-of-lock fluctuations don't goof up the auto vertical scaling in the data viewer display). ------------------------------------------------------------------------------ SegGener - John Zweizig A brief summary html file will be written that includes information on segment length and integrated time. ------------------------------------------------------------------------------ General Issues - All Szabi is working on a global DMT summary web page for which his new alarm summary page is the first installment. Part of the revamped DMT web structure is a new "spi" page that shows at a glance all of the DMT host machines and DMT processes running at LHO, LLO, CIT and MIT. The new structure will also support a set of cycling, highly abbreviated summary pages from several key monitors. Szabi has made requests of several authors for concise summary info for iconizing into the global summary page with links to more detailed info. Alarm-setting monitors have also been requested to provide an html page corresponding to each alarm type. ***** Peter Saulson provided specific suggestions for key information to be displayed on the global summary page, all based on the AS_Q and/or DAMR-CTRL channels: * Interferometer spectrum * Inspiral sensitivity monitor * Noise levels: broad band, narrow band, special band from some admixture of blrms, RMSBands, ServoMon, LineMon, MTLineMoN * Outliers and other non-Gaussian features from histogramming tool, and glitch rates from glitch-finding tools * RayleighMonitor Peter also suggested that there be a set of S1 "diagnostic challenges" (similar to previous engineering run investigations) focussed on learning how best to * Track violin modes * Track gain peaking * Track non-stationary broad-band noise * Track glitch rates * Find causal links between glitches in different channels * Characterize bilinear coupling between channels (In cases where more than one tool can do the job, investigation teams should be specifically charged with using all available tools, and comparing their utility.) ***** In response to complaints in the elog, KR proposed standardizing the time scales used for DMT viewer time series plots. No one objected to the following mandate: Every quantity that is plotted as a time series should provide a version that is 12 hours long. Additional choices that are strongly encouraged include ---------- 2 hours (driven by empirically determined upper limit of 8K bins for time series with 1-second binning) 6 hours 12 hours 24 hours In addition, monitors displaying long-term history in the global summary page should conform to a standard total interval of two weeks with 1000-second binning. ***** It was decided that all DMT glitch trigger rates (and similar quantities) should be provided in Hz. ***** After discussions with Daniel and John, KR proposed that DMT trend channels have names that are very similar to the channels to which they refer. For example, a quantity based solely on the AS_Q channel should have a name with the prefix H2:LSC-AS_Q rather than a prefix based on the monitor's name. An abbreviation of the monitor name should be included as a final suffix in the name. Authors should keep in mind that channel names tend to get truncated in GUI windows if they exceed 35-40 characters. A "meta-channel" quantity like locked state based on more than one data channel should still carry the appropriate subsystem prefix (e.g., H2:LSC). ***** As DMT monitor authors make improvements to their code in response to reports in the elog, they should append summaries of those changes to the elog entries. (Click on the time stamp at the upper left of the given entry; authors needing a userid/password to make Hanford elog entries can use KR's account - send an e-mail to learn the password; contact Bruce Sears at Caltech to get your own account for the longer term.) ***** DMT trends are now available in the data viewer for about half a dozen DMT monitors. It is not possible presently, however, to display both a DMT trend and an ordinary data channel trend on the same page, or even to display DMT trends together from different monitors. Dave Barker has contacted Alex Ivanov to get this fixed. John will talk to Rolf Bork about segregating the channel selection of DMT trends in the data viewer from the selection of ordinary data channels (without segregation of the resulting plots!) ***** Requests for missing DMT infrastructure features: 1) Peter requested the ability to display two-dimensional histograms, including scatterplots. A long discussion ensued on the benefits/costs of providing 2-D histograms via the DMT viewer (or a new tool modelled on it). In the end, the consensus was that an interactive tool using root graphics and a GUI control makes more sense. This too might be a good project for a SURF student and might even be folded into the histogram storage/retrieval project discussed above. 2) Fred requested a general capability for filtering a time series before plotting. There are some nice filter tools in the DMT library, but no handy utility for plotting the filtered data. Daniel pointed out the desirability of a filter design tool where poles/zeroes would be computed and the filter created, based on requested properties of the filter (e.g., elliptic with given rolloff at given frequency). KR believed Ed Daw has this in mind as a long-term project, but didn't know the time scale for completion. 3) Fred also requested the ability to histogram data channels and check for consistency with an expected shape as the histogram accumulates. He also wanted to be able to filter the data before histogramming. John described the features of HistCompr and DMT filtering which appear to address Fred's needs, once a filtered reference histogram for a chi**2 comparison has been created. ***** Next meeting: Detector Characterization teleconference Friday June 14 @ 8:00 a.m. PDT