Nof S.Y. Springer Handbook of Automation

Подождите немного. Документ загружается.

Collaborative Analytics for Astrophysics Explorations 93.5 Sunfall Architecture and Components 1655

have been sent to a human for evaluation under the old

parameters. The fraction of those 1094 which would be

rejected (i. e., not sent for human scanning) under vari-

ous settings of R1 was calculated; the results are shown

in Table 93.1.

A similar test was then performed for R2. On the

basis of the combined results, thresholds for R1andR2

were chosen at 0.1and0.15, respectively. At those set-

tings, the number of images passing the thresholds and

sent to human scanners decreased to 647 (versus 1094

under the old criteria). This is a scanning load reduc-

tion of 41%, representing a saving of ≈ 6 h of manual

scanning labor each day.

Figures 93.7–93.9 show examples of subimages

viewed during this process. Each ﬁgure depicts a triplet

of reference/new/subtraction images (the reference im-

age is from before the supernova explosion, the new

image is the current one including the supernova, and

the subtraction depicts the new bright spot alone). All

of these subimages were passed by the search software

before our new roundness features were added. The

ﬁrst ﬁgure (Fig.93.7) depicts a good candidate (note

the compact circular form), but the next two ﬁgures

(Figs 93.8 and 93.9) display bad subtractions which

should have been culled out. After our algorithm had

been implemented (features R1andR2 added to the

threshold criteria), Fig. 93.8 failed the R1 cut and would

not have been passed for human scanning. Figure 93.9

failed the R2 cut and likewise would not have been

passed.

93.5.2 Workﬂow Status Monitor

The Workﬂow StatusMonitor is a web-baseddashboard

application, designed to improve project scientists’ sit-

uational awareness of the data ﬂow by synthesizing

diverse information ﬂows from various systems and dis-

playing critical workﬂow status data on a single site

(Fig.93.10).

The supernova search software is highly paral-

lelized as 30000images are queued for processing in

a multistage pipeline that runs on NERSC’s 300node

computing cluster, the Parallel Distributed Systems

Facility (PDSF). Nodes frequently go down or jobs

fail, and failures must be detected promptly and jobs

resubmitted quickly due to the time-critical nature

of the search. Detection of such failures was chal-

lenging and time-consuming before the deployment

of the Workﬂow Status Monitor. The monitor dis-

plays graphs and visual displays of job completion

times on PDSF, job queues, PDSF node uptimes,

and disk vault loads. The interface was prototyped,

evaluated for efﬁcacy by project scientists, and imple-

mented via Hypertext Preprocessor (PHP) on a web-

site hosted on an SNfactory server running SuSE

Linux.

The dashboard also displays date and time infor-

mation at project locations, in several formats relevant

to astronomers. Computer scientists developing a sta-

tus monitor for the Mars exploration rovers [93.40]

demonstrated that apparently minor features, such as

a clock that displayed the time on Mars and at var-

ious project locations on Earth, yielded tremendous

improvements in scientists’ situational awareness and

may very well have prevented critical errors. Additional

information displayed includes telescope scheduling,

supernova candidates found, and current operational

status of telescopes, spectrographs, and cameras used

in processing SNfactory data.

93.5.3 Data Forklift

Each night, over 50GB of heterogeneous, distributed

supernova data arrive at the SNfactory and need to be

processed, managed, analyzed, queried, and displayed

– all within a time period of 24h or less (ideally 12h).

The data are distributed overwide geographicdistances;

some of it is hosted on systems outside the SNfactory’s

control, and some on unreliable systems with frequent

downtime.

To solve these data management problems, we

designed and implemented the Sunfall Data Forklift,

a suite of services for automated data retrieval, storage,

movement, querying, and staging for display. The Data

Forklift consists of a coordinator and a set of individ-

ual services, each customized for its particular task, but

having key attributes in common.

The Data Forklift supports different place, asyn-

chronous collaborative scientiﬁc work by facilitating

data and information transfer amongst a geographically

separated team. Due to the time-critical nature of the

data collection (telescopes must be operated at night

and are located in different time zones), tasks must take

place at distinct, speciﬁed times.

All Data Forklift services share the following prop-

erties:

•

The mechanism resides at a central location (an

SNfactory server), but processes data remotely from

widely separate geographic locations, including

making connections to external machines not under

the SNfactory’s control.

Part I 93.5

1656 Part I Home, Ofﬁce, and Enterprise Automation

Fig. 93.10 Sunfall Workﬂow Status Monitor. This web-based dashboard presents a single-page overview of many

relevant information ﬂows from multiple systems and, according to project scientists, increases operator efﬁciency

signiﬁcantly

•

All data transfers are secure, utilizing encrypted

channels and authentication.

•

The Forklift services and coordinator are reliable

and self-restarting. The Forklift coordinator restarts

itself if its process dies or if the server is rebooted.

•

The mechanism is fault tolerant; services detect un-

reliable connections and recover from errors in data

transfer. Partially transferred ﬁles are detected and

retransferred, and the process restarts the transfer at

the point of failure.

•

The mechanism is extensible (new tasks can easily

be added). The Forklift coordinator was designed so

that new services can easily be added to a central

table.

•

The Data Forklift enables data retrieval from ex-

ternal sources. Many astronomical databases are

web-based, and designed for interactive rather than

automated retrieval. Forklift services handle such

interactions in a fault-tolerant manner. SNfactory

spectral data are stored at a central processing

database in France, and must be retrieved according

to external requirements.

•

User actions can trigger Data Forklift requests.

For example, whenever a supernova candidate is

Part I 93.5

Collaborative Analytics for Astrophysics Explorations 93.5 Sunfall Architecture and Components 1657

saved to the Supernova Warehouse (SNwarehouse)

database, several Forklift services are automatically

started, retrieving related image data from tape stor-

age, accessing external web-based asteroid, galaxy,

and other astronomical databases, converting im-

age ﬁles into display format, and staging data for

display.

•

Forklift services act as middleware for the SNware-

house, retrieving data from internal and external

databases, performing program logic, and staging

data for display by the SNwarehouse GUI.

Fig. 93.11 SNwarehouse overview table. The sky visualization at the top plots declination versus hour angle (astronom-

ical sky coordinates) in a view preferred by domain experts. The green lines represent air mass (the thickness of the

atmosphere) at particular coordinates at the speciﬁed time

•

The Data Forklift is written in Perl, and runs under

SuSE Linux 9.3 on an SNfactory server.

93.5.4 Supernova Warehouse

The SNwarehouse is a comprehensive supernova data

management, workﬂow visualization, and collaborative

scientiﬁc analysis application. It consists of a Postgre-

SQL databasehosted ona dedicated SNfactory database

serverrunning SuSE Linux9.3, DataForklift services as

middleware, and a graphical user interface (GUI) writ-

Part I 93.5

1658 Part I Home, Ofﬁce, and Enterprise Automation

ten in Java. SNwarehouse supports collaborative remote

asynchronous work in several different ways.

Collaboration members can access the GUI from

any networked computer worldwide via a Forklift re-

mote deployment mechanism. Security is provided via

password authentication and encrypted communica-

tion channels. SNwarehouse furnishes project scientists

with a shared workspace that enables easy distribution,

analysis, and access of data. Collaboration members

can view, modify, and annotate supernova data, add

comments, change a candidate’s state, and schedule

follow-up observations from work, home, while observ-

ing at the telescope, or when attending conferences.

This access is critical due to the 24/7natureof

SNfactory operations. All transactions are recorded in

the SNwarehouse database, and the change history and

provenance of the data are permanently stored (records

cannot be deleted in order to maintain the change his-

tory) and continuously visible to all authenticated users.

SNwarehousecentralizes data frommultiple sources

and supports task-oriented workﬂow. Project members

perform well-deﬁned tasks, such as vetting, scheduling,

and analyzing targets, which collectively accomplish

the goal of ﬁnding and following type Ia supernovae.

Typically, an individual or small group performs a given

task, and the results of the task provide inputs for the

next task in the workﬂow, often performed by another

set of group members. Thus, the inputs and outputs of

any task must be well deﬁned and easily recognizable.

SNwarehouse Overview

SNwarehouse facilitates ﬁve major tasks: overview,

supernova candidate vetting, observation scheduling

(determining the order of observation of targets for

a single night), data taking (capturing spectral data at

the telescope with SNIFS), and post mortem (spectral

data analysis and classiﬁcation). These are each de-

scribed in more detail in the sections ahead.

SNwarehouse’s interaction design takes the ap-

proach of overview, ﬁlter, and drill down to details. The

main overview page (Fig.93.11) displays two tightly

coupled representations of the list of targets registered

in the database. The top visualization plots the targets

in the sky; below is a sortable, tabular representation of

the same data. From here one can drill down as needed

based on the desired task.

Supernova Candidate Vetting

Vetting involves a process of scientiﬁc analytic dis-

course where the scientist must quickly decide, based

on limited data, how to allocate scarce and expensive

telescope time to the night’s supernova candidates. At

this point, before spectra are taken, it is often unknown

whether a target candidate is a supernova, so the sci-

entist must gather as much relevant data as rapidly as

possible to make an informed prediction. This task in-

volves retrieving any available images of the target’s

location in the sky prior to discovery and querying sev-

eral external astronomical databases.

Once a target is saved as a candidate supernova, the

target name will appear in the SNwarehouse overview

table, color-coded in red, indicating that the target is

newly discovered. On a rolling basis, a background pro-

cess checks if previous data products exist on disk.

If found, these data products are registered with the

database, and an exclamation mark appears next to the

target name, a ﬂag signaling newly found images. The

combination of these two visual clues allows the scien-

tist to quickly see which targets need evaluation. Once

this determination is made, the vetter will then open the

details view for the target (Fig.93.12).

At this stage, the vetter is trying to determine

whether this target is potentially a type Ia supernova,

based on how consistently the magnitude is rising in

comparison with standard type Ia supernovae. The de-

tails view helps the vetter make this determination in

two ways. First, a visualization of the magnitudes of

all target observations against a standard type Ia light

curve (Fig. 93.13) allows the vetter to determine quickly

if the magnitudesare risinglike thoseof a typical typeIa

supernova. In addition, the vetter can compare the dis-

covery image with prior images taken at this particular

set of coordinates in the sky (Fig.93.14).

Observation Scheduling

Prior to follow-up observation at the telescope, a sched-

ule is made based on the vetter’s assessments to order

and allocatetelescope time.In SNwarehouse,the sched-

uler starts by ﬁltering for only those targets with

observation requests. With this list, the scheduler must

order and assign exposure times for each target using

a variety of techniques. Exposure times are automat-

ically determined using a lookup table based on the

phase andredshift of the target. Considering these expo-

sure times, the scheduler must also order the target list

according to when the target will be visible in the sky

by the telescope. A custom visualization offers insight

for this task.

The Sky visualization (Fig.93.15) depicts the posi-

tions of targets in the sky at a given time and ground

location. The green lines representair mass(atmosphere

thickness) for target coordinates at the speciﬁed time.

Part I 93.5

Collaborative Analytics for Astrophysics Explorations 93.5 Sunfall Architecture and Components 1659

Fig. 93.12 SNwarehouse details view. Comparison with standard light curve is in upper right-hand corner.Theﬁve rows

at the bottom were imaged on different dates; note that the target supernova is visible at the time of discovery but not on

prior dates

Fig. 93.13 Supernova light curve. Here, we see how the

rise in magnitude of this target (squares) compares with

the light curve of a standard type Ia SN (curve)



The blue line is the horizon, and the red halo around the

moon is the lunar exclusion zone, wherelight castby the

moon makes it difﬁcult to view faint targets. The yel-

low circle depicts the sun. Major telescope names and

corresponding latitudes and longitudes are displayed on

a drop-down menu so the visualization can be used

worldwide. The time can be changed so the viewer can

285 290 295 300 305 310 315

SNF20061021-003 mag versus DOY 2006

17.5

18.5

19.5

Part I 93.5

1660 Part I Home, Ofﬁce, and Enterprise Automation

Fig. 93.14 The target is visible at the time of discovery but not visible on a prior date

Fig. 93.15 The Sky visualization shows the target’s position in the sky, the air mass (thickness of the atmosphere) at the

location, and the positions of the sun and moon. The visualization plots declination versus hour angle (astronomical sky

coordinates) in a view preferred by domain experts. The green lines represent air mass at particular coordinates at the

speciﬁed time, the yellow circle is the sun, and the red circle is the lunar exclusion zone

plan observations for the remainder of the night. If the

target appears within the blue horizon lines, it is visible

to the telescope, assuming good weather. The scheduler

must note the rise and set times of each target when

creating the schedule for the night.

It can be challenging for humans to imagine and

visualize the path of astronomical objects over time,

as they appear to move along the inside of the hemi-

sphere of the night sky. Further, even an astronomer

skilled in the art of visualizing celestial coordinates can

have difﬁculty predicting the paths of several objects in

the night sky, accurately estimating which of them will

avoid clouds, and calculating when they will reach the

optimal position for observing.

The key idea in the Sky visualization is the mapping

of the spherical night sky to a two-dimensional rectilin-

ear projection, where astronomical objects’ paths over

time move linearly from left to right. Although there

is some spatial distortion (similar to that of a Mercator

projection of the Earth [93.41]), it is much easier for ob-

servers to predict the future position of objects moving

along a linear path in a plane.

The Sky visualization is used throughout SNware-

house, and is especially helpful for the spectral data

collecting component of SNwarehouse, Data Taking.

Data Taking

During each night of spectral observations with SNIFS

at the University of Hawaii 2.2m telescope, an observer

needs to point the telescope at each scheduled target

in order. In the event of weather or mechanical prob-

lems, a rescheduling decision must be made quickly

and efﬁciently. The Sunfall data-taking interface facil-

itates this process. Much of the observation procedure

is automated, allowing the observer to concentrate

on troubleshooting, rescheduling, and determining the

success of each target observation (Fig.93.16). Nev-

ertheless, human input is crucial during the telescope

Part I 93.5

Collaborative Analytics for Astrophysics Explorations 93.5 Sunfall Architecture and Components 1661

Fig. 93.16 SNwarehouse data-taking window. The observer can follow the targets on the Sky visualization, take notes on

the success or failure of each observation, telescope status and weather conditions, and reschedule targets if necessary

observing process, as the scientists must evaluate large

amounts of dynamic data and make time-critical deci-

sions that cannot be automated, often in as little as 45s.

Participatory design was a natural choice for Sunfall

Data Taking, especially given that expert domain

knowledge was needed to deﬁne the use cases. Scien-

tists were involved in the design process from the start,

helping to deﬁne the problem space and outlining the

necessary functions. The design of the data-taking GUI

went through several prototypes prior to release, in-

cluding low-ﬁdelity paper prototypes and high-ﬁdelity

semifunctional prototypes. Each prototype was evalu-

ated by the scientists, who provided feedback on areas

where the interface was confusing or too tedious to use

(i.e., too many steps to perform a particular task).

Since telescope observation procedures are one of

the key elements of SNfactory science, and thus Data

Taking is an important component of SNwarehouse, we

will discuss this component’s design, implementation,

and use in the system in greater detail in the next few

Subsections.

Telescope Observing Procedures

Each night of spectral observations with SNIFS at the

University of Hawaii 2.2m telescope requires a com-

plex set of procedures involving collaboration among

a geographically distributed team, and the operation of

legacy software telescope controllers driving one-of-a-

kind hardware.

The scientist performing the night’s observations

is known as the shifter. The shifter may be located

anywhere in the world, although SNfactory collabora-

tion members attempt to distribute the operational tasks

among time zones so as to minimize nighttime work.

Part I 93.5

1662 Part I Home, Ofﬁce, and Enterprise Automation

Additional members of the team include a telescope

operator physically located in the Hilo, Hawaii, oper-

ations room, and senior scientist experts who may be

physically located on the other side of the Earth from

the current shifter. At the end of the shift, another sci-

entist evaluates each night’s set of observations using

a slightly different view of the same data.

Legacy Software Interface

The telescope interface is complex, involving a number

of shell scripts which must be run from the command

line at various times. Functions of the interface include

data monitoring as well as telescope control. The scripts

interface directly with the telescope and spectrograph

device drivers. There are several systems that directly

control the telescope.

The telescope control system (TCS) user interface

was written using curses, a Unix terminal control li-

brary originally developed in 1977 [93.42]. When TCS

was originally written, space was at such a premium on

the screen that the developers were forced to use every

Fig. 93.17 VNC telescope control window with chat client

possible shortcut; for example, the Scottish word hie is

used to describe the state of one of the mirrors when it

is moving between two positions.

There are also two directors, which monitor a large

amount of information from three CCD cameras, the

SNIFS instrument itself, as well as the TCS. These ﬁve

information streams are piped into two windows, result-

ing in an extremely rapid ﬂow of information past the

eye. The display is ﬁlled with blinking and scrolling

text, as numeric values are dynamically updated and

various internal program status messages rapidly scroll

across several text windows (Fig.93.17). High-priority

error messages are printed in red to distinguish them

from lower-priority status messages; however, all the

ﬂickering and movement on the screen can be dis-

tracting, especially since relatively unimportant status

messages are continuously scrolling past the window.

To start the night’s observing, the shifter must ﬁrst

establish an secure shell (SSH) tunnel to the Mauna Kea

summit computer (a Dell Dimension PC running Red

Hat Linux), then connect to a virtual network comput-

Part I 93.5

Collaborative Analytics for Astrophysics Explorations 93.5 Sunfall Architecture and Components 1663

ing (VNC) server on the summit machine, opening two

local windows to establish the interface. Two other ma-

chines at the summit are custom-built, possessing Tyan

motherboards that each enable operation of two cam-

era Peripheral Component Interconnect (PCI) interfaces

simultaneously. The shifter must also connect to a pre-

arranged chat room where all scientists involved in the

observation participate, ask questions, provide advice,

and use an informal protocol to transfer positive control

of telescope operation.

To a novice user, the telescope interface is daunt-

ingly complex. The learning curve is extremely steep

at the beginning of training. The scientists concluded

that it was critical to reduce the shifter’s decision-

making load.They initially decidedthat the best method

of doing this was to create an autopilot, an auto-

mated instrument control program that would run the

entire observing process without human intervention.

However, unexpected hardware interactions as well as

operational changes ensured that the automated in-

strument control program could not run autonomously

throughout the night. Additionally, it proved impossibly

complex to modify the program to reliably compensate

for the plethora of unexpected events that could oc-

cur during a night’s observing; human intervention was

consistently required.

Based on this initial experience, it was determined

that the shifters needed a decision support system to

supplement the automation. Although some level of au-

Fig. 93.18 The Visible Time display depicts object visibility and exposure times

tomation was necessary to reduce drudgery and lighten

the shifter’s workload, it became clear that a system was

needed that combined automation with a visual inter-

face that facilitated human intervention. The scientists

instituted various protocol changes, and the data-taking

system was designed and implemented by an interdisci-

plinary team of computer scientists and astrophysicists,

alleviating many sources of shifter error. Tasks that the

shifters performed with data taking included viewing

spectral images, exploring data, keeping a standard-

ized notebook on sky and instrument conditions, and

decision-making. The intent was to have the shifters al-

locate their attention to higher-level tasks, in particular:

evaluating their rate of performance on theschedule, de-

termining whether or not conditions sufﬁced to perform

lower-priority observations, and assessing data quality.

Data-Taking Components

The data-taking GUI is depicted in Fig.93.16. This

interface runs as a separate window on the summit com-

puter and is intended to augment rather than supplement

the existing telescope control interface. Operational

constraints during software development precluded

a complete redesign of the entiretelescope control inter-

face, although incremental improvements were made.

The automated instrument control program was left

in place; if nothing goes wrong during the night, it

will automatically slew the telescope to each target in

turn, start an exposure of the correct length, record the

Part I 93.5

1664 Part I Home, Ofﬁce, and Enterprise Automation

data, and move on to the next target. However, in the

event of weather or mechanical problems, quick and ac-

curate decision-making is required. Data Taking was

designed to support such decisions, with the express

goal of improving the shifter’s situational awareness. To

this end, a number of components were designed to pro-

vide supplemental information about critical observing

conditions, as well as provide predictive tools to en-

able the shifter to make scheduling decisions easily and

correctly.

Since the shifter is operating the telescope remotely,

they may be unaware of deteriorating weather con-

Fig. 93.19 SNwarehouse details view depicting spectral data observations. The small postage stamp images on the left-

hand side of each observation subwindow indicate the accuracy and signal strength of the received data

ditions or the approach of sunrise. A software agent

running in the chat room broadcasts key astronomical

events such as sunrise, and provides information on cur-

rent telescope operation, including the current exposure

or the process of slewing to new coordinates [93.43].

Prior to each night’s observation at the telescope,

a schedule is made in order to allocate telescope time

to each target. In SNwarehouse, the scheduler starts

by ﬁltering for only those targets with observation re-

quests. With this list, the scheduler must order and

assign exposure times for each target using a variety

of techniques. Exposure times are automatically deter-

Part I 93.5