Surveying the (Computing) Landscape

If you ask a question, you should be prepared to hear a candid answer. And if you are serious about wanting frank information, you should be open to changing your behavior or policies based on the answer.

In late spring, NCSA administered a web-based survey asking Alliance-allocated users to provide constructive feedback on all aspects of the computational environment and support services - from allocations to consulting to training. More than 175 users took NCSA up on the chance to comment. And, although the questionnaire was intended as an Alliance instrument, fewer than 5 respondents considered a site other than NCSA as their primary computing site, making the survey a referendum on NCSA's environment.

"This wasn't just an exercise for us," said John Towns, senior associate director of Scientific Computing and Visualization. "We really want to apply what we learned from the responses to changing the way we do business. It's already had an impact."

Overall Computing Environment

Asked to rate the quality of the computing environment, more than 80% found NCSA to have one that is good or excellent. Responses to subsequent related questions prevented staff from resting on these laurels, however.

Many users cited the long queues and slow turnaround times as barriers to their productivity. More than 60% chose additional capacity when asked to prioritize future computer acquisitions. Anticipating both of these responses, NCSA in August increased its SGI Origin2000 system to 1,548 processors, which increased the center's SGI computing capacity by 50 percent, and reduced turnaround time for short and medium jobs (the st-sj and mt-mj queues) where these new resources were applied.

Software

Selecting software to purchase for a center as diverse as NCSA is a balancing act. The center has finite resources to satisfy demand yet users logically expect to have the most common packages available. More than 70% indicated they had access to needed third-party or commercial software. The center's Software Committee constantly evaluates software license status and discusses usage patterns to attempt to satisfy the greatest number of users.

Training

Respondents were clear in their desire for more training and more web-based training. This just-in-time approach makes scheduling training convenient and is cost-effective for users. In June 1999, the Alliance Partners for Advanced Computational Services (PACS) organized a training committee that will coordinate the Alliance's HPC training program. This inter-institutional team hopes to coordinate schedules, minimize overlap of course content, offer courses in geographically distributed locations whenever possible, and develop standards for Alliance online training courses. Using the Access Grid, courses can be offered simultaneously around the country.

Users were quite adamant in requesting an online MPI course, so this is the first one under development by the PACS training team. A prototype is expected to be available at SC 99.

Getting Help

NCSA users generally are advised to contact the Consulting Services office for help or advice in using the center's systems. The staff in that office are not, however, the only ones charged with providing user assistance. Three other NCSA teams provide help but their services are not directly advertised to users. Because of this distributed structure, survey responses about assistance cannot be traced directly to the Consulting Services personnel.

Most users contact NCSA via email several times a year (no more than 50 indicated a more frequent contact rate). The NCSA consulting teams are rated highly for their courtesy, professionalism, and friendliness, with 66% of those responding considering this aspect excellent. The accuracy of the information provided and the timeliness of the responses are rated less highly, but ratings were still good. With a relatively complex computing environment, the staff on all four response teams deal with a large number of variables when answering user questions: system, software, scientific discipline, user computing sophistication and experience, as well as the completeness of the problem description. Choosing between an educated guess or asking for more information is a daily dilemma. The first choice could result in an incorrect solution; the second increases response time.

Web-based Information

Most respondents have accessed some of the information provided on the NCSA website and almost all found the online materials useful. That was the good news. Given the opportunity to evaluate the information and offer specific suggestions, readers were less complimentary. In general, they felt information was missing, lacked depth, and was, in some cases, out-of-date. Several comment that the presentation was too "fancy."

Although the survey specifically requested feedback on the HPC information provided by the center, the responses do not clearly indicate what branches of the NCSA web tree were the subject of the individual comments. This is a point of clarification in future surveys. User services personnel will be evaluating these comments with an eye to improving the quality and organization of the information.

Reacting to All the Feedback

The staff reviewing the survey results experienced their ups and downs. For each "Great job all around!" there was a "The answer 'it's broken' is not acceptable." John Towns noted it was hard to hear negative comments. "But we really appreciate how candid our users were in answering our questions."

This fall, NCSA user services managers, lead by Associate Director Bruce Loftis, are formulating plans to address the most pressing issues identified in the responses. Included in the follow-up process are phone calls to users who volunteered contact information in their survey responses.

The survey will be repeated in 2000. Staff hope for increased, more widely distributed participation and for a continuation of the frank commentary that has already changed NCSA's computing environment for the better. Users who want to provide input in the meantime can use the Alliance online suggestion form.

--Ginny Hudak-David, NCSA