From perlman@mailhost Wed Aug 7 14:03:48 1996 Return-Path: Received: from dev1.NISDEV (dev1.dev.oclc.org) by fssun09.dev.oclc.org (4.1/SMI-4.1) id AA26841; Wed, 7 Aug 96 14:01:01 EDT Received: by dev1.NISDEV (SMI-8.6/SMI-SVR4) id OAA20279; Wed, 7 Aug 1996 14:01:01 -0400 From: perlman@mailhost (Gary PERLMAN) Message-Id: <199608071801.OAA20279@dev1.NISDEV> Subject: Re: tutorial submission/graphic design help To: epfran1@PacBell.COM (Ellen Francik) Date: Wed, 7 Aug 1996 14:01:00 -0400 (EDT) Cc: perlman@fssun09.dev.oclc.org, Jim_A_Larson@ccm.jf.intel.com In-Reply-To: from "Ellen Francik" at Aug 7, 96 10:37:44 am X-Mailer: ELM [version 2.4 PL23beta3] Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Status: O Last year the papers chairs got volunteer mentors to help with papers, particularly international papers, I think. Ask Robin Jeffries (jeffries.chi@xerox.com) about possibilities. As an instructor, here is what I would like: I provide some pages in print or online. I get some subset back with makeover ideas. I implement these ideas as best I can. A professional makeover of many individual pages is probably out of the question, but some professional ideas would be useful. Of course, my interpretation and implementation might not work, but it would probably be an improvement. Volunteer designers might be able to get quantitiative feedbback on the success of their suggestions if they help improve subjective ratings about the tutorial notes for previously offered tutorials. They could use that to promote the consulting services. I don't want to think about what would happen if a graphic designer boasted big gains when 4/5 of their designs got lower ratings. Okay, that's a complete expansion of my little idea. Gary > Gary: good idea about the graphic designer. I have the same problem > personally. Jim, should we think of this as a volunteer opportunity, or do > we want to alert Paul that we want to hire somebody and have them on-call, > in the same way that we might plan for technical writing support for > tutorial instructors? > > Ellen > > (Jim: original message below) > > >I just thought the summary information, such as the > >title (ok, I'll admit it's not much work to retype > >my 3-word title, but others might be more difficult), > >abstract, etc. might have been useful for assigning > >to reviewerers. It would be presumptuous for me to > >volunteer the description for the advance program, > >but I guess that's the next phase where you > >might request an online version. > > > > > >Here is the review you will probably get for my tutorial. > >"This tutorial has been offered many times (1992/94/95/96) and has > >been inceasingly-well-attended and well-received each time. > >Based on the comments from attendees, there appears to be good > >balance among the topics and mode of presentation (lecture, demos, > >exercises), but the handout should be made more visually interesting." > > > >I know my handout is dull, but I don't have any ideas about > >how to jazz it up. I am very open to suggestions along that line, > >including conversion to an online presentation format (e.g., PowerPoint). > >But I would really like to get some concrete suggestions. > >Maybe the tutorials chairs should recruit some graphic designers > >to serve as mentors for tutorial instructors who could use some help. > > > >Gary > > > From perlman@mail.cis.ohio-state.edu Tue Sep 3 22:47:45 1996 Return-Path: Received: from mailer.oclc.org (mailer.dev.oclc.org) by oclc.org (4.1/SMI-4.1) id AA03668; Tue, 3 Sep 96 22:47:44 EDT Received: from mail.cis.ohio-state.edu by mailer.oclc.org (5.0/SMI-4.1) id AA11896; Tue, 3 Sep 1996 22:47:42 +0500 Received: from hammond.cis.ohio-state.edu (hammond.cis.ohio-state.edu [164.107.17.6]) by mail.cis.ohio-state.edu (8.6.7/8.6.4) with ESMTP id WAA18796 for ; Tue, 3 Sep 1996 22:47:29 -0400 From: Gary Perlman Received: (perlman@localhost) by hammond.cis.ohio-state.edu (8.6.7/8.6.4) id WAA16095 for perlman; Tue, 3 Sep 1996 22:47:28 -0400 Date: Tue, 3 Sep 1996 22:47:28 -0400 Message-Id: <199609040247.WAA16095@hammond.cis.ohio-state.edu> To: perlman@cis.ohio-state.edu Content-Length: 2183 Status: RO Does High Tech = Low Science? Do the ends justify the means, or lack thereof? The rise and fall of empiricism in HCI research. Decline of cited stats in CHI conference, HCI Jl. Some quites on empiricism: Popper Keppler on Brahe Epicycles talk by Geoffrey @ washington Can't play 20 questions with nature and win - Newell 73? During the selection of papers for HF Perspectives on HCI, there was an ongoing discussion among the editors about the relationship of the conference papers of the HFES (particularly the CSTG) to SIGCHI. Generally, the HFES papers were shorter, more applied, solving specific problems, and more empirical. This brought me to wonder: To what degree was the HFES/CSTG more empirical than SIGCHI? How had this changed over the years? We already had some data on the number of empirical papers in the HFES proceedings, at least in the papers reviewed and selected for the HF Percpectives in HCI collection. But that is not what is really of interest. What is of interest to me is my perception that the role of empirical validation has seemed to be declining I surveyed the CHI proceedings for data: graphs significance tests No more Fitts' Law old chi = little studies -> systems w/eval -> systems w/out eval Smnall steps vs. big strides Pro-Empiricism Wayne Gray Ben Shneiderman - non-psych Boehm-Davis - HFES Anti-Empiricism Tom Malone Dave Woods if it requires a statistical test, then it is not important enough to be studied Anti-HFES Peter Polson John Karat Systems People - don't know better PARC Discussant Don Norman AUTHOR: Symposium on Cognition (8th : 1972 : Carnegie-Mellon University) TITLE: Visual information processing; proceedings. PLACE: New York, PUBLISHER: Academic Press, YEAR: 1973 PUB TYPE: Book FORMAT: xiv, 555 p. illus. 24 cm. NOTES: Includes bibliographies. ISBN: 0121701506 SUBJECT: Human information processing -- Congresses. Imagery (Psychology) -- Congresses. Visual perception -- Congresses. OTHER: Chase, William G., ed. Carnegie-Mellon University. From jeroenb@design.nl Mon Aug 12 09:11:33 1996 Return-Path: Received: from mailer.oclc.org (mailer.dev.oclc.org) by fssun09.dev.oclc.org (4.1/SMI-4.1) id AA02562; Mon, 12 Aug 96 09:11:33 EDT Received: from mail.cis.ohio-state.edu by mailer.oclc.org (5.0/SMI-4.1) id AA24060; Mon, 12 Aug 1996 09:11:33 +0500 Received: from design.nl (morris.design.nl [193.78.77.130]) by mail.cis.ohio-state.edu (8.6.7/8.6.4) with ESMTP id JAA07800 for ; Mon, 12 Aug 1996 09:11:16 -0400 Received: from radon.design.nl by design.nl via ESMTP (940816.SGI.8.6.9/940406.SGI) for id PAA04114; Mon, 12 Aug 1996 15:11:13 +0200 From: jeroenb@design.nl (Jeroen Bouwman) Received: by radon.design.nl id MAA15842; Mon, 12 Aug 1996 12:11:13 -0100 Date: Mon, 12 Aug 1996 12:11:13 -0100 Message-Id: <199608121311.MAA15842@radon.design.nl> To: perlman@cis.ohio-state.edu Subject: CHI97 Invitation to Review Content-Length: 2233 Status: RO Dear Gary Perlman, The members of the CHI Technical Program Committee each year work hard to develop a program that is of high quality and reflects the latest innovations in HCI research and practice. One of the most important aspects of developing a Technical Program is soliciting reviews from qualified experts in the field. We are sending you this note because you have been a CHI reviewer in the past or have indicated interest in doing so at some point. This year we are seeking volunteer reviewers for all categories of Technical Program submissions, including the Early Submissions (reviewed late September to mid October) and the Late-Breaking Submissions (reviewed late January to early February). If you are willing to be considered as a reviewer for any category of CHI 97 submissions, please visit our new Web-based reviewer database to update your reviewer record; it has been initialized with your contact information from past years, but we need you to check this information and to complete an expertise profile for matching against submissions. (Please update your record even if you do not wish to review this year, for use by future conferences.) The information in this database is private; only you and the relevant Technical Chairs will have access to your personal record. In the next few weeks, the various chairs will be contacting volunteers to build their review committees. To retrieve and edit your record, use your Web browser to connect to: http://www.acm.org/sigchi/chi97/call/review/database.html You will be prompted for a name and password. To access your existing record, you will need to enter the following name and password information: Name: rev0423 Password: pwd15231 If you have any problems retrieving or updating this information, please contact the CHI97 Hypermedia Chair (Eddy Boeve), chi97-hypermedia@acm.org. If you are unable to access this material over the Web, we will send you an appropriate ASCII form to fill out and return by email. Thank you very much in advance for working as part of the CHI97 team in constructing an outstanding Technical Program!! Mary Beth Rosson and Jenny Preece CHI97 Technical Program Co-Chairs chi97-tech-program@acm.org From CHI@omg.unb.ca Fri Sep 27 14:59:45 1996 Return-Path: Received: from mailer.oclc.org (mailer.dev.oclc.org) by oclc.org (4.1/SMI-4.1) id AA22719; Fri, 27 Sep 96 14:59:45 EDT Received: from mail.cis.ohio-state.edu by mailer.oclc.org (5.0/SMI-4.1) id AA23293; Fri, 27 Sep 1996 14:59:40 +0500 Received: from mail.acm.org (mail.acm.org [199.222.69.4]) by mail.cis.ohio-state.edu (8.6.7/8.6.4) with ESMTP id OAA20041 for ; Fri, 27 Sep 1996 14:59:24 -0400 From: CHI@omg.unb.ca Received: from unb.ca (hermes.csd.unb.ca [131.202.3.20]) by mail.acm.org (8.7.5/8.7.5) with ESMTP id PAA76188 for ; Fri, 27 Sep 1996 15:00:08 -0400 Received: (from omg.unb.ca [131.202.34.1]) by unb.ca (8.7.6/960430-08:40) id PAA21034 for ; Fri, 27 Sep 1996 15:59:17 -0300 (ADT) Received: from caspian.omg.unb.ca by omg.unb.ca (4.1/SMI-4.1-950806) id AA14768; Fri, 27 Sep 96 15:54:16 ADT Received: by caspian.omg.unb.ca (950413.SGI.8.6.12) id QAA18637; Fri, 27 Sep 1996 16:25:14 -0300 Date: Fri, 27 Sep 1996 16:25:14 -0300 Message-Id: <199609271925.QAA18637@caspian.omg.unb.ca> Apparently-To: perlman@acm.org Content-Length: 7292 Status: RO Dear Gary, You should soon be receiving a package of papers for review. These papers have been assigned to you by a program that assigns reviewers to papers on the basis of matching phrases. You may not feel fully qualified to review all the papers you receive. However, we ask that you do your best. In general it is often useful to have the opinion of someone who is not in the immediate area of research. You should, in the near future, be hearing from the Associate Chairs who are responsible for the papers on your list. Associate Chairs are responsible for consolidating sets of reviews, writing meta-reviews, based on the reviews and representing the papers at the Papers Committee Meeting in early November. We have had some difficulty in identifying conflicts. Naturally, you should not be reviewing your own papers, or papers by close colleages. We also define a conflict as anyone who is working for the same company or university. If you think that you are in conflict with a paper you have been assigned, let your associate chair know and also send a message to us: chi97-papers@acm.org Here is the list of papers you should have. Let us know if there is a mismatch. 1036 User Reactions to Agents with Faces 1098 Learning Personal Preferences on Online Newspaper Articles from User behaviors 1109 Speech Browsing the World Wide Web 1185 A empirical comparison of three usability evaluation methods 1199 Crowded collaborative virtual environments **************************************************************** Regards, Colin Ware Dennis Wixon Here is the review form. =========================================================================== This review form will be processed automatically. This means that it is very important to LEAVE THE FORM AS-IS EXCEPT WHERE WE ASK YOU TO ENTER INFORMATION. Information-entry points are marked with "{:". It is also important that you email us an ASCII version of your filled-in forms, so that they won't have funny control characters in them and so on. Probably the easiest thing to do is to save this template as a file and edit a copy of it for each review, then bring it in as part of a mail message when you're ready to send the review back. If you use your mail-reply function to do this, don't worry if special characters (e.g., ">") are inserted at the beginning of lines in the template; we can strip these out. You may combine reviews into a single mail message if you like, just make sure that EACH review begins with the form-begin banner and ends with the form-end banner. If you do choose to work on the form within a word processor (e.g., MS Word), please remember to save it TEXT-ONLY before emailing it back to us. If you believe that creating an ASCII version will be a problem for you, please immediately contact us at chi97-papers@acm.org. NOTE: Question 2 asks you to evaluate the paper in terms of the criteria specified for each of the six paper categories described in the official call for participation. To summarize, -- EMPIRICAL: judged primarily according to the appropriateness and rationale for the methods of data collection analysis, and significance of conclusions -- EXPERIENCE: judged primarily according to usefulness to practitioners of lessons learned, i.e., are they able to learn something new from it and can they apply it -- SYSTEMS: judged primarily according to the originality (i.e., novelty and interest value) of the design idea and soundness of rationale behind it. -- THEORY: judged primarily according to the originality (i.e., novelty and interest value) and soundness of the theoretical arguments, and relevance to research or practice -- METHODOLOGY: judged primarily according to the originality (i.e., novelty and interest value) and soundness of the proposed method, and likely usefulness to intended audience. -- OPINION: judged primarily on the impact and quality of the argument, including the data (research or practice) used to support the argument, and on the likelihood of this argument to have a stimulating effect on the CHI community. Email your completed reviews NO LATER THAN 5PM EDT OCTOBER 17 to: your Associate Chair and to CHI@omg.unb.ca THE FORM BEGINS WITH THE NEXT LINE: +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +++++++++++ CHI97 Paper Review Form-Begin Banner +++++++++++ +++++++++++++ INCLUDE AT THE TOP OF EACH REVIEW +++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ All your input should appear after a curly bracket followed by a colon. PAPER NUMBER {: } PAPER TITLE {: } REVIEWER NUMBER {: }1. Briefly SUMMARIZE the paper {: }2. What TYPE OF SUBMISSION is appropriate for this paper (see bottom of Cover Sheet Two for the authors' selection)? If your assessment of type differs from the Authors, please explain why you made your selection. In the next two questions, use the criteria specific to this Type of Submission (see above) to assess the paper. {: }3. Describe the IMPORTANCE OF THE PROBLEM, addressing the following concerns: Is it new and significant? Is it a useful and relevant problem? Do they sufficiently motivate the problem? Did you learn anything? Does the issue apply to CHI? {: }4. Describe the QUALITY OF THE ARGUMENT, addressing the following: Does the analysis use appropriate methodology? Is the argument well-structured? Do they provide sufficient data and/or well-supported arguments? Do they cite relevant work? Is the paper appropriately focused? Does the analysis cover all the important issues at the appropriate level of detail? {: } 5. Describe the QUALITY OF THE MECHANICS, addressing the following: Do they clearly describe what was done and/or how it was studied? Is the writing clear and concise? Do they provide the right level of detail? Do they use too much jargon? Do the figures support the text? {: }6. Provide any OTHER COMMENTS you believe would be useful to the author (including pointers to missing relevant work). {: }7. Using the scale 1 = Definitely reject 2 = Probably reject 3 = Could go either way 4 = Probably accept 5 = Definitely accept provide a numerical rating of the paper's ACCEPTABILITY {: }8. SUMMARIZE your ASSESSMENT of the paper, pointing out which aspects described above you weighted most heavily in your rating in question 7? {: }9. Using the scale 1 = Very unconfident, really just a guess 2 = Rather unconfident, but I know a bit 3 = Moderately confident, I know as much as most 4 = Pretty confident, I know this area well 5 = Extremely confident, I consider myself an expert rate your EXPERTISE in the area addressed by the paper. {: }10. Use this space to provide comments that you feel are relevant to the review process but that you do NOT want FORWARDED to the author(s) {: } +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ++++++++++++ CHI97 Paper Review Form-End Banner ++++++++++++ ++++++++++++ INCLUDE AT THE BOTTOM OF EACH FORM +++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ From l.tweedie@ic.ac.uk Wed Oct 2 19:39:43 1996 Return-Path: Received: from mailer.oclc.org (mailer.dev.oclc.org) by oclc.org (4.1/SMI-4.1) id AA00238; Wed, 2 Oct 96 19:39:42 EDT Received: from mail.cis.ohio-state.edu by mailer.oclc.org (5.0/SMI-4.1) id AA00844; Wed, 2 Oct 1996 19:39:38 +0500 Received: from mail.acm.org (mail.acm.org [199.222.69.4]) by mail.cis.ohio-state.edu (8.6.7/8.6.4) with ESMTP id SAA13749 for ; Wed, 2 Oct 1996 18:42:09 -0400 From: l.tweedie@ic.ac.uk Received: from punch.ic.ac.uk (punch.ic.ac.uk [155.198.5.17]) by mail.acm.org (8.7.5/8.7.5) with ESMTP id SAA60018 for ; Wed, 2 Oct 1996 18:42:47 -0400 Received: from judy.ic.ac.uk by punch.ic.ac.uk with SMTP (PP); Wed, 2 Oct 1996 23:42:04 +0100 Received: from dinghy.ee.ic.ac.uk (dinghy.ee.ic.ac.uk [155.198.123.8]) by judy.ic.ac.uk (8.7.5/8.7.5) with SMTP id XAA04085 for ; Wed, 2 Oct 1996 23:41:47 +0100 (BST) Received: from by dinghy.ee.ic.ac.uk (4.1/4.0) id AB17159; Wed, 2 Oct 96 23:41:43 BST Message-Id: <9610022241.AB17159@dinghy.ee.ic.ac.uk> Date: Wed, 2 Oct 1996 23:42:15 +0100 To: perlman@acm.org Subject: CHI paper 1036 Content-Length: 662 Status: RO Hi I understand that you are a CHI reviewer for the paper: 1036 User Reactions to Agents with Faces I am the associate chair responsible for meta-reviewing this paper. I would be pleased to have the review as soon as you can send it to me! Any problems please get in touch Lisa _____________________________________________________ Lisa Tweedie, Information Engineering Tel: +44 171 594 6261 Imperial College, London, SW7. Fax: +44 171 581 4419 _____________________________________________________ From rvirzi@gte.com Fri Oct 4 15:10:14 1996 Return-Path: Received: from mailer.oclc.org (mailer.dev.oclc.org) by oclc.org (4.1/SMI-4.1) id AA03323; Fri, 4 Oct 96 15:10:14 EDT Received: from mail.cis.ohio-state.edu by mailer.oclc.org (5.0/SMI-4.1) id AA19788; Fri, 4 Oct 1996 15:10:10 +0500 Received: from mail.acm.org (mail.acm.org [199.222.69.4]) by mail.cis.ohio-state.edu (8.6.7/8.6.4) with ESMTP id OAA12783 for ; Fri, 4 Oct 1996 14:23:49 -0400 Received: from ns.gte.com (ns.gte.com [132.197.8.9]) by mail.acm.org (8.7.5/8.7.5) with ESMTP id OAA166136 for ; Fri, 4 Oct 1996 14:24:23 -0400 Received: from [132.197.104.26] by ns.gte.com (8.7.5/) X-Sender: rv01@pophost.gte.com Message-Id: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Date: Fri, 4 Oct 1996 14:19:00 +0100 To: rvirzi@ns.gte.com From: rvirzi@gte.com (Robert A. Virzi) Subject: Paper Review - Speech Browsing the World Wide Web Content-Length: 1169 Status: RO Dear Reviewer: I am the associate chair for the paper Speech Browsing the World Wide Web and you are shown as one of the reviewers for this paper. I expect to have your review of this paper by OCTOBER 17th, via email. Please be sure to use the reviewer form already sent to you (I can forward a new one if you lost the one mailed earlier). If you don't have the paper, the review form, or if you won't be able to review the paper by OCTOBER 17th, please let me know immediately. You can reach me using any of the contact methods shown below, although email is preferred. Reviews MUST be returned via email. You should expect confirmation from me that I've recieved your review. I'll send out reminders as the deadline approaches, if need be. Please, help the process run smoothly by writing persuasive reviews and getting them in on time. That's all for now. I'm looking forward to hearing what you all have to say about this paper. -Bob Robert Virzi rvirzi@gte.com voice: +1 617.466.2881 GTE Labs, MS-38 fax: +1.617.466.4035 40 Sylvan Rd Waltham, MA, USA 02254 From rvirzi@gte.com Fri Oct 4 15:10:16 1996 Return-Path: Received: from mailer.oclc.org (mailer.dev.oclc.org) by oclc.org (4.1/SMI-4.1) id AA03353; Fri, 4 Oct 96 15:10:15 EDT Received: from mail.cis.ohio-state.edu by mailer.oclc.org (5.0/SMI-4.1) id AB19788; Fri, 4 Oct 1996 15:10:14 +0500 Received: from mail.acm.org (mail.acm.org [199.222.69.4]) by mail.cis.ohio-state.edu (8.6.7/8.6.4) with ESMTP id OAA12930 for ; Fri, 4 Oct 1996 14:28:59 -0400 Received: from ns.gte.com (ns.gte.com [132.197.8.9]) by mail.acm.org (8.7.5/8.7.5) with ESMTP id OAA184244 for ; Fri, 4 Oct 1996 14:29:33 -0400 Received: from [132.197.104.26] by ns.gte.com (8.7.5/) X-Sender: rv01@pophost.gte.com Message-Id: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Date: Fri, 4 Oct 1996 14:28:28 +0100 To: rvirzi@ns.gte.com From: rvirzi@gte.com (Robert A. Virzi) Subject: Paper Review - A empirical comparison of three usability evaluation methods Content-Length: 1195 Status: O Dear Reviewer: I am the associate chair for the paper A empirical comparison of three usability evaluation methods and you are shown as one of the reviewers for this paper. I expect to have your review of this paper by OCTOBER 17th, via email. Please be sure to use the reviewer form already sent to you (I can forward a new one if you lost the one mailed earlier). If you don't have the paper, the review form, or if you won't be able to review the paper by OCTOBER 17th, please let me know immediately. You can reach me using any of the contact methods shown below, although email is preferred. Reviews MUST be returned via email. You should expect confirmation from me that I've recieved your review. I'll send out reminders as the deadline approaches, if need be. Please, help the process run smoothly by writing persuasive reviews and getting them in on time. That's all for now. I'm looking forward to hearing what you all have to say about this paper. -Bob Robert Virzi rvirzi@gte.com voice: +1 617.466.2881 GTE Labs, MS-38 fax: +1.617.466.4035 40 Sylvan Rd Waltham, MA, USA 02254 From rvirzi@gte.com Fri Oct 11 11:39:59 1996 Return-Path: Received: from mailer.oclc.org (mailer.dev.oclc.org) by oclc.org (4.1/SMI-4.1) id AA20892; Fri, 11 Oct 96 11:39:58 EDT Received: from mail.cis.ohio-state.edu by mailer.oclc.org (5.0/SMI-4.1) id AA19481; Fri, 11 Oct 1996 11:39:55 +0500 Received: from mail.acm.org (mail.acm.org [199.222.69.4]) by mail.cis.ohio-state.edu (8.6.7/8.6.4) with ESMTP id KAA19341 for ; Fri, 11 Oct 1996 10:42:32 -0400 Received: from ns.gte.com (ns.gte.com [132.197.8.9]) by mail.acm.org (8.7.5/8.7.5) with ESMTP id KAA32532 for ; Fri, 11 Oct 1996 10:42:48 -0400 Received: from [132.197.104.26] by ns.gte.com (8.7.5/) X-Sender: rv01@pophost.gte.com Message-Id: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Date: Fri, 11 Oct 1996 10:40:06 +0100 To: (Recipient list suppressed) From: rvirzi@gte.com (Robert A. Virzi) Subject: Paper Review - Speech Browsing the World Wide Web Content-Length: 513 Status: O Dear Reviewer: I am the associate chair for the paper Speech Browsing the World Wide Web This is just a reminder that your review is due to me by next Thursday, 17-Oct. If you get this and you already sent me your review, please contact me or send the review again. Thanks. -Bob Robert Virzi rvirzi@gte.com voice: +1 617.466.2881 GTE Labs, MS-38 fax: +1.617.466.4035 40 Sylvan Rd Waltham, MA, USA 02254 From rvirzi@gte.com Fri Oct 11 12:18:51 1996 Return-Path: Received: from mailer.oclc.org (mailer.dev.oclc.org) by oclc.org (4.1/SMI-4.1) id AA25589; Fri, 11 Oct 96 12:18:51 EDT Received: from mail.cis.ohio-state.edu by mailer.oclc.org (5.0/SMI-4.1) id AA21301; Fri, 11 Oct 1996 12:18:48 +0500 Received: from mail.acm.org (mail.acm.org [199.222.69.4]) by mail.cis.ohio-state.edu (8.6.7/8.6.4) with ESMTP id KAA19207 for ; Fri, 11 Oct 1996 10:38:42 -0400 Received: from ns.gte.com (ns.gte.com [132.197.8.9]) by mail.acm.org (8.7.5/8.7.5) with ESMTP id KAA91446 for ; Fri, 11 Oct 1996 10:38:58 -0400 Received: from [132.197.104.26] by ns.gte.com (8.7.5/) X-Sender: rv01@pophost.gte.com Message-Id: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Date: Fri, 11 Oct 1996 10:36:02 +0100 To: (Recipient list suppressed) From: rvirzi@gte.com (Robert A. Virzi) Subject: Paper Review - A empirical comparison of three usability evaluation methods Content-Length: 539 Status: O Dear Reviewer: I am the associate chair for the paper A empirical comparison of three usability evaluation methods This is just a reminder that your review is due to me by next Thursday, 17-Oct. If you get this and you already sent me your review, please contact me or send the review again. Thanks. -Bob Robert Virzi rvirzi@gte.com voice: +1 617.466.2881 GTE Labs, MS-38 fax: +1.617.466.4035 40 Sylvan Rd Waltham, MA, USA 02254 From betsy.comstock@ljo.dec.com Mon Oct 14 15:50:23 1996 Return-Path: Received: from mailer.oclc.org (mailer.dev.oclc.org) by oclc.org (4.1/SMI-4.1) id AA04349; Mon, 14 Oct 96 15:50:22 EDT Received: from mail.cis.ohio-state.edu by mailer.oclc.org (5.0/SMI-4.1) id AA23416; Mon, 14 Oct 1996 15:49:50 +0500 Received: from mail.acm.org (mail.acm.org [199.222.69.4]) by mail.cis.ohio-state.edu (8.6.7/8.6.4) with ESMTP id OAA23511 for ; Mon, 14 Oct 1996 14:06:03 -0400 Received: from mail12.digital.com (mail12.digital.com [192.208.46.20]) by mail.acm.org (8.7.5/8.7.5) with ESMTP id OAA21490 for ; Mon, 14 Oct 1996 14:06:10 -0400 Received: from whyvms.ako.dec.com by mail12.digital.com (8.7.5/UNX 1.5/1.0/WV) id NAA04335; Mon, 14 Oct 1996 13:56:26 -0400 (EDT) Received: from isbu-mail.Syntax: by whyvms.ako.dec.com (5.65/MS-010395) id AA06558; Mon, 14 Oct 1996 13:54:53 -0400 Received: from lkgdhcp-20-244-39.lkg.dec.com [16.20.244.39] (HELO bcomstock.ibg.ljo.dec.com) by isbu-mail.ljo.dec.com (AltaVista Mail V1.0/1.0 BL17 listener) id 0000_0042_3262_7edb_992a; Mon, 14 Oct 1996 13:56:43 -0400 Received: by bcomstock.ibg.ljo.dec.com with Microsoft Mail id <01BBB9D8.3AA9C660@bcomstock.ibg.ljo.dec.com>; Mon, 14 Oct 1996 14:01:43 -0400 Message-Id: <01BBB9D8.3AA9C660@bcomstock.ibg.ljo.dec.com> From: Betsy Comstock To: "'c.mcknight@lboro.ac.uk'" , "'hbrunne@uswest.com'" , "'lnormore@cas.org'" , "'paul.kearney@sharp.co.uk'" , "'perlman@acm.org'" , "'scooter@gene.com'" Subject: CHI Paper 1098 Date: Mon, 14 Oct 1996 14:01:41 -0400 Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Content-Length: 322 Status: RO Hi CHI Reviewer! According to the notes I've received from Colin Ware and Dennis Wixon, I = am the Associate Chair responsible for the review of paper 1098. So, = when you've finished your review, please Email it NO LATER THAN 5PM EDT = OCTOBER 17 to: CHI@omg.unb.ca &=09 betsy.comstock@ljo.dec.com Thanks! Betsy From l.tweedie@ic.ac.uk Mon Oct 14 17:23:03 1996 Return-Path: Received: from mailer.oclc.org (mailer.dev.oclc.org) by oclc.org (4.1/SMI-4.1) id AA15186; Mon, 14 Oct 96 17:23:02 EDT Received: from mail.cis.ohio-state.edu by mailer.oclc.org (5.0/SMI-4.1) id AA27436; Mon, 14 Oct 1996 17:22:51 +0500 Received: from mail.acm.org (mail.acm.org [199.222.69.4]) by mail.cis.ohio-state.edu (8.6.7/8.6.4) with ESMTP id MAA20510 for ; Mon, 14 Oct 1996 12:59:18 -0400 From: l.tweedie@ic.ac.uk Received: from punch.ic.ac.uk (punch.ic.ac.uk [155.198.5.17]) by mail.acm.org (8.7.5/8.7.5) with ESMTP id MAA20744 for ; Mon, 14 Oct 1996 12:59:25 -0400 Received: from judy.ic.ac.uk by punch.ic.ac.uk with SMTP (PP); Mon, 14 Oct 1996 17:45:18 +0100 Received: from dinghy.ee.ic.ac.uk (dinghy.ee.ic.ac.uk [155.198.123.8]) by judy.ic.ac.uk (8.7.5/8.7.5) with SMTP id RAA22597 for ; Mon, 14 Oct 1996 17:44:50 +0100 (BST) Received: from [155.198.123.20] (pixie) by dinghy.ee.ic.ac.uk (4.1/4.0) id AA27362; Mon, 14 Oct 96 17:44:46 BST X-Sender: lisat@155.198.123.8 Message-Id: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Date: Mon, 14 Oct 1996 17:42:03 +0100 To: perlman@acm.org Subject: CHI paper 1036 Content-Length: 457 Status: RO Hi Don't forget - I need your CHI review as soon as possible but definitely by 17th (this Thursday) Lisa ________________________________________________ Lisa Tweedie Tel: 0171 589 5111 ext:56210 Postgrad Student Supervisor: Professor Bob Spence Dept. of Electrical Eng. Imperial College Exhibition Road South Kensington London SW7 http://www.ee.ic.ac.uk/research/information/www/lisat.html From tam@comp.lancs.ac.uk Tue Oct 15 16:39:52 1996 Return-Path: Received: from mailer.oclc.org (mailer.dev.oclc.org) by oclc.org (4.1/SMI-4.1) id AA19625; Tue, 15 Oct 96 16:39:51 EDT Received: from mail.cis.ohio-state.edu by mailer.oclc.org (5.0/SMI-4.1) id AB17067; Tue, 15 Oct 1996 16:39:43 +0500 Received: from mail.acm.org (mail.acm.org [199.222.69.4]) by mail.cis.ohio-state.edu (8.6.7/8.6.4) with ESMTP id PAA01361 for ; Tue, 15 Oct 1996 15:46:01 -0400 Received: from milou.comp.lancs.ac.uk (milou.comp.lancs.ac.uk [194.80.34.7]) by mail.acm.org (8.7.5/8.7.5) with SMTP id PAA18440 for ; Tue, 15 Oct 1996 15:46:06 -0400 Received: from tina.comp.lancs.ac.uk by milou.comp.lancs.ac.uk; Tue, 15 Oct 1996 20:40:19 +0100 Received: from [148.88.9.252] by tina.comp.lancs.ac.uk; Tue, 15 Oct 1996 20:41:32 +0100 X-Sender: tam@milou.comp.lancs.ac.uk Message-Id: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Date: Tue, 15 Oct 1996 20:57:03 +0000 To: tam@comp.lancs.ac.uk From: Tom Rodden Subject: CHI'97 Review 1199 Content-Length: 256 Status: RO Hi, I am the associate chair responsible for Paper #: 1199 Title: Crowded collaborative virtual environments Could I remind you that reviews are due by 17th October 1996. Please email me if you will have problems meeting this deadline. Tom Rodden From jalarson@COLBY.EDU Wed Oct 16 09:54:40 1996 Return-Path: Received: from mailer.oclc.org (mailer.dev.oclc.org) by oclc.org (4.1/SMI-4.1) id AA23562; Wed, 16 Oct 96 09:54:40 EDT Received: from host-04.colby.edu by mailer.oclc.org (5.0/SMI-4.1) id AA18915; Wed, 16 Oct 1996 09:54:34 +0500 Received: from [137.146.66.54] (dip-07.mudd.colby.edu) by host-04.colby.edu with SMTP (1.40.112.8/Colby 1.1) id AA092584065; Wed, 16 Oct 1996 09:54:25 -0400 Date: Wed, 16 Oct 1996 09:54:25 -0400 Message-Id: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" To: perlman@mailer.oclc.org From: jalarson@COLBY.EDU (Jim Larson) Subject: chi97 tutorial review 101 reviewer 16 Content-Length: 5809 Status: O Notifications of acceptance/rejection were snail-mailed on Friday 16 Oct. > > Your reviewer number: 16 > Proposal number: 101 > Proposal author(s): Gary Perlman > > > 1. How familiar are you with the subject matter of the tutorial? > (1 = not at all, 5 = expert) >5 > > 2. Does the tutorial proposal explain the content of the tutorial > clearly and in detail? If not, what areas need further clarification > or detail? > >The author included a complete copy of the handouts used at HFES'96. >It's assumed that the same, or substantially similar, handouts would >be used for CHI'97. Based on these handouts, it's quite clear what >will be covered. > > 3. Does the tutorial proposal explain the activities (lecture, > exercises, discussion, demonstrations, etc.) that will take place > during the tutorial clearly and in detail? If not, what areas need > further clarification or detail? > >It's clear that there will be lectures, with supporting examples, >followed by an exercise for each of the major categories of evaluation >techniques. The details of the exercises are not all clear (primarily >for the first and third exercises), but I do not believe this is a >problem. The author has given this tutorial successfully a number of >times and I believe he has an assortment of useful exercises. > > 4. How appropriate are the activities (lecture, exercises, > discussion, demonstrations, etc.) for presenting the content of the > tutorial? Why? > >The basic plan seems reasonable. From my experiences, I believe these >techniques can be conveyed reasonably in the amount of time allocated. >Assuming the exercises are carefully chosen, participants should gain >a reasonable understanding of the issues discussed. > > 5. The two-page camera-ready summary is a draft of what will appear > in the Conference Companion. Is it clear, and does it match the > content of the tutorial? Why or why not? > >Yes - this summarizes the tutorial. > > 6. The four-paragraph description is a draft of what will appear > in the Advance Program. Is it clear, and does it match the content of > the tutorial? Will the right students end up in the tutorial? What > would make it more clear? > >Yes, this clearly describes the content and activities. Based on this >description, people new to the area of usability evaluation should >sign up - which is the goal. > > 7. How well does the content of the tutorial cover the topic? > What material should be added/dropped? Is the content current, or > does it need updating? Can you recommend sources that the authors > should consult? > >The content seems reasonable. The author may want to update his >references for the cognitive walkthrough. I see that he includes the >"Usability Inspection Methods" book by Nielsen & Mack in the >references. This includes the best description of the most recent >version of the cognitive walkthrough that I have seen, I'd suggest >mentioning it when describing the cognitive walkthrough. > > 8. How well does the amount of material fit the length of the > tutorial? Does it try to cover too much or too little for the time > requested? What would you recommend adding or deleting? > >I think these topics can be covered reasonably well in the alloted >time. > > 9. How well will the topic appeal to CHI attendees? > (1 = little appeal, 5 = broad interest) > >4 - Would be higher, but it is an introductory tutorial which > eliminates people with more experience. > > 10. To what specialities or backgrounds is this tutorial likely > to appeal? (Examples: software developers, usability specialists, > industrial designers, etc.) Why? > >Software developers are probably the primary audience. Managers may >be interested. > > 11. What is your estimate of the number of people who are likely > to take this tutorial? (At CHI '96, tutorials ranged from 16 to 351 > people, with a median of 63.) > > ____ very small (fewer than 20) > ____ small (20-39) > ____ medium (40-69) > xxxx large (70-100) > ____ very large (more than 100) > > 12. What is the level of this tutorial? > > xxxx introductory (no prerequisites; a person with no > background in CHI could understand it) > ____ intermediate (assumes a general background in CHI, > but no knowledge of the specific topic) > ____ advanced (assumes some knowledge of the specific > topic) > > 13. Is this tutorial: > > ____ a survey of a broad topic > ____ an in-depth treatment of a specialized topic > xxxx other (explain) > >It goes into more detail than a survey would, but it doesn't provide a >detailed treatment of all of the issues. I'd say the amount of detail >is appropriate for this tutorial. > > 14. Does the title reflect the tutorial content? Can you suggest > a better or simpler title? > >The title is fine. > > 15. Have you heard other talks or tutorials by the author(s)? > (If there is more than one author, indicate which you have heard.) If so, > do you feel that the author(s) will do a good job of presenting this > material? > >Yes, I believe the author will do a good job with this tutorial. > > 16. Do you think that the equipment and supplies requested are reasonable, > given the content of the tutorial? If not, what would be more reasonable? > >Yes, the equipment is reasonable. > > 17. Would you like to take this tutorial yourself? Why? > >No, I teach these topics myself, so I wouldn't be interested in an >introductory tutorial on these issues. > > 18. What is your overall rating of this tutorial? Why? > (1 = don't include, 5 = "must have" for CHI 97) > >5 > > 19. Additional comments for the author(s): > From jalarson@COLBY.EDU Wed Oct 16 11:59:42 1996 Return-Path: Received: from mailer.oclc.org (mailer.dev.oclc.org) by oclc.org (4.1/SMI-4.1) id AA12631; Wed, 16 Oct 96 11:59:41 EDT Received: from host-04.colby.edu by mailer.oclc.org (5.0/SMI-4.1) id AA24278; Wed, 16 Oct 1996 11:59:34 +0500 Received: from [137.146.66.54] (dip-07.mudd.colby.edu) by host-04.colby.edu with SMTP (1.40.112.8/Colby 1.1) id AA180181568; Wed, 16 Oct 1996 11:59:28 -0400 Date: Wed, 16 Oct 1996 11:59:28 -0400 Message-Id: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" To: perlman@mailer.oclc.org From: jalarson@COLBY.EDU (Jim Larson) Subject: CHI 97 tutorial review 101 reviewer 14 Content-Length: 5593 Status: RO Acceptance/rejection notifications were snail-mailed on Friday. One of the reviews of your tutorial follows: > > Your reviewer number: 14 > Proposal number: 101 > Proposal author(s): Gary Perlman > > > 1. How familiar are you with the subject matter of the tutorial? > (1 = not at all, 5 = expert) > > 4 > > 2. Does the tutorial proposal explain the content of the tutorial > clearly and in detail? If not, what areas need further clarification > or detail? > > Yes. > > 3. Does the tutorial proposal explain the activities (lecture, > exercises, discussion, demonstrations, etc.) that will take place > during the tutorial clearly and in detail? If not, what areas need > further clarification or detail? > > Mostly, especially given the supplementary materials. It would > be useful to include more description of hands-on exercises and > activities in the proposal, but this is an experienced presenter > whose work is well-known. > > 4. How appropriate are the activities (lecture, exercises, > discussion, demonstrations, etc.) for presenting the content of the > tutorial? Why? > > Quite appropriate. My experience teaching usability evaluation > (in a University course and short course) suggests that this is a > good mix. I find about 50% lecture is appropriate for large groups > (25% lecture for groups small enough to teach during exercises). > > One thing not obviously included that may be useful is a set of > "demonstration--critique" activities (e.g., having the class critique > a bad test plan or a flawed user test). > > 5. The two-page camera-ready summary is a draft of what will appear > in the Conference Companion. Is it clear, and does it match the > content of the tutorial? Why or why not? > > It is generally clear and matches to tutorial content. The use of > highlighting in the bibliography list is unclear. I think the > description would be more compelling (though its purpose is not > necessarily marketing) if the bibliography were reduced and the > evaluation method subsections in the topics outline were extended. > > 6. The four-paragraph description is a draft of what will appear > in the Advance Program. Is it clear, and does it match the content of > the tutorial? Will the right students end up in the tutorial? What > would make it more clear? > > It is clear, targets the right audience, and has the right focus. > > 7. How well does the content of the tutorial cover the topic? > What material should be added/dropped? Is the content current, or > does it need updating? Can you recommend sources that the authors > should consult? > > Content is still current and relevant. Author's notes show > impressive familiarity with the topic. One useful addition would > be to include a sample usability report in the notes packet. > > 8. How well does the amount of material fit the length of the > tutorial? Does it try to cover too much or too little for the time > requested? What would you recommend adding or deleting? > > Contents are appropriate, if slightly ambitious. I generally > cover this material in eight hours (with more hands-on activities), > but I think it is the right set of material and it is better to > cover it somewhat quickly than to omit parts. > > 9. How well will the topic appeal to CHI attendees? > (1 = little appeal, 5 = broad interest) > > 4 > > 10. To what specialities or backgrounds is this tutorial likely > to appeal? (Examples: software developers, usability specialists, > industrial designers, etc.) Why? > > Certainly to software developers and beginning usability specialists. > Also, management and newcomers to HCI. > > 11. What is your estimate of the number of people who are likely > to take this tutorial? (At CHI '96, tutorials ranged from 16 to 351 > people, with a median of 63.) > > ____ very small (fewer than 20) > ____ small (20-39) > _XX_ medium (40-69) > ____ large (70-100) > ____ very large (more than 100) > > 12. What is the level of this tutorial? > > _XX_ introductory (no prerequisites; a person with no > background in CHI could understand it) > ____ intermediate (assumes a general background in CHI, > but no knowledge of the specific topic) > ____ advanced (assumes some knowledge of the specific > topic) > > 13. Is this tutorial: > > _XX_ a survey of a broad topic > ____ an in-depth treatment of a specialized topic > ____ other (explain) > > 14. Does the title reflect the tutorial content? Can you suggest > a better or simpler title? > > Very good title. > > 15. Have you heard other talks or tutorials by the author(s)? > (If there is more than one author, indicate which you have heard.) If so, > do you feel that the author(s) will do a good job of presenting this > material? > > Have heard him talk (not tutorial) and expect he will do well. > > 16. Do you think that the equipment and supplies requested are reasonable, > given the content of the tutorial? If not, what would be more reasonable? > > Yes/NA. > > 17. Would you like to take this tutorial yourself? Why? > > No. Already familiar with and teach the material. > > 18. What is your overall rating of this tutorial? Why? > (1 = don't include, 5 = "must have" for CHI 97) > > 4 > > 19. Additional comments for the author(s): > From jalarson@COLBY.EDU Wed Oct 16 17:28:26 1996 Return-Path: Received: from mailer.oclc.org (mailer.dev.oclc.org) by oclc.org (4.1/SMI-4.1) id AA22900; Wed, 16 Oct 96 17:28:25 EDT Received: from host-04.colby.edu by mailer.oclc.org (5.0/SMI-4.1) id AA03245; Wed, 16 Oct 1996 17:28:24 +0500 Received: from [137.146.66.54] (dip-07.mudd.colby.edu) by host-04.colby.edu with SMTP (1.40.112.8/Colby 1.1) id AA132081300; Wed, 16 Oct 1996 17:28:20 -0400 Date: Wed, 16 Oct 1996 17:28:20 -0400 Message-Id: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" To: perlman@mailer.oclc.org (Gary PERLMAN) From: jalarson@COLBY.EDU (Jim Larson) Subject: Re: chi97 tutorial review 101 reviewer 16 Cc: chi97-tutorials@acm.org Content-Length: 472 Status: RO Sorry for the confusion. Ellen will be your liaison. -Jim >> Notifications of acceptance/rejection were snail-mailed on Friday 16 Oct. > >Hi, > >I got my acceptance, but it did not indicate who >was my liaison. Is it you or Ellen? > >I'll get to the checklist soon, >but one thing I can tell you... >I ain't gonna be sending my electronic >document via postal mail or courier! >Some bits might get dropped :-) >I'll submit them according to the instructions. > >Gary From perlman@oclc.org Thu Oct 17 11:04:55 1996 Return-Path: Received: from dev1.NISDEV (dev1.dev.oclc.org) by oclc.org (4.1/SMI-4.1) id AA29505; Thu, 17 Oct 96 11:04:22 EDT Received: by dev1.NISDEV (SMI-8.6/SMI-SVR4) id LAA12861; Thu, 17 Oct 1996 11:04:21 -0400 Date: Thu, 17 Oct 1996 11:04:21 -0400 From: perlman@oclc.org (Gary PERLMAN) Message-Id: <199610171504.LAA12861@dev1.NISDEV> To: CHI@omg.unb.ca, betsy.comstock@ljo.dec.com, perlman@dev1 Subject: CHI97 Review Paper 1098 Reviewer 293 Status: O +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +++++++++++ CHI97 Paper Review Form-Begin Banner +++++++++++ +++++++++++++ INCLUDE AT THE TOP OF EACH REVIEW +++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ All your input should appear after a curly bracket followed by a colon. PAPER NUMBER {: 1098 } PAPER TITLE {: Learning Personal Preferences on Online Newspaper Articles from User behaviors } REVIEWER NUMBER {: 293 }1. Briefly SUMMARIZE the paper {: The abstract is a good summary of the paper. The work compared creating a user interest profile (as represented by weighted keywords) by required explicit ratings of all articles, implicit (automated) assignment by monitoring browsing behavior, and combining implicit and optional explicit ratings. The result was that a combination of implicit with optional explicit ratings quickly approached the best rating scheme (required explicit), but with less effort. }2. What TYPE OF SUBMISSION is appropriate for this paper (see bottom of Cover Sheet Two for the authors' selection)? If your assessment of type differs from the Authors, please explain why you made your selection. In the next two questions, use the criteria specific to this Type of Submission (see above) to assess the paper. {: EMPIRICAL + experience + systems The work is primarily empirical, but the methods are a little shaky because the analysis methods seemed to be ad hoc. }3. Describe the IMPORTANCE OF THE PROBLEM, addressing the following concerns: Is it new and significant? Is it a useful and relevant problem? Do they sufficiently motivate the problem? Did you learn anything? Does the issue apply to CHI? {: The work is novel in that it attempts to find a balance of implicit methods (providing inferior results) but requiring no work by users with explicit methods. I am not sure I learned anything because of two problems: (1) There is no comparison of the methods to known methods, so I don't know if their BEST method is better or worse than other approaches. (2) Their analysis was not explained in enough detail for me to understand it fully, so I am not sure how much of their results I trust. But, this is a useful area to explore. Network access to information is clearly a problem of interest. }4. Describe the QUALITY OF THE ARGUMENT, addressing the following: Does the analysis use appropriate methodology? Is the argument well-structured? Do they provide sufficient data and/or well-supported arguments? Do they cite relevant work? Is the paper appropriately focused? Does the analysis cover all the important issues at the appropriate level of detail? {: The methods seemed to be made up on the fly. The methods they used for attaching weights did not seem to be related to existing models in IR (e.g., vector space). Their rating scale was not a standard one, ranging from "very relevant" to "not relevant at all", which had poorly chosen end anchors and which they converted arbitrarily to a ratio scale. I did not understand the definition of the "rank order" measure, making it difficult for me to have high confidence in their results. They used no inferential statistics, and in some cases, it did not matter (e.g., in 6a, A is clearly better than B), but in some cases it did (e.g., in 6a, is B or C declining?). The authors approached some analyses intuitively, but that can be subjective, and some inferential stats on some subtle trends would have provided an objective view. In 6b, the authors put a line (I am not sure it is a regression line) through their points to show the correlation between subjective ranks and anticipated scores, but a closer view of the data suggest that such an analysis is inappropriate. For ratings A|B, the anticipated scores were highly ranked, and maybe even correlated within that subset (i.e., As were ranked higher than Bs). But for ratings C|D|E, there appears to be no correlation between rating and rank. This may be in part due to the lack of psychometric properties of their rating scale. Their conclusions were well stated, but I had to question one aty the top of page six, column 1. When bonus points are set properly... IT DOES NOT WORK AT ALL WITH 30 BONUS POINTS (my caps) To my eye, the curve seems to be moving down over time, and perhaps with more data, it would asymptote much lower. the authors did not check for this trend, so they should not have such a strong statement. (This is independent of the issue of modeling using arbitrary values like 10 or 30 bonus points.) } 5. Describe the QUALITY OF THE MECHANICS, addressing the following: Do they clearly describe what was done and/or how it was studied? Is the writing clear and concise? Do they provide the right level of detail? Do they use too much jargon? Do the figures support the text? {: As mentioned, some measures andf procedures were not well specified. The writing was adequate. The figures and overall organization were okay. }6. Provide any OTHER COMMENTS you believe would be useful to the author (including pointers to missing relevant work). {: Table 1: "Number of Subjects" -- change to "Subject Number" }7. Using the scale 1 = Definitely reject 2 = Probably reject 3 = Could go either way 4 = Probably accept 5 = Definitely accept provide a numerical rating of the paper's ACCEPTABILITY {: 2 }8. SUMMARIZE your ASSESSMENT of the paper, pointing out which aspects described above you weighted most heavily in your rating in question 7? {: Methodological problems in data collection and analysis make this paper of dubious value. The lack of references to the IR filtering literature is also a problem, but I believe the combination of implicit and explicit ratings to be novel, regardless. }9. Using the scale 1 = Very unconfident, really just a guess 2 = Rather unconfident, but I know a bit 3 = Moderately confident, I know as much as most 4 = Pretty confident, I know this area well 5 = Extremely confident, I consider myself an expert rate your EXPERTISE in the area addressed by the paper. {: 4 }10. Use this space to provide comments that you feel are relevant to the review process but that you do NOT want FORWARDED to the author(s) {: Quaint. The ad hoc methods reminded me of Donders' (1876) work. Suggest they explain their methods more clearly and submit to INTERACT'97. } +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ++++++++++++ CHI97 Paper Review Form-End Banner ++++++++++++ ++++++++++++ INCLUDE AT THE BOTTOM OF EACH FORM +++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ From perlman@oclc.org Thu Oct 17 15:28:40 1996 Return-Path: Received: from dev1.NISDEV (dev1.dev.oclc.org) by oclc.org (4.1/SMI-4.1) id AA02306; Thu, 17 Oct 96 15:28:33 EDT Received: by dev1.NISDEV (SMI-8.6/SMI-SVR4) id PAA06264; Thu, 17 Oct 1996 15:28:33 -0400 Date: Thu, 17 Oct 1996 15:28:33 -0400 From: perlman@oclc.org (Gary PERLMAN) Message-Id: <199610171928.PAA06264@dev1.NISDEV> To: CHI@omg.unb.ca, perlman@dev1, rvirzi@gte.com Subject: CHI97 Review Paper 1185 Reviewer 293 Status: O +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +++++++++++ CHI97 Paper Review Form-Begin Banner +++++++++++ +++++++++++++ INCLUDE AT THE TOP OF EACH REVIEW +++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ All your input should appear after a curly bracket followed by a colon. PAPER NUMBER {: 1185 } PAPER TITLE {: An empirical comparison of three usability evaluation methods } REVIEWER NUMBER {: 293 }1. Briefly SUMMARIZE the paper {: 102 students evaluated one or two systems using an inspection method or empirical evaluation followed by empirical evaluation. Students also provided ratings about the methods. The authors pooled all the problems found and analysed the ability of the different methods to find usability problems of different severities. }2. What TYPE OF SUBMISSION is appropriate for this paper (see bottom of Cover Sheet Two for the authors' selection)? If your assessment of type differs from the Authors, please explain why you made your selection. In the next two questions, use the criteria specific to this Type of Submission (see above) to assess the paper. {: EMPIRICAL Methodology }3. Describe the IMPORTANCE OF THE PROBLEM, addressing the following concerns: Is it new and significant? Is it a useful and relevant problem? Do they sufficiently motivate the problem? Did you learn anything? Does the issue apply to CHI? {: The authors provide a nice little section on what's new and important. I think that understanding the effectiveness of combined methods is highly relevant to actual practice, where practitioners usually apply inspection methods before testing users. I thought that the collection of evaluator satisfaction ratings was of interest, although when gathered from student evaluators, I am not confident the data have much to offer. Because I had problems understanding what data were being presented, I am not sure if I learned anything -- I kept on wanting to see their data because I could not follow their analyses. }4. Describe the QUALITY OF THE ARGUMENT, addressing the following: Does the analysis use appropriate methodology? Is the argument well-structured? Do they provide sufficient data and/or well-supported arguments? Do they cite relevant work? Is the paper appropriately focused? Does the analysis cover all the important issues at the appropriate level of detail? {: The literature review and contrast is done well. The argument could be motivated more by a theoretical basis than they used -- for example, at the very end they note that there are more similarities between Cognitive Walkthroughs and Think Aloud than between Heuristic Evaluation and Think Aloud. This might have been stated as a hypothesis to make a prediction, but the work was more presented as: let's compare the methods in combination and see how they do, without a theoretical motivation (which is not all that bad). I have serious reservations about the validity of the experiment because of the subject population -- students, and what they evaluated (only the results for the text retrieval system were reported, I think). So we have non-experts in evaluation and in the domain, which might still be of interest if their data were compared to double-expert evaluators. Instead, the student data is compared to the pooled student data, which itself is not validated to a great extent (the authors make some judgements on the ratings of severity, as well as deleted data for some types of ratings (another indication of the lack of experience of the students)). My experience, which is supported in published studies, is that novice evaluators identify many non-problems (that do not affect performance) and sometimes miss important ones, which makes me even more wary of comparing group evaluations to the union of the evaluations. I was left wondering how well the students followed the methods. For example, what tasks/scenarios were designed (and by whom)? What forms did the student use to gather data? The use of the students themselves as SUBJECTS (as well as evaluators) makes me uneasy about the realism of the results. Even if a student is unfamiliar with a tool, the fact that they may be in the process of doing a usability inspection of another tool, and will soon be doing or just did a thinking aloud evaluation of another tool, introduces many variables that make me wonder what the data mean. Any study like this one has a lot of ad hoc data manipulation, and this is no exception. Many readers would be uncomfortable with the idea that 34+14=48 of 155 problems are removed by the authors. The authors do make some attempts to analyse the false and neutralized problems separately, however. The data analysis generally lacked rigor. I think one t-test was mentioned, but many of their tests required something to test more than two conditions. No degrees of freedom were reported, which would have helped me understand what comparisons were actually being made -- in most cases, I was not sure. No t-values or significance levels were reported, and there were few measures of variation reported. The analyses in the tables could have been clearer, especially in the discussion of MARGINAL differences of problems found with TA. For example, Tables II and IV could be combined to highlight the increase in % problems found: CW-TA = CW + 5% HE-TA = HE + 6% Perhaps remarkably, but certainly not significantly, CW-TA = TA - 3% The scale used for satisfaction was unusual, ranging from -100 (disagree) to 100 (agree). a 200-point scale is far more than the number of distinctions that can be made. The questions were worded in both positive and negative forms, making the ratings difficult for the reader and possibly confusing to the rater. For example, 2. The procedure of the method was unclear. CW=-53 HE=25 TA=-70 means that TA was very not unclear, and CW was somewhat not unclear but HE was a little unclear. I would like to see standard errors (or standard deviations and Ns) for the ratings. } 5. Describe the QUALITY OF THE MECHANICS, addressing the following: Do they clearly describe what was done and/or how it was studied? Is the writing clear and concise? Do they provide the right level of detail? Do they use too much jargon? Do the figures support the text? {: I'd prefer to read actual percentages or values instead of de-quantified references like "One fourth" and "one third" or "approximately the same total number". Give me the numbers and I'll decide if they have the inferred properties. The first abbreviation of the cryptic LUP was LUB. I found this abbreviation a little too much jargony, but the use of acronyms in the paper is probably better than longer terms. This paper would need an English-speaking editor to fix the grammar in about 20 places. Reformat Table XI with the ratings so that things line up and so that the question numbers are in their own column. Avoid the term Diverge to indicate statistical significance; indicate the significance level instead. Indicate which pairs of values are significantly different. It was confusing that the authors advertised the 102 number and then noted that one some of the data were to be presented. It would be easier to understand if the data actually used were reported throughout, and the authors' other publishing plans be placed in a footnote. The experimental conditions could have been much better described, such as by a diagram showing how many groups were in one condition and then another. From the procedure, I could tell which tools the groups evaluated, but I could not understand the design of the control condition to TA; I could only assume that it was as I hoped. The description is especially frustrating because only the data from one tool is presented, so much of the complexity is there for no reason. The statement "heuristic evaluation has invited the evaluators to take a broader scope of awareness upon the usability problems" should probably be replaced by a quantified statement like "evaluators using Heuristic Evaluation found 19% of the problems compared to 10% for Cognitive Walkthroughs". It's more accurate, although it does highlight the extremely low percentage of problems found by the double-non-expert evaluators. Reference 18 has a typo: "og" }6. Provide any OTHER COMMENTS you believe would be useful to the author (including pointers to missing relevant work). {: I'd like to see a paper on the class organization process, maybe for the SIGCHI Bulletin in the education section. Contact Andrew Sears, the Education Editor. }7. Using the scale 1 = Definitely reject 2 = Probably reject 3 = Could go either way 4 = Probably accept 5 = Definitely accept provide a numerical rating of the paper's ACCEPTABILITY {: 2 }8. SUMMARIZE your ASSESSMENT of the paper, pointing out which aspects described above you weighted most heavily in your rating in question 7? {: The subject population creates some validity questions that will not go away. As a case study in education, I would give this a higher rating, but as a study to guide practitioners, I am not confident about the source (subjects) of the data (double-non-experts). The experimental procedure needs some clarification, and the data analysis needs to be made more rigorous. On the positive side, it's a lot of data with some interesting twists, so I think it's worth pursuing. I think the authors have to be very careful about generalizing the results beyond the scope of the study, though. }9. Using the scale 1 = Very unconfident, really just a guess 2 = Rather unconfident, but I know a bit 3 = Moderately confident, I know as much as most 4 = Pretty confident, I know this area well 5 = Extremely confident, I consider myself an expert rate your EXPERTISE in the area addressed by the paper. {: 4.5 }10. Use this space to provide comments that you feel are relevant to the review process but that you do NOT want FORWARDED to the author(s) {: I started out liking this paper and was very excited, but my opinion got lower and lower as I read the paper. } +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ++++++++++++ CHI97 Paper Review Form-End Banner ++++++++++++ ++++++++++++ INCLUDE AT THE BOTTOM OF EACH FORM +++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ From perlman@oclc.org Thu Oct 17 15:28:43 1996 Return-Path: Received: from dev1.NISDEV (dev1.dev.oclc.org) by oclc.org (4.1/SMI-4.1) id AA02343; Thu, 17 Oct 96 15:28:40 EDT Received: by dev1.NISDEV (SMI-8.6/SMI-SVR4) id PAA06290; Thu, 17 Oct 1996 15:28:40 -0400 Date: Thu, 17 Oct 1996 15:28:40 -0400 From: perlman@oclc.org (Gary PERLMAN) Message-Id: <199610171928.PAA06290@dev1.NISDEV> To: CHI@omg.unb.ca, perlman@dev1, rvirzi@gte.com Subject: CHI97 Review Paper 1109 Reviewer 293 Status: O +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +++++++++++ CHI97 Paper Review Form-Begin Banner +++++++++++ +++++++++++++ INCLUDE AT THE TOP OF EACH REVIEW +++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ All your input should appear after a curly bracket followed by a colon. PAPER NUMBER {: 1109 } PAPER TITLE {: Speech Browsing the World Wide Web } REVIEWER NUMBER {: 293 }1. Briefly SUMMARIZE the paper {: One study with a custom WWW browser gathered stats about what commands people would use to control a Wizard of Oz controlled WWW browser by voice. People used short commands. A second study compared speech input to mouse input by having half the subjects use mouse then voice, and half use voice then mouse, followed by a set of 12 tasks in which they could use either. The results were ambiguous, but more subjects used voice than the mouse. }2. What TYPE OF SUBMISSION is appropriate for this paper (see bottom of Cover Sheet Two for the authors' selection)? If your assessment of type differs from the Authors, please explain why you made your selection. In the next two questions, use the criteria specific to this Type of Submission (see above) to assess the paper. {: EMPIRICAL Input Devices (the keywords on the WWW are largely irrelevant) }3. Describe the IMPORTANCE OF THE PROBLEM, addressing the following concerns: Is it new and significant? Is it a useful and relevant problem? Do they sufficiently motivate the problem? Did you learn anything? Does the issue apply to CHI? {: The applicability of speech input is clearly a CHI area, but the authors provide little motivation for the current study. The experiments had no hypotheses, and little information was found, certainly nothing new or significant. The results on brief commands have been known since the first Wizard of Oz studies or word processors. The results from the second experiment are muddled. }4. Describe the QUALITY OF THE ARGUMENT, addressing the following: Does the analysis use appropriate methodology? Is the argument well-structured? Do they provide sufficient data and/or well-supported arguments? Do they cite relevant work? Is the paper appropriately focused? Does the analysis cover all the important issues at the appropriate level of detail? {: I did not understand why the studies were done. The first study found that people use short commands with computers, which is not a new result. The second study had a nice counter-balanced design, but it was more of an exploratory experiment than one to test a hypothesis. The result was that the data do not clearly provide anything useful, expect for some questions. The authors write: "The most interesting result of this test is that 13/20 users used voice as their PREFERRED input modality." [caps mine] That users preferred what they used more is an unwarranted conclusion because the speech modality was novel to the subjects. People may have used it more because they found it novel, fun. It would have been useful to ask (i.e., in a questionnaire) which method was preferred, possibly for which types of commands. A set of 12 tasks is probably not enough experience, though. The discussions of possible confoundings, such as order effects and extra time for Wizard of Oz, were especially welcome. They might look at: Observations on Using Speech Input for Window Navigation Schmandt et al, Interact'90, pp. 787-793. as well as the larger literature on speech input. } 5. Describe the QUALITY OF THE MECHANICS, addressing the following: Do they clearly describe what was done and/or how it was studied? Is the writing clear and concise? Do they provide the right level of detail? Do they use too much jargon? Do the figures support the text? {: }6. Provide any OTHER COMMENTS you believe would be useful to the author (including pointers to missing relevant work). {: At the end of the second results section, just before the CONCLUSIONS, on page 8: I think that voice input would interfere cognitively more with reading than mouse (or other manual) input. See the Schmandt reference above, but also check out Gould's work and others. Add Wizard of Oz to the keywords. Avoid judgemental terms like "no-nonsense" and use more precise terms like "brief" or X % 1-2 word commands. Avoid the term "human subjects" -- it's so obvious that it seems silly. }7. Using the scale 1 = Definitely reject 2 = Probably reject 3 = Could go either way 4 = Probably accept 5 = Definitely accept provide a numerical rating of the paper's ACCEPTABILITY {: 1 }8. SUMMARIZE your ASSESSMENT of the paper, pointing out which aspects described above you weighted most heavily in your rating in question 7? {: The results are not novel. The inferences are flawed. }9. Using the scale 1 = Very unconfident, really just a guess 2 = Rather unconfident, but I know a bit 3 = Moderately confident, I know as much as most 4 = Pretty confident, I know this area well 5 = Extremely confident, I consider myself an expert rate your EXPERTISE in the area addressed by the paper. {: 5 }10. Use this space to provide comments that you feel are relevant to the review process but that you do NOT want FORWARDED to the author(s) {: I suppose it does not hurt to indicate that the subjects using voice and the mouse were human, but it still made me laugh. If I had the chance to spit at the screen to select items, I'd use that as my "dominant" modality for a while, but I don't think a brief concentration indicates preference. Please feel free to ask me for more reasons to reject this. } +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ++++++++++++ CHI97 Paper Review Form-End Banner ++++++++++++ ++++++++++++ INCLUDE AT THE BOTTOM OF EACH FORM +++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ From perlman@oclc.org Thu Oct 17 16:42:15 1996 Return-Path: Received: from dev1.NISDEV (dev1.dev.oclc.org) by oclc.org (4.1/SMI-4.1) id AA11726; Thu, 17 Oct 96 16:42:14 EDT Received: by dev1.NISDEV (SMI-8.6/SMI-SVR4) id QAA15793; Thu, 17 Oct 1996 16:42:12 -0400 Date: Thu, 17 Oct 1996 16:42:12 -0400 From: perlman@oclc.org (Gary PERLMAN) Message-Id: <199610172042.QAA15793@dev1.NISDEV> To: CHI@omg.unb.ca, l.tweedie@ic.ac.uk, perlman@dev1 Subject: CHI97 Review Paper 1036 Reviewer 293 Status: O +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +++++++++++ CHI97 Paper Review Form-Begin Banner +++++++++++ +++++++++++++ INCLUDE AT THE TOP OF EACH REVIEW +++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ All your input should appear after a curly bracket followed by a colon. PAPER NUMBER {: 1036 } PAPER TITLE {: User Reactions to Agents with Faces } REVIEWER NUMBER {: 293 }1. Briefly SUMMARIZE the paper {: The author created a computerized poker game in which the opponents were depicted by agents with various facial expressions. Realistic and cartoon male/female faces, a cartoon dog, a smiley, an the invisible man (woman?), were possible players. After a short poker game, users filled out a short questionnaire about their impressions of the players. Results were summarized. }2. What TYPE OF SUBMISSION is appropriate for this paper (see bottom of Cover Sheet Two for the authors' selection)? If your assessment of type differs from the Authors, please explain why you made your selection. In the next two questions, use the criteria specific to this Type of Submission (see above) to assess the paper. {: EMPIRICAL Agents User Studies }3. Describe the IMPORTANCE OF THE PROBLEM, addressing the following concerns: Is it new and significant? Is it a useful and relevant problem? Do they sufficiently motivate the problem? Did you learn anything? Does the issue apply to CHI? {: I don't know if it's important, but it's sort of fun. I don't know anything about this area, but I have trouble taking it too seriously. I did not learn anything. Perhaps people in the area would find the data useful as it is a reasonable survey collecting some basic data. Relevant to CHI? Sure. There are motivational issues, and there are issues in using facial expressions to provide some information. }4. Describe the QUALITY OF THE ARGUMENT, addressing the following: Does the analysis use appropriate methodology? Is the argument well-structured? Do they provide sufficient data and/or well-supported arguments? Do they cite relevant work? Is the paper appropriately focused? Does the analysis cover all the important issues at the appropriate level of detail? {: I thought the study and analysis was well done. That is an area in which I am qualified. The author collected some reasonable data (i.e., the questions used and the rating scale), although the subject population is unknown, and so generalization to general populations is questionable. } 5. Describe the QUALITY OF THE MECHANICS, addressing the following: Do they clearly describe what was done and/or how it was studied? Is the writing clear and concise? Do they provide the right level of detail? Do they use too much jargon? Do the figures support the text? {: I thought the paper was well written. The tables could be made more regular by adopting one statistical test (i.e., always use F) and report it in the standard format (i.e., F(1,df)=XXX.) Don't make up a new format for F tests (i.e., F(XXX,df2)); the standard format has been used 99.999% of the time for decades. The labels for the graphs (Fig 5 - 9) could be clearer, but I liked the 95% CI bars (although I don't know which of the several common methods was used to compute them). }6. Provide any OTHER COMMENTS you believe would be useful to the author (including pointers to missing relevant work). {: }7. Using the scale 1 = Definitely reject 2 = Probably reject 3 = Could go either way 4 = Probably accept 5 = Definitely accept provide a numerical rating of the paper's ACCEPTABILITY {:3 }8. SUMMARIZE your ASSESSMENT of the paper, pointing out which aspects described above you weighted most heavily in your rating in question 7? {: I do not take this area of research too seriously, especially when it is applied to an unidentified population. If it were targeted at a specific population, say kids, then I think it would have more merit. The sample population has unknown characteristics, so it is difficult to make any generalizations. Still, the analysis was well done, and the neatness factor is high. }9. Using the scale 1 = Very unconfident, really just a guess 2 = Rather unconfident, but I know a bit 3 = Moderately confident, I know as much as most 4 = Pretty confident, I know this area well 5 = Extremely confident, I consider myself an expert rate your EXPERTISE in the area addressed by the paper. {: I'll pass on this rating. I give me a 5 on methodology, and 1 on content area. }10. Use this space to provide comments that you feel are relevant to the review process but that you do NOT want FORWARDED to the author(s) {: It's new, it's fun. Why not accept it? I accessed the site to try it out. That dog bet way too much money on bad hands! } +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ++++++++++++ CHI97 Paper Review Form-End Banner ++++++++++++ ++++++++++++ INCLUDE AT THE BOTTOM OF EACH FORM +++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ From perlman@oclc.org Thu Oct 17 16:55:53 1996 Return-Path: Received: from dev1.NISDEV (dev1.dev.oclc.org) by oclc.org (4.1/SMI-4.1) id AA13712; Thu, 17 Oct 96 16:55:42 EDT Received: by dev1.NISDEV (SMI-8.6/SMI-SVR4) id QAA18497; Thu, 17 Oct 1996 16:55:40 -0400 Date: Thu, 17 Oct 1996 16:55:40 -0400 From: perlman@oclc.org (Gary PERLMAN) Message-Id: <199610172055.QAA18497@dev1.NISDEV> To: CHI@omg.unb.ca, perlman@dev1, tam@comp.lancs.ac.uk Subject: CHI97 Review Paper 1199 Reviewer 293 Status: O +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +++++++++++ CHI97 Paper Review Form-Begin Banner +++++++++++ +++++++++++++ INCLUDE AT THE TOP OF EACH REVIEW +++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ All your input should appear after a curly bracket followed by a colon. PAPER NUMBER {: 1199 } PAPER TITLE {: Crowded collaborative virtual environments } REVIEWER NUMBER {: 293 }1. Briefly SUMMARIZE the paper {: This paper is so far removed from my areas of expertise that I don't feel qualified. I did not follow the paper. }2. What TYPE OF SUBMISSION is appropriate for this paper (see bottom of Cover Sheet Two for the authors' selection)? If your assessment of type differs from the Authors, please explain why you made your selection. In the next two questions, use the criteria specific to this Type of Submission (see above) to assess the paper. {: SYSTEMS }3. Describe the IMPORTANCE OF THE PROBLEM, addressing the following concerns: Is it new and significant? Is it a useful and relevant problem? Do they sufficiently motivate the problem? Did you learn anything? Does the issue apply to CHI? {: I really don't have a feel for the importance of the area. The authors motivate their crowd mechanism using the same arguments as are used to support decluttering in complex displays. }4. Describe the QUALITY OF THE ARGUMENT, addressing the following: Does the analysis use appropriate methodology? Is the argument well-structured? Do they provide sufficient data and/or well-supported arguments? Do they cite relevant work? Is the paper appropriately focused? Does the analysis cover all the important issues at the appropriate level of detail? {: I could not follow the paper. It's not in my area. Perhaps it assumes too much knowledge on the part of the reader. } 5. Describe the QUALITY OF THE MECHANICS, addressing the following: Do they clearly describe what was done and/or how it was studied? Is the writing clear and concise? Do they provide the right level of detail? Do they use too much jargon? Do the figures support the text? {: The writing was fine, at least at the micro level. But I never got the big picture of what they were presenting. }6. Provide any OTHER COMMENTS you believe would be useful to the author (including pointers to missing relevant work). {: }7. Using the scale 1 = Definitely reject 2 = Probably reject 3 = Could go either way 4 = Probably accept 5 = Definitely accept provide a numerical rating of the paper's ACCEPTABILITY {: I refuse to provide a rating. I should not have received this paper. }8. SUMMARIZE your ASSESSMENT of the paper, pointing out which aspects described above you weighted most heavily in your rating in question 7? {: }9. Using the scale 1 = Very unconfident, really just a guess 2 = Rather unconfident, but I know a bit 3 = Moderately confident, I know as much as most 4 = Pretty confident, I know this area well 5 = Extremely confident, I consider myself an expert rate your EXPERTISE in the area addressed by the paper. {: I feel really confident that I should not have received this paper. }10. Use this space to provide comments that you feel are relevant to the review process but that you do NOT want FORWARDED to the author(s) {: } +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ++++++++++++ CHI97 Paper Review Form-End Banner ++++++++++++ ++++++++++++ INCLUDE AT THE BOTTOM OF EACH FORM +++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ From rvirzi@gte.com Mon Oct 21 15:54:35 1996 Return-Path: Received: from mailer.oclc.org (mailer.dev.oclc.org) by oclc.org (4.1/SMI-4.1) id AA16536; Mon, 21 Oct 96 15:54:35 EDT Received: from ns.gte.com by mailer.oclc.org (5.0/SMI-4.1) id AA02598; Mon, 21 Oct 1996 15:54:34 +0500 Received: from [132.197.104.26] by ns.gte.com (8.7.5/) X-Sender: rv01@pophost.gte.com Message-Id: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Date: Mon, 21 Oct 1996 15:55:02 +0100 To: perlman@mailer.oclc.org (Gary PERLMAN) From: rvirzi@gte.com (Robert A. Virzi) Subject: Re: CHI97 Review Paper 1109 Reviewer 293 Content-Length: 536 Status: RO Gary- In your original review you said the following about the paper on speech cruising the WWW: >{: >Please feel free to ask me for more reasons to reject this. >} Well, you may be interested in knowing that this paper got three ratings of 5 out of seven reviewers. It also got two 1s and a 2. The 5s are NOT reversed scales. If you choose to revise your review, I would be happy to revise my recommendation. -Bob rvirzi@gte.com Think Globally. === +1(617)466-2881 === Act Locally! From perlman@oclc.org Mon Oct 21 16:29:03 1996 Return-Path: Received: from dev1.NISDEV (dev1.dev.oclc.org) by oclc.org (4.1/SMI-4.1) id AA21101; Mon, 21 Oct 96 16:29:03 EDT Received: by dev1.NISDEV (SMI-8.6/SMI-SVR4) id QAA06750; Mon, 21 Oct 1996 16:29:02 -0400 From: perlman@oclc.org (Gary PERLMAN) Message-Id: <199610212029.QAA06750@dev1.NISDEV> Subject: Re: CHI97 Review Paper 1109 Reviewer 293 To: rvirzi@gte.com (Robert A. Virzi) Date: Mon, 21 Oct 1996 16:29:02 -0400 (EDT) Cc: perlman@mailhost (Gary Perlman) In-Reply-To: from "Robert A. Virzi" at Oct 21, 96 03:55:02 pm X-Mailer: ELM [version 2.4 PL23beta3] Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Status: O > Gary- > In your original review you said the following about the paper on speech > cruising the WWW: > >{: > >Please feel free to ask me for more reasons to reject this. > >} > > > Well, you may be interested in knowing that this paper got three ratings of > 5 out of seven reviewers. It also got two 1s and a 2. The 5s are NOT > reversed scales. > > If you choose to revise your review, I would be happy to revise my > recommendation. > -Bob Getting a divergence of opinions is not surprising in empirical papers. In such cases, to me at least, if there are serious problems, then they override any good ones (which may not have detected the problems). You are considering recommending acceptance? I think would be a mistake, given my summary criticisms. The results are not novel. The inferences are flawed. If you think the results are novel, say, compared to Gould's work on speech input to word processors, then I would willing to change that part of my opinion. But I do not think that any criticisms will change my opinion that the usage during a very _brief_ exposure to any novel medium, here speech input, is an indication of "preference". The preference inference from usage in a novel situation reminds me of an experiment reported in CHI83: The researchers created a help facility and advertised it on their system -- and lots of people used it, from which they inferred that it was useful. Over a period of two weeks, the help facilty use dropped to nil, and the researchers suggested that the reason was that the help was so good, that no one needed help anymore. My opinion was that the help facility was useless and even annoying, although my own data went into the "evidence" for the help facility being useful. Overall, maybe it was, but usage over a short period (in the CHI83 paper, two weeks instead of less than an hour) is not a valid indicator of preference, IMHO. I would present my arguments to the positive ratings and see if they want to change their ratings. Gary From epfran1@PacBell.COM Thu Oct 24 19:16:10 1996 Return-Path: Received: from mailer.oclc.org (mailer.dev.oclc.org) by oclc.org (4.1/SMI-4.1) id AA13085; Thu, 24 Oct 96 19:16:10 EDT Received: from gw3.pacbell.com by mailer.oclc.org (5.0/SMI-4.1) id AA01185; Thu, 24 Oct 1996 19:16:07 +0500 Received: from srv.PacBell.COM (mother.srv.PacBell.COM) by gw3.pacbell.com (5.x/PacBell-10/18/95) id AA12233; Thu, 24 Oct 1996 16:16:03 -0700 Received: from zeus.srv.PacBell.COM by srv.PacBell.COM (4.1/Mother-7/26/95) id AA28713; Thu, 24 Oct 96 14:30:36 PDT Received: from [129.245.90.15] by zeus.srv.PacBell.COM (4.1/SMI-4.1) id AA01439; Thu, 24 Oct 96 14:30:34 PDT X-Sender: epfran1@zeus.srv.pacbell.com Message-Id: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Date: Thu, 24 Oct 1996 14:28:48 -0700 To: perlman@mailer.oclc.org, dray.chi@xerox.com, nardi@apple.com, smetros@utk.edu, isensee@austin.ibm.com, ellen.isaacs@eng.sun.com, pete@dcs.qmw.ac.uk, jspool@uie.com, karen@acm.org, sarahb@werple.mira.net.au, nass@leland.stanford.edu, poynton@poynton.com, gershon@mitre.org From: epfran1@PacBell.COM (Ellen Francik) Subject: CHI 97/Msg 1 of 4, Tutorials overview Content-Length: 3820 Status: RO OVERVIEW OF CHI 97 TUTORIAL PROGRAM This is the first of four messages e-mailed to you by Ellen Francik, CHI 97 tutorials co-chair. This message does not require a response. However, each of the other three messages ask that you respond to specific questions. Congratulations on the acceptance of your tutorial at CHI 97 in Atlanta! We expect a record-breaking attendance at the conference itself and record-breaking attendance at the pre-conference tutorials. We are excited about the excellent tutorial program planned for CHI 97, listed below. For your information, the following tutorials will be presented at CHI 97. THE TIMES AND DAYS OF THE TUTORIALS HAVE NOT BEEN DETERMINED YET. Author: Jakob Nielsen Title: User Interface Design for the WWW Author: Wayne Neale and Cindy McCombe Title: (working title) Web Design Options and Guidelines Author: Alison Lee and Andreas Girgensohn Title: Developing Collaborative Applications Using the World Wide Web Shell Author: Rob Jacobs and Keith Butler Title: Introduction to HCI Author: Thomas T. Hewett Title: Cognitive Factors in Design: Basic Phenomena in Human Memory and Problem Solving Author: Gary Perlman Title: Practical Usability Evaluation Author: Charles A. Poynton Title: Color and Type in Information Design Author: Aaron Marcus Title: Metaphor Design in User Interfaces: How to Effectively Mange Expectations, Surprise, Comprehension, and Delight Author: William Horton Title: Designing Icons and Visual Symbols Author: Susan E. Metros and John G. Hedberg Title: Visualizing Interfaces for Multimedia and the World Wide Web Author: Nahum Gershon, Stuart Card, and Stephen G. Eick Title: Information Visualization Author: Thom Gillespie Title: Digital Storytelling and Computer Game Design Author: Bruce Damer Title: Interacting and Designing in Virtual Worlds on the Internet (* This is a hands on PC course *) Author: Susan LuperFoy Title: Spoken Dialogue Interfaces Author: Clifford Nass and Byron Reeves Title: Social and Natural Interfaces: Theory and Design Author: Tandy Trower Title: Creating Conversational Interfaces Author: Mark Millier Title: Software Agents Author: Ian Smith Title: Java-Based UI Development and Web Application Deployment Author: Karen L. McGraw Title: Wizards, Coaches, Advisors, & More: A Performance Support Primer Author: Jared Spool, Carolyn Snyder, and Tara Scanlon Title: Product Usability: Survival Techniques Author: Debra Herschmann Title: Practical Interface Design: Developing Software within Real-World Constraints Author: Sarah Bloomer, Rachel Croft, and Helen Kieboom Title: Strategic Usability: Introducing Usability into Organisations Author: Deborah Mayhew Title: Managing the Design of the User Interface Author: Bonnie Nardi and Victor Kaptelinin Title: Activity Theory: Basic Concepts and Applications Author: Ellen Isaacs Title: Interviewing Customers : Discovering What They Can't Tell You Author: Karen Holtzblatt and Hugh Beyer Title: Getting Started on a Contextual Project Author: Tony Salvador and Michael Mateas Title: Introduction to Engineering Ethnography Author: Susan M. Dray Title: Structured Observation: Practical methods for understanding users and their work in context Author: Karen Holtzblatt and Hugh Beyer Title: Contextual Design: User Customer Work Models to Drive System Design Author: Peter Johnson, Hilary Johnson, Stephanie Wilson Title: Task Analysis and Task-Based User Interface Design Author: Scott Isensee, Dick Berry & Dave Roberts Title: Object View and Interaction Design Author: Betty Edwards Title: Drawing (working title only) Ellen Francik Pacific Bell CHI 97 Tutorials Co-Chair 2600 Camino Ramon, Rm. 3E050H chi97-tutorials@acm.org San Ramon, CA 94583 USA +1 510-867-8662 From epfran1@PacBell.COM Thu Oct 24 17:41:00 1996 Return-Path: Received: from mailer.oclc.org (mailer.dev.oclc.org) by oclc.org (4.1/SMI-4.1) id AA02982; Thu, 24 Oct 96 17:40:59 EDT Received: from gw3.pacbell.com by mailer.oclc.org (5.0/SMI-4.1) id AA29049; Thu, 24 Oct 1996 17:40:59 +0500 Received: from srv.PacBell.COM (mother.srv.PacBell.COM) by gw3.pacbell.com (5.x/PacBell-10/18/95) id AA15010; Thu, 24 Oct 1996 14:38:02 -0700 Received: from zeus.srv.PacBell.COM by srv.PacBell.COM (4.1/Mother-7/26/95) id AA28985; Thu, 24 Oct 96 14:37:55 PDT Received: from [129.245.90.15] by zeus.srv.PacBell.COM (4.1/SMI-4.1) id AA01938; Thu, 24 Oct 96 14:37:54 PDT X-Sender: epfran1@zeus.srv.pacbell.com Message-Id: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Date: Thu, 24 Oct 1996 14:36:08 -0700 To: perlman@mailer.oclc.org, dray.chi@xerox.com, nardi@apple.com, smetros@utk.edu, isensee@austin.ibm.com, ellen.isaacs@eng.sun.com, pete@dcs.qmw.ac.uk, jspool@uie.com, karen@acm.org, sarahb@werple.mira.net.au, nass@leland.stanford.edu, poynton@poynton.com, gershon@mitre.org From: epfran1@PacBell.COM (Ellen Francik) Subject: CHI 97/Msg 2 of 4, Submit Publicity Form Cc: jalarson@colby.edu Content-Length: 1977 Status: RO SUBMIT PUBLICITY FORM FOR YOUR CHI-97 TUTORIAL This is the second of four messages e-mailed to you by Ellen Francik. The CHI-97 author kits did not include the TUTORIAL PUBLICITY FORM included below. Based on feedback from the reviewers, you may want to change the publicity information submitted with your original proposal. This form also asks for information that many of you did not include in your original proposal, and asks for additional information that will attract attendees. This information is needed to generate publicity information about your tutorial for the advance program, the final program, and other flyers and brochures. Please complete and return this electronic form to me by Tuesday, October 29, so that we may put this information into the Advance Program. Thank you for your rapid response, Ellen chi97-tutorials@acm.org TUTORIAL PUBLICITY FORM (Do not use more than 250 words when filling out this form) 1. Tutorial Title: 2. Authors and affiliations: 3. Length (half day or full day): 4. Benefits of taking this tutorial (e.g., You will obtain a solid foundation for conducting field research with customers and incorporating your findings into product development. You will learn about Contextual Inquiry methods and when and how to apply them.) 5. Origins of this tutorial (e.g., This is an updated version of a highly rated tutorial from CHI'96.) 6. What you will learn in this tutorial: 7. Tutorial level (e.g., introductory, intermediate, advanced): 8. Expected background of attendee: 8. Presentation Style (e.g. lecture, hands-on, small group exercises, etc.): 9. Why instructor is qualified to teach this tutorial (instructor's background): 10: Other information that will encourage attendance at this tutorial: Ellen Francik Pacific Bell CHI 97 Tutorials Co-Chair 2600 Camino Ramon, Rm. 3E050H chi97-tutorials@acm.org San Ramon, CA 94583 USA +1 510-867-8662 From epfran1@PacBell.COM Thu Oct 24 17:42:13 1996 Return-Path: Received: from mailer.oclc.org (mailer.dev.oclc.org) by oclc.org (4.1/SMI-4.1) id AA03030; Thu, 24 Oct 96 17:42:13 EDT Received: from gw3.pacbell.com by mailer.oclc.org (5.0/SMI-4.1) id AA29055; Thu, 24 Oct 1996 17:42:13 +0500 Received: from srv.PacBell.COM (mother.srv.PacBell.COM) by gw3.pacbell.com (5.x/PacBell-10/18/95) id AA15381; Thu, 24 Oct 1996 14:39:16 -0700 Received: from zeus.srv.PacBell.COM by srv.PacBell.COM (4.1/Mother-7/26/95) id AA29026; Thu, 24 Oct 96 14:39:07 PDT Received: from [129.245.90.15] by zeus.srv.PacBell.COM (4.1/SMI-4.1) id AA02032; Thu, 24 Oct 96 14:39:05 PDT X-Sender: epfran1@zeus.srv.pacbell.com Message-Id: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" X-Priority: 1 (Highest) Date: Thu, 24 Oct 1996 14:37:19 -0700 To: perlman@mailer.oclc.org, dray.chi@xerox.com, nardi@apple.com, smetros@utk.edu, isensee@austin.ibm.com, ellen.isaacs@eng.sun.com, pete@dcs.qmw.ac.uk, jspool@uie.com, karen@acm.org, sarahb@werple.mira.net.au, nass@leland.stanford.edu, poynton@poynton.com, gershon@mitre.org From: epfran1@PacBell.COM (Ellen Francik) Subject: CHI 97/Msg 3 of 4, Author Kit received? Cc: jalarson@colby.edu Content-Length: 249 Status: RO HAVE YOU RECEIVED YOUR CHI-97 AUTHOR KIT? This is the second of four messages e-mailed to you by Ellen Francik. Please reply to this e-mail message immediately and tell me if you have received the CHI-97 tutorial instructor author kit. Thanks, From epfran1@PacBell.COM Thu Oct 24 19:16:11 1996 Return-Path: Received: from mailer.oclc.org (mailer.dev.oclc.org) by oclc.org (4.1/SMI-4.1) id AA13089; Thu, 24 Oct 96 19:16:11 EDT Received: from gw3.pacbell.com by mailer.oclc.org (5.0/SMI-4.1) id AA01184; Thu, 24 Oct 1996 19:16:05 +0500 Received: from srv.PacBell.COM (mother.srv.PacBell.COM) by gw3.pacbell.com (5.x/PacBell-10/18/95) id AA12157; Thu, 24 Oct 1996 16:15:49 -0700 Received: from zeus.srv.PacBell.COM by srv.PacBell.COM (4.1/Mother-7/26/95) id AA29175; Thu, 24 Oct 96 14:42:48 PDT Received: from [129.245.90.15] by zeus.srv.PacBell.COM (4.1/SMI-4.1) id AA02329; Thu, 24 Oct 96 14:42:46 PDT X-Sender: epfran1@zeus.srv.pacbell.com Message-Id: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" X-Priority: 1 (Highest) Date: Thu, 24 Oct 1996 14:41:00 -0700 To: perlman@mailer.oclc.org, dray.chi@xerox.com, nardi@apple.com, smetros@utk.edu, isensee@austin.ibm.com, ellen.isaacs@eng.sun.com, pete@dcs.qmw.ac.uk, jspool@uie.com, karen@acm.org, sarahb@werple.mira.net.au, nass@leland.stanford.edu, poynton@poynton.com, gershon@mitre.org From: epfran1@PacBell.COM (Ellen Francik) Subject: CHI 97/Msg 4 of 4, Send AV Information Cc: jalarson@colby.edu Content-Length: 1025 Status: RO SUBMIT A-V AND COMPUTER REQUEST FORM FOR YOUR CHI-97 TUTORIAL This is the fourth of four messages e-mailed to you by Ellen Francik. By carefully scheduling the day and room of your tutorial, we can avoid unnecessary and expensive equipment set-up and tear-down operations. While we have your initial A-V and computer requirement estimates from your proposals, you may want to change them based on feedback from the reviewers. We need your final A-V and computer requirements as soon as possible so that the day and room of your tutorial can be scheduled. Please select the CHI 97 AUDIO-VISUAL AND COMPUTER REQUEST FORM FOR TUTORIALS from your CHI 97 tutorial author kit. Mail the completed form to me by Tuesday, October 29: Ellen Francik CHI 97 Tutorials Co-Chair 471 Viking Drive Pleasant Hill, CA 94523 USA If you do not live in the United States, please send the form express mail. I will collect the forms and pass them on to Rodney Fuller, the CHI 97 Technical Support Chair. Ellen chi97-tutorials@acm.org From jalarson@COLBY.EDU Tue Nov 5 09:28:28 1996 Return-Path: Received: from mailer.oclc.org (mailer.dev.oclc.org) by oclc.org (4.1/SMI-4.1) id AA22563; Tue, 5 Nov 96 09:28:27 EST Received: from host-04.colby.edu by mailer.oclc.org (5.0/SMI-4.1) id AA01720; Tue, 5 Nov 1996 09:28:26 +0500 Received: from [137.146.66.54] (dip-07.mudd.colby.edu) by host-04.colby.edu with SMTP (1.40.112.8/Colby 1.1) id AA027594102; Tue, 5 Nov 1996 09:28:22 -0500 Date: Tue, 5 Nov 1996 09:28:22 -0500 Message-Id: Mime-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable To: perlman@mailer.oclc.org From: jalarson@COLBY.EDU (Jim Larson) Subject: Review tutorial description a.s.a.p. Content-Length: 9701 Status: RO Dear tutorial author, The Advanced Program is about to go to press, and I haven't received your feedback on your tutorial description. Please review the following general introduction to CHI 97 tutorials, followed by a careful review of the wording for your tutorial. Please forward any changes to chi97-tutorials@acm.chi today so that your changes can be incorporated into the advanced program. Thanks in advance for your prompt reply. -Jim Larson (Ellen is out for the rest of the week so I am receiving feedback from all tutorial authors.) ------------------------------------------ Tutorials are full-day (9:00 AM to 5:30 PM) and half-day (9:00 AM to 12:30 PM and 2:00 PM to 5:30 PM) courses that enable you to learn new skills and knowledge through extended interaction with expert instructors. The 32 tutorials available at CHI 97 represent the leading edge of current practice and research in human computer interaction. Not only do the tutorials provide you with a solid grounding in where the field of HCI is today, but they also point the way to where our field will be in the future. The tutorials are offered on Saturday evening, Sunday and Monday. Saturday tutorials Chances are good that you will find yourself arriving in Atlanta on Saturday, since many favorable airfares require a Saturday night stay. Spend Saturday evening with Jakob Nielsen (2 User Interface Design for the World Wide Web) or with Keith Butler and Rob Jacob (1 Human-Computer Interaction: Introduction and Overview). Both tutorials start at 6:00 PM and end at 9:30 PM. There will be a food and beverage break for the benefit of attendees who have traveled from distant time zones. Or if you arrive Monday, join Stuart Card, Stephen Eick and Nahum Gershon for an exciting look at how to visualize information by attending their Monday afternoon tutorial (30 Information Visualization). Earn CEUs for CHI tutorials This year again you can earn Continuing Education Units (CEUs) for taking CHI tutorials. SIGCHI has been approved as an Authorized CEU Sponsor by the International Association for Continuing Education and Training (IACET). SIGCHI's approval to award CEUs is a recognition of the outstanding quality of the CHI tutorial program. The CEU is a recognized unit of measure for continuing education and professional training programs. Each CEU requires the successful completion of 10 hours of instruction. Successful completion of a CHI tutorial means that you attend the entire session and participate fully in the learning activities of the tutorial. If you choose to earn CEUs for your CHI tutorials, you will earn 0.6 CEUs for each full-day tutorial that you complete (6 hours of instruction), or 0.3 CEUs for each half-day (3 hours). Because you can take a total of 2 1/2 days of instruction (15 hours), you can earn as much as 1.5 CEUs during the tutorial program. IACET will send you a confirmation letter, maintain a record of your CEUs, and issue transcripts upon your request. To obtain CEU credit, you must register for CEUs at the same time that you register for your tutorials. There is a small administrative charge. If you work for a manager who does not already know that CHI tutorials offer outstanding professional training, be sure to tell him or her about the Continuing Education Units. Many tutorials are new for CHI 97 CHI 97 offers 20 tutorials that are "new for 97." These brand new tutorials on exciting topics have never been offered at the CHI conference before. In addition, CHI 97 offers 12 "classic" tutorials, updated versions of highly-rated tutorials from previous CHI conferences. If you are new to the CHI conference and are just getting started in the field of HCI, the Newcomers tutorials will give you a solid background and prepare you to get the most out of the rest of the conference. Good tutorial choices for newcomers include: 1 Human-Computer Interaction: Introduction and Overview (Saturday evening) 3 Cognitive Factors in Design: Basic Phenomena in Human Memory and Problem Solving (Sunday) 16 Practical Usability Evaluation (Monday) Want to know how to design and implement user interfaces for the World Wide Web? These tutorials will get you started: 2 User Interface Design for the World Wide Web (Saturday evening) 4 Developing Collaborative Applications Using the World Wide Web "Shell" (Sunday) 17 Designing Usable and Visually Appealing Web Sites (Monday) =46our great tutorials showing you how to design great graphics and visuals = are: 5 Color and Type in Information Design (Sunday) 6 Designing Icons and Visual Symbols (Sunday) 18 Metaphor Design in User Interfaces: How to Effectively Manage Expectation, Surprise, Comprehension, and Delight (Monday) 27 Multimedia Visual Interface Design (Monday) Also, see how graphics are used for powerful information display in: 30 Information Visualization (Monday afternoon). User interfaces for emerging applications are presented in two forward-looking tutorials: 7 MediaJazz: Digital Storytelling and Computer Game Design (Sunday) 19 Interacting and Designing in Virtual Worlds on the Internet (Monday) Interacting with computers should be as easy as talking or interacting with other people. Three tutorials that lead the way in interacting with computers in a more natural way are: 8 Spoken Dialogue Interfaces (Sunday) 28 Social and Natural Interfaces: Theory and Design (Monday morning) 31 Creating Conversational Interfaces for Interactive Software Agents (Monday afternoon) Software agents is an emerging technology. Learn what agents can do, how to interact with agents, and how to implement agents using Java with these tutorials: 9 Wizards, Coaches, Advisors, and More: A Performance Support Primer (Sunday= ) 29 Software Agents (Monday morning) 31 Creating Conversational Interfaces for Interactive Software Agents (Monday afternoon) 32 Java-Based User Interface Development and Web Application Deployment (Monday afternoon) Two classic tutorials on usability engineering are: 10 Product Usability: Survival Techniques (Sunday) 20 Practical Interface Design: Developing Software Within Real-World Constraints (Monday) Having trouble managing the user interface process or convincing your managers that user interface design and evaluation is necessary? Then enroll in: 11 Strategic Usability: Introducing Usability into Organizations (Sunday) 21 Managing the Design of the User Interface (Monday) Understanding users is fundamental to designing great user interfaces. We offer six tutorials on gathering requirements and feedback from users: 12 Activity Theory: Basic Concepts (Sunday) 14 Getting Started on a Contextual Project (Sunday morning) 15 Design Ethnography: Using Custom Ethnographic Techniques To Develop New Product Concepts (Sunday afternoon) 22 Interviewing Customers: Discovering What They Can't Tell You (Monday) 23 Structured Observation: Practical Methods for Understanding Users and Their Work in Context (Monday) 24 Contextual Design: Using Customer Work Models to Drive Systems Design (Monday) Want to know about user interface design? Try these tutorials: 13 Designing User Interfaces from Analyses of Users' Work Tasks (Sunday) 25 Object, View, and Interaction Design (Monday) The speaker at last year's closing session, Betty Edwards, will be back to present a tutorial on drawing and how artistic training can improve user interfaces. This tutorial has limited enrollment, so sign up early: 26 Drawing on the Right Side of the Brain (Monday) Whether you are new to the CHI conference or have been attending for years, there are courses designed for you in the CHI 97 tutorials program. ------------------ 16 Practical Usability Evaluation Gary Perlman Ohio State University Monday, March 24, introductory full-day tutorial Benefits You will learn how to use cost-effective methods for evaluating interactive systems and gain enough experience to apply the methods on your own. Origins This tutorial is an update of a highly-rated CHI 96 tutorial. =46eatures =85 goals for interactive system evaluation =85 cost-benefit analysis of usability evaluation =85 choosing the right evaluation method =85 usability inspection methods (e.g., heuristic evaluation) =85 observational skills and video =85 data collection by software logging =85 questionnaires for structured feedback =85 a step-by-step guide to usability testing =85 sources of information about usability evaluation, including a usability bibliography =85 several sample data collection forms are provided Audience Managers interested in increasing usability testing in their organizations. Software engineers interested in practical methods for usability evaluation during - not after - development. Human factors specialists who want to learn more about usability evaluation methods. No background in usability evaluation is assumed. Presentation Lecture, interspersed with exercises and discussion. Instructors Gary Perlman is a professor in the Department of Computer and Information Science at Ohio State University. He is the author of statistical and hypertext software used widely for user interface evaluation. Gary has consulted extensively for major information technology companies on user interface development process. He is also the creator of the "HCI Bibliography" project, the largest free-access bibliography on human-computer interaction. Related Tutorials 1 Human-Computer Interaction: Introduction and Overview (Saturday evening) 3: Cognitive Factors in Design: Basic Phenomena in Human Memory and Problem Solving (Sunday) 11 Strategic Usability: Introducing Usability into Organizations (Sunday) From perlman@oclc.org Tue Nov 5 09:49:36 1996 Return-Path: Received: from dev1.NISDEV (dev1.dev.oclc.org) by oclc.org (4.1/SMI-4.1) id AA24628; Tue, 5 Nov 96 09:49:35 EST Received: by dev1.NISDEV (SMI-8.6/SMI-SVR4) id JAA26471; Tue, 5 Nov 1996 09:49:35 -0500 From: perlman@oclc.org (Gary PERLMAN) Message-Id: <199611051449.JAA26471@dev1.NISDEV> Subject: Re: Review tutorial description a.s.a.p. To: jalarson@COLBY.EDU (Jim Larson) Date: Tue, 5 Nov 1996 09:49:35 -0500 (EST) Cc: perlman@mailhost (Gary Perlman) In-Reply-To: from "Jim Larson" at Nov 5, 96 09:28:22 am X-Mailer: ELM [version 2.4 PL23beta3] Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Status: O I am sorry for not responding, but I am out sick. I did glance at the description and except where it conflicts with the description I sent as part of the proposal, it seems fine. One place where it is out of date is my affiliation, which has changed since last year (this was part of my submission). So, please change the affiliation: Gary Perlman OCLC, Online Computer Library Center and please change the description of the instructor: Gary Perlman is a consulting research scientist at OCLC, the Online Computer Library Center. > Dear tutorial author, > > The Advanced Program is about to go to press, and I haven't received your > feedback on your tutorial description. Please review the following general > introduction to CHI 97 tutorials, followed by a careful review of the > wording for your tutorial. Please forward any changes to > > chi97-tutorials@acm.chi > > today so that your changes can be incorporated into the advanced program. > > Thanks in advance for your prompt reply. > > -Jim Larson > (Ellen is out for the rest of the week so I am receiving feedback from all > tutorial authors.) > ------------------------------------------ > > Tutorials are full-day (9:00 AM to 5:30 PM) and half-day (9:00 AM to 12:30 > PM and 2:00 PM to 5:30 PM) courses that enable you to learn new skills and > knowledge through extended interaction with expert instructors. The 32 > tutorials available at CHI 97 represent the leading edge of current > practice and research in human computer interaction. Not only do the > tutorials provide you with a solid grounding in where the field of HCI is > today, but they also point the way to where our field will be in the > future. > > The tutorials are offered on Saturday evening, Sunday and Monday. > > Saturday tutorials > > Chances are good that you will find yourself arriving in Atlanta on > Saturday, since many favorable airfares require a Saturday night stay. > > Spend Saturday evening with Jakob Nielsen (2 User Interface Design for the > World Wide Web) or with Keith Butler and Rob Jacob (1 Human-Computer > Interaction: Introduction and Overview). > > Both tutorials start at 6:00 PM and end at 9:30 PM. There will be a food > and beverage break for the benefit of attendees who have traveled from > distant time zones. > > Or if you arrive Monday, join Stuart Card, Stephen Eick and Nahum Gershon > for an exciting look at how to visualize information by attending their > Monday afternoon tutorial (30 Information Visualization). > > Earn CEUs for CHI tutorials > > This year again you can earn Continuing Education Units (CEUs) for taking > CHI tutorials. > > SIGCHI has been approved as an Authorized CEU Sponsor by the International > Association for Continuing Education and Training (IACET). SIGCHI's > approval to award CEUs is a recognition of the outstanding quality of the > CHI tutorial program. > > The CEU is a recognized unit of measure for continuing education and > professional training programs. Each CEU requires the successful completion > of 10 hours of instruction. Successful completion of a CHI tutorial means > that you attend the entire session and participate fully in the learning > activities of the tutorial. > > If you choose to earn CEUs for your CHI tutorials, you will earn 0.6 CEUs > for each full-day tutorial that you complete (6 hours of instruction), or > 0.3 CEUs for each half-day (3 hours). Because you can take a total of 2 > 1/2 days of instruction (15 hours), you can earn as much as 1.5 CEUs during > the tutorial program. IACET will send you a confirmation letter, maintain a > record of your CEUs, and issue transcripts upon your request. > > To obtain CEU credit, you must register for CEUs at the same time that you > register for your tutorials. There is a small administrative charge. > > If you work for a manager who does not already know that CHI tutorials > offer outstanding professional training, be sure to tell him or her about > the Continuing Education Units. > > Many tutorials are new for CHI 97 > > CHI 97 offers 20 tutorials that are "new for 97." These brand new tutorials > on exciting topics have never been offered at the CHI conference before. In > addition, CHI 97 offers 12 "classic" tutorials, updated versions of > highly-rated tutorials from previous CHI conferences. > > If you are new to the CHI conference and are just getting started in the > field of HCI, the Newcomers tutorials will give you a solid background and > prepare you to get the most out of the rest of the conference. Good > tutorial choices for newcomers include: > > 1 Human-Computer Interaction: Introduction and Overview (Saturday evening) > 3 Cognitive Factors in Design: Basic Phenomena in Human Memory and Problem > Solving (Sunday) > 16 Practical Usability Evaluation (Monday) > > Want to know how to design and implement user interfaces for the World Wide > Web? These tutorials will get you started: > > 2 User Interface Design for the World Wide Web (Saturday evening) > 4 Developing Collaborative Applications Using the World Wide Web "Shell" > (Sunday) > 17 Designing Usable and Visually Appealing Web Sites (Monday) > > =46our great tutorials showing you how to design great graphics and visuals = > are: > > 5 Color and Type in Information Design (Sunday) > 6 Designing Icons and Visual Symbols (Sunday) > 18 Metaphor Design in User Interfaces: How to Effectively Manage > Expectation, Surprise, Comprehension, and Delight (Monday) > 27 Multimedia Visual Interface Design (Monday) > > Also, see how graphics are used for powerful information display in: > > 30 Information Visualization (Monday afternoon). > > User interfaces for emerging applications are presented in two > forward-looking tutorials: > > 7 MediaJazz: Digital Storytelling and Computer Game Design (Sunday) > 19 Interacting and Designing in Virtual Worlds on the Internet (Monday) > > Interacting with computers should be as easy as talking or interacting with > other people. Three tutorials that lead the way in interacting with > computers in a more natural way are: > > 8 Spoken Dialogue Interfaces (Sunday) > 28 Social and Natural Interfaces: Theory and Design (Monday morning) > 31 Creating Conversational Interfaces for Interactive Software Agents > (Monday afternoon) > > Software agents is an emerging technology. Learn what agents can do, how to > interact with agents, and how to implement agents using Java with these > tutorials: > > 9 Wizards, Coaches, Advisors, and More: A Performance Support Primer (Sunday= > ) > 29 Software Agents (Monday morning) > 31 Creating Conversational Interfaces for Interactive Software Agents > (Monday afternoon) > 32 Java-Based User Interface Development and Web Application Deployment > (Monday afternoon) > > Two classic tutorials on usability engineering are: > > 10 Product Usability: Survival Techniques (Sunday) > 20 Practical Interface Design: Developing Software Within Real-World > Constraints (Monday) > > Having trouble managing the user interface process or convincing your > managers that user interface design and evaluation is necessary? Then > enroll in: > > 11 Strategic Usability: Introducing Usability into Organizations (Sunday) > 21 Managing the Design of the User Interface (Monday) > > Understanding users is fundamental to designing great user interfaces. We > offer six tutorials on gathering requirements and feedback from users: > > 12 Activity Theory: Basic Concepts (Sunday) > 14 Getting Started on a Contextual Project (Sunday morning) > 15 Design Ethnography: Using Custom Ethnographic Techniques To Develop New > Product Concepts (Sunday afternoon) > 22 Interviewing Customers: Discovering What They Can't Tell You (Monday) > 23 Structured Observation: Practical Methods for Understanding Users and > Their Work in Context (Monday) > 24 Contextual Design: Using Customer Work Models to Drive Systems Design > (Monday) > > Want to know about user interface design? Try these tutorials: > > 13 Designing User Interfaces from Analyses of Users' Work Tasks (Sunday) > 25 Object, View, and Interaction Design (Monday) > > The speaker at last year's closing session, Betty Edwards, will be back to > present a tutorial on drawing and how artistic training can improve user > interfaces. This tutorial has limited enrollment, so sign up early: > > 26 Drawing on the Right Side of the Brain (Monday) > > Whether you are new to the CHI conference or have been attending for years, > there are courses designed for you in the CHI 97 tutorials program. > > > ------------------ > > 16 Practical Usability Evaluation > > Gary Perlman > Ohio State University > > Monday, March 24, introductory full-day tutorial > > Benefits > > You will learn how to use cost-effective methods for evaluating interactive > systems and gain enough experience to apply the methods on your own. > > Origins > > This tutorial is an update of a highly-rated CHI 96 tutorial. > > =46eatures > > =85 goals for interactive system evaluation > =85 cost-benefit analysis of usability evaluation > =85 choosing the right evaluation method > =85 usability inspection methods (e.g., heuristic evaluation) > =85 observational skills and video > =85 data collection by software logging > =85 questionnaires for structured feedback > =85 a step-by-step guide to usability testing > =85 sources of information about usability evaluation, including a usability > bibliography > =85 several sample data collection forms are provided > > Audience > > Managers interested in increasing usability testing in their organizations. > Software engineers interested in practical methods for usability evaluation > during - not after - development. Human factors specialists who want to > learn more about usability evaluation methods. No background in usability > evaluation is assumed. > > Presentation > > Lecture, interspersed with exercises and discussion. > > Instructors > > Gary Perlman is a professor in the Department of Computer and Information > Science at Ohio State University. He is the author of statistical and > hypertext software used widely for user interface evaluation. Gary has > consulted extensively for major information technology companies on user > interface development process. He is also the creator of the "HCI > Bibliography" project, the largest free-access bibliography on > human-computer interaction. > > Related Tutorials > > 1 Human-Computer Interaction: Introduction and Overview (Saturday evening) > 3: Cognitive Factors in Design: Basic Phenomena in Human Memory and Problem > Solving (Sunday) > 11 Strategic Usability: Introducing Usability into Organizations (Sunday) > > > > From jalarson@COLBY.EDU Tue Nov 5 10:01:35 1996 Return-Path: Received: from mailer.oclc.org (mailer.dev.oclc.org) by oclc.org (4.1/SMI-4.1) id AA25913; Tue, 5 Nov 96 10:01:34 EST Received: from host-04.colby.edu by mailer.oclc.org (5.0/SMI-4.1) id AA02856; Tue, 5 Nov 1996 10:01:31 +0500 Received: from [137.146.66.54] (dip-07.mudd.colby.edu) by host-04.colby.edu with SMTP (1.40.112.8/Colby 1.1) id AA049386054; Tue, 5 Nov 1996 10:00:54 -0500 Date: Tue, 5 Nov 1996 10:00:54 -0500 Message-Id: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" To: hewett@duvm.ocs.drexel.edu, alee@nynexst.com, luperfoy@mitre.org, nardi@apple.com, perlman@mailer.oclc.org, neale@kodak.com, karen@acm.org, aaron@amanda.com, bdamer@ccon.org, dray.chi@xerox.com, isensee@austin.ibm.com, nass@leland.stanford.edu, mmillier@ibeam.intel.com, gershon@mitre.org, tandyt@microsoft.com, debra_herschmann@uscmail.lehman.com From: jalarson@COLBY.EDU (Jim Larson) Subject: Plz return AV/computer requirements for CHI a.s.a.p. Content-Length: 4105 Status: RO Dear tutorial author, We have not yet received your AV/Computer Request form for tutorials. We need this form to schedule your tutorial into a room which can support your AV/Computer needs. Please take a minute and fill in the following electronic form and forward it to chi97-tutorials@acm.org -Jim Larson (Ellen is out for the rest of the week, so I'll be receiving and forwarding these forms.) -------------------------- CHI 97 Audio-Visual and Computer Request Form for Tutorials PAGE 1 Presenter: Address: Telephone: Alternate Phone: Fax: Email: Presentation Title: [table w/3 mics] [podium w/1 mic] [extra table] [screen 1] [screen 2] I wish to use Screen 1 for: __ I do not wish to use Screen 1 __ 35mm Slide Projector __ Overhead Projector I wish to use Screen 2 for: __ I do not wish to use Screen 2 __ 35mm Slide Projector __ Video player: __ 1/2" VHS __ 3/4" UMatic Format: __ NTSC __ PAL __ SECAM __ Mac with __ CD-ROM drive __ PC compatible with __ CD-ROM drive __ My own (specify details on page 2) I wish to use the following Audio Sources: __ NONE __ Cassette Player __ Compact Disk __ Sound from Video Player __ Sound from Computer I have the following Special Equipment Requests: PAGE 2 Any query about this form, contact Rodney Fuller, CHI 97 Technology Support Chair. What you will bring. ------------------- Please describe, in as much detail as possible, any computer hardware, software, cables, or other electrical equipment that you intend to bring with you. If you are bringing a computer and expect the display to be shown to the audience, please give details of its display interface. If your software requires any specific operating system version, please give details. If you plan on bringing your laptop/digital device you MUST tell us what model it is and how you intend to mirror video out (e.g., which port). Any intended use of CDI or DVD requires a 3 month advance notification. You are responsible for knowing how to mirror video out of your laptop--and you should plan on showing up no more than 3, but no less than 1, hour before your session to test projection. No more than 5 laptops will be connected to a video projector in any one session. At this time, projection from Sun Voyagers will not be supported because of non-standard scan rates. Special Requirements. -------------------- Please describe any special audio-visual or staging needs you may have (e.g., power requirements, wheelchair access to stage, unusual lighting, synchronized use of equipment). If you have asked for a computer or video player you can expect video projection of 256 colors; video player and slide projector controls next to speaker podium; a computer operating system with NO special system extension; a copy of Powerpoint Presenter (only the most recent version); internet access; a frames capable web browser (the most recent stable non-beta version) but NO plug-ins. A laser pointer and sound amplification will be provided. Please specify below if you prefer to use a podium or wireless microphone. If you require internet access please specify that below. If you would like computers on your students' desks please specify that below, as well as how your students will interact with them. Because of the cost, it is unlikely that student computers will be provided unless advance registration justifies expanding the tutorial AV budget. Last Chance! ------------ Is there anything else we need to know to help your presentation run smoothly? From perlman@oclc.org Tue Nov 5 10:22:01 1996 Return-Path: Received: from dev1.NISDEV (dev1.dev.oclc.org) by oclc.org (4.1/SMI-4.1) id AA27875; Tue, 5 Nov 96 10:22:01 EST Received: by dev1.NISDEV (SMI-8.6/SMI-SVR4) id KAA00137; Tue, 5 Nov 1996 10:22:00 -0500 From: perlman@oclc.org (Gary PERLMAN) Message-Id: <199611051522.KAA00137@dev1.NISDEV> Subject: Plz return AV/computer requirements for CHI a.s.a.p. (fwd) To: chi97-tutorials@acm.org Date: Tue, 5 Nov 1996 10:22:00 -0500 (EST) Cc: perlman@mailhost (Gary Perlman) X-Mailer: ELM [version 2.4 PL23beta3] Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Status: O Sorry, I thought I sent this in already. Gary > CHI 97 Audio-Visual and Computer Request Form for Tutorials > > PAGE 1 > > Presenter: Gary Perlman > > Address: OCLC, Online Computer Library Center 6565 Frantz Road Dublin, Ohio 43017 USA > > > > > Telephone: Voice: +1-614-761-5058 > > Alternate Phone: > > Fax: Fax: +1-614-793-0915 > > Email: Email: perlman@oclc.org WWW: www.acm.org/~perlman > > Presentation Title: Practical Usability Evaluation > > audience's left to right, the picture has a place for one or more > presenters to sit, a podium for the current presenter to stand, a small > table for extra material for the presenter. All of these are on a dais. Off > the dais further to the right are two projection screens, 1 and 2:> > > [table w/3 mics] [podium w/1 mic] [extra table] [screen 1] [screen 2] > > I wish to use Screen 1 for: > __ I do not wish to use Screen 1 > __ 35mm Slide Projector > _X Overhead Projector > > I wish to use Screen 2 for: > __ I do not wish to use Screen 2 > __ 35mm Slide Projector > _X Video player: > _X 1/2" VHS __ 3/4" UMatic > Format: > _X NTSC __ PAL __ SECAM > __ Mac with __ CD-ROM drive > __ PC compatible with __ CD-ROM drive > __ My own (specify details on page 2) > > I wish to use the following Audio Sources: > __ NONE > __ Cassette Player > __ Compact Disk > _X Sound from Video Player > __ Sound from Computer > > I have the following Special Equipment Requests: no other requirements From jalarson@COLBY.EDU Fri Nov 1 17:57:15 1996 Return-Path: Received: from mailer.oclc.org (mailer.dev.oclc.org) by oclc.org (4.1/SMI-4.1) id AA26533; Fri, 1 Nov 96 17:57:13 EST Received: from host-04.colby.edu by mailer.oclc.org (5.0/SMI-4.1) id AA01179; Fri, 1 Nov 1996 17:57:07 +0500 Received: from [137.146.66.54] (dip-07.mudd.colby.edu) by host-04.colby.edu with SMTP (1.40.112.8/Colby 1.1) id AA048798566; Fri, 1 Nov 1996 17:49:26 -0500 Date: Fri, 1 Nov 1996 17:49:26 -0500 Message-Id: Mime-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable To: alee@nynexst.com, poynton@inforamp.net, luperfoy@mitre.org, thom@copper.ucs.indiana.edu, tony_salvador@ccm.jf.intel.com, tandyt@microsoft.com, gershon@mitre.org, mmillier@ibeam.intel.com, isensee@austin.ibm.com, aaron@amanda.com, neale@kodak.com, jacob@cs.tufts.edu, kmcgraw@clark.net, iansmith@cc.gatech.edu, nass@leland.stanford.edu, smetros@utk.edu, dray.chi@xerox.com, drdeb@vineyard.net, bdamer@ccon.org, perlman@mailer.oclc.org, pete@cdx.qmw.ac.uk, nardi@apple.com, sarahb@werple.mira.net.au, jspool@uie.com, william@horton.com, hewett@duvm.ocs.drexel.edu, jakob@eng.sun.com, karen@acm.org, debra_herschmann@uscmail.lehman.com, rbower2154@aol.com, ellen@communities.com From: jalarson@COLBY.EDU (Jim Larson) Subject: CHI 97 tutorial intro + tutorials 1-9 Cc: chi97-tutorials@acm.org, jalarson@host-04.COLBY.EDU Content-Length: 23711 Status: RO Dear Author, Please review the tutorial intro to see how your tutorial fits into the overall tutorial program. Then carefully review the material for your tutorial. Please send any changes to me by Monday evening, November 4 so I can incorporate changes into the final copy. Thanks (especially for the short turnaround) Jim Larson chi97-tutorials@acm.org ----------------------------------------------- TUTORIALS ADVANCE PROGRAM DRAFT 6c, November 1, 1996 --------------------------------------- TUTORIALS Tutorials are full-day (9:00 AM to 5:30 PM) and half-day (9:00 AM to 12:30 PM and 2:00 PM to 5:30 PM) courses that enable you to learn new skills and knowledge through extended interaction with expert instructors. The 32 tutorials available at CHI 97 represent the leading edge of current practice and research in human computer interaction. Not only do the tutorials provide you with a solid grounding in where the field of HCI is today, but they also point the way to where our field will be in the future. The tutorials are offered on Saturday evening, Sunday and Monday. Saturday tutorials Chances are good that you will find yourself arriving in Atlanta on Saturday, since many favorable airfares require a Saturday night stay. Spend Saturday evening with Jakob Nielsen (2 User Interface Design for the World Wide Web) or with Keith Butler and Rob Jacob (1 Human-Computer Interaction: Introduction and Overview). Both tutorials start at 6:00 PM and end at 9:30 PM. There will be a food and beverage break for the benefit of attendees who have traveled from distant time zones. Or if you arrive Monday, join Stuart Card, Stephen Eick and Nahum Gershon for an exciting look at how to visualize information by attending their Monday afternoon tutorial (30 Information Visualization). Earn CEUs for CHI tutorials This year again you can earn Continuing Education Units (CEUs) for taking CHI tutorials. SIGCHI has been approved as an Authorized CEU Sponsor by the International Association for Continuing Education and Training (IACET). SIGCHI's approval to award CEUs is a recognition of the outstanding quality of the CHI tutorial program. The CEU is a recognized unit of measure for continuing education and professional training programs. Each CEU requires the successful completion of 10 hours of instruction. Successful completion of a CHI tutorial means that you attend the entire session and participate fully in the learning activities of the tutorial. If you choose to earn CEUs for your CHI tutorials, you will earn 0.6 CEUs for each full-day tutorial that you complete (6 hours of instruction), or 0.3 CEUs for each half-day (3 hours). Because you can take a total of 2 1/2 days of instruction (15 hours), you can earn as much as 1.5 CEUs during the tutorial program. IACET will send you a confirmation letter, maintain a record of your CEUs, and issue transcripts upon your request. To obtain CEU credit, you must register for CEUs at the same time that you register for your tutorials. There is a small administrative charge. If you work for a manager who does not already know that CHI tutorials offer outstanding professional training, be sure to tell him or her about the Continuing Education Units. Many tutorials are new for CHI 97 CHI 97 offers 20 tutorials that are "new for 97." These brand new tutorials on exciting topics have never been offered at the CHI conference before. In addition, CHI 97 offers 12 "classic" tutorials, updated versions of highly-rated tutorials from previous CHI conferences. If you are new to the CHI conference and are just getting started in the field of HCI, the Newcomers tutorials will give you a solid background and prepare you to get the most out of the rest of the conference. Good tutorial choices for newcomers include: 1 Human-Computer Interaction: Introduction and Overview (Saturday evening) 3 Cognitive Factors in Design: Basic Phenomena in Human Memory and Problem Solving (Sunday) 16 Practical Usability Evaluation (Monday) Want to know how to design and implement user interfaces for the World Wide Web? These tutorials will get you started: 2 User Interface Design for the World Wide Web (Saturday evening) 4 Developing Collaborative Applications Using the World Wide Web "Shell" (Sunday) 17 Designing Usable and Visually Appealing Web Sites (Monday) =46our great tutorials showing you how to design great graphics and visuals = are: 5 Color and Type in Information Design (Sunday) 6 Designing Icons and Visual Symbols (Sunday) 18 Metaphor Design in User Interfaces: How to Effectively Manage Expectation, Surprise, Comprehension, and Delight (Monday) 27 Multimedia Visual Interface Design (Monday) Also, see how graphics are used for powerful information display in: 30 Information Visualization (Monday afternoon). User interfaces for emerging applications are presented in two forward-looking tutorials: 7 MediaJazz: Digital Storytelling and Computer Game Design (Sunday) 19 Interacting and Designing in Virtual Worlds on the Internet (Monday) Interacting with computers should be as easy as talking or interacting with other people. Three tutorials that lead the way in interacting with computers in a more natural way are: 8 Spoken Dialogue Interfaces (Sunday) 28 Social and Natural Interfaces: Theory and Design (Monday morning) 31 Creating Conversational Interfaces for Interactive Software Agents (Monday afternoon) Software agents is an emerging technology. Learn what agents can do, how to interact with agents, and how to implement agents using Java with these tutorials: 9 Wizards, Coaches, Advisors, and More: A Performance Support Primer (Sunday= ) 29 Software Agents (Monday morning) 31 Creating Conversational Interfaces for Interactive Software Agents (Monday afternoon) 32 Java-Based User Interface Development and Web Application Deployment (Monday afternoon) Two classic tutorials on usability engineering are: 10 Product Usability: Survival Techniques (Sunday) 20 Practical Interface Design: Developing Software Within Real-World Constraints (Monday) Having trouble managing the user interface process or convincing your managers that user interface design and evaluation is necessary? Then enroll in: 11 Strategic Usability: Introducing Usability into Organizations (Sunday) 21 Managing the Design of the User Interface (Monday) Understanding users is fundamental to designing great user interfaces. We offer six tutorials on gathering requirements and feedback from users: 12 Activity Theory: Basic Concepts (Sunday) 14 Getting Started on a Contextual Project (Sunday morning) 15 Design Ethnography: Using Custom Ethnographic Techniques To Develop New Product Concepts (Sunday afternoon) 22 Interviewing Customers: Discovering What They Can't Tell You (Monday) 23 Structured Observation: Practical Methods for Understanding Users and Their Work in Context (Monday) 24 Contextual Design: Using Customer Work Models to Drive Systems Design (Monday) Want to know about structured design? Try these tutorials: 13 Designing User Interfaces from Analyses of Users' Work Tasks (Sunday) 25 Object, View, and Interaction Design (Monday) The speaker at last year's closing session, Betty Edwards, will be back to present a tutorial on drawing and how artistic training can improve user interfaces. This tutorial has limited enrollment, so sign up early: 26 (Drawing tutorial) (Monday) Whether you are new to the CHI conference or have been attending for years, there are courses designed for you in the CHI 97 tutorials program. [Descriptions of each tutorial follow:] 1 Human-Computer Interaction: Introduction and Overview Keith A. Butler, Boeing Information and Support Services Robert J.K. Jacob, Tufts University Saturday, March 22, half-day evening introductory tutorial Benefits If you are a newcomer to the HCI field, this tutorial will give you the background you need to get the most out of the CHI conference. Origins This tutorial is a tried-and-true introduction to the field of human-computer interaction (HCI). It has become a CHI conference tradition. =46eatures =85 what is HCI and why is it important? =85 introduction to building usable systems =85 introduction to the psychology of HCI =85 introduction to computer technologies for HCI =85 psychological data for design =85 design methods and tools =85 user interface media and tools =85 introduction to user interface architecture =85 future directions of HCI =85 where to learn more about HCI during the conference =85 where to learn more in the published HCI literature Audience Professionals from computing-related fields who are new to the field of human-computer interaction or new to the CHI conference. No background in HCI is assumed. Presentation Lecture, plus videos and demos of relevant tools. Instructors Keith Butler is a senior principal scientist for user-centered design at Boeing Information and Support Services. Before joining Boeing, he was a member of the technical staff at Bell Labs. Rob Jacob is on the faculty of the Electrical Engineering and Computer Science Department at Tufts University. He is a member of the editorial board of ACM Transactions on Computer-Human Interaction and former Vice Chair of SIGCHI. Related Tutorials 3 Cognitive Factors in Design: Basic Phenomena in Human Memory and Problem Solving (Sunday) 16 Practical Usability Evaluation (Monday) 2 User Interface Design for the World Wide Web Jakob Nielsen, Sun Microsystems Saturday, March 22, half-day evening introductory tutorial Benefits You will learn how to design compelling, easy to use, well-structured, and attractive web sites. You will also learn simple and efficient methods for testing web sites for usability. Origins This tutorial was the most highly attended at CHI 96. Earlier versions of this tutorial were also presented at the Fourth and Fifth International WWW Conferences and won awards for excellence. =46eatures =85 major user interface considerations for WWW design =85 competitive usability analysis of other Web sites =85 use of customer surveys =85 discovering the users' mental models of the information space =85 user testing of WWW designs, including exploratory and directed tasks =85 using paper prototyping techniques to test early homepage designs =85 testing the intuitiveness of your icons =85 usability inspection methods to evaluate a large body of Web pages =85 issues in redesigning a Web site =85 utilizing non-standard elements, such as Netscape extensions and Java ap= plets =85 designing for an international audience Audience Anyone who authors or designs WWW page content, especially those with responsibility for the overall design and usability engineering of an entire site. You should have some experience using the WWW and understand the basic nature of Web pages. Presentation Lecture using in-depth case studies and examples. Instructor Jakob Nielsen is a Sun Microsystems Distinguished Engineer. He was the user interface lead for the recent redesign of Sun's WWW pages (www.sun.com). Jakob Nielsen's recent books include Multimedia and Hypertext: The Internet and Beyond, Usability Engineering, Usability Inspection Methods" (with Bob Mack) and International User Interfaces (with Elisa del Galdo). He writes the monthly "Alert Box" column on Internet usability: http://www.sun.com/columns/alertbox Related Tutorials 4 Development of Collaborative Applications with the WWW Shell (Sunday) 17 Designing Usable and Visually Appealing Web Sites (Monday) 32 Java-Based User Interface Development and Web Application Deployment (Monday PM) 3 Cognitive Factors in Design: Basic Phenomena in Human Memory and Problem Solving Thomas T. Hewett, Drexel University Sunday, March 23, full-day introductory tutorial Benefits You will learn the theoretical underpinnings of how people remember and how they solve problems. You will learn how to use that knowledge during product design to interpret user interface guidelines and also to go beyond the guidelines. Origins This was a top-rated tutorial at CHI 95 and CHI 96. =46eatures =85 clustering and organization in recall =85 characteristics of short-term memory =85 how information is transferred from short-term to long-term memory =85 characteristics of long-term memory =85 basic processes and phenomena in human problem-solving =85 rules of thumb for effective problem-solving =85 how memory and problem-solving affect computer use =85 memory and problem-solving issues in HCI design =85 suggestions for self-directed study Audience Interface designers whose applications require users to interpret complex instructions and perform detailed procedures. Developers who have found that there are users who have trouble using their products without training. Anyone who needs to teach psychological aspects of human-computer interaction. Not intended for those with coursework in cognitive psychology. Presentation Lecture, demonstration, and exercises. Instructor Tom Hewett, Professor of Psychology at Drexel University, is a cognitive psychologist with considerable classroom experience. He has offered variants of this tutorial successfully to hundreds of interface designers at a variety of national and international conferences. Tom is a published courseware author and has worked on the development and evaluation of several projects. Related Tutorials 1 Human-Computer Interaction: Introduction and Overview (Saturday evening) 16 Practical Usability Evaluation (Monday) 4 Developing Collaborative Applications Using the World Wide Web "Shell" Alison Lee and Andreas Girgensohn NYNEX Science & Technology Sunday, March 23, full-day intermediate tutorial Benefits You will understand how to use the Web as more than a mechanism for distributing documents over the Internet. You will learn how to use commonly available Web tools and technologies (e.g., Java, JavaScript, CGI scripts, tables, helper applications) in combination with Web browsers to design, develop, deploy and execute interactive and collaborative Web-based applications. Origins This is an update of a CSCW 96 tutorial. =46eatures =85 software concepts and architecture of the World Wide Web =85 tools and technologies that underlie the World Wide Web and form buildin= g blocks of the WWW "Shell" =85 how to use the building blocks in the WWW Shell to develop collaborative applications =85 adding application data and behavior to the WWW Shell =85 issues in designing and developing collaborative applications =85 advantages and limitations of using the WWW Shell for application develo= pment Audience Participants should be familiar with the World Wide Web and have experience using a Web browser. Although familiarity with C/C++, Perl, and Java are not required, participants should be comfortable with scripting and programming languages. Experience designing and developing collaborative applications is useful. Presentation Lecture, small-group exercises, discussions, and demonstrations. Instructors Both instructors have several years of experience with building Web-based applications for communication and collaboration within their organization. They have developed a tool, Web Dynamic Forms, that enables Web developers to create interactive and domain-specific form interfaces. Related Tutorials 2 User Interface Design for the World Wide Web (Saturday evening) 17 Designing Usable and Visually Appealing Web Sites (Monday) 32 Java-Based User Interface Development and Web Application Deployment (Monday PM) 5 Color and Type in Information Design Sunday, March 23, full-day advanced tutorial Charles A. Poynton, Poynton Vector Corporation Mary Mooney, Sun Microsystems Benefits You will learn the perceptual, color science, and engineering principles that underlie effective information presentation. You will learn to apply these principles to the design of graphical user interfaces and information displays. Origins This tutorial is based on the SIGGRAPH 96 course "Digital Color," with material on information display and design added for CHI 97. =46eatures =85 designing for human visual perception, including colorblindness =85 designing for different viewing conditions =85 choosing effective color combinations for print and for computer monitor= s =85 understanding tone reproduction and creating the illusion of more tones or colors than you have =85 managing color palettes across applications and platforms =85 typography; choosing serif or sans-serif fonts =85 illustration =85 information design Audience Graphic designers, interface designers, and developers of online information. You should have experience in developing user interfaces, experience in creating and manipulating digital imagery, or experience in writing or illustration. Presentation Lecture, demonstration, and exercises. Instructors Charles A. Poynton is the founder and principal of Poynton Vector Corporation, where he works to integrate video technology -particularly high definition television and accurate color reproduction - into computer workstations. His book, "A Technical Introduction to Digital Video," was recently published by John Wiley & Sons. Mary Mooney has worked as Information Architect for Ikonic Interactive designing Web sites and interactive TV applications for Time Warner. She previously designed multimedia applications, computer based training, and prototypes for IBM, =46ord, and World Cup Soccer while at Sybase. She has a degree in Fine Arts with post graduate study in advertising design and illustration. Related Tutorials 18 Metaphor Design in User Interfaces: How to Effectively Manage Expectations, Surprise, Comprehension, and Delight (Monday) 27 Multimedia Visual Interface Design (Monday AM) 6 Designing Icons and Visual Symbols William Horton, William Horton Consulting Sunday, March 23, full-day introductory tutorial Benefits You will learn how to design icons that users recognize, understand fully, and remember reliably. Origins This is an update of a highly rated tutorial from CHI 95 and CHI 96. =46eatures =85 where and when to use icons, words, or both =85 how to represent conceptual and nonvisual concepts with icons =85 how to draw icons - limitations of size and colors =85 how to design icons for an international products =85 how to systematically design large number of icons Audience This tutorial is for anyone - graphic artist, user interface designer, human factors specialist, technical writer, product designer - who designs icons and visual symbols for use in computer displays, technical documents, and other media where a concept or idea must be communicated in a restricted area or to an international audience. You won't need artistic talent, just a willingness to try creative approaches to solving communication problems. Presentation Lecture, interactive discussion, and individual and group exercises. Instructor William Horton, author of The Icon Book, is an expert in communicating business and technical information. He is a graduate of MIT and a registered professional engineer. His other books include Designing and Writing Online Documentation (2nd ed.), Illustrating Computer Documentation, Secrets of User-Seductive Documents, and The Web Page Design Cookbook with CD-ROM. Related Tutorials 18 Metaphor Design in User Interfaces: How to Effectively Manage Expectations, Surprise, Comprehension, and Delight (Monday) 26 Drawing on the Right Side of the Brain (Monday) 27 Multimedia Visual Interface Design (Monday AM) 7 MediaJazz: Digital Storytelling and Computer Game Design Thom Gillespie, Indiana University Sunday, March 23, full-day advanced tutorial Benefits You will practice digital storytelling and computer game design from concept development to business proposal and prototype development. Origins This tutorial is new for CHI 97. =46eatures =85 understanding information as art, entertainment, and fun =85 historical, aesthetic, and cognitive basis of digital storytelling and computer game design =85 learning the relationship between interaction and storytelling =85 in-class evaluation of popular interactive titles =85 rapid brainstorming and critique of new ideas =85 creating a design specification and paper prototype of a product to submit to a commercial publisher, complete with market analysis and battle plan Audience Creative thinkers interested in digital storytelling and computer game design. People who can use a mouse with either hand, can create slightly goofy art in Photoshop, and have a creative streak which always got them in trouble when they were children. Please bring at least 20 photographs or images that are important to you. Presentation Lecture, small group development, prototype presentation and critique. The morning is focused on individual concept development and the afternoon on group work. Instructor Thom Gillespie is a practicing artist who works in both digital and analog. He has taught just north of the Arctic Circle at University of Alaska, and just south of the Equator for the United Nations in Central Java. Currently, Thom teaches at Indiana University in the School of Library and Information Studies. He is also a founding member of DigitalMuses, a Midwest design company specializing in digital storytelling. Related Tutorials 18 Metaphor Design in User Interfaces: How to Effectively Manage Expectations, Surprise, Comprehension, and Delight (Monday) 19 Interacting and Designing in Virtual Worlds on the Internet (Monday) 26 Drawing on the Right Side of the Brain (Monday) 27 Multimedia Visual Interface Design (Monday AM) 8 Spoken Dialogue Interfaces Susann LuperFoy The MITRE Corporation Sunday, March 23, full-day introductory tutorial Benefits What can speech systems do today? You will learn about recent advances and current efforts to design, construct, and evaluate complete conversational systems that integrate speech recognition and synthesis with other enabling technologies. This tutorial draws upon work in user-system dialogue design, speech synthesis and recognition, natural language processing, machine translation, planning and plan recognition, gesture analysis, computational discourse, and usability evaluation. Origins This tutorial is new for CHI 97. =46eatures =85 assessing the state of the art =85 introduction to the underlying technologies =85 tools for developing spoken dialogue systems =85 examples of projects, including software agents that communicate via spoken language =85 how to evaluate spoken dialogue interfaces Audience Commercial and government managers of technology will learn about potential applications of spoken dialogue systems in their organizations. Students, faculty, and other researchers working in component fields will get an overview of the broader discipline. Presentation Lecture and guided discussion. Instructor Susann LuperFoy is a Lead Scientist at the MITRE Corporation in Washington, DC and adjunct faculty at Georgetown University. She has worked in computational linguistics and human-system interaction for over ten years, collaborating with leading researchers in voice-to-voice machine translation and spoken dialogue systems in the United States, Japan, and Europe. She is currently editing an MIT Press book, "Automated Spoken Dialogue Systems," which includes papers from all areas of spoken dialogue research and development. Related Tutorials 28 Social and Natural Interfaces: Theory and Design (Monday AM) 29 Software Agents (Monday AM) 31 Creating Conversational Interfaces for Interactive Software Agents (Monday PM) From jalarson@COLBY.EDU Fri Nov 1 17:57:16 1996 Return-Path: Received: from mailer.oclc.org (mailer.dev.oclc.org) by oclc.org (4.1/SMI-4.1) id AA26538; Fri, 1 Nov 96 17:57:15 EST Received: from host-04.colby.edu by mailer.oclc.org (5.0/SMI-4.1) id AA01180; Fri, 1 Nov 1996 17:57:08 +0500 Received: from [137.146.66.54] (dip-07.mudd.colby.edu) by host-04.colby.edu with SMTP (1.40.112.8/Colby 1.1) id AA049048614; Fri, 1 Nov 1996 17:50:14 -0500 Date: Fri, 1 Nov 1996 17:50:14 -0500 Message-Id: Mime-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable To: alee@nynexst.com, poynton@inforamp.net, luperfoy@mitre.org, thom@copper.ucs.indiana.edu, tony_salvador@ccm.jf.intel.com, tandyt@microsoft.com, gershon@mitre.org, mmillier@ibeam.intel.com, isensee@austin.ibm.com, aaron@amanda.com, neale@kodak.com, jacob@cs.tufts.edu, kmcgraw@clark.net, iansmith@cc.gatech.edu, nass@leland.stanford.edu, smetros@utk.edu, dray.chi@xerox.com, drdeb@vineyard.net, bdamer@ccon.org, perlman@mailer.oclc.org, pete@cdx.qmw.ac.uk, nardi@apple.com, sarahb@werple.mira.net.au, jspool@uie.com, william@horton.com, hewett@duvm.ocs.drexel.edu, jakob@eng.sun.com, karen@acm.org, debra_herschmann@uscmail.lehman.com, rbower2154@aol.com, ellen@communities.com From: jalarson@COLBY.EDU (Jim Larson) Subject: chi tutorials 9-20 Cc: jalarson@host-04.COLBY.EDU, chi97-tutorials@acm.org Content-Length: 22309 Status: RO Dear Author, Please review the introductory section to see how your tutorial fits into the overall tutorial program. Then review the material for your tutorial. Please send any changes to me by Monday evening, November 1 so I can incorporate them into the final copy. Thanks Jim Larson chi97-tutorials@acm.org -------------------------------------------- 9 Wizards, Coaches, Advisors, and More: A Performance Support Primer Karen L. McGraw Cognitive Technologies Sunday, March 23, full-day introductory tutorial Benefits This tutorial provides you with a solid foundation in performance support system components to help you enhance the use, acceptability, and performance of interactive systems. Origins This tutorial is new for CHI 97. =46eatures =85 method and depth of performance support required, based on client/user n= eeds =85 types of performance support which may be embedded within or accessed from an interactive system =85 types and uses of performance support components =85 examples of performance support system components and development templa= te =85 process of performance support design, development, and evaluation that enables you to work more effectively on a multidisciplinary team =85 how to identify and address hurdles and feasibility issues Audience Designers of organization-critical interactive systems, including software analysts and engineers, project champions, human factors engineers, and trainers. Presentation Lecture with small group exercises and activities. Instructor Karen McGraw has over 15 years of experience solving performance problems through the careful definition, development, and deployment of technology. She has conducted or managed requirements engineering, design, development, and usability testing for human-computer interfaces, performance support systems, intelligent tutoring and expert systems, and information systems. Karen is the author of numerous texts and articles, including "User-centered Requirements: A Scenario-based Engineering Process" published by Lawrence Erlbaum. Related Tutorials 29 Software Agents (Monday AM) 31 Creating Conversational Interfaces for Interactive Software Agents (Monday PM) 10 Product Usability: Survival Techniques Jared M. Spool, Carolyn Snyder, and Tara Scanlon User Interface Engineering Sunday, March 23, full-day introductory tutorial Benefits You will learn how to produce a more usable product by prototyping and testing a design in a matter of hours, using readily available materials. Origins This is an update of a tutorial presented at several CHI conferences including CHI 96. More emphasis is placed on discount usability methods and less on design principles and user interface guidelines. =46eatures =85 develop products using a process of iterative refinement with frequent measurements to ensure they are still "on course" =85 build a paper mock-up of a product interface using common office supplie= s =85 usability test a paper mock-up and improve it very quickly based on feedback from users =85 apply the concepts of affordances and mental models to create a successful design Audience All members of the development team, including engineers, designers, technical writers, and managers. Experience in developing commercial products is highly recommended. Participants are not assumed to have experience with usability testing. Presentation Small team design exercises, plus lecture and discussion. Instructors User Interface Engineering has extensive experience in teaching paper prototyping and usability evaluation to development teams. Jared Spool is also on the faculty of the Tufts University Gordon Institute. Carolyn Snyder holds an MBA from the University of Chicago. Tara Scanlon previously worked for Dun and Bradstreet Software and Digital Equipment Corporation. She holds a Masters in Technical Communication from Rensselaer Polytechnic Institute. Related Tutorials 16 Practical Usability Evaluation (Monday) 20 Practical Interface Design: Developing Software within Real-World Constraints (Monday) 21 Managing the Design of the User Interface (Monday) 11 Strategic Usability: Introducing Usability into Organizations Sarah Bloomer, Rachel Croft, and Helen Kieboom The Hiser Group Sunday, March 23, full-day intermediate tutorial Benefits You will learn techniques for convincing management and development teams of the value of usability in a way meaningful to your organization. Origins This tutorial is new for CHI 97. =46eatures =85 barriers to and opportunities for usability activities =85 techniques for communicating the value of usability to your organization =85 aligning usability activities with business goals and objectives =85 translating usability information into terms each target audience unders= tands =85 practice in developing usability strategy for an organization =85 analysis of real-world cases Audience Experienced usability professionals, user interface designers, developers, and managers who want to improve the acceptance of usability activities within their organizations. Consultants who wish to convince potential clients. Presentation Lecture with team exercises. Instructor The Hiser Group has consulted to a number of large clients in Australia over the past 3 years, facilitating the introduction of usability at the organizational level. Sarah Bloomer, Rachel Croft, and Helen Kieboom have been leading the development of this service and have worked directly on the case studies which will be presented. Their usability expertise also includes styleguide development, cost-justification, ethnographic methods, and sociological issues. Related Tutorials 16 Practical Usability Evaluation (Monday) 21 Managing the Design of the User Interface (Monday) 12 Activity Theory: Basic Concepts and Applications Victor Kaptelinin Ume=E5 University Bonnie A. Nardi Apple Computer Sunday, March 23, full-day intermediate tutorial Benefits You will acquire a unifying theoretical framework covering many commonly used methods of studying users and the context of their work. You will learn practical ways of applying activity theory to problems of human-computer interaction. Origins This tutorial is new for CHI 97. =46eatures =85 the five basic principles of activity theory =85 how activity theory relates to other approaches such as cognitive science, distributed cognition, scenario-based design, participatory design, and situativity theory =85 how to apply activity theory to real-world problems of design and evalua= tion =85 how to select a contextual methodology for your problem Audience Researchers, designers, engineers, and managers who are interested in theoretical approaches to contextual design and evaluation. Presentation Lecture, demonstration, and exercises. Instructors Victor Kaptelinin is a Research Associate at the Department of Informatics, Ume=E5 University. He has held various research and teaching positions and has published extensively on the topics of the psychology of perception, computer-assisted learning, and human-computer interaction. His current research is on contextual factors of human-computer interaction and on skill automatization in computer use. Bonnie Nardi is an anthropologist in the Apple Research Laboratories, the editor of "Context and Consciousness: Activity Theory and Human-Computer Interaction" and the author of "A Small Matter of Programming: Perspectives on End User Computing." She was Technical Program Co-Chair for CHI 96. Related Tutorials 22 Interviewing Customers: Discovering What They Can't Tell You (Monday) 23 Structured Observation: Practical Methods for Understanding Users and Their Work in Context (Monday) 24 Contextual Design: Using Customer Work Models to Drive System Design (Mon= day) 13 Designing User Interfaces from Analyses of Users' Work Tasks Peter Johnson, Stephanie Wilson, and Hilary Johnson Queen Mary and Westfield College, University of London Sunday, March 23, full-day intermediate tutorial Benefits You will learn how to study, analyze, and model users' work tasks. You will understand how to progress from this analysis to envisioning users' future work and designing the user interface to support it. Origins This tutorial has been highly rated at international conferences such as INTERACT95 and BCS HCI 95. =46eatures =85 advantages and limitations of task analysis and task modeling =85 detailed presentation of methods for analyzing and modeling tasks =85 how the content of models influences interaction design =85 practice in using task analysis and modeling for user interface design Audience Designers, developers, human factors practitioners, and other HCI professionals. No knowledge of task analysis or user interface programming is assumed. Presentation Lecture with small group analysis and design practical exercises. Instructors Peter Johnson is Professor of Human Computer Interaction and teaches courses in user interface modeling, interactive systems design and graphical user interfaces. He developed the concepts of work-task analysis and task-based user interface design. He is currently investigating the role of representations in design, user involvement in analysis and design, and principles of HCI. Hilary Johnson is a member of faculty at QMW and teaches courses in human factors and user modeling. She developed the Task Knowledge Structures approach; her research includes task modeling, explanation, and evaluation. Stephanie Wilson is a senior researcher investigating user involvement in interactive system design. She developed the ADEPT tools for task modeling and user interface design. Related Tutorials 25 Object, View, and Interaction Design (Monday) 14 Getting Started on a Contextual Project Karen Holtzblatt and Hugh R. Beyer InContext Enterprises, Inc. Sunday, March 23, half-day morning intermediate tutorial Benefits You will be able to plan and conduct a project using contextual techniques to gather customer data. Origins This tutorial is new for CHI 97. =46eatures =85 technical and organizational issues to consider before introducing contextual techniques =85 choosing a project and setting its focus =85 choosing customers to visit =85 structuring a project and gathering data =85 communicating the project's findings Audience Anyone interested in putting contextual or customer-centered techniques into practice on their own projects. Some familiarity with contextual techniques is desirable. Presentation Lecture, demonstration, and hands-on exercises. Instructors Karen Holtzblatt and Hugh Beyer are co-founders of InContext Enterprises, Inc., a firm that works with companies such as Microsoft and WordPerfect coaching teams to design products, product strategies, and information systems from customer data. Karen and Hugh are developers of contextual design, a customer-centered design process that extends the contextual inquiry data gathering technique. Karen is an originator of the contextual inquiry approach to field data collection and has pioneered the introduction of this technique into working engineering teams. Hugh has worked in industry for the past 12 years as a programmer, architect, and consultant. He has developed processes for using customer data to drive object-oriented design. Related Tutorials 22 Interviewing Customers: Discovering What They Can't Tell You (Monday) 23 Structured Observation: Practical Methods for Understanding Users and Their Work in Context (Monday) 24 Contextual Design: Using Customer Work Models to Drive System Design (Mon= day) 15 Design Ethnography: Using Custom Ethnographic Techniques to Develop New Product Concepts Tony Salvador, Intel Corporation Michael Mateas, Carnegie Mellon University Sunday, March 23, half-day afternoon intermediate tutorial Benefits Engineering Ethnography is a set of data collection and analysis perspectives, assumptions and skills that can be used effectively and efficiently to understand a particular environment of people for the express purpose of identifying and designing a range of new technology products. You will learn the parameters for initiating and conducting an ethnographic study, and review the results from several studies conducted at Intel Corporation. Origins This tutorial is new for CHI 97. =46eatures =85 practical and corporate benefits of this work for new product development =85 how design ethnography is qualitatively different than ethnography =85 how to initiate and conduct a study for a particular user domain =85 results from our studies of families with young children, teens, and/or business communications Audience Participants should have experience with some techniques for determining product requirements. Presentation Lecture with extensive discussion. Instructors Tony Salvador and Michael Mateas have evolved this methodology in the course of their work at Intel and Tektronix. Both have experience with a wide range of requirement analysis techniques for business and consumer products. Related Tutorials 22 Interviewing Customers: Discovering What They Can't Tell You (Monday) 23 Structured Observation: Practical Methods for Understanding Users and Their Work in Context (Monday) 24 Contextual Design: Using Customer Work Models to Drive System Design (Mon= day) 16 Practical Usability Evaluation Gary Perlman Ohio State University Monday, March 24, introductory full-day tutorial Benefits You will learn how to use cost-effective methods for evaluating interactive systems and gain enough experience to apply the methods on your own. Origins This tutorial is an update of a highly-rated CHI 96 tutorial. =46eatures =85 goals for interactive system evaluation =85 cost-benefit analysis of usability evaluation =85 choosing the right evaluation method =85 usability inspection methods (e.g., heuristic evaluation) =85 observational skills and video =85 data collection by software logging =85 questionnaires for structured feedback =85 a step-by-step guide to usability testing =85 sources of information about usability evaluation, including a usability bibliography =85 several sample data collection forms are provided Audience Managers interested in increasing usability testing in their organizations. Software engineers interested in practical methods for usability evaluation during - not after - development. Human factors specialists who want to learn more about usability evaluation methods. No background in usability evaluation is assumed. Presentation Lecture, interspersed with exercises and discussion. Instructors Gary Perlman is a professor in the Department of Computer and Information Science at Ohio State University. He is the author of statistical and hypertext software used widely for user interface evaluation. Gary has consulted extensively for major information technology companies on user interface development process. He is also the creator of the "HCI Bibliography" project, the largest free-access bibliography on human-computer interaction. Related Tutorials 1 Human-Computer Interaction: Introduction and Overview (Saturday evening) 3: Cognitive Factors in Design: Basic Phenomena in Human Memory and Problem Solving (Sunday) 11 Strategic Usability: Introducing Usability into Organizations (Sunday) 17 Designing Usable and Visually Appealing Web Sites [Still having trouble distinguishing this from Nielsen.] Monday, March 24, full-day intermediate tutorial Wayne Neale, Ph.D. and Cindy McCombe Eastman Kodak Company Benefits You will learn to use design methods confidently to craft well-engineered and highly usable web sites. Origins This tutorial is new for CHI 97. =46eatures =85 structuring information to maintain user orientation and facilitate access to information =85 prototyping and evaluating the information structure =85 design techniques unique to the Web environment =85 specific techniques to achieve consistency and support usability =85 graphics production and management techniques that ensure high quality and rapid system response Audience Participants should have basic familiarity with the web and HTML. Presentation Lecture, demonstration, and exercises. Instructors Wayne Neale has been involved with the Web since 1993 and has designed, built, and maintained the 6000+ page Kodak Web site over the past three years. His educational background is in human-computer interaction, cognitive psychology, business, and industrial engineering. Wayne has taught classes on human-computer interaction and Web site design at Virginia Tech and Kodak. Cindy McCombe is a visual interaction designer supporting the Kodak Web site. Her background is in communications, journalism, graphic design, graphic arts publishing, and typography. Cindy has taught a courses in publishing ranging from Web design to newswriting at the Rochester Institute of Technology and Kodak. Related Tutorials 2 User Interface Design for the World Wide Web (Saturday evening) 4 Developing Collaborative Applications Using the World Wide Web "Shell" (Sunday) 18 Metaphor Design in User Interfaces: How to Effectively Manage Expectation, Surprise, Comprehension, and Delight Aaron Marcus Aaron Marcus and Associates, Inc. Monday, March 24, full-day intermediate tutorial Benefits You will learn how to plan, analyze, design, and implement user interface metaphors Origins This is an update of a tutorial presented at CHI 95 and various SIGGRAPH conferences. =46eatures =85 what metaphores are and how they convey structures and processes =85 how to explore and identify metaphores for target user communities =85 how to analyze and design metaphores based on logical paradigms, task analysis, and cultural sterotypes =85 practical guidance for research and commercial product development =85 apply principles using hands-on experience through simple pen-and-paper exercises Audience User interface designers; researchers and developers of productivity tools, multimedia products, and Web-based documents. Presentation Illustrated lectures with pen-and-paper sketch exercises and group critiques= . Instructor Aaron Marcus and his staff have designed and evaluated more than 120 user interfaces and information display environments. They have been contracted to undertake metaphor design projects for Kaiser Permanente Health Systems, Motorola, Oracle, and SABRE Travel Information Network. In 1992, Aaron received the National Computer Graphics Association Industry Achievement award for his contributions to the field. Related Tutorials 5 Color and Type in Information Display (Sunday) 6 Designing Icons and Visual Symbols (Sunday) 19 Interacting and Designing in Virtual Worlds on the Internet Bruce Damer Contact Consortium Monday, March 24, full-day intermediate tutorial Benefits You will be introduced to the new Internet medium of "avatar" virtual environments. You will learn about this new area for research and applications of interaction design and virtual community. Origins This tutorial is an expansion of a highly rated demonstration at CHI 96 and was also featured at various conferences, including CSCW 96. =46eatures =85 virtual communities =85 background on 3D interfaces =85 cooperative and participatory interaction design =85 CSCW and CSCL applications of virtual worlds =85 hands-on experience using an avatar in a virtual world =85 participation of people at remote sites within the virtual world during the tutorial Audience Participants should have some experience with use of graphical interfaces (Windows 95), and in navigating in 3D spaces. No programming experience is assumed. Presentation Lecture followed by hands-on exercises. Instructor Bruce Damer is co-director of the Contact Consortium, a non-profit research membership organization dedicated to the development of the virtual worlds medium. The Consortium has engaged in extensive usability testing of virtual worlds provided by its member companies, which include Worlds Incorporated, Intel, Black Sun Interactive, IDS, Nippon Telephone and Telegraph (Software Division), Philips, Microsoft, Onlive, The Palace and others. The collaborative construction and staffing of a virtual town (Sherwood Forest), a virtual university (the U) and the hosting of a major international conference has given the Consortium good experience in this medium. Related Tutorials 7 Media Jazz: Digital Storytelling and Computer Game Design (Sunday) 20 Practical Interface Design: Developing Software within Real-World Constra= ints Debra Herschmann Wang Software Monday, March 24, full-day advanced tutorial Benefits You will learn techniques for designing a user interface within real-world development constraints (budget, time, staff) while preserving the usability of the system. Origins This is an update of a CHI 96 tutorial. =46eatures =85 factors influencing user interface design: budget, schedule, choice of development tools, conflicting agendas of the development team, and skill level of the development team =85 defining the best user interface that can be implemented within project resource constraints =85 communication within and across project teams =85 how to revise, scale down, or even eliminate features while preserving the integrity of the user interface =85 tracking the user interface through the development process Audience User interface designers, project managers, and developers involved with application design and implementation in production environments. =46amiliarity with concepts of graphical user interface design and team software development is assumed. Presentation Lecture, real-life examples, and individual and group exercises. Instructors Debra Herschmann is a Senior Designer at Wang Software and is responsible for developing graphical user interfaces for a variety of workflow and document imaging applications. She has extensive experience designing interactive applications for businesses, museums, and educational institutions. Debra also teaches Interactive Design at the School of Visual Arts in New York City. Related Tutorials 10 Product Usability: Survival Techniques (Sunday) 11 Strategic Usability: Introducing Usability into Organizations (Sunday) From jalarson@COLBY.EDU Fri Nov 1 17:59:04 1996 Return-Path: Received: from mailer.oclc.org (mailer.dev.oclc.org) by oclc.org (4.1/SMI-4.1) id AA26597; Fri, 1 Nov 96 17:59:03 EST Received: from host-04.colby.edu by mailer.oclc.org (5.0/SMI-4.1) id AA01303; Fri, 1 Nov 1996 17:58:57 +0500 Received: from [137.146.66.54] (dip-07.mudd.colby.edu) by host-04.colby.edu with SMTP (1.40.112.8/Colby 1.1) id AA049198634; Fri, 1 Nov 1996 17:50:34 -0500 Date: Fri, 1 Nov 1996 17:50:34 -0500 Message-Id: Mime-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable To: alee@nynexst.com, poynton@inforamp.net, luperfoy@mitre.org, thom@copper.ucs.indiana.edu, tony_salvador@ccm.jf.intel.com, tandyt@microsoft.com, gershon@mitre.org, mmillier@ibeam.intel.com, isensee@austin.ibm.com, aaron@amanda.com, neale@kodak.com, jacob@cs.tufts.edu, kmcgraw@clark.net, iansmith@cc.gatech.edu, nass@leland.stanford.edu, smetros@utk.edu, dray.chi@xerox.com, drdeb@vineyard.net, bdamer@ccon.org, perlman@mailer.oclc.org, pete@cdx.qmw.ac.uk, nardi@apple.com, sarahb@werple.mira.net.au, jspool@uie.com, william@horton.com, hewett@duvm.ocs.drexel.edu, jakob@eng.sun.com, karen@acm.org, debra_herschmann@uscmail.lehman.com, rbower2154@aol.com, ellen@communities.com From: jalarson@COLBY.EDU (Jim Larson) Subject: chi tutorials 21-32 Cc: chi97-tutorials@acm.org Content-Length: 22143 Status: RO Dear Author, Please review the introductory section to see how your tutorial fits into the overall tutorial program. Then review the material for your tutorial. Please send any changes to me by Monday evening, November 1 so I can incorporate them into the final copy. Thanks Jim Larson chi97-tutorials@acm.org -------------------------------------- 21 Managing the Design of the User Interface Deborah Mayhew Deborah J. Mayhew and Associates Monday, March 24, full-day introductory tutorial Benefits You will learn what techniques and methods are available for designing good user interfaces and when and why to apply them. You will learn not only design and evaluation techniques, but also organizational and managerial strategies. Origins This CHI "classic" consistently receives high ratings from enthusiastic participants. The tutorial has been updated to reflect new ideas, approaches, and methods in the field of usability engineering. =46eatures =85 analyzing the costs and benefits of applying human factors methods =85 organizational structures and processes that foster good interface desig= n =85 planning for and manage the use of usability techniques within the overall project plan =85 evaluating design alternatives by defining and prioritizing objective goals and criteria =85 making interface design decisions by applying quick and inexpensive interface evaluation techniques =85 information gathering methods that support preliminary design and specification =85 testing and evaluation methods for simulations and prototypes Audience Experience with software development methodologies will provide a useful context. No experience with usability engineering is necessary. This tutorial will be most relevant to development managers, developers with an interest in or responsibility for usability, and usability engineering practitioners. Presentation The tutorial is organized around a sample development life cycle and presents an overview of usability methods which can be applied at different points in the development process. Instructor Deborah Mayhew holds a Ph.D. in Cognitive Psychology and has worked 18 years in software development organizations, 11 years as a usability consultant. Deborah is the author of two books on usability. Her clients have included IBM, AT&T, American Airlines, Hewlett-Packard, Ford Motor Co., GTE, and American Express. Often her work involves introducing usability techniques and methods into software development organizations. Deborah Mayhew has been offering tutorials at CHI since 1986. Related Tutorials 10 Product Usability: Survival Techniques (Sunday) 11 Strategic Usability: Introducing Usability Into Organizations (Sunday) 22 Interviewing Customers: Discovering What They Can't Tell You Ellen Isaacs Electronic Communities Monday, March 24, full-day introductory tutorial Benefits You will learn an approach to discovering customers' underlying needs for computer support. You will gain first-hand experience in developing good interview questions, interviewing people, and analyzing the responses you receive. Origins This tutorial is an update of a highly-rated CHI 96 tutorial. =46eatures =85 contrasting interviews with other means of learning about customers =85 basic interviewing skills, including ways to ask open, unbiased question= s =85 preparing for interviews =85 recording and analyzing the information you receive =85 many examples of good and poor interview questions Audience Designers, developers, writers, managers, testers, marketers, and anyone else who needs to identify customer requirements and end-user needs. Presentation Exercises, demonstrations, and lecture. Highly interactive; be ready to participate. Instructor Ellen Isaacs is a user interface designer at Electric Communities where she is designing virtual worlds. Prior to that she designed multimedia-based collaboration applications at Sun Microsystems. She has been interviewing customers on a regular basis for the past seven years and has made it a standard component of her design process. She received her Ph.D. from Stanford University in cognitive psychology, with a focus on language and conversation. Before that, she worked as a professional newspaper reporter. Ellen has combined her background in human information processing with her skills in interviewing to develop a technique for learning about customers' needs and translating that information into design requirements. Related Tutorials 12 Activity Theory: Basic Concepts and Applications (Sunday) 14 Getting Started on a Contextual Project (Sunday AM) 15 Introduction to Engineering Ethnography (Sunday PM) 23 Structured Observation: Practical Methods for Understanding Users and Their Work in Context Susan M. Dray Dray & Associates Monday, March 24, full-day introductory tutorial Benefits You will learn powerful techniques for gathering information about users and their work. Specifically, you will learn when and how to apply the techniques of structured observation to the software development process. Origins This highly rated tutorial was presented at CHI 96. =46eatures =85 introduction to the structured observation techniques of: naturalistic observation, contextual inquiry, usability roundtable, and artifact walk-through =85 how to plan for and carry out observations of users at their work sites =85 what to do with the data =85 next steps for using the data in system design =85 applied tools for use "in the trenches" and practice using them Audience Developers, designers, and managers responsible for customer needs analysis and identifying user requirements, as well as anyone who wants to understand how users work in order to do a better job of system design. Presentation Lecture, group discussion, and small group exercises to obtain practical experience performing structured observation. Instructor Susan M. Dray has over 15 years experience working as researcher, manager, and consultant in the design and use of technology. She has worked at Honeywell, Inc., IDS Financial Services, and, most recently, as an independent consultant. Related Tutorials 12 Activity Theory: Basic Concepts and Applications (Sunday) 14 Getting Started on a Contextual Project (Sunday AM) 15 Introduction to Engineering Ethnography (Sunday PM) 24 Contextual Design: Using Customer Work Models to Drive Systems Design Karen Holtzblatt and Hugh R. Beyer InContext Enterprises, Inc. Monday, March 24, full-day introductory tutorial Benefits Learn how to develop work models for representing data from field studies in a comprehensible form suitable for system design. Origins This highly rated tutorial from CHI 95 and CHI 96 has been updated for CHI 9= 7. =46eatures =85 contextual influences on people in their work environments =85 modeling customers' work practices =85 choosing types of work models to represent different aspects of users' tasks, environments, communication paths, and organizational culture =85 how to derive design decisions from work models =85 consolidating work models to reveal common structure, work strategies, intents, and concepts =85 techniques to generate, evaluate, and iterate innovative designs from work models Audience Anyone interested in customer-centered design, requirements analysis, or tailoring products and systems to people's work. Especially valuable for those who have experience collecting field data about users and who want to learn how to use that information to influence system design. Presentation Lecture, video, demonstration, and exercises. Instructors Karen Holtzblatt and Hugh Beyer are co-founders of InContext Enterprises, Inc., a firm that works with companies such as Microsoft and WordPerfect coaching teams to design products, product strategies, and information systems from customer data. Karen and Hugh are developers of contextual design, a customer-centered design process that extends the contextual inquiry data gathering technique. Karen is an originator of the contextual inquiry approach to field data collection and has pioneered the introduction of this technique into working engineering teams. Hugh has worked in industry for the past 12 years as a programmer, architect, and consultant. He has developed processes for using customer data to drive object-oriented design. Related Tutorials 12 Activity Theory: Basic Concepts and Applications (Sunday) 14 Getting Started on a Contextual Project (Sunday AM) 15 Introduction to Engineering Ethnography (Sunday PM) 25 Object, View, and Interaction Design [ "features" need a second look; make them concrete] Dick Berry, Dave Roberts, and Scott Isensee IBM Corporation Monday, March 24, full-day intermediate tutorial Benefits You will learn techniques for turning user requirements into design for an object-oriented user interface. The tools and techniques described will help to ensure the model's accuracy and completeness and will generate output that feeds into object-oriented code design methodologies. Origins This tutorial is new for CHI 97. =46eatures. You will learn how to =85 identify objects relevant to a task and relationships among among the ob= jects =85 find the needed views of objects =85 synthesize tasks using objects and views =85 use state diagrams and state tables to provide detailed interaction descriptions for objects and their views =85 prototype and evaluate the interface design Audience HCI professionals responsible for turning user requirements into interface design. Background in at least one (but not all) of the following areas is recommended: requirements gathering, user interface design, and object-oriented analysis. This tutorial is on object-oriented user interface design rather than object-oriented programming. Presentation Lecture and exercises. Instructors The instructors are user interface architects at IBM. They develop user interface guidelines, interface design methodologies, and new user interaction techniques. They developed the Common User Access object-oriented user interface style and hold numerous patents related to user interface design. Related Tutorials 13 Designing User Interfaces from Analyses of Users' Work Tasks (Sunday) 26 Drawing on the Right Side of the Brain Betty Edwards California State University Monday, March 24, full-day introductory tutorial Benefits Participants will learn basic strategies for accessing the visual perceptual mode of thinking. Origins Betty Edwards' CHI 96 plenary speech covered some elements of her work on drawing and accessing the visual perceptual mode of thinking. She is returning to conduct a hands-on tutorial. =46eatures * Very basic drawing skills. * An understanding of the nature of drawing. Audience This seminar is intended for a wide audience. That is, no specific group or profession. Presentation Lecture with hands-on exercises. No previous experience is required. Instructor Betty Edwards is a creativity consultant and author of Drawing on the Right Side of the Brain and Drawing on the Artist Within. Her educational background is in art education and psychology. She is Professor Emeritus at California State University, Long Beach. Related Tutorials 6 Designing Icons and Visual Symbols (Sunday) 7 MediaJazz: Digital Storytelling and Computer Game Design (Sunday) 27 Multimedia Visual Interface Design Susan E. Metros University of Tennessee/Knoxville John G. Hedberg University of Wollongong, Australia Monday, March 24, half-day morning introductory tutorial Benefits You will understand how the components of visual interface design work with the cognitive demands of an interface. As a result, you will be able to design or direct the design of interfaces for multimedia, Web sites, courseware, and training modules. Origins International conferences in media education (AACE Ed Media), 1994 - 1996. =46eatures =85 components of effective, stimulating, appropriate visual interface desig= n =85 demands those components make on the user's attention, memory, or other cognitive skills =85 translating written materials and other media into visual components =85 creating visual frameworks =85 graphically prototyping screens, nodes, and links =85 case studies of successful interfaces Audience Designers and developers of graphic or multimedia products, or anyone interested in using visual communication more effectively. No background in art is assumed. Presentation Lectures, small group exercises, and discussions. Instructors Susan Metros teaches computer-enhanced graphic design courses and does research on improved interfaces for multimedia applications and Web spaces. At the University of Tennessee/Knoxville, she is establishing a university-wide Instructional Technology Center. Susan has also designed the infrastructure and graphical interface for several interactive multimedia projects. John Hedberg's research is in navigation, cognition, and design in interactive multimedia. He has written and consulted on instructional design, including projects with government departments and industry in the US and in Southeast Asia. Both Susan and John helped create the award-winning CD-ROM "Investigating Lake Iluka," an ecology simulation. Related Tutorials 5 Color and Type in Information Display (Sunday) 7 MediaJazz: Digital Storytelling and Computer Game Design (Sunday) 30 Information Visualization (Monday PM) 28 Social and Natural Interfaces: Theory and Design Monday, March 24, half-day morning introductory tutorial Benefits You will learn the theoretical and empirical framework for social and natural interfaces, a rapidly growing category of interfaces. You will obtain over 100 specific guidelines for the design of interfaces that conform with social and natural rules. You will learn how to apply social science methods to assess these interfaces. Origins This tutorial is new for CHI 97. =46eatures =85 examples of how users treat computer interfaces in a social manner =85 social science concepts including manners, personality, emotion, roles, and form, and their implications for interface design =85 how to design interfaces that meet users' expectations for social intera= ction =85 using questionnaires, reaction time data, and physiological measures to assess these interfaces =85 practice in evaluating social and natural interfaces Audience Designers, usability specialists, and others interested in creating or assessing interfaces that conform with social and natural rules. No knowledge of programming is necessary. Presentation Lecture, discussion, and exercises. Instructors The instructors are co-authors of "The Media Equation: How People Treat Computers, Television, and New Media Like Real People and Places." They are the directors of the Social Responses to Communication Technology industrial affiliates program at Stanford. Their theories and research have been used in the design of numerous software products. Related Tutorials 8 Spoken Dialogue Interfaces (Sunday) 9 Wizards, Coaches, Advisors, and More: A Performance Support Primer (Sunday= ) 31 Creating Conversational Interfaces for Interactive Software Agents (Monday PM) 29 Software Agents [Features section could be stronger] Marc Millier Intel Architecture Labs, Intel Corporation Monday, March 24, half-day morning intermediate tutorial Benefits You will understand the structure and architectures of current agent technology and understand the user interface issues of software agents. Origins This tutorial is new for CHI 97. =46eatures =85 what is and isn't a software agent =85 roles agents play and architectures to support these activities =85 how agents observe and recognize, learn and adapt, plan and infer =85 how users interact with agents =85 trusting and controlling agents =85 is it an agent or a person? Audience Software engineers or managers who are interested in agent technologies and architectures. A familiarity with software architecture and the software development process is assumed. Presentation Lectures, demonstrations, and discussion. Instructor Marc Millier has been a working practitioner in software engineering for 25 years. His background includes databases, information retrieval, intelligent filtering, and object-oriented design and implementation. His current activity includes leading a team of software engineers in the research and development of a practical multi-agent architecture targeted for the business desktop. Related Tutorials 9 Wizards, Coaches, Advisors, and More: A Performance Support Primer (Sunday= ) 31 Creating Conversational Interfaces for Interactive Software Agents (Monday PM) 30 Information Visualization Stuart Card, Xerox PARC Stephen G. Eick, Bell Laboratories Nahum Gershon, The MITRE Corporation Monday, March 24, half-day afternoon intermediate tutorial Benefits You will gain a working knowledge of how to effectively visualize abstract information (in contrast with scientific data which is often spatial). You will learn how to apply this knowledge to specific areas such as the World Wide Web, text databases, and network information. Origins This tutorial was first presented at SIGGRAPH 96. =46eatures =85 what is information visualization and how it relates to scientific data visualization =85 user tasks where information visuatization helps =85 visualization and interaction techniques, including node and link diagrams, hierarchical data, data landscapes, and representations for text, focusing, filtering, and linking =85 perceptual basis for information visualization, including information processing principles, the dual perception system, and focus+periphery dynamic displays =85 case studies illustrating these principles, including browsing and organizing infrormation in the WWW, and querying and searching databases and the WWW Audience Participants should have some basic knowledge in graphics and visualization and interest to understand this emerging and significant area. Presentation Lecture with questions Instructors Stuart Card is a Xerox Research Fellow and manager of the User Interface Research Group at Xerox PARC. He and his team created the Information Visualizer, an animated 3D interactive information workspace program. Stephen G. Eick is a technical manager of the Data Visualization Research Group at Bell Laboratories, a division of Lucent Technologies. His research focuses on extracting information latent in large databases using novel interactive visualizations. His group has developed a suite of visualizations, including tools for visualizing abstract networks, software source code, and text corpora. Nahum Gershon is Principal Scientist at The MITRE Corporation. His work is concerned with information and data visualization, network browsers, image processing, data organization, and analysis of medical, environment, and other multidimensional data. Related Tutorials 5 Color and Type in Information Display (Sunday) 31 Creating Conversational Interfaces for Interactive Software Agents Tandy Trower Microsoft Corporation Monday, March 24, half-day afternoon intermediate tutorial Benefits You will learn practical design principles in designing conversational interfaces with visible interactive agents. Origins This tutorial is new for CHI 97. =46eatures =85 overview of agent and speech research and implementation =85 psychological and social research on media design =85 guidelines on basic princiles for creating effective conversational inte= rfaces =85 demonstration of agent application using conversational dsign principles Audience CHI professionals and product designers interested in designing interactive agent interfaces with conversational (speech-enabled) interfaces. Should have a basic understanding of HCI design and practice, but noeed not have a background in speech or agent interface design Presentation Lecture with demonstrations and video materials. Instructor Tandy Trower is the Director of the Advanced User Interface Design Group at Microsoft. He managed the development and introduction of a variety of products, including Microsoft C Compiler, Microsoft Flight Simulator, and Microsoft Windows. He is author of The Windows Interface Guidelines for Software Design. Related Tutorials 8 Spoken Dialogue Interfaces (Sunday) 28 Social and Natural Interfaces: Theory and Design (Monday AM) 29 Software Agents (Monday AM) 32 Java-Based User Interface Development and Web Application Deployment Ian Smith GVU Center, Georgia Tech Monday, March 24, half-day afternoon intermediate tutorial Benefits You will understand the possibilities provided by the World Wide Web for application deployment. You will gain a more detailed understanding of the issues involved in developing user interfaces for the WWW in Java. Origins This tutorial is new for CHI 97. Benefits =85 how Java applets work within the context of Web servers and browsers =85 Abstract Window Toolkit (AWT), the primary mechanism by which graphical applications can be constructed in Java =85 capabilities and problems of interfaces developed within cross-platform settings =85 emerging technologies and how they will affect your ability to deliver graphical applications via the Web Audience Participants should be well-versed in using the web; experience with developing HTML documents is helpful. A software development background would help in understanding some of the details, but is not required. Presentation Lecture plus demonstrations. Instructor Ian Smith is one of the principal developers of SubArctic, a new UI toolkit written in Java that is used by developers around the world, and has developed numerous Java applications. Ian has taught courses about Java and has lectured extensively on the subject. He has experience working on Java-related technologies at Xerox PARC and JavaSoft. Related Tutorials 2 User Interface Design for the World Wide Web (Saturday evening) 4 Developing Collaborative Applications Using the World Wide Web "Shell" (Sunday) From rvirzi@gte.com Tue Nov 19 13:19:53 1996 Received: from ns.gte.com (ns.gte.com [132.197.8.9]) by mail.acm.org (8.7.5/8.7.5) with ESMTP id NAA18054 for ; Tue, 19 Nov 1996 13:19:49 -0500 Received: from [132.197.104.26] by ns.gte.com (8.7.5/) X-Sender: rv01@pophost.gte.com Message-Id: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Date: Tue, 19 Nov 1996 13:16:50 -0500 To: (Recipient list suppressed) From: rvirzi@gte.com (Robert A. Virzi) Subject: CHI97 META 1185 - 1 Status: RO Dear CHI Reviewer- This and the next message contain my meta review and each of the reviewers individual comments on this paper. The CHI program committee asked me to distribute these as part of a feedback process for reviewers. You will be getting additional feedback from the CHI office, including a list of accepted papers. Please treat this information as confidential, as not all of the authors may be aware of their acceptance status at this time. If you have specific questions, feel free to contact me. Thank you very much for all your help in selecting the best papers for CHI and for the thoughtful comments we will be passing back to the authors. -Bob Virzi +++++++++++++ Meta reviewer: +++++++++++++++++++++++ ++++++++++++++++++++++++++++++++++++++++++++++++++++++ ++++++ CHI97 Meta-Review Form-Begin Banner ++++++++++ +++++++ INCLUDE AT THE TOP OF EACH FORM ++++++++++++++ ++++++++++++++++++++++++++++++++++++++++++++++++++++++ PAPER NUMBER {: 1185 } PAPER TITLE {: A empirical comparison of three usability methods } AUTHORS {: Frokjaer }1. Briefly SUMMARIZE the paper {: The paper reports a study comparing three usability methods, cognitive walkthrough, heuristic evaluation, and thinking aloud. }2. Summarize the reviewers' comments on the IMPORTANCE OF THE PROBLEM underlying this paper (e.g., significance, methodology, relevance to CHI). {: This is a critical problem. Experimental comparisons of methods has the potential of big paybacks for the field. }3. Summarize the reviewers' comments on the QUALITY OF THE ARGUMENT in this paper (e.g. completeness, focus, rationale, data analysis, connections to related work). {: None of the reviewers felt the paper was without flaws. The flaws group into three major categories. (1) Methodological problems. These include throwing out data, confounded conditions, subjective data massaging without IRR scores, and other basic methodological flaws were detected. (2) Analysis problems. Many differences are discussed, few are tested. Of the few that are tested, it is hard to tell how the tests were performed. For example, degrees of freedom are not reported. (3) Validity problems. Students are treated as both naive participants and expert evaluators. Lists of problems are constructed without justification. Practice effect not controlled for. }4. Summarize the reviewers' comments on the QUALITY OF THE MECHANICS of this paper (e.g., writing clarity, level of detail, figures). {: The paper needs the help of a copy editor. }5. Provide YOUR OWN ASSESSMENT of the paper, in particular, any issues you wish to raise that the reviewers did not. {: This paper has far too many methodological flaws to consider publishing in the CHI proceedings. My detailed comments appear below. The authors describe a convoluted set of tasks that they had undergraduate computer science students perform. They alternately claim that the students are experts, hence good subjects for a heuristic evaluation, and naive, hence good as thinking aloud subjects. Moreover, the same subjects fill in many of the conditions without controlling for practice effects. Many confounds appear in the design. The authors throw out lots of data from the student evaluations, and then rate the remaining data without deriving reliability measures. Some differences in the data are examined using t-tests, others are not, but are still treated as real. Moreover, although many t-tests are done on the same data, no bonferonni or similar corrections are made. Relevant work not included: This paper has a great degree of overlap with a 1993 paper (or so) by Virzi, Sorce, and Herbert that appeared in HFES conf proc. Comparing the current work to this would have been useful. Work of Smilowitz, Darnell, and Benson regarding lab testing, beta testing and forum testing is relevant to this work, and not cited. Also, Prail and colleagues should be referenced. These comparisons need to be drawn because the current work's results are inconsistent with earlier work. Why? It is interesting that the authors call thinking aloud an empirical method. I believe others who have written in this area (Karat) would disagree, as TA does not collect empirical traditional measures, namely time and accuracy. "Both individual and group oriented effects are included in this study, but not reported here." This is bothersome. Why aren't they reported? Are they confounds in the study? The authors say that TA used after HE is more effective than ... TA alone. Well, since they have the same people doing both evaluations, how do they know it is not just a learning effect or a practice effect? The experimental procedure section is very difficult to read. I cannot tell what they did with any certainty. This is partly a language problem, I think. However, from what I can tell, the study has many confounds. Why is the data from Elm tossed away? In scoring the LUPs, they authors discount (throw out) about 1/3 of all the usability problems identified by the students. Post hoc data tossing is another experimental no-no. LUPs were compared to the A-LUP to score the experimental conditions. The authors do not say who did this. More importantly, they do not say how it was done. Were they done blind? Were any reliability statistics calculated? Basically, how do we know that the study is replicable? }6. Give the average of the reviewers acceptability rating {: 2.8 without revision }7. Using the scale [1.0,5.0] 1.0 = Definitely reject 2.0 = Probably reject 3.0 = Could go either way 4.0 = Probably accept 5.0 = Definitely accept provide a numeric RECOMMENDATION for this paper. Please base this on the original reviewers' ratings, weighting individual reviewer's recommendations, if needed, so that the assessments you consider most important have the most influence. (If you believe the original average rating accurately assesses this paper, please copy it here). {: 1 }7. Describe your RATIONALE for the recommendation in question 6. How did you adjust the weights of the original reviews? What factors described above did you weight the most heavily? What factors did you discount? {: This paper has too many methodological problems. One cannot tell which, if any, results are reliable. }8. Put here any COMMENTS you want the committee to see, but NOT the AUTHORS of the paper {: } +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ++++++++++++++ CHI96 Meta-Review Form-End Banner +++++++++++++++ ++++++++++++++ INCLUDE AT THE BOTTOM OF EACH FORM +++++++++++++++ ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +++++++++++ CHI97 Paper Review Form-Begin Banner +++++++++++ +++++++++++++ INCLUDE AT THE TOP OF EACH REVIEW +++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ All your input should appear after a curly bracket followed by a colon. PAPER NUMBER {:1185 } PAPER TITLE {: An Empirical Comparison of Three Usability Evaluation Methods } REVIEWER NUMBER {: }1. Briefly SUMMARIZE the paper {: A comparison of cognitive walkthrough and heuristic evaluation followed by thinking aloud methods in the detection of usability problems, severity of those problems, detection of 'false positives' and evaluator satisfaction in a text retrieval system. }2. What TYPE OF SUBMISSION is appropriate for this paper (see bottom of Cover Sheet Two for the authors' selection)? If your assessment of type differs from the Authors, please explain why you made your selection. In the next two questions, use the criteria specific to this Type of Submission (see above) to assess the paper. {:Paper }3. Describe the IMPORTANCE OF THE PROBLEM, addressing the following concerns: Is it new and significant? Is it a useful and relevant problem? Do they sufficiently motivate the problem? Did you learn anything? Does the issue apply to CHI? {: The problem they have investigated is an interesting one. It is not new, but they have put a new twist on it by attempting to investigate the effect of using two evaluation methods on the same user interface. In theory, this would be a useful problem to study in that it has relevance to practitioners interested in how to maximize the effectiveness of a usability evaluation program. Starting the paper, I thought I would learn something new from it, but by the end, I feel as though methodological problems (and perhaps problems in communication) led me to doubt whether my learnings would really be useful. If some of the methodologies and results are clarified, this paper can be useful for the CHI community. }4. Describe the QUALITY OF THE ARGUMENT, addressing the following: Does the analysis use appropriate methodology? Is the argument well-structured? Do they provide sufficient data and/or well-supported arguments? Do they cite relevant work? Is the paper appropriately focused? Does the analysis cover all the important issues at the appropriate level of detail? {:Overall the methodology was good. The authors probably could have provided less information up front about the overall experiment design due to the fact that results & analysis were only provided for the evaluation if the TeSS system. The methodology chosen is appropriate to the question they are trying to answer but there are isolated weaknesses in their methods that are a concern. It is unclear to the reader how exactly the A-LUP was constructed. The authors do not describe the method they use for compiling the information from all 51 evaluators into the A-LUP or the ways in which they come up with severity ratings for each problem (averages? expert opinions of the authors?). This is a major problem because all statistics are done based on this A-LUP. A related issue is the comparison of evaluator severity ratings to those of the authors. The authors do not describe the method used in order to compile the authors estimates of problem severity. The work cited is appropriate to the problem they are investigating. There are examples, however of questionable conclusions from comparisons with others' work. For example, the authors discuss Jeffries et al and note that their results differ from Jeffries' results in whether or not usability tests and cognitive walkthroughs were the most expensive methods. But prior to this comparisons the authors state that "the specific form of the usability testing in the study by Jeffries et al. is not identified by references or described in the paper, but presumably it had similarities with our thinking aloud tests." This seems to be a big leap and I would discount any conclusions based on such a lightweight presumption of similarity. These methods could actually have differed greatly so as to make the comparison invalid. Overall the paper is appropriately focused and covers a good amount of ground with respect to the data reported and analysis done on the data. Other isolated comments: the number of groups for each method is different in the 'Experimental Procedure' section than it is in Table III. The introduction of LUP says LUB instead. It is unclear in Table IV whether the data reported are for CW or HE *combined* with TA or for TA alone when it follows CW or HE. } 5. Describe the QUALITY OF THE MECHANICS, addressing the following: Do they clearly describe what was done and/or how it was studied? Is the writing clear and concise? Do they provide the right level of detail? Do they use too much jargon? Do the figures support the text? {: The writing of the paper was often grammatically flawed and difficult to get through. This may simply be a matter of getting a native English speaker to edit the paper, but something certainly needs to be done to 'clean it up' in this regard. There are multiple examples of non-sentences and sentences that do not make sense. The communication of which effects were statistically significant and which were not was not consistent. I often found myself wondering if an effect was significant and if so, where did post-hocs say the differences lay? Additionally, the authors should pick a consistent terminology for significance (instead of using 'significant' and 'diverge') Other examples of ambiguity in this area: significance was not reported in the 'Revealing the problem severity' section and at what level were differences significant in table VIIIa or Table X? }6. Provide any OTHER COMMENTS you believe would be useful to the author (including pointers to missing relevant work). {:The discussion should be more focused and directed. It seems to lack structure and solid conclusions or have a logical flow. }7. Using the scale 1 = Definitely reject 2 = Probably reject 3 = Could go either way 4 = Probably accept 5 = Definitely accept provide a numerical rating of the paper's ACCEPTABILITY {:4 }8. SUMMARIZE your ASSESSMENT of the paper, pointing out which aspects described above you weighted most heavily in your rating in question 7? {:Overall the paper could be good but it needs some work done first (as described above). With these clarification and 'clean ups' I think the paper could offer something to the CHI community. I weighted my answer to 7 on the fact that the premise of the paper is good and the basic methodology is good. A lot of the improvements that need to be made are in clarification of methods, reporting of results, tightening of the discussion and heavy editing for grammar and usage. }9. Using the scale 1 = Very unconfident, really just a guess 2 = Rather unconfident, but I know a bit 3 = Moderately confident, I know as much as most 4 = Pretty confident, I know this area well 5 = Extremely confident, I consider myself an expert rate your EXPERTISE in the area addressed by the paper. {:3 }10. Use this space to provide comments that you feel are relevant to the review process but that you do NOT want FORWARDED to the author(s) {: } +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ++++++++++++ CHI97 Paper Review Form-End Banner ++++++++++++ ++++++++++++ INCLUDE AT THE BOTTOM OF EACH FORM +++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +++++++++++ CHI97 Paper Review Form-Begin Banner +++++++++++ +++++++++++++ INCLUDE AT THE TOP OF EACH REVIEW +++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ All your input should appear after a curly bracket followed by a colon. PAPER NUMBER {: 1185 } PAPER TITLE {: An empirical comparison of three usability evaluation methods } REVIEWER NUMBER {: 0174 }1. Briefly SUMMARIZE the paper {: This paper compares three usability evaluation methods: Cognitive Walkthrough (CW), Heuristic Evaluation (HE) and Thinking Aloud (TA). 102 evaluators, third year computer science students, used the three methods and the two first in combination with TA to evaluate three computer programs (only one is reported in this study). HE and TA was the best alternative, revealing 25% of all usability problems diagnosed by the researchers. This result was explained by smaller over-lap between these two methods than between CW and TA. The HE instructions were the most difficult to understand measured in time in by the evaluators' evaluations of the three methods. The difference in results between this study and other studies are partly explained in terms of differences in the evaluators' degree of expertise and the tasks the products evaluated were used for. }2. What TYPE OF SUBMISSION is appropriate for this paper (see bottom of Cover Sheet Two for the authors' selection)? If your assessment of type differs from the Authors, please explain why you made your selection. In the next two questions, use the criteria specific to this Type of Submission (see above) to assess the paper. {: Empirical }3. Describe the IMPORTANCE OF THE PROBLEM, addressing the following concerns: Is it new and significant? Is it a useful and relevant problem? Do they sufficiently motivate the problem? Did you learn anything? Does the issue apply to CHI? {: Very important problem. More research is need in this area. The issue is central to HCI. I learnt new things through reading the study. }4. Describe the QUALITY OF THE ARGUMENT, addressing the following: Does the analysis use appropriate methodology? Is the argument well-structured? Do they provide sufficient data and/or well-supported arguments? Do they cite relevant work? Is the paper appropriately focused? Does the analysis cover all the important issues at the appropriate level of detail? {: The paper tries to take a larger grip of the situation than done in many previous studies by including also "false problems", "gradation of problems" and estimated costs of repair for real and false problems. It would be a good thing if the authors gave a short description of each of the methods at the beginning of the paper. The analysis appears to be using appropriate methodology and the argument is well-structured and sufficient, but the paper is too long, partly due to that the writing is too verbose. The paper does well at covering the important issues at the appropriate level of detail. P. 7 col 2 When commenting on the Coumo and Bowen paper the authors ought to make clear how the difference between their evaluators and those of Coumo and Bowen leads to the difference in results found. } 5. Describe the QUALITY OF THE MECHANICS, addressing the following: Do they clearly describe what was done and/or how it was studied? Is the writing clear and concise? Do they provide the right level of detail? Do they use too much jargon? Do the figures support the text? {:No figures in this paper. Is the writing clear and concise? The writing is too verbose. It is difficult to point out specific places but in general I feel that the paper could be made 1-2 pages shorter without loss of too many of the important details. Some sentences are hard to understand (for ex p3 col 2, 9 lines from below "they were equally blank" or p. 4 col 1, line 13-14 "A thinking Aloud experiment was made by three tests that were summarized into one report"). P. 6 col 2, Table X.1. The row with the headings to this Table (starting "Method HE KG ..." ) needs explanation. P.8 col 1 line 15-16 unclear. }6. Provide any OTHER COMMENTS you believe would be useful to the author (including pointers to missing relevant work). {: Reference list not written in a standardized way. }7. Using the scale 1 = Definitely reject 2 = Probably reject 3 = Could go either way 4 = Probably accept 5 = Definitely accept provide a numerical rating of the paper's ACCEPTABILITY {: 4 if revision possible, 3 if not. }8. SUMMARIZE your ASSESSMENT of the paper, pointing out which aspects described above you weighted most heavily in your rating in question 7? {: The writing would benefit by some clarification and shortening }9. Using the scale 1 = Very unconfident, really just a guess 2 = Rather unconfident, but I know a bit 3 = Moderately confident, I know as much as most 4 = Pretty confident, I know this area well 5 = Extremely confident, I consider myself an expert rate your EXPERTISE in the area addressed by the paper. {: 4 }10. Use this space to provide comments that you feel are relevant to the review process but that you do NOT want FORWARDED to the author(s) {: } +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ++++++++++++ CHI97 Paper Review Form-End Banner ++++++++++++ ++++++++++++ INCLUDE AT THE BOTTOM OF EACH FORM +++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ rvirzi@gte.com Think Globally. === +1(617)466-2881 === Act Locally! From rvirzi@gte.com Tue Nov 19 13:20:15 1996 Received: from ns.gte.com (ns.gte.com [132.197.8.9]) by mail.acm.org (8.7.5/8.7.5) with ESMTP id NAA14746 for ; Tue, 19 Nov 1996 13:20:08 -0500 Received: from [132.197.104.26] by ns.gte.com (8.7.5/) X-Sender: rv01@pophost.gte.com Message-Id: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Date: Tue, 19 Nov 1996 13:16:48 -0500 To: (Recipient list suppressed) From: rvirzi@gte.com (Robert A. Virzi) Subject: CHI97 META 1185 - 2 Status: RO +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +++++++++++ CHI97 Paper Review Form-Begin Banner +++++++++++ +++++++++++++ INCLUDE AT THE TOP OF EACH REVIEW +++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ All your input should appear after a curly bracket followed by a colon. PAPER NUMBER {: 1185 } PAPER TITLE {: An empirical comparison of three usability evaluation methods } REVIEWER NUMBER {: 293 }1. Briefly SUMMARIZE the paper {: 102 students evaluated one or two systems using an inspection method or empirical evaluation followed by empirical evaluation. Students also provided ratings about the methods. The authors pooled all the problems found and analysed the ability of the different methods to find usability problems of different severities. }2. What TYPE OF SUBMISSION is appropriate for this paper (see bottom of Cover Sheet Two for the authors' selection)? If your assessment of type differs from the Authors, please explain why you made your selection. In the next two questions, use the criteria specific to this Type of Submission (see above) to assess the paper. {: EMPIRICAL Methodology }3. Describe the IMPORTANCE OF THE PROBLEM, addressing the following concerns: Is it new and significant? Is it a useful and relevant problem? Do they sufficiently motivate the problem? Did you learn anything? Does the issue apply to CHI? {: The authors provide a nice little section on what's new and important. I think that understanding the effectiveness of combined methods is highly relevant to actual practice, where practitioners usually apply inspection methods before testing users. I thought that the collection of evaluator satisfaction ratings was of interest, although when gathered from student evaluators, I am not confident the data have much to offer. Because I had problems understanding what data were being presented, I am not sure if I learned anything -- I kept on wanting to see their data because I could not follow their analyses. }4. Describe the QUALITY OF THE ARGUMENT, addressing the following: Does the analysis use appropriate methodology? Is the argument well-structured? Do they provide sufficient data and/or well-supported arguments? Do they cite relevant work? Is the paper appropriately focused? Does the analysis cover all the important issues at the appropriate level of detail? {: The literature review and contrast is done well. The argument could be motivated more by a theoretical basis than they used -- for example, at the very end they note that there are more similarities between Cognitive Walkthroughs and Think Aloud than between Heuristic Evaluation and Think Aloud. This might have been stated as a hypothesis to make a prediction, but the work was more presented as: let's compare the methods in combination and see how they do, without a theoretical motivation (which is not all that bad). I have serious reservations about the validity of the experiment because of the subject population -- students, and what they evaluated (only the results for the text retrieval system were reported, I think). So we have non-experts in evaluation and in the domain, which might still be of interest if their data were compared to double-expert evaluators. Instead, the student data is compared to the pooled student data, which itself is not validated to a great extent (the authors make some judgements on the ratings of severity, as well as deleted data for some types of ratings (another indication of the lack of experience of the students)). My experience, which is supported in published studies, is that novice evaluators identify many non-problems (that do not affect performance) and sometimes miss important ones, which makes me even more wary of comparing group evaluations to the union of the evaluations. I was left wondering how well the students followed the methods. For example, what tasks/scenarios were designed (and by whom)? What forms did the student use to gather data? The use of the students themselves as SUBJECTS (as well as evaluators) makes me uneasy about the realism of the results. Even if a student is unfamiliar with a tool, the fact that they may be in the process of doing a usability inspection of another tool, and will soon be doing or just did a thinking aloud evaluation of another tool, introduces many variables that make me wonder what the data mean. Any study like this one has a lot of ad hoc data manipulation, and this is no exception. Many readers would be uncomfortable with the idea that 34+14=48 of 155 problems are removed by the authors. The authors do make some attempts to analyse the false and neutralized problems separately, however. The data analysis generally lacked rigor. I think one t-test was mentioned, but many of their tests required something to test more than two conditions. No degrees of freedom were reported, which would have helped me understand what comparisons were actually being made -- in most cases, I was not sure. No t-values or significance levels were reported, and there were few measures of variation reported. The analyses in the tables could have been clearer, especially in the discussion of MARGINAL differences of problems found with TA. For example, Tables II and IV could be combined to highlight the increase in % problems found: CW-TA = CW + 5% HE-TA = HE + 6% Perhaps remarkably, but certainly not significantly, CW-TA = TA - 3% The scale used for satisfaction was unusual, ranging from -100 (disagree) to 100 (agree). a 200-point scale is far more than the number of distinctions that can be made. The questions were worded in both positive and negative forms, making the ratings difficult for the reader and possibly confusing to the rater. For example, 2. The procedure of the method was unclear. CW=-53 HE=25 TA=-70 means that TA was very not unclear, and CW was somewhat not unclear but HE was a little unclear. I would like to see standard errors (or standard deviations and Ns) for the ratings. } 5. Describe the QUALITY OF THE MECHANICS, addressing the following: Do they clearly describe what was done and/or how it was studied? Is the writing clear and concise? Do they provide the right level of detail? Do they use too much jargon? Do the figures support the text? {: I'd prefer to read actual percentages or values instead of de-quantified references like "One fourth" and "one third" or "approximately the same total number". Give me the numbers and I'll decide if they have the inferred properties. The first abbreviation of the cryptic LUP was LUB. I found this abbreviation a little too much jargony, but the use of acronyms in the paper is probably better than longer terms. This paper would need an English-speaking editor to fix the grammar in about 20 places. Reformat Table XI with the ratings so that things line up and so that the question numbers are in their own column. Avoid the term Diverge to indicate statistical significance; indicate the significance level instead. Indicate which pairs of values are significantly different. It was confusing that the authors advertised the 102 number and then noted that one some of the data were to be presented. It would be easier to understand if the data actually used were reported throughout, and the authors' other publishing plans be placed in a footnote. The experimental conditions could have been much better described, such as by a diagram showing how many groups were in one condition and then another. From the procedure, I could tell which tools the groups evaluated, but I could not understand the design of the control condition to TA; I could only assume that it was as I hoped. The description is especially frustrating because only the data from one tool is presented, so much of the complexity is there for no reason. The statement "heuristic evaluation has invited the evaluators to take a broader scope of awareness upon the usability problems" should probably be replaced by a quantified statement like "evaluators using Heuristic Evaluation found 19% of the problems compared to 10% for Cognitive Walkthroughs". It's more accurate, although it does highlight the extremely low percentage of problems found by the double-non-expert evaluators. Reference 18 has a typo: "og" }6. Provide any OTHER COMMENTS you believe would be useful to the author (including pointers to missing relevant work). {: I'd like to see a paper on the class organization process, maybe for the SIGCHI Bulletin in the education section. Contact Andrew Sears, the Education Editor. }7. Using the scale 1 = Definitely reject 2 = Probably reject 3 = Could go either way 4 = Probably accept 5 = Definitely accept provide a numerical rating of the paper's ACCEPTABILITY {: 2 }8. SUMMARIZE your ASSESSMENT of the paper, pointing out which aspects described above you weighted most heavily in your rating in question 7? {: The subject population creates some validity questions that will not go away. As a case study in education, I would give this a higher rating, but as a study to guide practitioners, I am not confident about the source (subjects) of the data (double-non-experts). The experimental procedure needs some clarification, and the data analysis needs to be made more rigorous. On the positive side, it's a lot of data with some interesting twists, so I think it's worth pursuing. I think the authors have to be very careful about generalizing the results beyond the scope of the study, though. }9. Using the scale 1 = Very unconfident, really just a guess 2 = Rather unconfident, but I know a bit 3 = Moderately confident, I know as much as most 4 = Pretty confident, I know this area well 5 = Extremely confident, I consider myself an expert rate your EXPERTISE in the area addressed by the paper. {: 4.5 }10. Use this space to provide comments that you feel are relevant to the review process but that you do NOT want FORWARDED to the author(s) {: } +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ++++++++++++ CHI97 Paper Review Form-End Banner ++++++++++++ ++++++++++++ INCLUDE AT THE BOTTOM OF EACH FORM +++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +++++++++++ CHI97 Paper Review Form-Begin Banner +++++++++++ +++++++++++++ INCLUDE AT THE TOP OF EACH REVIEW +++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ All your input should appear after a curly bracket followed by a colon. PAPER NUMBER {:1185 } PAPER TITLE {:An Empirical Comparison of Three Usability Evaluation Methods } REVIEWER NUMBER {: 288 }1. Briefly SUMMARIZE the paper {:This paper compares three usability evaluation, although the principle comparison is between two of them, cognitive walkthrough and heuristic evaluation. Computer science students learned the methods and used them to evaluate several systems. A large number of measures were used in comparing the methods. In general heuristic evaluation looked better. }2. What TYPE OF SUBMISSION is appropriate for this paper (see bottom of Cover Sheet Two for the authors' selection)? If your assessment of type differs from the Authors, please explain why you made your selection. In the next two questions, use the criteria specific to this Type of Submission (see above) to assess the paper. {:empircal }3. Describe the IMPORTANCE OF THE PROBLEM, addressing the following concerns: Is it new and significant? Is it a useful and relevant problem? Do they sufficiently motivate the problem? Did you learn anything? Does the issue apply to CHI? {:This is by now an established area of CHI research, and this contribution must be assessed in relation to the existing literature. To me the primary innovation is the wider range of performance measures that were examined. The usefulness of these methods is of course a major CHI concern. }4. Describe the QUALITY OF THE ARGUMENT, addressing the following: Does the analysis use appropriate methodology? Is the argument well-structured? Do they provide sufficient data and/or well-supported arguments? Do they cite relevant work? Is the paper appropriately focused? Does the analysis cover all the important issues at the appropriate level of detail? {:In general, OK. The biggest weakness in the paper (and one shared with a number of the earlier papers in this genre) is the use of relatively inexperienced students as subjects in the study. I of course understand the practical issues involved in getting a large number of subjects. But the experience of the evaluators is clearly an important issue. The report is very detailed. Many aspects of the "success" and "acceptance" of the evaluation methods are examined. This is useful in that there are tradeoffs involved, such as finding true interface problems versus false ones, or the time taken to learn or use a method. One concern in these kinds of studies is how well the subjects learned the methods. The authors discuss this, and qualify some of their results because of concerns about how the teaching worked. } 5. Describe the QUALITY OF THE MECHANICS, addressing the following: Do they clearly describe what was done and/or how it was studied? Is the writing clear and concise? Do they provide the right level of detail? Do they use too much jargon? Do the figures support the text? {:The paper is reasonably readable to both the CHI researcher and practitioner. Some of the details of the data presentation seem excessive, and if the paper is accepted I'd recommend a more concise presentation of the results. }6. Provide any OTHER COMMENTS you believe would be useful to the author (including pointers to missing relevant work). {: }7. Using the scale 1 = Definitely reject 2 = Probably reject 3 = Could go either way 4 = Probably accept 5 = Definitely accept provide a numerical rating of the paper's ACCEPTABILITY {:4 }8. SUMMARIZE your ASSESSMENT of the paper, pointing out which aspects described above you weighted most heavily in your rating in question 7? {:While this study has limitations it does provide some additional information for the growing body of evidence about the relative efficacy of usability evaluation methods. }9. Using the scale 1 = Very unconfident, really just a guess 2 = Rather unconfident, but I know a bit 3 = Moderately confident, I know as much as most 4 = Pretty confident, I know this area well 5 = Extremely confident, I consider myself an expert rate your EXPERTISE in the area addressed by the paper. {:4 }10. Use this space to provide comments that you feel are relevant to the review process but that you do NOT want FORWARDED to the author(s) {: } +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ++++++++++++ CHI97 Paper Review Form-End Banner ++++++++++++ ++++++++++++ INCLUDE AT THE BOTTOM OF EACH FORM +++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +++++++++++ CHI97 Paper Review Form-Begin Banner +++++++++++ +++++++++++++ INCLUDE AT THE TOP OF EACH REVIEW +++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ All your input should appear after a curly bracket followed by a colon. PAPER NUMBER {: 1185 }: PAPER TITLE {An Empirical Comparison of Three Usability Evaluation Methods } REVIEWER NUMBER {:184 }1. Briefly SUMMARIZE the paper {:This paper evaluates the effectiveness of three usability tests, alone and in combination. The authors argue for combining Heuristic Evaluation and Thinking Aloud usability methods.: }2. What TYPE OF SUBMISSION is appropriate for this paper (see bottom of Cover Sheet Two for the authors' selection)? If your assessment of type differs from the Authors, please explain why you made your selection. In the next two questions, use the criteria specific to this Type of Submission (see above) to assess the paper. {:paper }3. Describe the IMPORTANCE OF THE PROBLEM, addressing the following concerns: Is it new and significant? Is it a useful and relevant problem? Do they sufficiently motivate the problem? Did you learn anything? Does the issue apply to CHI? {:Identifying adequate, effective methods for evaluating the usability of computer systems is an important problem and warrants careful study. }4. Describe the QUALITY OF THE ARGUMENT, addressing the following: Does the analysis use appropriate methodology? Is the argument well-structured? Do they provide sufficient data and/or well-supported arguments? Do they cite relevant work? Is the paper appropriately focused? Does the analysis cover all the important issues at the appropriate level of detail? {:The findings of this study are so overburdened by questions of experimental design that it is difficult to assess their validity. The outcomes are highly dependent on who the evaluators were (computer science students), how authorized usability problems were determined (by the authors), limitations imposed by sample size (even though larger than in many comparable studies), order in which combined tests were administered, etc. It would have been helpful to learn more about the experiences of the evaluators in using the various methods. It also would have been helpful if the authors had provided examples of the particular problems uncovered by the each method. As a reader I felt very detached from the systems evaluated, the methods used, the experiences of the evaluators, and the problems uncovered. } 5. Describe the QUALITY OF THE MECHANICS, addressing the following: Do they clearly describe what was done and/or how it was studied? Is the writing clear and concise? Do they provide the right level of detail? Do they use too much jargon? Do the figures support the text? {:Although I had no problem understanding the text, it is apparent that the authors' first language is not English. It would not take too much effort to correct the more serious errors.: }6. Provide any OTHER COMMENTS you believe would be useful to the author (including pointers to missing relevant work). {: }7. Using the scale 1 = Definitely reject 2 = Probably reject 3 = Could go either way 4 = Probably accept 5 = Definitely accept provide a numerical rating of the paper's ACCEPTABILITY {:2 }8. SUMMARIZE your ASSESSMENT of the paper, pointing out which aspects described above you weighted most heavily in your rating in question 7? {:Findings are more the result of experimental design than they reflect differences in evaluation methods: }9. Using the scale 1 = Very unconfident, really just a guess 2 = Rather unconfident, but I know a bit 3 = Moderately confident, I know as much as most 4 = Pretty confident, I know this area well 5 = Extremely confident, I consider myself an expert rate your EXPERTISE in the area addressed by the paper. {:2 }10. Use this space to provide comments that you feel are relevant to the review process but that you do NOT want FORWARDED to the author(s) {: } +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ++++++++++++ CHI97 Paper Review Form-End Banner ++++++++++++ ++++++++++++ INCLUDE AT THE BOTTOM OF EACH FORM +++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ++++++++++++ CHI97 Paper Review Form-Begin Banner +++++ +++++++++++++ INCLUDE AT THE TOP OF EACH REVIEW ++++++ ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ All your input should appear after a curly bracket followed by a colon. PAPER NUMBER {:1185 } PAPER TITLE {: An empirical comparison of three usability evaluation methods } REVIEWER NUMBER {: 233 }1. Briefly SUMMARIZE the paper {: Compares and contrasts three evaluation techniques. Study used 102 computer science students as evaluators. }2. What TYPE OF SUBMISSION is appropriate for this paper (see bottom of Cover Sheet Two for the authors' selection)? If your assessment of type differs from the Authors, please explain why you made your selection. In the next two questions, use the criteria specific to this Type of Submission (see above) to assess the paper. {: Empirical and methodology }3. Describe the IMPORTANCE OF THE PROBLEM, addressing the following concerns: Is it new and significant? Is it a useful and relevant problem? Do they sufficiently motivate the problem? Did you learn anything? Does the issue apply to CHI? {: This paper forms part of a current active area in HCI- evaluating evaluation techniques. This is a vital task of critical importance to our discipline. It is certainly CHI; it is important; it is current rather than new. The outcomes are interesting without being startling. }4. Describe the QUALITY OF THE ARGUMENT, addressing the following: Does the analysis use appropriate methodology? Is the argument well-structured? Do they provide sufficient data and/or well-supported arguments? Do they cite relevant work? Is the paper appropriately focused? Does the analysis cover all the important issues at the appropriate level of detail? {: In general the paper's argument reads well. Some further work could be cited (eg. Henderson paper that appeared in BIT recently). The rationale for choosing the methods they did seems reasonable. } 5. Describe the QUALITY OF THE MECHANICS, addressing the following: Do they clearly describe what was done and/or how it was studied? Is the writing clear and concise? Do they provide the right level of detail? Do they use too much jargon? Do the figures support the text? {: I would like to have seen more consideration of the data analysis- which in this case is non-trivial. For example, the problems found are expressed as a percentage of all problems in the system. How can we know what all the problems were? I don't believe we ever can. }6. Provide any OTHER COMMENTS you believe would be useful to the author (including pointers to missing relevant work). {: There are some small problems with language and expression that could be tightened up through the support of a mentor. Occasionally slips into colloquialism (eg. 'They were equally blank'). }7. Using the scale 1 = Definitely reject 2 = Probably reject 3 = Could go either way 4 = Probably accept 5 = Definitely accept provide a numerical rating of the paper's ACCEPTABILITY {: 3 }8. SUMMARIZE your ASSESSMENT of the paper, pointing out which aspects described above you weighted most heavily in your rating in question 7? {: There are problems with the methodology- there always will be with such an ambitious study- but not in my view problems that are fatal. I think the outcome is interesting and the objective vital. If the paper is not accepted for CHI I urge the authors to resubmit to a journal where the additional space would allow them to deal in more detail with methodological issues. }9. Using the scale 1 = Very unconfident, really just a guess 2 = Rather unconfident, but I know a bit 3 = Moderately confident, I know as much as most 4 = Pretty confident, I know this area well 5 = Extremely confident, I consider myself an expert rate your EXPERTISE in the area addressed by the paper. {: 4 }10. Use this space to provide comments that you feel are relevant to the review process but that you do NOT want FORWARDED to the author(s) {: } +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ++++++++++++ CHI97 Paper Review Form-End Banner ++++++++ ++++++++++++ INCLUDE AT THE BOTTOM OF EACH FORM +++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ rvirzi@gte.com Think Globally. === +1(617)466-2881 === Act Locally! From rvirzi@gte.com Tue Nov 19 13:22:47 1996 Received: from ns.gte.com (ns.gte.com [132.197.8.9]) by mail.acm.org (8.7.5/8.7.5) with ESMTP id NAA15694 for ; Tue, 19 Nov 1996 13:22:40 -0500 Received: from [132.197.104.26] by ns.gte.com (8.7.5/) X-Sender: rv01@pophost.gte.com Message-Id: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Date: Tue, 19 Nov 1996 13:18:16 -0500 To: (Recipient list suppressed) From: rvirzi@gte.com (Robert A. Virzi) Subject: CHI97 META 1109 - 1 Status: RO Dear CHI Reviewer- This and the next message contain my meta review and each of the reviewers individual comments on this paper. The CHI program committee asked me to distribute these as part of a feedback process for reviewers. You will be getting additional feedback from the CHI office, including a list of accepted papers. Please treat this information as confidential, as not all of the authors may be aware of their acceptance status at this time. If you have specific questions, feel free to contact me. Thank you very much for all your help in selecting the best papers for CHI and for the thoughtful comments we will be passing back to the authors. -Bob Virzi ++++++++++++++ Meta reviewer: +++++++++++++++++++++++ ++++++++++++++++++++++++++++++++++++++++++++++++++++++ ++++++ CHI97 Meta-Review Form-Begin Banner ++++++++++ +++++++ INCLUDE AT THE TOP OF EACH FORM ++++++++++++++ ++++++++++++++++++++++++++++++++++++++++++++++++++++++ PAPER NUMBER {: 1109 } PAPER TITLE {: Speech browsing the WWW } AUTHORS {: Borges }1. Briefly SUMMARIZE the paper {: An experiment is reported in which subjects carry out tasks using the WWW using either speech input or a mouse to control the UI. Speech is simulated using the wizard of Oz technique. Performance and preference was measured. }2. Summarize the reviewers' comments on the IMPORTANCE OF THE PROBLEM underlying this paper (e.g., significance, methodology, relevance to CHI). {: All agreed this paper addressed an important topic. }3. Summarize the reviewers' comments on the QUALITY OF THE ARGUMENT in this paper (e.g. completeness, focus, rationale, data analysis, connections to related work). {: Reviewers were split right down the middle on this paper. Of the 7 reviewers, this paper received three ratings of 5. It also had two 1s and a 2. I will first summarize the positive aspects of the paper and then move to the negative criticisms. All of the reviewers thought the combination of speech access and the WWW was a timely topic and relevant to CHI. This is an example of a multi-modal interface, also relevant to CHI. The argument is well thought out and presented. Some relevant work is cited. The structure of the study, including counter-balancing, etc, was well done. One reviewer complimented them on NOT drawing unsupportable conclusions. The primary negative aspect of the article was that, although their study was well designed and thought out, it was completely lacking the use of inferential statistics. Three of the reviewers were very surprised that this was the case given that all the data was collected and all they had to do was analyze it. This camp of reviewers felt that none of the conclusions were supported, given the lack of appropriate statistical treatment. The authors assumed that the selected mode in experiment 2 indicated preference. Several reviewers point out that this need not necessarily be the case. One reviewer thought the combination of speech input and the web was gratuitous. That is, s/he felt other technologies could solve the problems better/faster/easier than speech. This same reviewer also felt the comparison to a non-standard browser hurt the generalizability of the results. }4. Summarize the reviewers' comments on the QUALITY OF THE MECHANICS of this paper (e.g., writing clarity, level of detail, figures). {: References are not done correctly. Check a style guide on ACM references. Some typos and odd phrases that need to be fixed. ("Human subjects" seems a bit redundant.) Table 4 is not usable as is. What is shading supposed to mean? There is no reason to retain the subject ordering in the table. Why not sort the table by decreasing percentage of commands given by mouse? This will order the table from heaviest mouse users to heaviest voice users. Alternatively, shade mouse-predominant users and leave voice-predominant users white. }5. Provide YOUR OWN ASSESSMENT of the paper, in particular, any issues you wish to raise that the reviewers did not. {: I liked the paper overall, but fall in with the group of reviewers who think the lack of statistics severely hampers this paper. I was extremely frustrated given the obviously high quality work in other aspects of the paper. They collected all the relevant data, report the means, but omit the Chi-squares and t-tests that could have given foundation to their claims. }6. Give the average of the reviewers acceptability rating {: 3.1. }7. Using the scale [1.0,5.0] 1.0 = Definitely reject 2.0 = Probably reject 3.0 = Could go either way 4.0 = Probably accept 5.0 = Definitely accept provide a numeric RECOMMENDATION for this paper. Please base this on the original reviewers' ratings, weighting individual reviewer's recommendations, if needed, so that the assessments you consider most important have the most influence. (If you believe the original average rating accurately assesses this paper, please copy it here). {: 2 }7. Describe your RATIONALE for the recommendation in question 6. How did you adjust the weights of the original reviews? What factors described above did you weight the most heavily? What factors did you discount? {: In the end, I think that a paper which bears all the trappings of an experimental paper, should be treated as an experimental paper. Some of the reviewers were more forgiving, noting the deficiencies in the statistical analysis and treating it as more of an Experience paper than a Experimental paper. Given that it can't be re-written, and it appears in the form of an experimental paper, I think we have to reject it. The authors should be able to add the statistics and resubmit. They have done an otherwise outstanding job. }8. Put here any COMMENTS you want the committee to see, but NOT the AUTHORS of the paper {: } +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ++++++++++++++ CHI96 Meta-Review Form-End Banner +++++++++++++++ ++++++++++++++ INCLUDE AT THE BOTTOM OF EACH FORM +++++++++++++++ ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +++++++++++ CHI97 Paper Review Form-Begin Banner +++++++++++ +++++++++++++ INCLUDE AT THE TOP OF EACH REVIEW +++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ All your input should appear after a curly bracket followed by a colon. PAPER NUMBER {:1109 } PAPER TITLE {:Speech Browsing the World-Wide Web } REVIEWER NUMBER {:124 }1. Briefly SUMMARIZE the paper {:Spoken commands are helpful and satisfying to users for browsing the WWW. }2. What TYPE OF SUBMISSION is appropriate for this paper (see bottom of Cover Sheet Two for the authors' selection)? If your assessment of type differs from the Authors, please explain why you made your selection. In the next two questions, use the criteria specific to this Type of Submission (see above) to assess the paper. {:Empirical }3. Describe the IMPORTANCE OF THE PROBLEM, addressing the following concerns: Is it new and significant? Is it a useful and relevant problem? Do they sufficiently motivate the problem? Did you learn anything? Does the issue apply to CHI? {:Useful and relevant, and very timely. Applies to CHI. }4. Describe the QUALITY OF THE ARGUMENT, addressing the following: Does the analysis use appropriate methodology? Is the argument well-structured? Do they provide sufficient data and/or well-supported arguments? Do they cite relevant work? Is the paper appropriately focused? Does the analysis cover all the important issues at the appropriate level of detail? {:Good job. Of particular note: They avoided drawing unsupportable conclusions. } 5. Describe the QUALITY OF THE MECHANICS, addressing the following: Do they clearly describe what was done and/or how it was studied? Is the writing clear and concise? Do they provide the right level of detail? Do they use too much jargon? Do the figures support the text? {:A clear description of a well-designed and well-executed study. I would have liked to see a graph of the results, rather than just tables. }6. Provide any OTHER COMMENTS you believe would be useful to the author (including pointers to missing relevant work). {:No matching phrase was ranked higher than 3 (there should have been at least one 4), and the phrase "speech and voice" was omitted. I would also recommend including a brief discussion of the issues and relevance of speech browsing to users with visual or motor disabilities or impairments. }7. Using the scale 1 = Definitely reject 2 = Probably reject 3 = Could go either way 4 = Probably accept 5 = Definitely accept provide a numerical rating of the paper's ACCEPTABILITY {:5 }8. SUMMARIZE your ASSESSMENT of the paper, pointing out which aspects described above you weighted most heavily in your rating in question 7? {:Good design; good execution; clear description. I weighted 4, 5, and 6 about the same. }9. Using the scale 1 = Very unconfident, really just a guess 2 = Rather unconfident, but I know a bit 3 = Moderately confident, I know as much as most 4 = Pretty confident, I know this area well 5 = Extremely confident, I consider myself an expert rate your EXPERTISE in the area addressed by the paper. {:3 }10. Use this space to provide comments that you feel are relevant to the review process but that you do NOT want FORWARDED to the author(s) {: } +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ++++++++++++ CHI97 Paper Review Form-End Banner ++++++++++++ ++++++++++++ INCLUDE AT THE BOTTOM OF EACH FORM +++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +++++++++++ CHI97 Paper Review Form-Begin Banner +++++++++++ +++++++++++++ INCLUDE AT THE TOP OF EACH REVIEW +++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ All your input should appear after a curly bracket followed by a colon. PAPER NUMBER {:1109 } PAPER TITLE {:Speech Browsing the World Wide Web } REVIEWER NUMBER {:117 }1. Briefly SUMMARIZE the paper {:This paper discussed a study done to determine how people would use speech to navigate the world wide web. A Wizard of Oz study was performed where users navigated the web one of three ways: speech only, mouse only, multimodal - speech and/or mouse combination. The interesting finding was that the user's choice of modality was determined by user satisfaction as opposed to efficiency of use. }2. What TYPE OF SUBMISSION is appropriate for this paper (see bottom of Cover Sheet Two for the authors' selection)? If your assessment of type differs from the Authors, please explain why you made your selection. In the next two questions, use the criteria specific to this Type of Submission (see above) to assess the paper. {:empirical }3. Describe the IMPORTANCE OF THE PROBLEM, addressing the following concerns: Is it new and significant? Is it a useful and relevant problem? Do they sufficiently motivate the problem? Did you learn anything? Does the issue apply to CHI? {:The problem is very appropriate for CHI. Many folks are interested in the area of speech and its applications and certainly navigating the web is an important one. It also discusses a multimodal interface which is becoming more prevalent. }4. Describe the QUALITY OF THE ARGUMENT, addressing the following: Does the analysis use appropriate methodology? Is the argument well-structured? Do they provide sufficient data and/or well-supported arguments? Do they cite relevant work? Is the paper appropriately focused? Does the analysis cover all the important issues at the appropriate level of detail? {:The argument is well structured. The research is well done. The data supports the arguments. Relevant work is cited. The analysis could also discuss the 100% cases in Table 4. } 5. Describe the QUALITY OF THE MECHANICS, addressing the following: Do they clearly describe what was done and/or how it was studied? Is the writing clear and concise? Do they provide the right level of detail? Do they use too much jargon? Do the figures support the text? {:The writing is clear and concise. There was little jargon. The figures (tables) support the text very well. Usually references are cited with last names only, this author used initials (i.e. R. VanBuskirk and M. Lalomia, L. Catledge and J. Pitkow). }6. Provide any OTHER COMMENTS you believe would be useful to the author (including pointers to missing relevant work). {:Results section, 3rd paragraph - two typos. 3rd sentence - test should be tests. 4th sentence - users comments should be users' comments. }7. Using the scale 1 = Definitely reject 2 = Probably reject 3 = Could go either way 4 = Probably accept 5 = Definitely accept provide a numerical rating of the paper's ACCEPTABILITY {:5 }8. SUMMARIZE your ASSESSMENT of the paper, pointing out which aspects described above you weighted most heavily in your rating in question 7? {:It is well researched. The area should be of interest to the CHI community. The methodology is good and the results are presented well. }9. Using the scale 1 = Very unconfident, really just a guess 2 = Rather unconfident, but I know a bit 3 = Moderately confident, I know as much as most 4 = Pretty confident, I know this area well 5 = Extremely confident, I consider myself an expert rate your EXPERTISE in the area addressed by the paper. {:4 }10. Use this space to provide comments that you feel are relevant to the review process but that you do NOT want FORWARDED to the author(s) {: } +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ++++++++++++ CHI97 Paper Review Form-End Banner ++++++++++++ ++++++++++++ INCLUDE AT THE BOTTOM OF EACH FORM +++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ rvirzi@gte.com Think Globally. === +1(617)466-2881 === Act Locally! From rvirzi@gte.com Tue Nov 19 13:22:57 1996 Received: from ns.gte.com (ns.gte.com [132.197.8.9]) by mail.acm.org (8.7.5/8.7.5) with ESMTP id NAA16232 for ; Tue, 19 Nov 1996 13:22:53 -0500 Received: from [132.197.104.26] by ns.gte.com (8.7.5/) X-Sender: rv01@pophost.gte.com Message-Id: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Date: Tue, 19 Nov 1996 13:18:18 -0500 To: (Recipient list suppressed) From: rvirzi@gte.com (Robert A. Virzi) Subject: CHI97 META 1109 - 2 Status: RO +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +++++++++++ CHI97 Paper Review Form-Begin Banner +++++++++++ +++++++++++++ INCLUDE AT THE TOP OF EACH REVIEW +++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ All your input should appear after a curly bracket followed by a colon. PAPER NUMBER {: 1109 } PAPER TITLE {: Speech Browsing the World Wide Web } REVIEWER NUMBER {: 293 }1. Briefly SUMMARIZE the paper {: One study with a custom WWW browser gathered stats about what commands people would use to control a Wizard of Oz controlled WWW browser by voice. People used short commands. A second study compared speech input to mouse input by having half the subjects use mouse then voice, and half use voice then mouse, followed by a set of 12 tasks in which they could use either. The results were ambiguous, but more subjects used voice than the mouse. }2. What TYPE OF SUBMISSION is appropriate for this paper (see bottom of Cover Sheet Two for the authors' selection)? If your assessment of type differs from the Authors, please explain why you made your selection. In the next two questions, use the criteria specific to this Type of Submission (see above) to assess the paper. {: EMPIRICAL Input Devices (the keywords on the WWW are largely irrelevant) }3. Describe the IMPORTANCE OF THE PROBLEM, addressing the following concerns: Is it new and significant? Is it a useful and relevant problem? Do they sufficiently motivate the problem? Did you learn anything? Does the issue apply to CHI? {: The applicability of speech input is clearly a CHI area, but the authors provide little motivation for the current study. The experiments had no hypotheses, and little information was found, certainly nothing new or significant. The results on brief commands have been known since the first Wizard of Oz studies or word processors. The results from the second experiment are muddled. }4. Describe the QUALITY OF THE ARGUMENT, addressing the following: Does the analysis use appropriate methodology? Is the argument well-structured? Do they provide sufficient data and/or well-supported arguments? Do they cite relevant work? Is the paper appropriately focused? Does the analysis cover all the important issues at the appropriate level of detail? {: I did not understand why the studies were done. The first study found that people use short commands with computers, which is not a new result. The second study had a nice counter-balanced design, but it was more of an exploratory experiment than one to test a hypothesis. The result was that the data do not clearly provide anything useful, expect for some questions. The authors write: "The most interesting result of this test is that 13/20 users used voice as their PREFERRED input modality." [caps mine] That users preferred what they used more is an unwarranted conclusion because the speech modality was novel to the subjects. People may have used it more because they found it novel, fun. It would have been useful to ask (i.e., in a questionnaire) which method was preferred, possibly for which types of commands. A set of 12 tasks is probably not enough experience, though. The discussions of possible confoundings, such as order effects and extra time for Wizard of Oz, were especially welcome. They might look at: Observations on Using Speech Input for Window Navigation Schmandt et al, Interact'90, pp. 787-793. as well as the larger literature on speech input. } 5. Describe the QUALITY OF THE MECHANICS, addressing the following: Do they clearly describe what was done and/or how it was studied? Is the writing clear and concise? Do they provide the right level of detail? Do they use too much jargon? Do the figures support the text? {: }6. Provide any OTHER COMMENTS you believe would be useful to the author (including pointers to missing relevant work). {: At the end of the second results section, just before the CONCLUSIONS, on page 8: I think that voice input would interfere cognitively more with reading than mouse (or other manual) input. See the Schmandt reference above, but also check out Gould's work and others. Add Wizard of Oz to the keywords. Avoid judgemental terms like "no-nonsense" and use more precise terms like "brief" or X % 1-2 word commands. Avoid the term "human subjects" -- it's so obvious that it seems silly. }7. Using the scale 1 = Definitely reject 2 = Probably reject 3 = Could go either way 4 = Probably accept 5 = Definitely accept provide a numerical rating of the paper's ACCEPTABILITY {: 1 }8. SUMMARIZE your ASSESSMENT of the paper, pointing out which aspects described above you weighted most heavily in your rating in question 7? {: The results are not novel. The inferences are flawed. }9. Using the scale 1 = Very unconfident, really just a guess 2 = Rather unconfident, but I know a bit 3 = Moderately confident, I know as much as most 4 = Pretty confident, I know this area well 5 = Extremely confident, I consider myself an expert rate your EXPERTISE in the area addressed by the paper. {: 5 }10. Use this space to provide comments that you feel are relevant to the review process but that you do NOT want FORWARDED to the author(s) {: } +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ++++++++++++ CHI97 Paper Review Form-End Banner ++++++++++++ ++++++++++++ INCLUDE AT THE BOTTOM OF EACH FORM +++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +++++++++++ CHI97 Paper Review Form-Begin Banner +++++++++++ +++++++++++++ INCLUDE AT THE TOP OF EACH REVIEW +++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ All your input should appear after a curly bracket followed by a colon. PAPER NUMBER {:1109 } PAPER TITLE {:Speech Browsing the World Wide Web } REVIEWER NUMBER {:282 }1. Briefly SUMMARIZE the paper {: Two empirical studies are presented concerning the use of speech as an interaction style for navigating on the WWW. The first study had 10 experienced web surfers use a speech-based Wizard of Oz prototype in a fairly unconstrained manner to analyze the types of commands and vocabulary that they would use. The findings were that a small set of simple spoken commands accounted for a majority of the commands issued with a fair amount of subject agreement. In the second study, also using the Wizard of Oz methodology, 20 subjects used a WWW browser in each of 3 conditions: speech only, mouse only, and mixed modality. While the reported results are weak and hard to interpret, people tended to settle into using one of the modalities predominantly instead of using mouse and speech in an efficient and complementary manner. The authors report (albeit without statistical support) that the users tended to favor the speech interaction even though they were more efficient (putatively) with the mouse. }2. What TYPE OF SUBMISSION is appropriate for this paper (see bottom of Cover Sheet Two for the authors' selection)? If your assessment of type differs from the Authors, please explain why you made your selection. In the next two questions, use the criteria specific to this Type of Submission (see above) to assess the paper. {:experience & empirical. The paper does not meet the standards of statistical support for an empirical paper in my opinion. Nevertheless, as an experience paper with some descriptive empirical support, I find it to be useful with a take home message about the set of commands that users are likely to issue when faced with a speech-based WWW browser. }3. Describe the IMPORTANCE OF THE PROBLEM, addressing the following concerns: Is it new and significant? Is it a useful and relevant problem? Do they sufficiently motivate the problem? Did you learn anything? Does the issue apply to CHI? {: This is an intriguing paper which stimulates ideas for further research and development. Definitely a CHI topic. The problem is fairly well motivated and has the appropriate caveats for the limitations of a speech-only interface for WWW interactions. I found the categorization of commands issued in the 1st study to be enlightening }4. Describe the QUALITY OF THE ARGUMENT, addressing the following: Does the analysis use appropriate methodology? Is the argument well-structured? Do they provide sufficient data and/or well-supported arguments? Do they cite relevant work? Is the paper appropriately focused? Does the analysis cover all the important issues at the appropriate level of detail? {:The methodology was pretty good. They appropriately counterbalanced the order of interaction modalities in exp. 2. As a research methodologist and statistician, I became a little queasy reading the results section of the second study. It is not up to acceptable reporting standards or style. The tone is somewhat meandering and imprecise, lacking any statistical support beyond the reporting of means and percentages. Claims are made that are not supported and that could be put to a statistical test. A case in point is the reporting of the distribution of commands in the multimodal interaction test. The authors state that a majority of users tended to prefer one modality, leaning toward a preference for the modality immediately preceding this test. Then they back pedal an say that we should not conclude that the results are biased by the order of unimodal testing. In fact, this can be tested. I conducted a 2 x 2 CHI square test looking at which modality preceded the test and if the majority of multimodal commands were issued by mouse or by speech and found statistical evidence for just such a bias (CHI square (1, N=20 ) = 5.5, p<.01). Other analyses could be done on their data if they had taken the time to do it. } 5. Describe the QUALITY OF THE MECHANICS, addressing the following: Do they clearly describe what was done and/or how it was studied? Is the writing clear and concise? Do they provide the right level of detail? Do they use too much jargon? Do the figures support the text? {:For the most part the writing is clear. The one problem with Tables 3 and 4 is the inconsistent use of shading. In table 3, the most efficient method is shaded, I have no idea what the shading means in Table 4. }6. Provide any OTHER COMMENTS you believe would be useful to the author (including pointers to missing relevant work). {: }7. Using the scale 1 = Definitely reject 2 = Probably reject 3 = Could go either way 4 = Probably accept 5 = Definitely accept provide a numerical rating of the paper's ACCEPTABILITY {:3 }8. SUMMARIZE your ASSESSMENT of the paper, pointing out which aspects described above you weighted most heavily in your rating in question 7? {:This paper is a hybrid experience/empirical paper. The topic is interesting but the take home message is interesting but the empirical results are not well supported. I would feel comfortable having this work presented at CHI but I am not an avid champion for it. }9. Using the scale 1 = Very unconfident, really just a guess 2 = Rather unconfident, but I know a bit 3 = Moderately confident, I know as much as most 4 = Pretty confident, I know this area well 5 = Extremely confident, I consider myself an expert rate your EXPERTISE in the area addressed by the paper. {:4 }10. Use this space to provide comments that you feel are relevant to the review process but that you do NOT want FORWARDED to the author(s) {: } +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ++++++++++++ CHI97 Paper Review Form-End Banner ++++++++++++ ++++++++++++ INCLUDE AT THE BOTTOM OF EACH FORM +++++++++++++ ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ++++++++++++++++++++++++++++++++++++++++++++++ +++++++++++ CHI97 Paper Review Form-Begin Banner +++++++++++ +++++++++++++ INCLUDE AT THE TOP OF EACH REVIEW +++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ All your input should appear after a curly bracket followed by a colon. PAPER NUMBER {: 1109 } PAPER TITLE {: speech browsing the world wide web } REVIEWER NUMBER {: 135 }1. Briefly SUMMARIZE the paper {: Empirical paper with focus on possible use of speech as input/control device on the internet. }2. What TYPE OF SUBMISSION is appropriate for this paper (see bottom of Cover Sheet Two for the authors' selection)? If your assessment of type differs from the Authors, please explain why you made your selection. In the next two questions, use the criteria specific to this Type of Submission (see above) to assess the paper. {: Empirical paper }3. Describe the IMPORTANCE OF THE PROBLEM, addressing the following concerns: Is it new and significant? Is it a useful and relevant problem? Do they sufficiently motivate the problem? Did you learn anything? Does the issue apply to CHI? {: The field is important and results can have useful application in different ways. Empirical data of this type is important although the number of subjects is a bit low for many applications (important in some situations though). }4. Describe the QUALITY OF THE ARGUMENT, addressing the following: Does the analysis use appropriate methodology? Is the argument well-structured? Do they provide sufficient data and/or well-supported arguments? Do they cite relevant work? Is the paper appropriately focused? Does the analysis cover all the important issues at the appropriate level of detail? {: One important comment - the selection criteria of the test subjects is not (from what I can see) clearly stated. Was this treated methodically good enough? With this low number of test subjects the question of whether the results could be used to say anything generally this is important. In a interesting paper this is a pity. } 5. Describe the QUALITY OF THE MECHANICS, addressing the following: Do they clearly describe what was done and/or how it was studied? Is the writing clear and concise? Do they provide the right level of detail? Do they use too much jargon? Do the figures support the text? {: }6. Provide any OTHER COMMENTS you believe would be useful to the author (including pointers to missing relevant work). {: }7. Using the scale 1 = Definitely reject 2 = Probably reject 3 = Could go either way 4 = Probably accept 5 = Definitely accept provide a numerical rating of the paper's ACCEPTABILITY {: 5 }8. SUMMARIZE your ASSESSMENT of the paper, pointing out which aspects described above you weighted most heavily in your rating in question 7? {: This is simply good, well timed and interesting work. }9. Using the scale 1 = Very unconfident, really just a guess 2 = Rather unconfident, but I know a bit 3 = Moderately confident, I know as much as most 4 = Pretty confident, I know this area well 5 = Extremely confident, I consider myself an expert rate your EXPERTISE in the area addressed by the paper. {: 5 }10. Use this space to provide comments that you feel are relevant to the review process but that you do NOT want FORWARDED to the author(s) {: } +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ++++++++++++ CHI97 Paper Review Form-End Banner ++++++++++++ ++++++++++++ INCLUDE AT THE BOTTOM OF EACH FORM +++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +++++++++++ CHI97 Paper Review Form-Begin Banner +++++++++++ +++++++++++++ INCLUDE AT THE TOP OF EACH REVIEW +++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ All your input should appear after a curly bracket followed by a colon. PAPER NUMBER {: 1109 } PAPER TITLE {: Speech browsing the World Wide Web } REVIEWER NUMBER {: 236 }1. Briefly SUMMARIZE the paper {: The authors explore the possibilities of a Web browser that accepts speech input. They describe two experiments that shed some light on the role of speech input and compare it to using a pointer. }2. What TYPE OF SUBMISSION is appropriate for this paper (see bottom of Cover Sheet Two for the authors' selection)? If your assessment of type differs from the Authors, please explain why you made your selection. In the next two questions, use the criteria specific to this Type of Submission (see above) to assess the paper. {: Empirical. I agree with the authors here. }3. Describe the IMPORTANCE OF THE PROBLEM, addressing the following concerns: Is it new and significant? Is it a useful and relevant problem? Do they sufficiently motivate the problem? Did you learn anything? Does the issue apply to CHI? {: There are a lot of "hot" and interesting things happening in Web technology. Speech input is not at the top of that list, however. Speech input in general is not new at all. The authors' attempts to make it relevant by relating it to current browser problems are not convincing because they do not demonstrate how speech input can *uniquely* solve those problems. In fact, they even admit that there are other possible solutions. }4. Describe the QUALITY OF THE ARGUMENT, addressing the following: Does the analysis use appropriate methodology? Is the argument well-structured? Do they provide sufficient data and/or well-supported arguments? Do they cite relevant work? Is the paper appropriately focused? Does the analysis cover all the important issues at the appropriate level of detail? {: My basic problem with the arguments in the paper stem from the proposed solutions to navigation problems. The authors suggest speech-based solutions without fully exploring "normal" solutions to the same problems. For example, the authors point out that some links are hidden from the user when the page must be scrolled. There are many ways to solve this without speech, such as having all of the links appear as a separate list, much like history or bookmarks. Plus, cases where the user has to "deduce the name of the link by observing the information on the pages" sounds like pretty bad design to me in the first place. This problem is exagerated when the authors create the "look for" functionality in their prototype. For an effective analysis of how and when speech input can be effective in a Web browser, one must stick to standard browser features. If the same old Netscape that the rest of the world is using was used in their Wizard of Oz study instead, then I would place more value in their results. But it is hard to separate the benefits of speech input from the features of their special browser. The frequency numbers in table 1 have limited value: it is reassuring to know that the simple, common actions that are awkward in current browsers (back and paging down) were common commands to speak. This is not enough for a CHI paper, but it does have some value. The results of the speech vs. pointer experiment are meaningless without a statistical test to determine if 1.35 is really bigger than 1.22, and so on. I must admit that I do not know a lot about speech recognition interfaces: it would have been really noce to know if the results found here correlate with those from other systems. Did the authors discover something unique to Web browsing usage of speech input? Or did they just rediscover the same old problems that others have found previously? It would be helpful to know in either case, but this paper's conclusions do not tell me which. The authors fail completely to mention one of the most important uses of speech input for Web browsers: the physically handicapped. The WebABLE! site could help lead them to information on this (http://www.webable.com/). } 5. Describe the QUALITY OF THE MECHANICS, addressing the following: Do they clearly describe what was done and/or how it was studied? Is the writing clear and concise? Do they provide the right level of detail? Do they use too much jargon? Do the figures support the text? {: All tables were useful except #4, where the shading was meaningless and confusing when one could also see shading in table 3. Overall, the mechanics were acceptable. }6. Provide any OTHER COMMENTS you believe would be useful to the author (including pointers to missing relevant work). {: Why no reference for the Wizard of Oz technique? }7. Using the scale 1 = Definitely reject 2 = Probably reject 3 = Could go either way 4 = Probably accept 5 = Definitely accept provide a numerical rating of the paper's ACCEPTABILITY {: 2 }8. SUMMARIZE your ASSESSMENT of the paper, pointing out which aspects described above you weighted most heavily in your rating in question 7? {: There are some interesting results here. But the results of the 2nd study are meaningless without statistical tests to determine if the differences are significant. The other results are difficult to interpret because usage of "normal" Netscape was not part of the experimental design. Finally, the results that are not specific enough to make concrete recommendations ("adding voice recognition of 'back' would decrease navigation time by x", for example). So, interesting results, but not that useful. }9. Using the scale 1 = Very unconfident, really just a guess 2 = Rather unconfident, but I know a bit 3 = Moderately confident, I know as much as most 4 = Pretty confident, I know this area well 5 = Extremely confident, I consider myself an expert rate your EXPERTISE in the area addressed by the paper. {: 3 }10. Use this space to provide comments that you feel are relevant to the review process but that you do NOT want FORWARDED to the author(s) {: } +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ++++++++++++ CHI97 Paper Review Form-End Banner ++++++++++++ ++++++++++++ INCLUDE AT THE BOTTOM OF EACH FORM +++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +++++++++++ CHI97 Paper Review Form-Begin Banner +++++++++++ +++++++++++++ INCLUDE AT THE TOP OF EACH REVIEW +++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ All your input should appear after a curly bracket followed by a colon. PAPER NUMBER {: 1109 } PAPER TITLE {: Speech Browsing the World Wide Web } REVIEWER NUMBER {: commissioned }1. Briefly SUMMARIZE the paper {: The authors carried out an experiment in which subjects carried out tasks on a WWW site using either speech or a mouse. The speech controlled interface was implemented using the Wizard-of-Oz technique. Subjects performance and preferences were measured and conclusions were drawn about the value of speech-based Web browsing. }2. What TYPE OF SUBMISSION is appropriate for this paper (see bottom of Cover Sheet Two for the authors' selection)? If your assessment of type differs from the Authors, please explain why you made your selection. In the next two questions, use the criteria specific to this Type of Submission (see above) to assess the paper. {: Empirical }3. Describe the IMPORTANCE OF THE PROBLEM, addressing the following concerns: Is it new and significant? Is it a useful and relevant problem? Do they sufficiently motivate the problem? Did you learn anything? Does the issue apply to CHI? {:I think the problem is important and the authors explain their rationale for doing the study very well. The issue of people's preferences for speech vs. other interface modalities has been studied before but is far from a settled issue. I didn't learn anything because the study had methodological problems, but if the data had been analyzed properly, the study could have produced valuable information. }4. Describe the QUALITY OF THE ARGUMENT, addressing the following: Does the analysis use appropriate methodology? Is the argument well-structured? Do they provide sufficient data and/or well-supported arguments? Do they cite relevant work? Is the paper appropriately focused? Does the analysis cover all the important issues at the appropriate level of detail? {: This is an odd study in that the logic of the study, its design, and the collection of the data are all solid. But the analysis of the data was very weak -- they used no inferential statistics at all. If they had, the reader could have drawn solid conclusions and the study would have been much more valuable. } 5. Describe the QUALITY OF THE MECHANICS, addressing the following: Do they clearly describe what was done and/or how it was studied? Is the writing clear and concise? Do they provide the right level of detail? Do they use too much jargon? Do the figures support the text? {:The study was described clearly and concisely; someone could replicate it with ease. }6. Provide any OTHER COMMENTS you believe would be useful to the author (including pointers to missing relevant work). {: This is a good paper except for the glaring omission of statistical analysis of the data. The authors should get statistical help, analyze the data properly, and (if the results are significant), submit a new paper. }7. Using the scale 1 = Definitely reject 2 = Probably reject 3 = Could go either way 4 = Probably accept 5 = Definitely accept provide a numerical rating of the paper's ACCEPTABILITY {:1 }8. SUMMARIZE your ASSESSMENT of the paper, pointing out which aspects described above you weighted most heavily in your rating in question 7? {:I like this paper except for the data analysis. It's an important research problem, the paper is well-written, and they reference the literature well. I would strongly urge the authors to do a proper analysis of the data so we know what can really be concluded from the data. }9. Using the scale 1 = Very unconfident, really just a guess 2 = Rather unconfident, but I know a bit 3 = Moderately confident, I know as much as most 4 = Pretty confident, I know this area well 5 = Extremely confident, I consider myself an expert rate your EXPERTISE in the area addressed by the paper. {:5 }10. Use this space to provide comments that you feel are relevant to the review process but that you do NOT want FORWARDED to the author(s) {: } +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ++++++++++++ CHI97 Paper Review Form-End Banner ++++++++++++ ++++++++++++ INCLUDE AT THE BOTTOM OF EACH FORM +++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ rvirzi@gte.com Think Globally. === +1(617)466-2881 === Act Locally! From epfran1@PacBell.COM Mon Dec 2 17:17:20 1996 Received: from mailer.oclc.org (mailer.dev.oclc.org) by oclc.org (4.1/SMI-4.1) id AA06751; Mon, 2 Dec 96 17:19:09 EST Received: from gw3.pacbell.com by mailer.oclc.org (5.0/SMI-4.1) id AA23219; Mon, 2 Dec 1996 17:19:07 +0500 Received: from srv.PacBell.COM (mother.srv.PacBell.COM) by gw3.pacbell.com (5.x/PacBell-10/18/95) id AA14420; Mon, 2 Dec 1996 14:19:00 -0800 Received: from zeus.srv.PacBell.COM by srv.PacBell.COM (4.1/Mother-7/26/95) id AA06718; Mon, 2 Dec 96 14:18:49 PST Received: from [129.245.90.15] by zeus.srv.PacBell.COM (4.1/SMI-4.1) id AA24516; Mon, 2 Dec 96 14:18:47 PST X-Sender: epfran1@zeus.srv.pacbell.com Message-Id: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Date: Mon, 2 Dec 1996 14:16:52 -0800 To: perlman@oclc.org, debra_herschmann@usccmail.lehman.com, dray.chi@xerox.com, nardi@apple.com, smetros@utk.edu, isensee@austin.ibm.com, ellen@communities.com, pete@dcs.qmw.ac.uk, csnyder@uie.com, karen@acm.org, sarahb@werple.mira.net.au, nass@leland.stanford.edu, poynton@poynton.com, gershon@mitre.org From: epfran1@PacBell.COM (Ellen Francik) Subject: CHI 97/Tutorial notes and other materials Cc: chi97-tutorials@acm.org Content-Length: 1324 Status: RO Dear CHI 97 Tutorial Instructors, Jim and I thank all of you for working through the rapid iterations of the Advance Program. We're excited about the CHI 97 Tutorials program and we're sure that once people see the Advance Program they'll be excited too. Some of you are coming back from the US Thanksgiving holiday and many of you are looking forward to celebrating winter holidays or taking breaks during December. It's becoming a busy time of year for many families. Without ruining your holidays (;-) we wanted to remind you that tutorial notes, release forms, and the document for the Conference Companion (in electronic and camera-ready formats) are due 3 January 1997. All of these should be sent to me, your Tutorials liaison, by postal or courier mail to my Pacific Bell address. After 3 January we'll be going through all the materials. If any revisions are needed, we only have three weeks to do them; final materials are due from us to the Conference Office on 24 January. So, if there's anything we can do now to help you and to minimize revisions, please let us know. Best regards and happy holidays, Ellen Ellen Francik Pacific Bell CHI 97 Tutorials Co-Chair 2600 Camino Ramon, Rm. 3E050H chi97-tutorials@acm.org San Ramon, CA 94583 USA +1 510-867-8662 From epfran1@PacBell.COM Mon Dec 30 15:36:15 1996 Received: from mailer.oclc.org (mailer.dev.oclc.org) by oclc.org (4.1/SMI-4.1) id AA07382; Mon, 30 Dec 96 15:39:09 EST Received: from gw3.pacbell.com ([129.245.2.24]) by mailer.oclc.org (5.0/SMI-4.1) id AA23622; Mon, 30 Dec 1996 15:38:59 +0500 Received: from srv.PacBell.COM (mother.srv.PacBell.COM) by gw3.pacbell.com (5.x/PacBell-10/18/95) id AA08810; Mon, 30 Dec 1996 12:35:03 -0800 Received: from zeus.srv.PacBell.COM by srv.PacBell.COM (4.1/Mother-7/26/95) id AA27671; Mon, 30 Dec 96 12:34:38 PST Received: from [129.245.90.15] by zeus.srv.PacBell.COM (4.1/SMI-4.1) id AA26490; Mon, 30 Dec 96 12:34:37 PST X-Sender: epfran1@zeus.srv.pacbell.com Message-Id: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Date: Mon, 30 Dec 1996 12:32:34 -0800 To: perlman@oclc.org, debra_herschmann@usccmail.lehman.com, dray.chi@xerox.com, nardi@apple.com, smetros@utk.edu, isensee@austin.ibm.com, ellen@communities.com, pete@dcs.qmw.ac.uk, csnyder@uie.com, karen@acm.org, sarahb@werple.mira.net.au, nass@leland.stanford.edu, poynton@poynton.com, gershon@mitre.org From: epfran1@PacBell.COM (Ellen Francik) Subject: CHI 97/Reminder: tutorial notes due Friday Content-Length: 663 Status: RO Dear CHI 97 Tutorial Instructors, I've already heard from many of you about how work on your tutorial notes is progressing. Thank you to those who have already turned your notes in! For those I haven't yet heard from, this is a reminder that tutorial notes, release forms, and the document for the Conference Companion (in electronic and camera-ready formats) are DUE FRIDAY, 3 JANUARY 1997. All of these should be sent to me, your Tutorials liaison, by postal or courier mail: Ellen Francik Pacific Bell 2600 Camino Ramon, Rm. 3E050H San Ramon, CA 94583 USA In the meantime, if there is anything I can do to help please let me know. Best regards, Ellen From epfran1@PacBell.COM Mon Dec 30 17:57:27 1996 Received: from srv.PacBell.COM (mother.srv.PacBell.COM) by gw3.pacbell.com (5.x/PacBell-10/18/95) id AA29937; Mon, 30 Dec 1996 15:00:21 -0800 Received: from zeus.srv.PacBell.COM by srv.PacBell.COM (4.1/Mother-7/26/95) id AA29789; Mon, 30 Dec 96 15:00:20 PST Received: from [129.245.90.15] by zeus.srv.PacBell.COM (4.1/SMI-4.1) id AA28781; Mon, 30 Dec 96 15:00:18 PST X-Sender: epfran1@zeus.srv.pacbell.com Message-Id: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Date: Mon, 30 Dec 1996 14:58:15 -0800 To: Gary PERLMAN From: epfran1@PacBell.COM (Ellen Francik) Subject: Your circumstances & tutorial offering Status: RO Dear Gary, I'm so sorry. Of course this is a great shock and of course figuring out what to do with your health care is your first priority. I have a close coworker and a dear friend both with MS. Knowing them is the closest I've come to experiencing it. They both have low vision and one of them has some mobility limitations. (They are also two of the most energetic, indomitable people I know!) I know you can't predict the course of your health at this point, but I'm committed to supporting you in presenting this tutorial IF that's what you want to do. And I know my cochair well enough to know he will feel strongly the same. I don't know exactly what your "sensory deficits" are right now, but here are some examples of what we might arrange together: - Have you do the talking and get a student volunteer to flip slides, fiddle with computers, and so on. (You've given this tutorial before; your notes are probably in good shape and won't need revising.) - Get a colleague to share presentation with you. - Get a colleague to do the bulk of the presentation but have you available as chief organizer, commenter, and presenter of special sections. - Get someone to type/write questions from the audience for you, if hearing is a problem that hearing aids can't solve. (If you're doing most of the talking, we probably don't need full-blown real time captioning.) And so on. We can handle letting people know that there is an additional instructor (if you wanted to bring a colleague in to help). There's the Web site, which can be updated, and notification through Registration. And email too. All of which is to say, I'd like to continue planning for you to give the tutorial, understanding that: - it'll take some time & conversation between us to figure out the best logistical solutions to the limitations you're now having; we'll want to alert the AV team as soon as possible to any changes we know about - you should feel free to cancel the tutorial at any time that makes sense for you. So, think about it. If you have the notes and other materials done, send them in. I don't send master copies off to New York until the 22nd or so. And either email me or call me, whichever is easier, and let me know how you'd like to proceed. All my best wishes, Ellen Ellen Francik Pacific Bell CHI 97 Tutorials Co-Chair 2600 Camino Ramon, Rm. 3E050H chi97-tutorials@acm.org San Ramon, CA 94583 USA +1 510-867-8662 From perlman Thu Jan 2 14:19:00 1997 Received: by turing.acm.org; id AA29657; Thu, 2 Jan 1997 14:18:50 -0500 Date: Thu, 2 Jan 1997 14:18:50 -0500 From: Gary PERLMAN Message-Id: <9701021918.AA29657@turing.acm.org> To: epfran1@PacBell.COM Subject: Re: Your circumstances & tutorial offering Cc: perlman Status: RO Hi Ellen, Thanks for the ideas about working around my problems. My main fear is that things will turn worse around the time I have to travel. Adapting to a known problem is doable, in my experience, humans being adaptable, but right now, I am struggling with daily sensory changes. More specifically, my deficits are that I get little or no sensory feedback from my feet, hands and torso; it's like they have fallen asleep, but they don't wake up. I have similar lesser problems in my legs and arms. If things stay the same, I am sure I can relearn how to do things at close to previous levels, but right now, touch typing and writing are the hardest. My understanding is that my symptoms are minor for MS, so my prognosis is good. I think it should be possible to arrange for help, even on short notice, given that the main physical task I have is to flip overheads, and even those can be mounted so they are easier to handle. It seems to be Thursday already, so it would be pretty hard for me to get the materials printed and sent by tomorrow. I forget if I mentioned that my wife had a baby on Dec 10, but had to go back in for emergency operations Dec. 19 AND Dec. 30, so I am still needed at home for her. Given your Dec. 22 date for NYC delivery, I'll try to get my materials sent to you by courier on Monday. Thanks for your support. Gary From pete@dcs.qmw.ac.uk Wed Dec 18 10:09:34 1996 Received: from mailer.oclc.org (mailer.dev.oclc.org) by oclc.org (4.1/SMI-4.1) id AA05720; Wed, 18 Dec 96 10:11:48 EST Received: from mangle.dcs.qmw.ac.uk ([138.37.95.250]) by mailer.oclc.org (5.0/SMI-4.1) id AA21358; Wed, 18 Dec 1996 10:11:44 +0500 Received: from petes-mac.dcs.qmw.ac.uk [138.37.88.79]; by mangle.dcs.qmw.ac.uk (8.8.2/8.8.4/S-3.3) with SMTP; id NAA03161; Wed, 18 Dec 1996 13:42:36 GMT Date: Wed, 18 Dec 1996 13:42:36 GMT X-Sender: pete@mailhost.dcs.qmw.ac.uk Message-Id: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" To: epfran1@PacBell.COM (Ellen Francik), perlman@oclc.org, debra_herschmann@usccmail.lehman.com, dray.chi@xerox.com, nardi@apple.com, smetros@utk.edu, isensee@austin.ibm.com, ellen@communities.com, csnyder@uie.com, karen@acm.org, sarahb@werple.mira.net.au, nass@leland.stanford.edu, poynton@poynton.com, gershon@mitre.org From: pete@dcs.qmw.ac.uk (Peter Johnson) Subject: Re: CHI 97/Tutorial notes and other materials Cc: chi97-tutorials@acm.org Content-Length: 789 Status: RO Dear Ellen and Colleagues, John Wiley are wanting to charge me $3.80 per copy to distribute copies of one of my own papers as part of the tutorial notes. Is there anything to be done about this? It seems grossly unfair and extremely avaricious. Very best wishes Pete. Professor Peter Johnson |JANET pete@dcs.qmw.ac.uk Department of Computer Science |ARPA/Internet pete@dcs.qmw.ac.uk Queen Mary & Westfield College |UUCP pete@qmw-dcs.UUCP University of London |phone + 44 (0)171-975-5224 Mile End Road |fax + 44 (0)181 980 6533 London | E1 4NS |hhtp://www.dcs.qmw.ac.uk/~pete/ England. | From perlman Mon Dec 30 16:34:49 1996 Received: by turing.acm.org; id AA19660; Mon, 30 Dec 1996 16:34:40 -0500 Date: Mon, 30 Dec 1996 16:34:40 -0500 From: Gary PERLMAN Message-Id: <9612302134.AA19660@turing.acm.org> To: epfran1@PacBell.COM Subject: Re: CHI 97/Reminder: tutorial notes due Friday Cc: perlman Status: RO > Dear CHI 97 Tutorial Instructors, > > I've already heard from many of you about how work on your tutorial notes > is progressing. Thank you to those who have already turned your notes in! > > For those I haven't yet heard from, this is a reminder that tutorial notes, > release forms, and the document for the Conference Companion (in electronic > and camera-ready formats) are DUE FRIDAY, 3 JANUARY 1997. All of these > should be sent to me, your Tutorials liaison, by postal or courier mail: > > Ellen Francik > Pacific Bell > 2600 Camino Ramon, Rm. 3E050H > San Ramon, CA 94583 > USA > > In the meantime, if there is anything I can do to help please let me know. > > Best regards, > > Ellen Hi Ellen, I don't want to sound any alarms, but I guess I have to. Please keep this confidential, but I have been diagnosed with Multiple Sclerosis and have a pile of symptoms, mainly sensory deficits. My crisis has been going on for 3 weeks, but I got my diagnosis today. My concern is that even if I get the notes to you, my condition might get worse by the time of CHI and I might not be able to give the tutorial. If my condition stays the same or gets better, then I could give the tutorial, although I can't really write too well and now type 5-10 words a minute. I'm pretty stunned right now, so I don't have any suggestions about how to proceed. One option is to minimize the downside and cancel right now. I think the worst case if for people to sign up and think they are going to get a tutorial, and then find out they did not. Even if the tutorial is in the reg materials, people can simply be denied entry and given their second choice to accept or bail out. It may mean losing a valuable slot, but perhaps it could be filled by a second offering of a tutorial that sells out. I am willing to sit here and see if I get better by CHI -- it's a coin toss with MS -- but I wanted to leave that up to you. I am very sorry to have to put this snag in your planning. Gary From epfran1@PacBell.COM Thu Jan 2 15:31:10 1997 Received: from srv.PacBell.COM (mother.srv.PacBell.COM) by gw3.pacbell.com (5.x/PacBell-10/18/95) id AA02277; Thu, 2 Jan 1997 12:34:10 -0800 Received: from zeus.srv.PacBell.COM by srv.PacBell.COM (4.1/Mother-7/26/95) id AA23257; Thu, 2 Jan 97 12:34:09 PST Received: from [129.245.90.15] by zeus.srv.PacBell.COM (4.1/SMI-4.1) id AA08609; Thu, 2 Jan 97 12:34:08 PST X-Sender: epfran1@zeus.srv.pacbell.com Message-Id: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Date: Thu, 2 Jan 1997 12:32:04 -0800 To: Gary PERLMAN From: epfran1@PacBell.COM (Ellen Francik) Subject: Re: Your circumstances & tutorial offering Status: RO Hi Gary, >Thanks for the ideas about working around my problems. >My main fear is that things will turn worse around the >time I have to travel. All we can do is plan for what we know. I mean (god/dess forbid!) *anybody* could break a limb or get pneumonia just before the tutorials program, but we plan as if that won't be a serious issue. >My understanding is that my symptoms are minor for MS, >so my prognosis is good. Great. >I think it should be possible >to arrange for help, even on short notice, given that >the main physical task I have is to flip overheads, >and even those can be mounted so they are easier >to handle. Oh, shoot, yes. Managing instructor AV is what student volunteers are for. We're starting to estimate how many and what type we need (estmate due 1/24) so we'll throw that into the bin. >It seems to be Thursday already, so it would be pretty >hard for me to get the materials printed and sent by >tomorrow. ... Given your >Dec. 22 date for NYC delivery, I'll try to get >my materials sent to you by courier on Monday. That would be fine. If you need a few days beyond that, take them. I can handle your notes arriving as late as the 10th (e-pubs are due from us to NYC by the 17th, paper tutorial notes by the 24th). >I forget if I mentioned that my wife >had a baby on Dec 10, but had to go back in for >emergency operations Dec. 19 AND Dec. 30, so I >am still needed at home for her. My goodness. Not enough going on in your life? Congratulations on the new baby and best wishes for the health and continued recovery of your wife. Ellen From perlman Mon Jan 6 01:40:18 1997 Received: by turing.acm.org; id AA10849; Mon, 6 Jan 1997 01:40:16 -0500 Date: Mon, 6 Jan 1997 01:40:16 -0500 From: Gary PERLMAN Message-Id: <9701060640.AA10849@turing.acm.org> To: chi97-ep-submission@acm.org Subject: esub: tutorial / perlman / practical usability evaluation Cc: epfran1@PacBell.COM, perlman Status: R Gary PERLMAN, OCLC Online Computer Library Center 6565 Frantz Road, Dublin, Ohio 43017 USA Voice: +1-614-761-5058 Fax: +1-614-793-0915 Email: perlman@acm.org WWW: www.acm.org/~perlman One HTML file: gp.htm which is my writeup for my tutorial "Practical Usability Evaluation" has been transferred via anonymous ftp to ftp.acm.org. It is also available at: http://www.acm.org/~perlman/gp.htm Note the cool links from the keywords into the ACM CRCS. From l.tweedie@ic.ac.uk Mon Jan 6 11:04:59 1997 Received: from romeo.ic.ac.uk (romeo.ic.ac.uk [155.198.5.9]) by mail.acm.org (8.7.5/8.7.5) with SMTP id LAA40182 for ; Mon, 6 Jan 1997 11:04:56 -0500 From: l.tweedie@ic.ac.uk Received: from judy.ic.ac.uk [155.198.5.5] by romeo.ic.ac.uk with esmtp (Exim 0.57 #1) id 0vhHYy-0004jo-00; Mon, 6 Jan 1997 16:06:12 +0000 Received: from dinghy.ee.ic.ac.uk (root@dinghy.ee.ic.ac.uk [155.198.123.8]) by judy.ic.ac.uk (8.7.5/8.7.5) with SMTP id QAA15372; Mon, 6 Jan 1997 16:06:02 GMT Received: from [155.198.123.20] (pixie) by dinghy.ee.ic.ac.uk (4.1/4.0) id AA23867; Mon, 6 Jan 97 16:05:59 GMT X-Sender: lisat@155.198.123.8 Message-Id: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Date: Mon, 6 Jan 1997 16:06:40 +0000 To: (Recipient list suppressed) Subject: CHI reviews paper 1036 Part 1 Status: R Happy New Year Below are all the reviews for papers 1036. It was not accepted. Thank you for your prompt review/s. I'm sorry that I have taken a little while to return these to you. Lisa Tweedie +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +++++++++++ CHI97 Paper Review Form-Begin Banner +++++++++++ +++++++++++++ INCLUDE AT THE TOP OF EACH REVIEW +++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ All your input should appear after a curly bracket followed by a colon. PAPER NUMBER {: 1036 } PAPER TITLE {: User Reactions to Agents with Faces } REVIEWER NUMBER {: rev0322 }1. Briefly SUMMARIZE the paper {: The paper reports the results of an empirical evaluation regarding people's impressions of faces in an interactive environment. It includes quantitative analysis of participants' impressions about personified agents in the interface. }2. What TYPE OF SUBMISSION is appropriate for this paper (see bottom of Cover Sheet Two for the authors' selection)? If your assessment of type differs from the Authors, please explain why you made your selection. In the next two questions, use the criteria specific to this Type of Submission (see above) to assess the paper. {: OK }3. Describe the IMPORTANCE OF THE PROBLEM, addressing the following concerns: Is it new and significant? Is it a useful and relevant problem? Do they sufficiently motivate the problem? Did you learn anything? Does the issue apply to CHI? {: The paper attempts to answer whether software agents should be personified in the interface. It is an important study because it attempts to answer this question empirical by conducting a well-designed experiment and by providing quantitative analysis that would indicate whether adding a face to an interface can improve the interaction between the user and the computer. I visited the Web page, reference number 4, and enjoyed participating in the experiment. }4. Describe the QUALITY OF THE ARGUMENT, addressing the following: Does the analysis use appropriate methodology? Is the argument well-structured? Do they provide sufficient data and/or well-supported arguments? Do they cite relevant work? Is the paper appropriately focused? Does the analysis cover all the important issues at the appropriate level of detail? {: I enjoyed reading the paper. I think that the paper is well structured, and the analysis of the empirical results is convincing. } 5. Describe the QUALITY OF THE MECHANICS, addressing the following: Do they clearly describe what was done and/or how it was studied? Is the writing clear and concise? Do they provide the right level of detail? Do they use too much jargon? Do the figures support the text? {: The tables of results are well presented. The aim of the study and the conclusions clearly address the relevant issues. }6. Provide any OTHER COMMENTS you believe would be useful to the author (including pointers to missing relevant work). {: In the first paragraph in the INTRODUCTION section, I suggest that references [9] and [7] be moved directly after the names Negroponte and Maes, respectively. In page number 4, in the subsection `Lab Experiment' (first paragraph): ``... and engagingness as an opponent poker...'' In the same page, in subsection `Questionnaire' (second paragraph): ``... each player.is'' - remove the period (.) }7. Using the scale 1 = Definitely reject 2 = Probably reject 3 = Could go either way 4 = Probably accept 5 = Definitely accept provide a numerical rating of the paper's ACCEPTABILITY {: 5 }8. SUMMARIZE your ASSESSMENT of the paper, pointing out which aspects described above you weighted most heavily in your rating in question 7? {: Interesting area, well designed experiment, good analysis, well structured paper. }9. Using the scale 1 = Very unconfident, really just a guess 2 = Rather unconfident, but I know a bit 3 = Moderately confident, I know as much as most 4 = Pretty confident, I know this area well 5 = Extremely confident, I consider myself an expert rate your EXPERTISE in the area addressed by the paper. {: 4 } +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ++++++++++++ CHI97 Paper Review Form-End Banner ++++++++++++ ++++++++++++ INCLUDE AT THE BOTTOM OF EACH FORM +++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +++++++++++ CHI97 Paper Review Form-Begin Banner +++++++++++ +++++++++++++ INCLUDE AT THE TOP OF EACH REVIEW +++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ All your input should appear after a curly bracket followed by a colon. PAPER NUMBER {: 1036 } PAPER TITLE {: User Reactions to Agents with Faces } REVIEWER NUMBER {: 284 }1. Briefly SUMMARIZE the paper {: The reactions of people to software agents with a variety of "face types" in a computer-based poker game were assessed. }2. What TYPE OF SUBMISSION is appropriate for this paper (see bottom of Cover Sheet Two for the authors' selection)? If your assessment of type differs from the Authors, please explain why you made your selection. In the next two questions, use the criteria specific to this Type of Submission (see above) to assess the paper. {: Empirical--appropriate }3. Describe the IMPORTANCE OF THE PROBLEM, addressing the following concerns: Is it new and significant? Is it a useful and relevant problem? Do they sufficiently motivate the problem? Did you learn anything? Does the issue apply to CHI? {: As the paper points out, faces have been used but seldom systematically tested. It is good that they be tested. }4. Describe the QUALITY OF THE ARGUMENT, addressing the following: Does the analysis use appropriate methodology? Is the argument well-structured? Do they provide sufficient data and/or well-supported arguments? Do they cite relevant work? Is the paper appropriately focused? Does the analysis cover all the important issues at the appropriate level of detail? {: I felt the problem to be well-conceived and the work competently carried out. They cited a number of relevant works. } 5. Describe the QUALITY OF THE MECHANICS, addressing the following: Do they clearly describe what was done and/or how it was studied? Is the writing clear and concise? Do they provide the right level of detail? Do they use too much jargon? Do the figures support the text? {: The writing style was clear. There were a few grammatical/orthographic errors including: Introduction, par. 3: "does not always make the experience [WHAT] for users"; Introduction, par. 4: "It is possible to make [the] advantages..."; and later in Facial Expression Design, "exited" should be "excited". The figures were pretty useful. Once into the data there was a LOT of detail, which makes it something of a "heavy read". It was better the second time through. }6. Provide any OTHER COMMENTS you believe would be useful to the author (including pointers to missing relevant work). {: }7. Using the scale 1 = Definitely reject 2 = Probably reject 3 = Could go either way 4 = Probably accept 5 = Definitely accept provide a numerical rating of the paper's ACCEPTABILITY {: 4 }8. SUMMARIZE your ASSESSMENT of the paper, pointing out which aspects described above you weighted most heavily in your rating in question 7? {: I think that the relative utility of different face types is an important question and the evaluation was appropriate. }9. Using the scale 1 = Very unconfident, really just a guess 2 = Rather unconfident, but I know a bit 3 = Moderately confident, I know as much as most 4 = Pretty confident, I know this area well 5 = Extremely confident, I consider myself an expert rate your EXPERTISE in the area addressed by the paper. {: 3 } +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ++++++++++++ CHI97 Paper Review Form-End Banner ++++++++++++ ++++++++++++ INCLUDE AT THE BOTTOM OF EACH FORM +++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +++++++++++ CHI97 Paper Review Form-Begin Banner +++++++++++ +++++++++++++ INCLUDE AT THE TOP OF EACH REVIEW +++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ All your input should appear after a curly bracket followed by a colon. PAPER NUMBER {: 1036 } PAPER TITLE {: User Reactions to Agents with Faces } REVIEWER NUMBER {: 293 }1. Briefly SUMMARIZE the paper {: The author created a computerized poker game in which the opponents were depicted by agents with various facial expressions. Realistic and cartoon male/female faces, a cartoon dog, a smiley, an the invisible man (woman?), were possible players. After a short poker game, users filled out a short questionnaire about their impressions of the players. Results were summarized. }2. What TYPE OF SUBMISSION is appropriate for this paper (see bottom of Cover Sheet Two for the authors' selection)? If your assessment of type differs from the Authors, please explain why you made your selection. In the next two questions, use the criteria specific to this Type of Submission (see above) to assess the paper. {: EMPIRICAL Agents User Studies }3. Describe the IMPORTANCE OF THE PROBLEM, addressing the following concerns: Is it new and significant? Is it a useful and relevant problem? Do they sufficiently motivate the problem? Did you learn anything? Does the issue apply to CHI? {: I don't know if it's important, but it's sort of fun. I don't know anything about this area, but I have trouble taking it too seriously. I did not learn anything. Perhaps people in the area would find the data useful as it is a reasonable survey collecting some basic data. Relevant to CHI? Sure. There are motivational issues, and there are issues in using facial expressions to provide some information. }4. Describe the QUALITY OF THE ARGUMENT, addressing the following: Does the analysis use appropriate methodology? Is the argument well-structured? Do they provide sufficient data and/or well-supported arguments? Do they cite relevant work? Is the paper appropriately focused? Does the analysis cover all the important issues at the appropriate level of detail? {: I thought the study and analysis was well done. That is an area in which I am qualified. The author collected some reasonable data (i.e., the questions used and the rating scale), although the subject population is unknown, and so generalization to general populations is questionable. } 5. Describe the QUALITY OF THE MECHANICS, addressing the following: Do they clearly describe what was done and/or how it was studied? Is the writing clear and concise? Do they provide the right level of detail? Do they use too much jargon? Do the figures support the text? {: I thought the paper was well written. The tables could be made more regular by adopting one statistical test (i.e., always use F) and report it in the standard format (i.e., F(1,df)=XXX.) Don't make up a new format for F tests (i.e., F(XXX,df2)); the standard format has been used 99.999% of the time for decades. The labels for the graphs (Fig 5 - 9) could be clearer, but I liked the 95% CI bars (although I don't know which of the several common methods was used to compute them). }6. Provide any OTHER COMMENTS you believe would be useful to the author (including pointers to missing relevant work). {: }7. Using the scale 1 = Definitely reject 2 = Probably reject 3 = Could go either way 4 = Probably accept 5 = Definitely accept provide a numerical rating of the paper's ACCEPTABILITY {:3 }8. SUMMARIZE your ASSESSMENT of the paper, pointing out which aspects described above you weighted most heavily in your rating in question 7? {: I do not take this area of research too seriously, especially when it is applied to an unidentified population. If it were targeted at a specific population, say kids, then I think it would have more merit. The sample population has unknown characteristics, so it is difficult to make any generalizations. Still, the analysis was well done, and the neatness factor is high. }9. Using the scale 1 = Very unconfident, really just a guess 2 = Rather unconfident, but I know a bit 3 = Moderately confident, I know as much as most1036 4 = Pretty confident, I know this area well 5 = Extremely confident, I consider myself an expert rate your EXPERTISE in the area addressed by the paper. {: I'll pass on this rating. I give me a 5 on methodology, and 1 on content area. } +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ++++++++++++ CHI97 Paper Review Form-End Banner ++++++++++++ ++++++++++++ INCLUDE AT THE BOTTOM OF EACH FORM +++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ________________________________________________ Lisa Tweedie Tel: 0171 589 5111 ext:56210 Dept. of Electrical Eng. Imperial College Exhibition Road South Kensington London SW7 http://www.ee.ic.ac.uk/research/information/www/lisat.html From l.tweedie@ic.ac.uk Mon Jan 6 11:05:53 1997 Received: from romeo.ic.ac.uk (romeo.ic.ac.uk [155.198.5.9]) by mail.acm.org (8.7.5/8.7.5) with SMTP id LAA39972 for ; Mon, 6 Jan 1997 11:05:29 -0500 From: l.tweedie@ic.ac.uk Received: from judy.ic.ac.uk [155.198.5.5] by romeo.ic.ac.uk with esmtp (Exim 0.57 #1) id 0vhHZS-0004jv-00; Mon, 6 Jan 1997 16:06:42 +0000 Received: from dinghy.ee.ic.ac.uk (root@dinghy.ee.ic.ac.uk [155.198.123.8]) by judy.ic.ac.uk (8.7.5/8.7.5) with SMTP id QAA15390; Mon, 6 Jan 1997 16:06:11 GMT Received: from [155.198.123.20] (pixie) by dinghy.ee.ic.ac.uk (4.1/4.0) id AA23871; Mon, 6 Jan 97 16:06:08 GMT X-Sender: lisat@155.198.123.8 Message-Id: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Date: Mon, 6 Jan 1997 16:06:50 +0000 To: (Recipient list suppressed) Subject: CHI reviews paper 1036 Part 2 Status: R +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +++++++++++ CHI97 Paper Review Form-Begin Banner +++++++++++ +++++++++++++ INCLUDE AT THE TOP OF EACH REVIEW +++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ All your input should appear after a curly bracket followed by a colon. PAPER NUMBER {: 1036 } PAPER TITLE {: User Reactions to Agents with Faces } REVIEWER NUMBER {: rev0322 }1. Briefly SUMMARIZE the paper {: The paper reports the results of an empirical evaluation regarding people's impressions of faces in an interactive environment. It includes quantitative analysis of participants' impressions about personified agents in the interface. }2. What TYPE OF SUBMISSION is appropriate for this paper (see bottom of Cover Sheet Two for the authors' selection)? If your assessment of type differs from the Authors, please explain why you made your selection. In the next two questions, use the criteria specific to this Type of Submission (see above) to assess the paper. {: OK }3. Describe the IMPORTANCE OF THE PROBLEM, addressing the following concerns: Is it new and significant? Is it a useful and relevant problem? Do they sufficiently motivate the problem? Did you learn anything? Does the issue apply to CHI? {: The paper attempts to answer whether software agents should be personified in the interface. It is an important study because it attempts to answer this question empirical by conducting a well-designed experiment and by providing quantitative analysis that would indicate whether adding a face to an interface can improve the interaction between the user and the computer. I visited the Web page, reference number 4, and enjoyed participating in the experiment. }4. Describe the QUALITY OF THE ARGUMENT, addressing the following: Does the analysis use appropriate methodology? Is the argument well-structured? Do they provide sufficient data and/or well-supported arguments? Do they cite relevant work? Is the paper appropriately focused? Does the analysis cover all the important issues at the appropriate level of detail? {: I enjoyed reading the paper. I think that the paper is well structured, and the analysis of the empirical results is convincing. } 5. Describe the QUALITY OF THE MECHANICS, addressing the following: Do they clearly describe what was done and/or how it was studied? Is the writing clear and concise? Do they provide the right level of detail? Do they use too much jargon? Do the figures support the text? {: The tables of results are well presented. The aim of the study and the conclusions clearly address the relevant issues. }6. Provide any OTHER COMMENTS you believe would be useful to the author (including pointers to missing relevant work). {: In the first paragraph in the INTRODUCTION section, I suggest that references [9] and [7] be moved directly after the names Negroponte and Maes, respectively. In page number 4, in the subsection `Lab Experiment' (first paragraph): ``... and engagingness as an opponent poker...'' In the same page, in subsection `Questionnaire' (second paragraph): ``... each player.is'' - remove the period (.) }7. Using the scale 1 = Definitely reject 2 = Probably reject 3 = Could go either way 4 = Probably accept 5 = Definitely accept provide a numerical rating of the paper's ACCEPTABILITY {: 5 }8. SUMMARIZE your ASSESSMENT of the paper, pointing out which aspects described above you weighted most heavily in your rating in question 7? {: Interesting area, well designed experiment, good analysis, well structured paper. }9. Using the scale 1 = Very unconfident, really just a guess 2 = Rather unconfident, but I know a bit 3 = Moderately confident, I know as much as most 4 = Pretty confident, I know this area well 5 = Extremely confident, I consider myself an expert rate your EXPERTISE in the area addressed by the paper. {: 4 } +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ++++++++++++ CHI97 Paper Review Form-End Banner ++++++++++++ ++++++++++++ INCLUDE AT THE BOTTOM OF EACH FORM +++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +++++++++++ CHI97 Paper Review Form-Begin Banner +++++++++++ +++++++++++++ INCLUDE AT THE TOP OF EACH REVIEW +++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ All your input should appear after a curly bracket followed by a colon. PAPER NUMBER {: 1036 } PAPER TITLE {: User Reactions to Agents with Faces } REVIEWER NUMBER {: 284 }1. Briefly SUMMARIZE the paper {: The reactions of people to software agents with a variety of "face types" in a computer-based poker game were assessed. }2. What TYPE OF SUBMISSION is appropriate for this paper (see bottom of Cover Sheet Two for the authors' selection)? If your assessment of type differs from the Authors, please explain why you made your selection. In the next two questions, use the criteria specific to this Type of Submission (see above) to assess the paper. {: Empirical--appropriate }3. Describe the IMPORTANCE OF THE PROBLEM, addressing the following concerns: Is it new and significant? Is it a useful and relevant problem? Do they sufficiently motivate the problem? Did you learn anything? Does the issue apply to CHI? {: As the paper points out, faces have been used but seldom systematically tested. It is good that they be tested. }4. Describe the QUALITY OF THE ARGUMENT, addressing the following: Does the analysis use appropriate methodology? Is the argument well-structured? Do they provide sufficient data and/or well-supported arguments? Do they cite relevant work? Is the paper appropriately focused? Does the analysis cover all the important issues at the appropriate level of detail? {: I felt the problem to be well-conceived and the work competently carried out. They cited a number of relevant works. } 5. Describe the QUALITY OF THE MECHANICS, addressing the following: Do they clearly describe what was done and/or how it was studied? Is the writing clear and concise? Do they provide the right level of detail? Do they use too much jargon? Do the figures support the text? {: The writing style was clear. There were a few grammatical/orthographic errors including: Introduction, par. 3: "does not always make the experience [WHAT] for users"; Introduction, par. 4: "It is possible to make [the] advantages..."; and later in Facial Expression Design, "exited" should be "excited". The figures were pretty useful. Once into the data there was a LOT of detail, which makes it something of a "heavy read". It was better the second time through. }6. Provide any OTHER COMMENTS you believe would be useful to the author (including pointers to missing relevant work). {: }7. Using the scale 1 = Definitely reject 2 = Probably reject 3 = Could go either way 4 = Probably accept 5 = Definitely accept provide a numerical rating of the paper's ACCEPTABILITY {: 4 }8. SUMMARIZE your ASSESSMENT of the paper, pointing out which aspects described above you weighted most heavily in your rating in question 7? {: I think that the relative utility of different face types is an important question and the evaluation was appropriate. }9. Using the scale 1 = Very unconfident, really just a guess 2 = Rather unconfident, but I know a bit 3 = Moderately confident, I know as much as most 4 = Pretty confident, I know this area well 5 = Extremely confident, I consider myself an expert rate your EXPERTISE in the area addressed by the paper. {: 3 } +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ++++++++++++ CHI97 Paper Review Form-End Banner ++++++++++++ ++++++++++++ INCLUDE AT THE BOTTOM OF EACH FORM +++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +++++++++++ CHI97 Paper Review Form-Begin Banner +++++++++++ +++++++++++++ INCLUDE AT THE TOP OF EACH REVIEW +++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ All your input should appear after a curly bracket followed by a colon. PAPER NUMBER {: 1036 } PAPER TITLE {: User Reactions to Agents with Faces } REVIEWER NUMBER {: 293 }1. Briefly SUMMARIZE the paper {: The author created a computerized poker game in which the opponents were depicted by agents with various facial expressions. Realistic and cartoon male/female faces, a cartoon dog, a smiley, an the invisible man (woman?), were possible players. After a short poker game, users filled out a short questionnaire about their impressions of the players. Results were summarized. }2. What TYPE OF SUBMISSION is appropriate for this paper (see bottom of Cover Sheet Two for the authors' selection)? If your assessment of type differs from the Authors, please explain why you made your selection. In the next two questions, use the criteria specific to this Type of Submission (see above) to assess the paper. {: EMPIRICAL Agents User Studies }3. Describe the IMPORTANCE OF THE PROBLEM, addressing the following concerns: Is it new and significant? Is it a useful and relevant problem? Do they sufficiently motivate the problem? Did you learn anything? Does the issue apply to CHI? {: I don't know if it's important, but it's sort of fun. I don't know anything about this area, but I have trouble taking it too seriously. I did not learn anything. Perhaps people in the area would find the data useful as it is a reasonable survey collecting some basic data. Relevant to CHI? Sure. There are motivational issues, and there are issues in using facial expressions to provide some information. }4. Describe the QUALITY OF THE ARGUMENT, addressing the following: Does the analysis use appropriate methodology? Is the argument well-structured? Do they provide sufficient data and/or well-supported arguments? Do they cite relevant work? Is the paper appropriately focused? Does the analysis cover all the important issues at the appropriate level of detail? {: I thought the study and analysis was well done. That is an area in which I am qualified. The author collected some reasonable data (i.e., the questions used and the rating scale), although the subject population is unknown, and so generalization to general populations is questionable. } 5. Describe the QUALITY OF THE MECHANICS, addressing the following: Do they clearly describe what was done and/or how it was studied? Is the writing clear and concise? Do they provide the right level of detail? Do they use too much jargon? Do the figures support the text? {: I thought the paper was well written. The tables could be made more regular by adopting one statistical test (i.e., always use F) and report it in the standard format (i.e., F(1,df)=XXX.) Don't make up a new format for F tests (i.e., F(XXX,df2)); the standard format has been used 99.999% of the time for decades. The labels for the graphs (Fig 5 - 9) could be clearer, but I liked the 95% CI bars (although I don't know which of the several common methods was used to compute them). }6. Provide any OTHER COMMENTS you believe would be useful to the author (including pointers to missing relevant work). {: }7. Using the scale 1 = Definitely reject 2 = Probably reject 3 = Could go either way 4 = Probably accept 5 = Definitely accept provide a numerical rating of the paper's ACCEPTABILITY {:3 }8. SUMMARIZE your ASSESSMENT of the paper, pointing out which aspects described above you weighted most heavily in your rating in question 7? {: I do not take this area of research too seriously, especially when it is applied to an unidentified population. If it were targeted at a specific population, say kids, then I think it would have more merit. The sample population has unknown characteristics, so it is difficult to make any generalizations. Still, the analysis was well done, and the neatness factor is high. }9. Using the scale 1 = Very unconfident, really just a guess 2 = Rather unconfident, but I know a bit 3 = Moderately confident, I know as much as most1036 4 = Pretty confident, I know this area well 5 = Extremely confident, I consider myself an expert rate your EXPERTISE in the area addressed by the paper. {: I'll pass on this rating. I give me a 5 on methodology, and 1 on content area. } +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ++++++++++++ CHI97 Paper Review Form-End Banner ++++++++++++ ++++++++++++ INCLUDE AT THE BOTTOM OF EACH FORM +++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ________________________________________________ Lisa Tweedie Tel: 0171 589 5111 ext:56210 Dept. of Electrical Eng. Imperial College Exhibition Road South Kensington London SW7 http://www.ee.ic.ac.uk/research/information/www/lisat.html From l.tweedie@ic.ac.uk Mon Jan 6 12:29:37 1997 Received: from romeo.ic.ac.uk (romeo.ic.ac.uk [155.198.5.9]) by mail.acm.org (8.7.5/8.7.5) with SMTP id MAA39452 for ; Mon, 6 Jan 1997 12:29:09 -0500 From: l.tweedie@ic.ac.uk Received: from judy.ic.ac.uk [155.198.5.5] by romeo.ic.ac.uk with esmtp (Exim 0.57 #1) id 0vhItH-0005Or-00; Mon, 6 Jan 1997 17:31:15 +0000 Received: from dinghy.ee.ic.ac.uk (root@dinghy.ee.ic.ac.uk [155.198.123.8]) by judy.ic.ac.uk (8.7.5/8.7.5) with SMTP id RAA20115; Mon, 6 Jan 1997 17:30:58 GMT Received: from [155.198.123.20] (pixie) by dinghy.ee.ic.ac.uk (4.1/4.0) id AA24425; Mon, 6 Jan 97 17:30:55 GMT X-Sender: lisat@155.198.123.8 Message-Id: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Date: Mon, 6 Jan 1997 17:31:36 +0000 To: (Recipient list suppressed) Subject: CHI reviews paper 1036 Part 1 Status: R Hi Sorry I have made an error Part 1 should read as below (I mistakenly repeated Part 2 twice) Lisa Tweedie ++++++++++++++ Meta reviewer: +++++++++++++++++++++++ ++++++++++++++++++++++++++++++++++++++++++++++++++++++ ++++++ CHI97 Meta-Review Form-Begin Banner ++++++++++ +++++++ INCLUDE AT THE TOP OF EACH FORM ++++++++++++++ ++++++++++++++++++++++++++++++++++++++++++++++++++++++ PAPER NUMBER {:1036 } PAPER TITLE {:User Reactions to Agents with Faces } AUTHORS {:Tomoko Koda, Pattie Maes }1. Briefly SUMMARIZE the paper {:The paper reports the results of an experiment into the use of personalized agents. The experiment was run over the Web and a pilot study was run in the lab. Subjects played poker games against personlised agents of various types. Qualitative and Empirical results are given. }2. Summarize the reviewers' comments on the IMPORTANCE OF THE PROBLEM underlying this paper (e.g., significance, methodology, relevance to CHI). {: The reviewers had a slightly varied view of the importance of this field. 4 seemed in favour (it was fun and important to do studies on) and the other two seemed to think it was not that important. Generally the reviewers seemed to feel that it was relevant to CHI and that the methodology was correct. }3. Summarize the reviewers' comments on the QUALITY OF THE ARGUMENT in this paper (e.g. completeness, focus, rationale, data analysis, connections to related work). {: Three of the reviewers thought this was excellent. The three other reviewers made some interesting points about the generalizablity and external validity of these results - if this paper were to be accepted these points should perhaps be explored in more detail. }4. Summarize the reviewers' comments on the QUALITY OF THE MECHANICS of this paper (e.g., writing clarity, level of detail, figures). {:The reviewers provided a number of suggestion of ways to improve the overall written English, the Results and the Tables. The results were in fact very confusing it would perhaps have been better to focus on just a few results that showed interested information and interpreted those results. I also feel that although you clearly defined the research questions at the beginning of the paper the related answers were not as clearly presented and carried through. }5. Provide YOUR OWN ASSESSMENT of the paper, in particular, any issues you wish to raise that the reviewers did not. {:Like some of the reviewers I would liked to have seen some discussion of the problems with using this way of getting subjects. Doing experiments over the Web is a very tempting proposition to many of us so we would like to learn from your experience! It is would have been better to edit down the number of results that you presented, it may be useful to provide a little more interpretation. Also there are a number of typos (e.g. spelling of excited on second page) which need to be corrected. There was also little reference in this work to other papers on emotion or facial experession. }6. Give the average of the reviewers acceptability rating {: 3.6 }7. Using the scale [1.0,5.0] 1.0 = Definitely reject 2.0 = Probably reject 3.0 = Could go either way 4.0 = Probably accept 5.0 = Definitely accept provide a numeric RECOMMENDATION for this paper. Please base this on the original reviewers' ratings, weighting individual reviewer's recommendations, if needed, so that the assessments you consider most important have the most influence. (If you believe the original average rating accurately assesses this paper, please copy it here). {: 3 }7. Describe your RATIONALE for the recommendation in question 6. How did you adjust the weights of the original reviews? What factors described above did you weight the most heavily? What factors did you discount? {: I lowered the rating because there seemed to be serious problems with the mechanics of this paper. } +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ++++++++++++++ CHI96 Meta-Review Form-End Banner +++++++++++++++ ++++++++++++++ INCLUDE AT THE BOTTOM OF EACH FORM +++++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +++++++++++ CHI97 Paper Review Form-Begin Banner +++++++++++ +++++++++++++ INCLUDE AT THE TOP OF EACH REVIEW +++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ All your input should appear after a curly bracket followed by a colon. PAPER NUMBER {: 1036 } PAPER TITLE {: User reactions to agents with faces } REVIEWER NUMBER {: 0010 }1. Briefly SUMMARIZE the paper {: Users reactions to agents (poker game players) with different types of faces are studied in the context of participation in a poker game with the agents. Personifications of agents were useful in the in the poker game context in the sense that most users considered the faces likable, engaging and comfortable. Evaluation of faces in isolation differed from evaluation of the faces as part of agents in the poker game. Users who liked "personification" of agents differed in their evaluation of face features from users who disliked personification. Men and women differed in their evaluation of faces. }2. What TYPE OF SUBMISSION is appropriate for this paper (see bottom of Cover Sheet Two for the authors' selection)? If your assessment of type differs from the Authors, please explain why you made your selection. In the next two questions, use the criteria specific to this Type of Submission (see above) to assess the paper. {: Empirical }3. Describe the IMPORTANCE OF THE PROBLEM, addressing the following concerns: Is it new and significant? Is it a useful and relevant problem? Do they sufficiently motivate the problem? Did you learn anything? Does the issue apply to CHI? {: The problem is new but not extremely significant. Do they sufficiently motivate the problem? Yes. Did you learn anything? Yes. Does the issue apply to CHI? Yes. }4. Describe the QUALITY OF THE ARGUMENT, addressing the following: Does the analysis use appropriate methodology? Is the argument well-structured? Do they provide sufficient data and/or well-supported arguments? Do they cite relevant work? Is the paper appropriately focused? Does the analysis cover all the important issues at the appropriate level of detail? {: Does the analysis use appropriate methodology? Is the argument well-structured? Do they provide sufficient data and/or well-supported arguments? Yes, but the representatives issue (how far do the results generalize?) is not well worked out neither with respect to users nor with respect to the categories used to describe the faces (since only one example of each category was used and conclusions were drawn about the whole category: male female caricature, human, non-human etc.) Do they cite relevant work? Not sure. Is the paper appropriately focused? Yes. Does the analysis cover all the important issues at the appropriate level of detail? Yes, except questions about external validity. Unclear to me why authors state (on p.1 col 2) that the paper has a focus on implementation rather than on user testing. The authors should provide background data for the respondents that agreed and those that did not agree. (P. 7 col 2) Not clear that subjects' opinions about personification affected their attitude towards the different faces since only group separation is the data background for this conclusion ("correlation" data). Authors should also discuss those users who deviated from the main (mean or majority) conclusion about faces and how large these groups were. Maybe these users are behind the conclusion in the last sentence of the paper. Apart from this the reasons for the conclusion in the last sentence of the paper is unclear } 5. Describe the QUALITY OF THE MECHANICS, addressing the following: Do they clearly describe what was done and/or how it was studied? Is the writing clear and concise? Do they provide the right level of detail? Do they use too much jargon? Do the figures support the text? {: I answer these questions in a positive way (i.e. not too much jargon) except the following: (p.1) explain/define agent on a general level. (on p.1 col 2, slightly above middle of column) "but does not always makes the experience for users if it is added..." word missing here? (P.3 col 2 line 9 under EXPRESSIVENESS) probably the word here should be "different" rather than "same" in "same facial expression". (p4 col 1) use "were" rather than "are" in the section on the questionnaire. (p4 col 2) Noface in heading should be No face. delete the abbreviations for the variables in the Tables. }6. Provide any OTHER COMMENTS you believe would be useful to the author (including pointers to missing relevant work). {: - }7. Using the scale 1 = Definitely reject 2 = Probably reject 3 = Could go either way 4 = Probably accept 5 = Definitely accept provide a numerical rating of the paper's ACCEPTABILITY {: 3 }8. SUMMARIZE your ASSESSMENT of the paper, pointing out which aspects described above you weighted most heavily in your rating in question 7? {: Of some interest. The weakest part is the generality aspect (i.e. unclear conclusions concerning the external validity of the results, both with respect to categories of people and with respect to the "pictures") }9. Using the scale 1 = Very unconfident, really just a guess 2 = Rather unconfident, but I know a bit 3 = Moderately confident, I know as much as most 4 = Pretty confident, I know this area well 5 = Extremely confident, I consider myself an expert rate your EXPERTISE in the area addressed by the paper. {: 3 } +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ++++++++++++ CHI97 Paper Review Form-End Banner ++++++++++++ ++++++++++++ INCLUDE AT THE BOTTOM OF EACH FORM +++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +++++++++++ CHI97 Paper Review Form-Begin Banner +++++++++++ +++++++++++++ INCLUDE AT THE TOP OF EACH REVIEW +++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ All your input should appear after a curly bracket followed by a colon. PAPER NUMBER {:1036 } PAPER TITLE {:User Reactions to Agents with Faces } REVIEWER NUMBER {:259 }1. Briefly SUMMARIZE the paper {:Quantitative study of how users experience agents depending on how they are presented i.e. natural faces, artificial faces, non human faces or no faces. Different experiments have been performed to measure the effects of both presentation of different faces and expressions as well as effects of the technology itself. }2. What TYPE OF SUBMISSION is appropriate for this paper (see bottom of Cover Sheet Two for the authors' selection)? If your assessment of type differs from the Authors, please explain why you made your selection. In the next two questions, use the criteria specific to this Type of Submission (see above) to assess the paper. {: }3. Describe the IMPORTANCE OF THE PROBLEM, addressing the following concerns: Is it new and significant? Is it a useful and relevant problem? Do they sufficiently motivate the problem? Did you learn anything? Does the issue apply to CHI? {:The problem is not new. It has been discussed before by Patti Maes from MIT Medialab. Although it is a useful and relevant problem. The problem could be better motivated and also the generalization of the results might be better discussed. I learned something about gender differences. The issue does apply to CHI. }4. Describe the QUALITY OF THE ARGUMENT, addressing the following: Does the analysis use appropriate methodology? Is the argument well-structured? Do they provide sufficient data and/or well-supported arguments? Do they cite relevant work? Is the paper appropriately focused? Does the analysis cover all the important issues at the appropriate level of detail? {:I think the authors should put in more in details discussing why they use quantitative methods and analysis and not qualitative ones. The response rate is quite low and I would like to see a discussion about how the authors think this is influencing their results. Relevant work is cited. } 5. Describe the QUALITY OF THE MECHANICS, addressing the following: Do they clearly describe what was done and/or how it was studied? Is the writing clear and concise? Do they provide the right level of detail? Do they use too much jargon? Do the figures support the text? {:The authors should restructure the results by connecting explanatory text directly after the tables. This would facilitate reading and understanding of the results. }6. Provide any OTHER COMMENTS you believe would be useful to the author (including pointers to missing relevant work). {: }7. Using the scale 1 = Definitely reject 2 = Probably reject 3 = Could go either way 4 = Probably accept 5 = Definitely accept provide a numerical rating of the paper's ACCEPTABILITY {:3 }8. SUMMARIZE your ASSESSMENT of the paper, pointing out which aspects described above you weighted most heavily in your rating in question 7? {:The structure of the text and tables in the results part. This could be worked out to facilitate the reader's understanding. Try to make it clearer. }9. Using the scale 1 = Very unconfident, really just a guess 2 = Rather unconfident, but I know a bit 3 = Moderately confident, I know as much as most 4 = Pretty confident, I know this area well 5 = Extremely confident, I consider myself an expert rate your EXPERTISE in the area addressed by the paper. {:4 } +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ++++++++++++ CHI97 Paper Review Form-End Banner ++++++++++++ ++++++++++++ INCLUDE AT THE BOTTOM OF EACH FORM +++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +++++++++++ CHI97 Paper Review Form-Begin Banner +++++++++++ +++++++++++++ INCLUDE AT THE TOP OF EACH REVIEW +++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ All your input should appear after a curly bracket followed by a colon. PAPER NUMBER {: 1036 } PAPER TITLE {: User reactions to agent with faces } REVIEWER NUMBER {: 278 }1. Briefly SUMMARIZE the paper {: The paper describes an experiment to assess the reaction of a user to interaction with agents represented by faces. The experiment is based on a simulation of a poker game. }2. What TYPE OF SUBMISSION is appropriate for this paper (see bottom of Cover Sheet Two for the authors' selection)? If your assessment of type differs from the Authors, please explain why you made your selection. In the next two questions, use the criteria specific to this Type of Submission (see above) to assess the paper. {:Empirical }3. Describe the IMPORTANCE OF THE PROBLEM, addressing the following concerns: Is it new and significant? Is it a useful and relevant problem? Do they sufficiently motivate the problem? Did you learn anything? Does the issue apply to CHI? {: The problem is an emerging one and it is matter of debate in research on agents. The paper sufficiently motivates the problem which is of interest for the CHI community. }4. Describe the QUALITY OF THE ARGUMENT, addressing the following: Does the analysis use appropriate methodology? Is the argument well-structured? Do they provide sufficient data and/or well-supported arguments? Do they cite relevant work? Is the paper appropriately focused? Does the analysis cover all the important issues at the appropriate level of detail? {: The paper extends previous, quoted experiments and appears to refine their results. The methodology is adequate, arguments are based on the outcomes of an experiment using appropriate statistics. The references are also adequate. A weakness is that the authors do not state what their model for an agent is. However the whole experiment appears to me questionable: in a poker scenario it is expected that the players do not manifest their emotion, so that the discussion is obscured by the improper situation. I am not a psychologist and a bad poker player, but as a visual system designer I am not able to assess the validity of the results and I would have preferred a more realistic and significant scenario. } 5. Describe the QUALITY OF THE MECHANICS, addressing the following: Do they clearly describe what was done and/or how it was studied? Is the writing clear and concise? Do they provide the right level of detail? Do they use too much jargon? Do the figures support the text? {: The mechanics are good in all respects }6. Provide any OTHER COMMENTS you believe would be useful to the author (including pointers to missing relevant work). {: The discussion on personification is linked to the model adopted for the agent and also, I believe, to the situation in which it is used. As a visual system designer, I wonder if personification can be of use to synthesize a complex situation for a person who is not in the condition to capture all the details behind the situation, but needs to get a quick indicator of what is going on. I would be interested in understanding for example if personification is useful in the control of a plant or in some other reasonable scenario. I am afraid that the results of these papers do not help me in responding to these questions. }7. Using the scale 1 = Definitely reject 2 = Probably reject 3 = Could go either way 4 = Probably accept 5 = Definitely accept provide a numerical rating of the paper's ACCEPTABILITY {: 3 }8. SUMMARIZE your ASSESSMENT of the paper, pointing out which aspects described above you weighted most heavily in your rating in question 7? {: My major concern was that the work did not provide any guidelines on how to carry out an evaluation if a task requires personification as a communication tool }9. Using the scale 1 = Very unconfident, really just a guess 2 = Rather unconfident, but I know a bit 3 = Moderately confident, I know as much as most 4 = Pretty confident, I know this area well 5 = Extremely confident, I consider myself an expert rate your EXPERTISE in the area addressed by the paper. {: 4 I know the agent literature professionally and I have an overall knowledge of personification } +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ++++++++++++ CHI97 Paper Review Form-End Banner ++++++++++++ ++++++++++++ INCLUDE AT THE BOTTOM OF EACH FORM +++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ________________________________________________ Lisa Tweedie Tel: 0171 589 5111 ext:56210 Dept. of Electrical Eng. Imperial College Exhibition Road South Kensington London SW7 http://www.ee.ic.ac.uk/research/information/www/lisat.html From epfran1@PacBell.COM Mon Jan 6 20:48:28 1997 Received: from mailer.oclc.org (mailer.dev.oclc.org) by oclc.org (4.1/SMI-4.1) id AA12487; Mon, 6 Jan 97 20:51:37 EST Received: from gw3.pacbell.com by mailer.oclc.org (5.0/SMI-4.1) id AA22645; Mon, 6 Jan 1997 20:51:36 +0500 Received: from srv.PacBell.COM (mother.srv.PacBell.COM) by gw3.pacbell.com (5.x/PacBell-10/18/95) id AA22841; Mon, 6 Jan 1997 17:51:17 -0800 Received: from zeus.srv.PacBell.COM by srv.PacBell.COM (4.1/Mother-7/26/95) id AA09064; Mon, 6 Jan 97 17:51:02 PST Received: from [129.245.90.15] by zeus.srv.PacBell.COM (4.1/SMI-4.1) id AA23119; Mon, 6 Jan 97 17:51:05 PST X-Sender: epfran1@zeus.srv.pacbell.com Message-Id: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Date: Mon, 6 Jan 1997 17:48:58 -0800 To: perlman@oclc.org, debra_herschmann@usccmail.lehman.com, dray.chi@xerox.com, nardi@apple.com, smetros@utk.edu, isensee@austin.ibm.com, ellen@communities.com, pete@dcs.qmw.ac.uk, csnyder@uie.com, karen@acm.org, sarahb@werple.mira.net.au, nass@leland.stanford.edu, poynton@poynton.com, gershon@mitre.org From: epfran1@PacBell.COM (Ellen Francik) Subject: Tutorial notes production details Cc: chi97-tutorials@acm.org Content-Length: 1759 Status: R For those of you who are still working on tutorial notes, you might want to watch out for the following: - Check the numbering. The Table of Contents should be page i if you put it before the Agenda (the production instructions are somewhat inconsistent on this point). Also, the Arabic numerals for the body of notes start over again at 1. - Please include an Abstract in the Roman-numbered pages of the notes. It's on p. 1 of "How to Prepare your Tutorial Notes," but for some reason didn't make the more detailed how-to checklist on p. 2. Sorry about that. - For your camera-ready and electronic Companion documents, make sure the columns on p. 2 are of even length. - Also, you may want to add your own copyright notice to the Companion document. CHI will add a copyright notice to the notes per se, but I don't believe they are planning to do so for the Companion 2-pager. - Be sure you download the template for the Companion publication. That will use Helvetica for the headers. If you just copy and paste from the Web site (or copy the paper document in your author kit) you'll end up with serif fonts for headers. - Please send 3, not 2, camera-ready copies of the Companion document. - Margins around your notes text need to be 1 inch - especially in the left margin, where the notes will be bound. Otherwise attendees will lose your great content. - It would be helpful if you confirmed with me that you've ftp'd your electronic publication to the ACM site. (If you've enclosed a disk, I'll take care of it.) Thanks for your attention, Ellen Ellen Francik Pacific Bell CHI 97 Tutorials Co-Chair 2600 Camino Ramon, Rm. 3E050H chi97-tutorials@acm.org San Ramon, CA 94583 USA +1 510-867-8662 From Betsy_Comstock@smtpnotes.pictel.com Fri Jan 10 09:44:53 1997 Received: from gateway-gw.pictel.com (gateway-gw.pictel.com [140.242.1.1]) by mail.acm.org (8.7.5/8.7.5) with SMTP id JAA26420 for ; Fri, 10 Jan 1997 09:44:50 -0500 Received: from roadrunner.pictel.com by gateway-gw.pictel.com (4.1/cf.gw.940128.1740) id AA05506; Fri, 10 Jan 97 09:47:32 EST Received: from smtpnotes.pictel.com by roadrunner.pictel.com (4.1/runner.910925.1) id AA29588; Fri, 10 Jan 97 09:47:43 EST Received: by smtpnotes.pictel.com (4.1/SMI-4.1) id AA00276; Fri, 10 Jan 97 09:47:18 EST Message-Id: <9701101447.AA00276@smtpnotes.pictel.com> Received: from PicTel with "Lotus Notes Mail Gateway for SMTP" id 742F173C752DB1908525641B0050BC7C; Fri, 10 Jan 97 14:47:14 To: Betsy Comstock/PicTel From: Betsy Comstock/PicTel Date: 10 Jan 97 9:43:36 EDT Subject: CHI meta review #1098 Mime-Version: 1.0 Content-Type: Text/Plain Status: RO To reviewers of CHI paper submission #1098: Thank you VERY much for your hard work in reviewing paper submissions for CHI. Appended is the meta review for paper submission #1098. I hope you'll find it useful to read other reviewers' comments. Please let me know if the mail system renders it unreadable or if for some other reason you'd prefer a hard copy. My sincere apologies for the delay in getting this meta review to you. My email access was interrupted when I changed jobs (I'm now happily working on making easy-to-use videoconferencing products at PictureTel). Thanks again, and I hope I see you at CHI97! Betsy Comstock comstock@pictel.com ++++++++++++++ Meta reviewer: +++++++++++++++++++++++ ++++++++++++++++++++++++++++++++++++++++++++++++++++++ ++++++ CHI97 Meta-Review Form-Begin Banner ++++++++++ +++++++ INCLUDE AT THE TOP OF EACH FORM ++++++++++++++ ++++++++++++++++++++++++++++++++++++++++++++++++++++++ PAPER NUMBER {: 1098 } PAPER TITLE {: Learning personal preferences on online newspaper articles from user behaviors } AUTHORS {: Hidekazu Sakagmi and Tomonari Kamba }1. Briefly SUMMARIZE the paper {: This paper presents methods for tailoring the delivery of WWW newpaper articles to individuals based on their interests. It demonstrates that an effective method was to score articles based on implicit cues (whether the user scrolled and/or expanded the article), combined with the option for users to adjust the score explicitly if they wanted. }2. Summarize the reviewers' comments on the IMPORTANCE OF THE PROBLEM underlying this paper (e.g., significance, methodology, relevance to CHI). {: Reviewers agree that this is a very important problem for the CHI community to address. The topic is not new, and reviewers pointed out several related lines of work. In this paper, the mechanisms employed for implicit feedback and the balance between implicit (but inferior) and explicit methods are interesting. }3. Summarize the reviewers' comments on the QUALITY OF THE ARGUMENT in this paper (e.g. completeness, focus, rationale, data analysis, connections to related work). {: The paper is structured fairly well and the general method appears sound. It is good that the conclusions are based on data. However, there are several distinct problems pointed out by the reviewers: Authors need to cite more of the related work in this general area. There should be a Method section describing the subjects and exact procedures used during the experiments. The methods used here should be compared with other approaches so readers can tell how they compare in effectiveness. The logic behind the calculation of scores needs to be described; as it is it seems arbitrary. Statistical analysis should be performed so that we can understand how robust and generalizable the results are and what conclusions are warranted. Please refer to the individual reviewer comments for the detailed suggestions they made for improvements. }4. Summarize the reviewers' comments on the QUALITY OF THE MECHANICS of this paper (e.g., writing clarity, level of detail, figures). {: The reviewers considered the paper generally well constructed and easy to read. They made valuable suggestions on more effective data presentation and description. }5. Provide YOUR OWN ASSESSMENT of the paper, in particular, any issues you wish to raise that the reviewers did not. {: I think the reviewers did a good job of pointing out the strengths and weaknesses of this paper. }6. Give the average of the reviewers acceptability rating {: 3.00 }7. Using the scale [1.0,5.0] 1.0 = Definitely reject 2.0 = Probably reject 3.0 = Could go either way 4.0 = Probably accept 5.0 = Definitely accept provide a numeric RECOMMENDATION for this paper. Please base this on the original reviewers' ratings, weighting individual reviewer's recommendations, if needed, so that the assessments you consider most important have the most influence. (If you believe the original average rating accurately assesses this paper, please copy it here). {: 3.00 }7. Describe your RATIONALE for the recommendation in question 6. How did you adjust the weights of the original reviews? What factors described above did you weight the most heavily? What factors did you discount? {: This is definitely a paper that could have gone either way. The topic is increasingly important and interesting. I really enjoyed the clarity of the presentation. I would strongly encourage the authors to incorporate the thoughtful comments the reviewers made as they continue this line of work. } +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ++++++++++++++ CHI96 Meta-Review Form-End Banner +++++++++++++++ ++++++++++++++ INCLUDE AT THE BOTTOM OF EACH FORM +++++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +++++++++++ CHI97 Paper Review Form-Begin Banner +++++++++++ +++++++++++++ INCLUDE AT THE TOP OF EACH REVIEW +++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ All your input should appear after a curly bracket followed by a colon. PAPER NUMBER {1098: } PAPER TITLE {Learning personal preferences on online newspaper articles from user behaviors: } REVIEWER NUMBER {190: }1. Briefly SUMMARIZE the paper {The authors explore the development of end user content preferences for texts retrieved over the Web. They do this by observing reading behaviors for materials presented with different preference monitoring methdologies. Indirect preference extraction never achieved the same level of material personalization as explicit preference indication methods, but was paritally effective on its own. Better performance was achieved through a hybrid approach, mixing indirect and explicit feedback techniques.: }2. What TYPE OF SUBMISSION is appropriate for this paper (see bottom of Cover Sheet Two for the authors' selection)? If your assessment of type differs from the Authors, please explain why you made your selection. In the next two questions, use the criteria specific to this Type of Submission (see above) to assess the paper. {This was an empirical submission. It could also be a systems submission.: }3. Describe the IMPORTANCE OF THE PROBLEM, addressing the following concerns: Is it new and significant? Is it a useful and relevant problem? Do they sufficiently motivate the problem? Did you learn anything? Does the issue apply to CHI? {Material filtering has become an extremely important business problem for companies seeking to provide internet end user services. The topic is not new, however; a number of doctoral dissertations have been produced on this topic since 1990. : }4. Describe the QUALITY OF THE ARGUMENT, addressing the following: Does the analysis use appropriate methodology? Is the argument well-structured? Do they provide sufficient data and/or well-supported arguments? Do they cite relevant work? Is the paper appropriately focused? Does the analysis cover all the important issues at the appropriate level of detail? {The general approach is sound; it exemplifies good, empirically informed system design. Distinct and clearly contrasting feedback approaches are laid out and exercised. The experimental design and measurement methdology appear to be sound. Conclusions are based on real data.: } 5. Describe the QUALITY OF THE MECHANICS, addressing the following: Do they clearly describe what was done and/or how it was studied? Is the writing clear and concise? Do they provide the right level of detail? Do they use too much jargon? Do the figures support the text? {The paper really suffers from poor presentation. In particular, it lacks a clear Methods section, with an understandable description of who the subjects were, how many subjects were employed, the specific experimental procedure followed, and so on. Also, in spite of the illustrations of group effects, the paper completely lacks any STATISTICS. Measurements without statistics pertain only to the current sample, and cannot be generalized to the population as a whole. And if this is true, then why should any of the rest of us care about this work? Also, the connection of text to figures is very poor. The text should more explicitly refer to what the figure is supposed to say about the current argument. The paper could be more concisely formatted; page three of the article is an example of very poort figure/text layout. Finally, Table 1 is extremely unclear and unnecessary: It is not clear from Row 1 whether these are subject numbers or numbers of subjects were column. Also, there is no need to present individual subject data; cell means with measures of variance is sufficient.: }6. Provide any OTHER COMMENTS you believe would be useful to the author (including pointers to missing relevant work). {: }7. Using the scale 1 = Definitely reject 2 = Probably reject 3 = Could go either way 4 = Probably accept 5 = Definitely accept provide a numerical rating of the paper's ACCEPTABILITY {3: }8. SUMMARIZE your ASSESSMENT of the paper, pointing out which aspects described above you weighted most heavily in your rating in question 7? {In general, this looks like a sound piece of work with a sound approach and experimental comparison. But it comes across like a piece of empirical experimental work done by a computer scientist without any experimental training. I think the topic is very important and very interesting. I think there is material to be learned from the work. But the presentation so significantly undermines our understanding of, and confidence in, the results, that I'm not sure it can be published in its current form.: }9. Using the scale 1 = Very unconfident, really just a guess 2 = Rather unconfident, but I know a bit 3 = Moderately confident, I know as much as most 4 = Pretty confident, I know this area well 5 = Extremely confident, I consider myself an expert rate your EXPERTISE in the area addressed by the paper. {3: } +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ++++++++++++ CHI97 Paper Review Form-End Banner ++++++++++++ ++++++++++++ INCLUDE AT THE BOTTOM OF EACH FORM +++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +++++++++++ CHI97 Paper Review Form-Begin Banner +++++++++++ +++++++++++++ INCLUDE AT THE TOP OF EACH REVIEW +++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ All your input should appear after a curly bracket followed by a colon. PAPER NUMBER {: 1098 } PAPER TITLE {: Learning personal preferences on online newspaper articles from user behaviors } REVIEWER NUMBER {: 249 }1. Briefly SUMMARIZE the paper {: Describes techniques used in a web-based personalised newspaper service. Uses a combination of explicit and implicit feedback to determine user's interest in a given article. Implicit feedback was derived from scrolling and zooming activities. }2. What TYPE OF SUBMISSION is appropriate for this paper (see bottom of Cover Sheet Two for the authors' selection)? If your assessment of type differs from the Authors, please explain why you made your selection. In the next two questions, use the criteria specific to this Type of Submission (see above) to assess the paper. {: Empirical, though could also be considered Systems }3. Describe the IMPORTANCE OF THE PROBLEM, addressing the following concerns: Is it new and significant? Is it a useful and relevant problem? Do they sufficiently motivate the problem? Did you learn anything? Does the issue apply to CHI? {: Problem of learning users' interest profile and using it to assemble relevant information is important. Many researchers are working on it using approaches related to the one described here. This work does have some distinctive features, however, notably the implictit feedback mechanisms employed. }4. Describe the QUALITY OF THE ARGUMENT, addressing the following: Does the analysis use appropriate methodology? Is the argument well-structured? Do they provide sufficient data and/or well-supported arguments? Do they cite relevant work? Is the paper appropriately focused? Does the analysis cover all the important issues at the appropriate level of detail? {: For an 'empirical' paper, I found the data analysis weak. Some results are presented, but the conclusions to be drawn from them are not clear. } 5. Describe the QUALITY OF THE MECHANICS, addressing the following: Do they clearly describe what was done and/or how it was studied? Is the writing clear and concise? Do they provide the right level of detail? Do they use too much jargon? Do the figures support the text? {: Writing clear and concise. Right level of detail. }6. Provide any OTHER COMMENTS you believe would be useful to the author (including pointers to missing relevant work). {: }7. Using the scale 1 = Definitely reject 2 = Probably reject 3 = Could go either way 4 = Probably accept 5 = Definitely accept provide a numerical rating of the paper's ACCEPTABILITY {: 4 }8. SUMMARIZE your ASSESSMENT of the paper, pointing out which aspects described above you weighted most heavily in your rating in question 7? {: An entertaining paper containing some potentially practical ideas applied to an important problem. }9. Using the scale 1 = Very unconfident, really just a guess 2 = Rather unconfident, but I know a bit 3 = Moderately confident, I know as much as most 4 = Pretty confident, I know this area well 5 = Extremely confident, I consider myself an expert rate your EXPERTISE in the area addressed by the paper. {: 4 } +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ++++++++++++ CHI97 Paper Review Form-End Banner ++++++++++++ ++++++++++++ INCLUDE AT THE BOTTOM OF EACH FORM +++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +++++++++++ CHI97 Paper Review Form-Begin Banner +++++++++++ +++++++++++++ INCLUDE AT THE TOP OF EACH REVIEW +++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ All your input should appear after a curly bracket followed by a colon. PAPER NUMBER {:1098 } PAPER TITLE {:Learning personal preferences on online newspaper articles from user behaviors. } REVIEWER NUMBER {:272 }1. Briefly SUMMARIZE the paper {:Paper describes an experiment to test the efficacy of deriving user preference profiles from reading behaviour. It concludes that the optimum combination is to allow direct user input to the process. }2. What TYPE OF SUBMISSION is appropriate for this paper (see bottom of Cover Sheet Two for the authors' selection)? If your assessment of type differs from the Authors, please explain why you made your selection. In the next two questions, use the criteria specific to this Type of Submission (see above) to assess the paper. {:Empirical. }3. Describe the IMPORTANCE OF THE PROBLEM, addressing the following concerns: Is it new and significant? Is it a useful and relevant problem? Do they sufficiently motivate the problem? Did you learn anything? Does the issue apply to CHI? {:The problem of information filtering is important and relevant. The problem of making system-level inferences on the basis of user behaviour is an enduring one for CHI. }4. Describe the QUALITY OF THE ARGUMENT, addressing the following: Does the analysis use appropriate methodology? Is the argument well-structured? Do they provide sufficient data and/or well-supported arguments? Do they cite relevant work? Is the paper appropriately focused? Does the analysis cover all the important issues at the appropriate level of detail? {:Generally well structured and well argued, although some detail of how the number of bonus points was derived would have been useful -- it seems completely arbitrary. } 5. Describe the QUALITY OF THE MECHANICS, addressing the following: Do they clearly describe what was done and/or how it was studied? Is the writing clear and concise? Do they provide the right level of detail? Do they use too much jargon? Do the figures support the text? {:Writing is clear and level of detail is appropriate. }6. Provide any OTHER COMMENTS you believe would be useful to the author (including pointers to missing relevant work). {: }7. Using the scale 1 = Definitely reject 2 = Probably reject 3 = Could go either way 4 = Probably accept 5 = Definitely accept provide a numerical rating of the paper's ACCEPTABILITY {:4 }8. SUMMARIZE your ASSESSMENT of the paper, pointing out which aspects described above you weighted most heavily in your rating in question 7? {:A reasonably solid paper, not exciting or even a surprising result, but would be OK. }9. Using the scale 1 = Very unconfident, really just a guess 2 = Rather unconfident, but I know a bit 3 = Moderately confident, I know as much as most 4 = Pretty confident, I know this area well 5 = Extremely confident, I consider myself an expert rate your EXPERTISE in the area addressed by the paper. {:3 } +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ++++++++++++ CHI97 Paper Review Form-End Banner ++++++++++++ ++++++++++++ INCLUDE AT THE BOTTOM OF EACH FORM +++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +++++++++++ CHI97 Paper Review Form-Begin Banner +++++++++++ +++++++++++++ INCLUDE AT THE TOP OF EACH REVIEW +++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ All your input should appear after a curly bracket followed by a colon. PAPER NUMBER {: 1098 } PAPER TITLE {: Learning Personal Preferences on Online Newspaper Articles from User behaviors } REVIEWER NUMBER {: 275 }1. Briefly SUMMARIZE the paper {: The authors present an experiment for determining the effectiveness of explicit vs. implicit feedback on a WWW-based newspaper. }2. What TYPE OF SUBMISSION is appropriate for this paper (see bottom of Cover Sheet Two for the authors' selection)? If your assessment of type differs from the Authors, please explain why you made your selection. In the next two questions, use the criteria specific to this Type of Submission (see above) to assess the paper. {: Empirical }3. Describe the IMPORTANCE OF THE PROBLEM, addressing the following concerns: Is it new and significant? Is it a useful and relevant problem? Do they sufficiently motivate the problem? Did you learn anything? Does the issue apply to CHI? {: The problem that the authors are addressing in its most global form is an extemely important problem: how can we provide better methods to determine the relevancy of information for a given user. The authors are looking at a more narrow view of this: how to determine relevancy of articles for readers of an on-line newspaper. The major novel approach the authors are taking is to compare and combine implicit and explicit methods of scoring relevance. }4. Describe the QUALITY OF THE ARGUMENT, addressing the following: Does the analysis use appropriate methodology? Is the argument well-structured? Do they provide sufficient data and/or well-supported arguments? Do they cite relevant work? Is the paper appropriately focused? Does the analysis cover all the important issues at the appropriate level of detail? {: The authors' experiments seem to show that both implicit and explicit approaches to scoring relevancy can improve the perceived relevancy of delivered articles. Some statistics would have been extremely useful: what was the standard error? Did ALL users follow the same pattern? Were there any individual anomolies? A control group would have also been helpful, e.g., a group which provided explicit feedback, but the feedback was not used in determining the delivery of papers. Another concern I have about the conclusions is that the graph (6a) shows group C and group B on day 5 as being very close in value, and the slope for group C between day 4 and day 5 being very steep, yet the authors discount group C -- why? The most interesting result that the authors' achieve is when they add group D, which gets implicit feedback as well some explicit feedback. Again, some sense of the standard deviation would have been helpful, as would some sense of the relationship between the number of articles scored explicitly and the difference bettwen the system and the user rankings. Its also not clear from the data whether the implicit scoring had any impact at all. This would be a very important result considering that most systems will not have built-in Java agents to collect implicit information. The authors also only cite a very small portion of the work in this area, focusing on the work done to filter delivery of NetNews articles, but not including any of the work on Latent Semantic Indexing (LSI), or the work going on at NCSA in support of the Mosaic Digital Library Project, for example. } 5. Describe the QUALITY OF THE MECHANICS, addressing the following: Do they clearly describe what was done and/or how it was studied? Is the writing clear and concise? Do they provide the right level of detail? Do they use too much jargon? Do the figures support the text? {: There are some very minor language lapses, but altogether this is a well-written, clear paper which was easy to read and follow. }6. Provide any OTHER COMMENTS you believe would be useful to the author (including pointers to missing relevant work). {: }7. Using the scale 1 = Definitely reject 2 = Probably reject 3 = Could go either way 4 = Probably accept 5 = Definitely accept provide a numerical rating of the paper's ACCEPTABILITY {: 3 }8. SUMMARIZE your ASSESSMENT of the paper, pointing out which aspects described above you weighted most heavily in your rating in question 7? {: In spite of the methodological problems noted above, I could be persuaded that this paper has merit because of the importance of the topic, but the lack of even rudimentary statistics makes the results hard to interpret. }9. Using the scale 1 = Very unconfident, really just a guess 2 = Rather unconfident, but I know a bit 3 = Moderately confident, I know as much as most 4 = Pretty confident, I know this area well 5 = Extremely confident, I consider myself an expert rate your EXPERTISE in the area addressed by the paper. {: 3 } +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ++++++++++++ CHI97 Paper Review Form-End Banner ++++++++++++ ++++++++++++ INCLUDE AT THE BOTTOM OF EACH FORM +++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +++++++++++ CHI97 Paper Review Form-Begin Banner +++++++++++ +++++++++++++ INCLUDE AT THE TOP OF EACH REVIEW +++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ All your input should appear after a curly bracket followed by a colon. PAPER NUMBER {: 1098 } PAPER TITLE {: Learning personal preferences on online newspaper articles from user behaviors } REVIEWER NUMBER {: 284 }1. Briefly SUMMARIZE the paper {: The paper compares two methods of creating a routing system for newspaper articles--either by explicit user input or by using agent-based software that observes user behavior and creates profiles. }2. What TYPE OF SUBMISSION is appropriate for this paper (see bottom of Cover Sheet Two for the authors' selection)? If your assessment of type differs from the Authors, please explain why you made your selection. In the next two questions, use the criteria specific to this Type of Submission (see above) to assess the paper. {: Empirical }3. Describe the IMPORTANCE OF THE PROBLEM, addressing the following concerns: Is it new and significant? Is it a useful and relevant problem? Do they sufficiently motivate the problem? Did you learn anything? Does the issue apply to CHI? {: It is an interesting problem--especially when taken in a wider context. However, Current Awareness services have been doing the "direct extraction" method for almost 20 years and others within the IR community have worked on automatic routing, usually of newspaper articles. So it's not particularly novel. }4. Describe the QUALITY OF THE ARGUMENT, addressing the following: Does the analysis use appropriate methodology? Is the argument well-structured? Do they provide sufficient data and/or well-supported arguments? Do they cite relevant work? Is the paper appropriately focused? Does the analysis cover all the important issues at the appropriate level of detail? {: The text analysis methodology they use is fairly standard. However, the paper lacks many of the standard references to IR work in this area. I am frankly bothered by the evaluation strategy: assigning numeric values to ranks and then calculating scores is not really methodologically valid. This is especially bothersome when the authors conclude that these "scores" need to be "set properly". } 5. Describe the QUALITY OF THE MECHANICS, addressing the following: Do they clearly describe what was done and/or how it was studied? Is the writing clear and concise? Do they provide the right level of detail? Do they use too much jargon? Do the figures support the text? {: The paper is acceptably written. }6. Provide any OTHER COMMENTS you believe would be useful to the author (including pointers to missing relevant work). {: The TREC and TIPSTER conferences provide many examples of the best work in document routing systems. }7. Using the scale 1 = Definitely reject 2 = Probably reject 3 = Could go either way 4 = Probably accept 5 = Definitely accept provide a numerical rating of the paper's ACCEPTABILITY {: 2 }8. SUMMARIZE your ASSESSMENT of the paper, pointing out which aspects described above you weighted most heavily in your rating in question 7? {: I don't believe that there is sufficient novelty to include this paper. I am also bothered by the appropriateness of the evaluation. }9. Using the scale 1 = Very unconfident, really just a guess 2 = Rather unconfident, but I know a bit 3 = Moderately confident, I know as much as most 4 = Pretty confident, I know this area well 5 = Extremely confident, I consider myself an expert rate your EXPERTISE in the area addressed by the paper. {: 4 } +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ++++++++++++ CHI97 Paper Review Form-End Banner ++++++++++++ ++++++++++++ INCLUDE AT THE BOTTOM OF EACH FORM +++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +++++++++++ CHI97 Paper Review Form-Begin Banner +++++++++++ +++++++++++++ INCLUDE AT THE TOP OF EACH REVIEW +++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ All your input should appear after a curly bracket followed by a colon. PAPER NUMBER {: 1098 } PAPER TITLE {: Learning Personal Preferences on Online Newspaper Articles from User behaviors } REVIEWER NUMBER {: 293 }1. Briefly SUMMARIZE the paper {: The abstract is a good summary of the paper. The work compared creating a user interest profile (as represented by weighted keywords) by required explicit ratings of all articles, implicit (automated) assignment by monitoring browsing behavior, and combining implicit and optional explicit ratings. The result was that a combination of implicit with optional explicit ratings quickly approached the best rating scheme (required explicit), but with less effort. }2. What TYPE OF SUBMISSION is appropriate for this paper (see bottom of Cover Sheet Two for the authors' selection)? If your assessment of type differs from the Authors, please explain why you made your selection. In the next two questions, use the criteria specific to this Type of Submission (see above) to assess the paper. {: EMPIRICAL + experience + systems The work is primarily empirical, but the methods are a little shaky because the analysis methods seemed to be ad hoc. }3. Describe the IMPORTANCE OF THE PROBLEM, addressing the following concerns: Is it new and significant? Is it a useful and relevant problem? Do they sufficiently motivate the problem? Did you learn anything? Does the issue apply to CHI? {: The work is novel in that it attempts to find a balance of implicit methods (providing inferior results) but requiring no work by users with explicit methods. I am not sure I learned anything because of two problems: (1) There is no comparison of the methods to known methods, so I don't know if their BEST method is better or worse than other approaches. (2) Their analysis was not explained in enough detail for me to understand it fully, so I am not sure how much of their results I trust. But, this is a useful area to explore. Network access to information is clearly a problem of interest. }4. Describe the QUALITY OF THE ARGUMENT, addressing the following: Does the analysis use appropriate methodology? Is the argument well-structured? Do they provide sufficient data and/or well-supported arguments? Do they cite relevant work? Is the paper appropriately focused? Does the analysis cover all the important issues at the appropriate level of detail? {: The methods seemed to be made up on the fly. The methods they used for attaching weights did not seem to be related to existing models in IR (e.g., vector space). Their rating scale was not a standard one, ranging from "very relevant" to "not relevant at all", which had poorly chosen end anchors and which they converted arbitrarily to a ratio scale. I did not understand the definition of the "rank order" measure, making it difficult for me to have high confidence in their results. They used no inferential statistics, and in some cases, it did not matter (e.g., in 6a, A is clearly better than B), but in some cases it did (e.g., in 6a, is B or C declining?). The authors approached some analyses intuitively, but that can be subjective, and some inferential stats on some subtle trends would have provided an objective view. In 6b, the authors put a line (I am not sure it is a regression line) through their points to show the correlation between subjective ranks and anticipated scores, but a closer view of the data suggest that such an analysis is inappropriate. For ratings A|B, the anticipated scores were highly ranked, and maybe even correlated within that subset (i.e., As were ranked higher than Bs). But for ratings C|D|E, there appears to be no correlation between rating and rank. This may be in part due to the lack of psychometric properties of their rating scale. Their conclusions were well stated, but I had to question one at the top of page six, column 1. When bonus points are set properly... IT DOES NOT WORK AT ALL WITH 30 BONUS POINTS (my caps) To my eye, the curve seems to be moving down over time, and perhaps with more data, it would asymptote much lower. the authors did not check for this trend, so they should not have such a strong statement. (This is independent of the issue of modeling using arbitrary values like 10 or 30 bonus points.) } 5. Describe the QUALITY OF THE MECHANICS, addressing the following: Do they clearly describe what was done and/or how it was studied? Is the writing clear and concise? Do they provide the right level of detail? Do they use too much jargon? Do the figures support the text? {: As mentioned, some measures and procedures were not well specified. The writing was adequate. The figures and overall organization were okay. }6. Provide any OTHER COMMENTS you believe would be useful to the author (including pointers to missing relevant work). {: Table 1: "Number of Subjects" -- change to "Subject Number" }7. Using the scale 1 = Definitely reject 2 = Probably reject 3 = Could go either way 4 = Probably accept 5 = Definitely accept provide a numerical rating of the paper's ACCEPTABILITY {: 2 }8. SUMMARIZE your ASSESSMENT of the paper, pointing out which aspects described above you weighted most heavily in your rating in question 7? {: Methodological problems in data collection and analysis make this paper of dubious value. The lack of references to the IR filtering literature is also a problem, but I believe the combination of implicit and explicit ratings to be novel, regardless. }9. Using the scale 1 = Very unconfident, really just a guess 2 = Rather unconfident, but I know a bit 3 = Moderately confident, I know as much as most 4 = Pretty confident, I know this area well 5 = Extremely confident, I consider myself an expert rate your EXPERTISE in the area addressed by the paper. {: 4 } +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ++++++++++++ CHI97 Paper Review Form-End Banner ++++++++++++ ++++++++++++ INCLUDE AT THE BOTTOM OF EACH FORM +++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ From perlman Fri Jan 24 21:10:18 1997 Received: by turing.acm.org; id AA01258; Fri, 24 Jan 1997 21:10:06 -0500 From: Gary PERLMAN Message-Id: <9701250210.AA01258@turing.acm.org> Subject: Re: Email for CHIkids registration (fwd) To: kiddie@millennianet.com (KiddieCorp) Date: Fri, 24 Jan 1997 21:10:06 -0500 (EST) Cc: perlman In-Reply-To: <199701131658.IAA12539@millennianet.com> from "KiddieCorp" at Jan 13, 97 08:58:20 am X-Mailer: ELM [version 2.4 PL25] Content-Type: text Content-Length: 1497 Status: O > >Dear CHI Parent, > > > >We understand you are looking for more information regarding CHIkids for > >CHI 97. Below describes the program and gives you more registration > >information. If you are interested in registering for CHIkids, please do > >so by sending back the forms below to KiddieCorp . > >You will not be fully registered however, until your registration fee is > >received by KiddiCorp. Thank you for the information about CHIkids for CHI 97. My son Mark will be 3 years 8 months by the time of CHI. I am interested in taking him to CHIkids as he enjoys some interactive multimedia games on his Mac and has become more interested recently in drawing with Kid Pix. My main concern is about backup in case he gets sick. We are fortunate that Mark has not missed a day due to illness for over a year, remarkable considering he is in daycare, but I am concerned about what I could do if he were to require my attention on Monday, March 24, when I am the only teacher of a full-day tutorial. The income to the conference of this tutorial has been over $50,000 for the past few years, so I would like to have backup arranged should Mark be sick that day. Are there any services coordinated through you to see to a sick child? Planning for the worst, like any parent, Gary PERLMAN, OCLC Online Computer Library Center 6565 Frantz Road, Dublin, Ohio 43017 USA Voice: +1-614-761-5058 Fax: +1-614-793-0915 Email: perlman@acm.org WWW: www.acm.org/~perlman From RCHARRON@mprgate.mpr.ca Sat Jan 25 11:32:51 1997 Received: from msgate.mpr.ca by mprgate.mpr.ca with SMTP id AA03598 (5.67b+/IDA-1.5 for ); Sat, 25 Jan 1997 08:33:09 -0800 Message-Id: <199701251629-3440@mpr.ca> Date: Sat, 25 Jan 1997 08:29:37 -0800 X-Mailer: Microsoft Mail with Intergate/SMTP (v1.Free) From: RCHARRON@mpr.ca (Rhona Charron) To: perlman@turing.acm.org Subject: RE: SIG at CHI'97 Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="_[INTERGATE-SMTP59248414]_" Status: RO --_[INTERGATE-SMTP59248414]_ Content-Type: text/plain; charset="us-ascii" I think I slotted you in on tuesday afternoon (tentatively OF COURSE) I do not think there will be any problem. We are reviewing all sigs ina couple of days. Stay tuned HEY where is your Canadian page eh? Resisting the push RHONA ---------- From: Gary PERLMAN[SMTP:perlman@turing.acm.org] Sent: January 24, 1997 6:23 PM To: rcharron@mprgate.mpr.ca Cc: perlman@turing.acm.org Subject: SIG at CHI'97 Well, you've had the proposals for over an hour, I'd like to know if the one on SIGCHI Information Infrastructure has been accepted and on what day an at what time it will take place. No, I'm not being pushy 'cause I went to Wagar. I am considering bring my son to CHIKids, and if the SIG will be late in the week, then I don't think I can muster the stamina to bring him for the whole week. I think my ideal time slot will be mid afternoon on Tuesday. By the way, the SIG will have information of special interest to Adjunct Chairs. Gary --_[INTERGATE-SMTP59248414]_ Content-Type: application/octet-stream; name="WINMAIL.DAT" Content-Transfer-Encoding: x-uuencode Content-Disposition: attachment; filename="WINMAIL.DAT" begin 644 WINMAIL.DAT M>)\^(A00`0:0"``$```````!``$``0>0!@`(````Y`0```````#H``$(@`<` M&````$E032Y-:6-R;W-O9G0@36%I;"Y.;W1E`#$(`0V`!``"`````@`"``$$ M@`$`$@```%)%.B!324<@870@0TA))SDW`%0$`06``P`.````S0``(P`0````4```!33510`````!X``S`!```` M%P```'!E M``$P`0````\````G1V%R>2!015),34%.)P```@$+,`$````<````4TU44#I0 M15),34%.0%154DE.1RY!0TTN3U)'``,``#D`````"P!`.@`````"`?8/`0`` M``0````````#$"X!`Y`&`"0%```0````"P`C```````#`"8```````L`*0`` M`````P`V``````!``#D`8+#N>=P*O`$>`'```0```!(```!213H@4TE'(&%T M($-(22``@0`0```&4```!)5$A)3DM)4TQ/5%1%1%E/ M54E.3TY454531$%9049415).3T].*%1%3E1!5$E614Q93T9#3U524T4I241/ M3D]45$A)3DM42$5215=)3$Q"14%.65!23T),14U714%215)%5DE%``````(! M"1`!````XP,``-\#``#R!@``3%I&=1+PRC'_``H!#P(5`J@%ZP*#`%`"\@D" M`&-H"L!S970R-P8`!L,"@S(#Q0(`<')"<1'B@ M(#8Z,C,U\$TX3]DRK51O.H\TZW(1<@-@E3=6)JQS"Q17\TZU-)1QT@13'X2$DG/4`L_QFO,'8C M(-,>,`>`'$!;$3\A$@&05$$+45D`(CU.;_L]`$V0;1_S(5`C\BQ2 M'1#>)T,P+&`@X!KP=QWQ5&+>5RH@"L`B-AKP85Y`!:!7`($$@2/R8F(#;1T0 M<]<<@A_@37%+8=!S4V990-]28F8S4R`E469R M+B3P&O7_3D5BL6'1!T!;=!N2(/=IH&]941U&3D4<@5008"-!9&HG0$4B$8#^:1&@(CTU MLB),3D]/7R*F!14Q`'P0``,`$!```````P`1$`````!````#T``0````4```!213H@`````+*$ ` end 1833 bytes --_[INTERGATE-SMTP59248414]_-- From kiddie@millennianet.com Mon Jan 13 11:53:26 1997 Received: from mailer.oclc.org (mailer.dev.oclc.org) by oclc.org (4.1/SMI-4.1) id AA28619; Mon, 13 Jan 97 11:56:51 EST Received: from millennianet.com by mailer.oclc.org (5.0/SMI-4.1) id AA00658; Mon, 13 Jan 1997 11:56:48 +0500 Received: from kiddie.millennianet.com (dialppp-1-27.millennianet.com [206.71.162.37]) by millennianet.com with SMTP id IAA12539 for ; Mon, 13 Jan 1997 08:58:20 -0800 (PST) Date: Mon, 13 Jan 1997 08:58:20 -0800 (PST) Message-Id: <199701131658.IAA12539@millennianet.com> X-Sender: kiddie@popmail.millennianet.com X-Mailer: Windows Eudora Light Version 1.5.2 Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" To: Gary PERLMAN From: KiddieCorp Subject: Email for CHIkids registration (fwd) Content-Length: 13047 Status: RO > > >Dear CHI Parent, > >We understand you are looking for more information regarding CHIkids for >CHI 97. Below describes the program and gives you more registration >information. If you are interested in registering for CHIkids, please do >so by sending back the forms below to KiddieCorp . >You will not be fully registered however, until your registration fee is >received by KiddiCorp. >____________________________________________________________________ > >CHI 97 CHIkids Information >____________________________________________________________________ >In association with the ACM SIGCHI, KiddieCorp is pleased to >offer an exciting new concept in convention child care. For children of >ages from 6 months to 13 years old, the CHIkids/KiddieCorp program is >intended to offer activities that are fun and entertaining yet >challenging and educational. > >The younger children will experience KiddieCorp's developmentally >appropriate, creative activities consisting of safe, fun-filled games, >toys, and arts & crafts projects. The older ones will be exposed to the >CHIkids Technology Center that includes multimedia story telling, CD-ROM >field trips, the CHIkids newsroom, and technology workouts (for a more >detailed description and registration information see the CHIkids >Technology Center Registration attached to this form). The middle age >groups will have the opportunity to experience both programs. > >KiddieCorp is in their eleventh year of providing high quality services >to conventions across the country. This will be their fourth year with >the CHI conference, and their second year collaborating on the CHIkids >program. The KiddieCorp team are bonded, qualified child care specialists >who are carefully selected and trained. In the CHIkids Technology Center >the KiddieCorp counselors work with CHI volunteers, and it will be this >team that will lead the children through an amazing range of advanced >computer activities. > >The child care program will be located in the Hyatt Regency Atlanta, >second floor. The deadline for CHIkids registration is March 1, 1997. >DUE TO THE PROGRAM'S POPULARITY THERE WILL BE NO ON-SITE REGISTRATION. >For parents with infants please bring diaper changing supplies, >formula/baby food, and a change of clothes. Please label all lunches and >personal belongings. KiddieCorp provides snacks and beverages only. >Lunches and a t-shirt are included for children three and over. >KiddieCorp staff does not administer medication. To ensure a safe and >fun-filled environment, any child that is ill will not be admitted to the >child care. > >________________________________________________________________________ > >CHIkids Technology Center Activities >________________________________________________________________________ >CHIkids challenges the traditional notion of child care and rolls summer >camp, technology, and CHI into one exciting new experience for children. >This is an opportunity for the next generation to explore computers, >technology and user interface design at the CHI 97 conference. Four >areas of exploration will be created especially for children (3-13 years >of age). They will have the opportunity to create multimedia stories, >try the latest educational multimedia titles, test emerging software >technologies with CHI researchers, and to be conference reporters using >desktop publishing tools and the World Wide Web: > >CD-ROM Field Trips >Would you like your children to go on a trip that takes them under the >ocean, inside a rainforest, across the world, or even out into space? In >this activity area children will use CD-ROMs as a way to learn about >geography, history or science in an engaging and exciting way. For >example, children can become "oceanographers", doing the things real >oceanographers do. Or they may be master "sleuths", traveling the world >to solve mysteries, learning geography and problem solving as they go. >Part of each activity will be exploring these simulated worlds, while >other parts will allow them to create multimedia presentations about >their experiences, which may be published in the CHIkids Newsletter. > >CHIkids Newsroom >Using current computer-based publishing tools, CHIkids will publish daily >news about the CHI conference. Children will research and report on >what is happening at the CHI conference's tutorials, panels, paper >discussions, or upcoming events. They may also review and write about >various software products that they use during their CHIkids experience. >CHIkids reporters will write, illustrate, and photograph news as they >see it for their stories to be produced in print and electronic form. >Both the printed and WWW versions will be distributed to the CHI >community during the CHI 97 conference. (For an example of the CHI 96 >CHIkids Newsletter see: http://www.acm.org/sigchi/chi96/CHI_WEB/web_fin/) > >Multimedia Storytelling >This area will offer children the chance to become multimedia >storytellers. They will be given opportunities to draw, animate, create >sounds, write content, and use digital photographs, all to develop their >own interactive multimedia project. Children will come to understand what >multimedia is, how it is developed by today's professionals, and what >tools are available for children to use. Depending on the children's >interests they may also be exposed to a range of authoring tools, >scripting languages and debugging strategies. Children will come away >from their multimedia storytelling experience with a finished multimedia >project. > >Technology Workouts >In this activity area children will examine the ways new technologies are >tested for usability. Researchers from the CHI community will bring >prototype and beta software for children to use. Together they will >develop a usability test plan and execute it in a focus group situation. >Older children will come to understand what it means to test software and >younger children will have fun playing with new technology experiences >not yet available commercially. Researchers will receive valuable >feedback from children on their work. > >======================================================================= >CHIkids REGISTRATION Forms: >======================================================================= > >Technology Activity Registration: > >Parents, we ask that you indicate your child's interest in the four areas >just described. Please mark the list below with either a #1, #2, #3, or >#4 (1 being of most interest, 4 being of least) We will make every >attempt to offer your child her or his first and second choices. If you >would like your child to be exposed to all four of the areas throughout >their stay at CHIkids, please mark a #1 next to each possible entry. If >at any time during the CHIkids experience that your child is unhappy with >the technology area they are in, we will make every attempt to place them >in a different area of their choosing. > >___ CD-ROM Field Trips >___ CHIkids Newsroom >___ Multimedia Storytelling >___ Technology Workouts > >Please write your child(ren)'s name and age below: > >Child Name: >Age: > >Child Name: >Age: > >Did your child(ren) participant last year in the CHIkids program? >Yes___ No___ > > > >If you feel that there are any special considerations we should know about >your child (e.g., allergies, medication, special needs however minor), >please feel free to share them with us below: > > > > >________________________________________________________________________ > >During the CHI 97 conference there will be an opportunity for children in >the CHIkids program to ask questions of adult CHI panelists who are >discussing the future of technologies for children. Would you be >interested in allowing your child to: > >Ask a question to the panelists using a microphone to be heard? yes / no > >Attend the presentation (in the audience) at the CHI 97 conference? yes / no >________________________________________________________________________ > >It is anticipated that on Monday afternoon of the CHIkids Program, there >will be an opportunity for up to 30 children to be taken on a tour of the >CNN Newsroom in Atlanta. There will be bus transportation and the tour >will be supervised by the CHIkids/KiddieCorp adult staff. The cost of >this trip is an additional $20 to the existing registration fee (see cost >chart at the end of this registration information). Only children 9 >years and older will be allowed to attend this trip. And there will only >be room for the first 30 children who register for this option. In >addition, your child must be registered for that Monday of the CHIkids >program to be eligible for this special option. If you would like your >child to attend this special field trip, please fill out the following: > >I give my permission to my child(ren): >to attend the CHIkids CNN tour. I understand that this tour will be >supervised by the CHIkids/KiddieCorp Program staff. If an emergency >does arise, I accept responsibility and agree to pay for any medical >treatment that my child(ren) may need to receive. >__________________________________________________________________ > >On the last morning of the conference the CHIkids program will be >presenting to the CHI community a summary of what was accomplished during >the CHIkids technology experience. Please respond to the following >questions pertaining to this event (for children ages 3 and over). > >--Would you mind if your child was photographed for this presentation? >Yes / No > >--Would you like your child to participate in this final presentation? >Yes / No > >--Would you like your child to attend the presentation (in the audience) >at the CHI 97 conference? Yes / No > > >_______________________________________________________________________ >CHIkids is from March 23 - 27, 1997 for ages 6 months to 13 years old. >The fees associated with the child care depend upon the age of the child. >Fees reflect the fact that younger children require a higher number of >caregivers than older children. The fee structure is listed below. The >daily program schedule will be 8:30am - 6:15pm, except on Sunday which >will be 1:30pm - 6:15pm. For children 3 years old and up **a lunch is >included** during all day sessions (choose from a selection of lunch >offerings each morning) and each child will receive a complimentary >t-shirt. > > >Group #1: 6 months to 2 years (Child care program only) >Child's Name_______________________ Age:__________ >(select desired shifts) >Sun* Mon Tues Wed Thurs > >$35__ $50__ $50__ $50__ $50__ > > > >Group #2: 3 to 5 years old (Child care program and technology center) >Child's Name_______________________ Age:__________ >(select desired shifts) >Sun* Mon Tues Wed Thurs > >$35__ $50__ $50__ $50__ $50__ > > > >Group #3: 6 to 8 years old (Child care program and technology center) >Child's Name_______________________ Age:__________ >(select desired shifts) >Sun* Mon Tues Wed Thurs > >$35__ $50__ $50__ $50__ $50__ > > > >Group #4: 9 to 11 years old (technology center only) >Child's Name_______________________ Age:__________ >(select desired shifts) >Sun* Mon Tues Wed Thurs > >$25__ $40__ $40__ $40__ $40__ > $60**__ > > > >Group #5: 12 years old and older (CHIkids Jr. Leaders) >Child's Name_______________________ Age:_________ >(select desired shifts) >Sun* Mon Tues Wed Thurs > >$25__ $35__ $35__ $35__ $35__ > $55**__ > > >TOTAL Registration Fee: __________ > >*Sunday afternoon program only. >**CNN tour - limited to first 30 children 9 years old and above > >Parent Name(s): >Home: # ( ) > >Address: > >Work: # ( ) >Fax: # ( ) >E-mail: > >Send all completed registration forms (via fax, mail, or e-mail) and >your check to: (make check payable to KIDDIECORP) > >KiddieCorp/CHIkids >5665 Oberlin Dr., Suite 102 >San Diego CA 92121 USA >Fax: +1-619-455-5841 >E-mail: kiddie@millennianet.com > >We suggest that you make a copy of your completed form as a reference. >Confirmations will not be sent. KiddieCorp reserves the right to limit >participation of any child whose presence or behavior may endanger the >health or safety of other children. If you have any questions >please contact KiddieCorp by phone at +1-619-455-1718, by fax at +1-619- >455-5841, or by e-mail at kiddie@millennianet.com. > > > *remember deadline: March 1, 1997* > >Note: You are not fully registered, nor is your child's space assured >until KiddieCorp actually receives your check in hand. Due to the >popularity of the program there will be no on-site registration. > > > > > > > Craig Leweck From RCHARRON@mprgate.mpr.ca Sat Feb 1 16:25:55 1997 Received: from mprgate.mpr.ca (mprgate.mpr.ca [134.87.131.13]) by mail.acm.org (8.7.5/8.7.5) with SMTP id QAA24946 for ; Sat, 1 Feb 1997 16:25:49 -0500 Received: from msgate.mpr.ca by mprgate.mpr.ca with SMTP id AA15230 (5.67b+/IDA-1.5 for ); Sat, 1 Feb 1997 13:25:50 -0800 Message-Id: <199702012134-23151@mpr.ca> Date: Sat, 01 Feb 1997 13:34:04 -0800 X-Mailer: Microsoft Mail with Intergate/SMTP (v1.Free) From: RCHARRON@mpr.ca (Rhona Charron) To: perlman@acm.org Cc: rcharron@mprgate.mpr.ca Subject: CHI97 SIG INFORMATION Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Status: RO Dear Gary, Thank you very much for submitting an application for the CHI 97 Special Interest Group (SIG) ACM SIGCHI Information Infrastructure. We are very happy to inform you that your submission has been accepted. The date, time and location for your SIG is:Wed, March26: 9:00-10:30 Cairo/Vancouver Room If you have not already registered for the CHI 97 Conference, registration materials are available in the Advanced Program and on the web at: http://www.acm.org/sigchi/chi97/ap/ If you have questions regarding registration, please forward them to: chi97-registration@acm.org Please note that the audio-visual equipment supplied for the SIG is one overhead projector and a flip chart. If you have any questions / comments about any of the above, please forward them to us at the email address below asap so that we can do what is required to ensure the success of the SIG. We look forward to working with you in preparing for your SIG at the Conference in Atlanta. Yours sincerely, Rhona Charron & Mark Leonard CHI97 SIG Co-Chairs From perlman Sun Feb 2 23:55:36 1997 Received: by turing.acm.org; id AA20646; Sun, 2 Feb 1997 23:55:35 -0500 Date: Sun, 2 Feb 1997 23:55:35 -0500 From: Gary PERLMAN Message-Id: <9702030455.AA20646@turing.acm.org> To: to_go@compuserve.com Subject: ACM SIGCHI CHI 97 hotel reservation Cc: perlman Status: O CHI 97 Housing Reservation Please print and mail, fax or email this form to: Commerce Travel, Inc. Fax: +1 410 484 6550 31 Walker Ave. Email: to_go@compuserve.com Baltimore, MD 21208 USA Personal Data Last Name: Perlman First Name: Gary Company/Organization: OCLC Online Computer Library Center Company Address: 6565 Frantz Road City: Dublin Prov/State: OH Postal/Zip Code: 43220 Country: USA Telephone: 614-761-5058 Fax: 614-793-0915 Name of person making reservation: Gary Perlman Hotel Data Arrival Sunday 23 March Time: about 6pm Departure Wednesday 26 March Time: about 12 noon 1st Choice: Hyatt Regency Atlanta Hotel 2nd Choice: Westin Peachtree Atlanta 3rd Choice: Days Inn Room Type: single, non-smoking Rooming with: Special requirements: Method of Payment Credit card type: Visa Name: Gary Perlman Card Number: 4356___ 0031____ 0627____ 2855____ (please fill in manually) Expiry Date: month 01____ year 99____ (please fill in manually) Signature: Gary Perlman________________________ (please fill in manually) From perlman@oclc.org Fri Jan 31 13:44:08 1997 Received: from oclc.org (fssun10.dev.oclc.org [132.174.19.11]) by mail.acm.org (8.7.5/8.7.5) with SMTP id NAA30824 for ; Fri, 31 Jan 1997 13:44:06 -0500 Received: from dev1.NISDEV (dev1.dev.oclc.org) by oclc.org (4.1/SMI-4.1) id AA24668; Fri, 31 Jan 97 13:21:12 EST Received: by dev1.NISDEV (SMI-8.6/SMI-SVR4) id NAA05707; Fri, 31 Jan 1997 13:21:12 -0500 From: perlman@oclc.org (Gary PERLMAN) Message-Id: <199701311821.NAA05707@dev1.NISDEV> Subject: CHI 97/Alert on "virtual tutorials" (fwd) To: perlman@acm.org Date: Fri, 31 Jan 1997 13:21:11 -0500 (EST) X-Mailer: ELM [version 2.4 PL23beta3] Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Status: RO Forwarded message: >From epfran1@PacBell.COM Fri Jan 31 13:17:44 1997 X-Sender: epfran1@zeus.srv.pacbell.com Message-Id: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Date: Fri, 31 Jan 1997 10:14:05 -0800 To: perlman@mailer.oclc.org, debra_herschmann@usccmail.lehman.com, dray.chi@xerox.com, nardi@apple.com, smetros@utk.edu, Dave_Roberts@aussmtp.austin.ibm.com, ellen@communities.com, pete@dcs.qmw.ac.uk, csnyder@uie.com, karen@acm.org, sarahb@werple.mira.net.au, nass@leland.stanford.edu, poynton@poynton.com, gershon@mitre.org From: epfran1@PacBell.COM (Ellen Francik) Subject: CHI 97/Alert on "virtual tutorials" Cc: chi97-tutorials@acm.org, chi97-mgt-team@acm.org Content-Length: 3530 Dear CHI 97 tutorial instructors, Many of you have received this message asking for your help in setting up an online version of your tutorials. (Message is below. Thanks, Susan, for passing it along.) This is not necessarily an ACM activity nor is it required that you respond to this. We're checking with SIGCHI to see if they know anything about this effort. It may be that: - this is a legitimate extension of the burgeoning "Tutorials on Tour" plan that SIGCHI is working on, OR... - this is not a SIGCHI initiative but an ACM initiative, OR... - this is being initiated by people who think ACM *should* do it, OR... - this is a freelance effort and the organizers are trolling for names. We'll let you know as soon as we find out anything. Ellen Francik and Jim Larson CHI 97 Tutorials Cochairs chi97-tutorials@acm.org >Date: Fri, 31 Jan 97 11:33:54 EST >From: "Susan E. Metros" >Subject: Is this on the up and up? >To: epfran1@PacBell.COM >Reply-To: pmay@beta.tricity.wsu.edu > >Ellen: > >Is this sanctioned by teh ACM? There web address is not operational sO I am >unable to find out additional information. Its a neat idea, but it seems a >little suspect to me... Do you have any insight into this? Thanks. Susan >----- Forwarded message follows ----- > >Return-Path: >Received: from beta.tricity.wsu.edu ([192.31.216.9]) by utkux.utcc.utk.edu >(5.x/2.8s-UTK.UTCC) > id AA16151; Fri, 31 Jan 1997 10:37:18 -0500 >Received: from localhost by beta.tricity.wsu.edu; >(5.65v3.2/1.1.8.2/15Jul96-0522PM) > id AA12055; Fri, 31 Jan 1997 07:35:19 -0800 >Date: Fri, 31 Jan 1997 07:35:19 -0800 (PST) >From: "Peggy S. May" >To: alee@nynexst.com, dray@mr.net, gershon@mitre.org, iansmith@cc.gatech.edu, > jacob@cs.tufts.edu, jhedberg@uow.edu.au, jspool@uie.com, > METROS@utkux.utcc.utk.edu, nardi@taurua.apple.com, > neal@pixel.kodak.com, poynton@poynton.com, tgillesp@indiana.edu, > Tony@intel.com, vklinin@informatik.ume.se, whorton@horton.com >Cc: Roy Rada >Message-Id: >Mime-Version: 1.0 >Content-Type: TEXT/PLAIN; charset=US-ASCII >Content-Length: 785 >X-UIDL: 854726462.003 > >We found your name at the CHI '97 web site as a conference tutor. > >We are hoping to work with ACM to create something like a Virtual >Computing School that will help extend some existing ACM conference >tutorials in online form. (A draft of this proposal is available at >http://134.121.242.55/acm.htm/.) Would you please help us with some >preliminary market analysis from the tutor's perspective. > >Do you think students would be willing to pay to participate in a virtual >form of your tutorial? > >Would you be willing to be somehow involved in online tutoring? > >If you are willing to help us further, we have many other questions we >would like to ask and also welcome any guidance from you. > >Thank for for you time. > > >Roy Rada >rada@eecs.wsu.edu > >Peggy S. Hults >pmay@beta.tricity.wsu.edu > > > >----- End of Forwarded message ----- > >_______________________________________________________________________________ >Susan E. Metros >Professor and Acting Director of Academic Technology > >The University of Tenneessee >2410 Dunford Hall >915 Volunteer Ave. >Knoxville, TN 37996-2410 > >Telephone: 423 974 7925 >Fax: 423 974 2050 >E-mail: smetros@utk.edu >URL: http://www.at.utk.edu/itc/ > > > From Darrow@hq.acm.org Mon Mar 10 20:35:44 1997 Received: from LAN-GATEWAY.hq.acm.org (lan-gateway.acm.org [199.222.74.87]) by mail.acm.org (8.8.4/8.7.5) with SMTP id UAA21228; Mon, 10 Mar 1997 20:35:42 -0500 Received: by LAN-GATEWAY.hq.acm.org with Microsoft Mail id <3324E11F@LAN-GATEWAY.hq.acm.org>; Mon, 10 Mar 97 20:35:43 PST From: Diane Darrow To: Mike Atwood , "'Barbee2'" , Clare-Marie Karat , Gene Lynch , Guy Boy , Jim Miller , Gary PERLMAN , Marian Williams Subject: Comps for CHI97 - Date: Mon, 10 Mar 97 20:33:00 PST Message-Id: <3324E11F@LAN-GATEWAY.hq.acm.org> Encoding: 14 TEXT X-Mailer: Microsoft Mail V3.0 Status: RO Hi all, My apologies for getting this information out late to you...I thought it had been sent to you by Alan Edwards when he did the complimenary allocation for CHI97 registrations. If you haven't registered you can do so at this special site: http://www.reg-master.com/CHI97comp.html or you can register by filling out the registration form and writing complimentary at the top of the form. If you have questions, please let me know. regards, diane From epfran1@PacBell.COM Mon Mar 10 20:03:02 1997 Received: from gw3.pacbell.com (gw3.PacBell.COM [129.245.2.24]) by mail.acm.org (8.8.4/8.7.5) with SMTP id UAA21070; Mon, 10 Mar 1997 20:01:49 -0500 Received: from srv.PacBell.COM (mother.srv.PacBell.COM) by gw3.pacbell.com (5.x/PacBell-10/18/95) id AA13785; Mon, 10 Mar 1997 16:57:40 -0800 Received: from zeus.srv.PacBell.COM by srv.PacBell.COM (4.1/Mother-7/26/95) id AA14932; Mon, 10 Mar 97 16:57:07 PST Received: from [129.245.90.15] (epfran1.srv.PacBell.COM) by zeus.srv.PacBell.COM (4.1/SMI-4.1) id AA10182; Mon, 10 Mar 97 16:57:05 PST X-Sender: epfran1@zeus.srv.pacbell.com Message-Id: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Date: Mon, 10 Mar 1997 16:54:46 -0800 To: perlman@acm.org, debra_herschmann@usccmail.lehman.com, dray.chi@xerox.com, nardi@apple.com, smetros@utk.edu, Dave_Roberts@aussmtp.austin.ibm.com, ellen@communities.com, pete@dcs.qmw.ac.uk, csnyder@uie.com, karen@acm.org, sarahb@werple.mira.net.au, nass@leland.stanford.edu, poynton@poynton.com, gershon@mitre.org, jakob@eng.sun.com, jacob@cs.tufts.edu, hewett@duvm.ocs.drexel.edu, alee@nynexst.com, william@horton.com, thom@copper.ucs.indiana.edu, luperfoy@mitre.org, kmcgraw@clark.net, tony_salvador@ccm.jf.intel.com, neale@kodak.com, aaron@amanda.com, bdamer@ccon.org, drdeb@vineyard.net, mmillier@ibeam.jf.intel.com, tandyt@microsoft.com, iansmith@cc.gatech.edu, Mary Mooney , Volker@amanda.com From: epfran1@PacBell.COM (Ellen Francik) Subject: CHI 97/Tutorial Instructor's FAQ Cc: chi97-tutorials@acm.org Status: RO Dear CHI 97 Tutorial Instructor, (1) Please acquaint yourself with the answers to these frequently-asked questions. (2) Please forward this message to your co-instructors. Thank you! Jim and Ellen chi97-tutorials@acm.org ------------------------------------------------------------------- CHI 97 TUTORIAL INSTRUCTOR'S FAQ 1. Where do I check in on the day of my tutorial? 2. When and where is the speakers' breakfast? 3. How do I go about getting paid? 4. How should I ship stuff to Atlanta? 5. How do I get my tutorial supplies? 6. How do I get a copy of my tutorial notes? 7. Can you confirm the schedule for the tutorial day? 8. What can I expect the Student Volunteers to do? 9. What should I do if there's a problem with my AV/CS equipment? 10. How do I arrange to load software or do other prep the night before my tutorial? 11. Can I hang things on the wall in my tutorial room? 12. Where are the speaker prep and speaker rehearsal rooms? 1. Where do I check in on the day of my tutorial? Check in with Jim and Ellen at the Tutorials Desk in the registration area. They will be worried sick until they see you on site. The Tutorials Desk will open at about 5:00 pm on Saturday and at about 8:00 am on Sunday and Monday. 2. When and where is the speakers' breakfast? At least one instructor from your tutorial must attend the speakers' breakfast on the day of your tutorial, in order to get last-minute information and instructions. It will be held in the Avanzare restaurant of the Hyatt Hotel at 7:30 am. 3. How do I go about getting paid? Fill out a couple of forms at the Tutorials Desk. Be sure to have an identification number like Social Security or passport number. In one form, you will need to tell us how the honorarium should be split among multiple instructors and where each check should be mailed after the conference (since we can't promise that you'll get your checks at the conference). The other form is a W-9 (US tax form), which each instructor who is receiving a check must fill out. 4. How should I ship stuff to Atlanta? If you are shipping tutorial supplies or handouts, you can ship them to yourself at your hotel. (Trust us: if you can handcarry the materials, this is the simplest way to ship them.) Schedule the delivery to arrive NO SOONER than one day prior to your arrival. Use a trackable method, such as FedEx. Mark the package "Hold for Guest Arrival" with your name and arrival date. Phone the hotel to make sure your package has arrived. Please allow 1-2 days in advance of your arrival for small parcels. Alternatively, you can ship to the Conference Decorator/Freight Handling Vendor, Freeman Decorating Company. (They are working through a company known as Sullivan Transfer, which name will show up in the address below.) Freeman will recieve crated, boxed, or skidded materials at the warehouse up to 30 days prior to the conference move-in. All freight must arrive to Freeman no later than Thursday, March 20, 1997. Shipments MUST be sent with freight charged PREPAID. Collect shipments will not be accepted. Please allow 1-2 days in advance of your arrival for small parcels. Large parcels may need up to 30 days. Address your freight as follows: Tutorial Hold for: ACM/CHI 97 Conference C/O Sullivan Transfer Company 260 University Avenue Atlanta, GA 30315 USA To check on the arrival of warehouse freight, please call +1 404 377 0061. Be sure to reference ACM/CHI 97. To pick up your shipment, go to the CHI 97 Conference Registration Desk in the Exhibit Lobby of the Hyatt and ask for the location of the Freeman freight pickup area. Freeman will bring your shipment to their storage area in the Exhibit Hall in the Hyatt and you can pay for the freight handling directly (by cash, check, or credit card). Freight should be delivered beginning on Saturday, 3/22/97 by 3 PM. As mentioned in your author kit, the costs related to shipping materials and equipment to CHI 97 are your responsibility. 5. How do I get my tutorial supplies? The Student Volunteers assigned to your tutorial will bring your tutorial supplies with them to the tutorial. 6. How do I get a copy of my tutorial notes? The Student Volunteers will bring a copy of the notes for each instructor. 7. Can you confirm the schedule for the tutorial day? The schedule for the tutorial day is exactly as described in your author kit (see, miracles do happen :-). Here's a recap. Saturday evening ---------------- 6:00 start 7:30-8:00 break 9:30 end Sunday and Monday, full day --------------------------- 9:00 start 10:30-11:00 break 12:30-2:00 lunch 3:30-4:00 break 5:30 end Sunday and Monday morning, half day ----------------------------------- 9:00 start 10:30-11:00 break 12:30 end Sunday and Monday afternoon, half day ------------------------------------- 2:00 start 3:30-4:00 break 5:30 end 8. What can I expect the Student Volunteers to do? The Student Volunteers will check badges at the door, run the AV equipment, distribute supplies and materials to the participants, call for help if you need it, and remind you when it's time for a break if you seem to be forgetting. You can ask them for other logistical help, too, but keep in mind that they are not auxiliary instructors. In the past, an occasional nervous instructor has taken out his or her anxiety on a poor Student Volunteer. Please make sure your SVs are treated with respect. If a student volunteer is especially helpful to you, please make a note of his or her name and pass it along to Jim or Ellen. They will see that the SV gets special recognition. A kind word from a luminary in the CHI field goes a long way :-). 9. What should I do if there's a problem with my AV/CS equipment? Tell one of your Student Volunteers. They know how to summon help. Don't try to fix the equipment yourself. 10. How do I arrange to load software or do other prep the night before my tutorial? Get in touch with Technology Support Chair Rodney Fuller (chi97-tech-support@acm.org) before the conference. Rodney will arrange a time for you to do your setting up. Please cc your email correspondence with Rodney to Jim and Ellen (chi97-tutorials@acm.org). By the way: Rodney is doing a work stint in Ireland, so don't expect him to be on US time! 11. Can I hang things on the walls in my tutorial room? If you mentioned in your supply request that you need to hang papers on the wall, we will provide an approved adhesive for you to use (we may substitute bulletin boards if the facility requires us to). Don't use any other method of hanging things on the wall, because it will violate our contracts with the Convention Center and the hotels and will potentially cost the conference a lot of money. 12. Where are the speaker prep and speaker rehearsal rooms? Ask at the Tutorials Desk or consult the Final Program to find out where the speaker prep and speaker rehearsal rooms are. -------- end of FAQ -------- From rcharron@mprgate.mpr.ca Tue Mar 4 15:23:00 1997 Received: from mprgate.mpr.ca (mprgate.mpr.ca [134.87.131.13]) by mail.acm.org (8.8.4/8.7.5) with SMTP id PAA28026; Tue, 4 Mar 1997 15:22:30 -0500 Received: from boson (boson.mpr.ca) by mprgate.mpr.ca with SMTP id AA15371 (5.67b+/IDA-1.5); Tue, 4 Mar 1997 12:11:51 -0800 Received: from madeira by boson; (5.65v3.2/1.1.8.2/31Jan96-1209PM) id AA31687; Tue, 4 Mar 1997 12:11:36 -0800 Received: by madeira.mpr.ca with Microsoft Mail id <01BC2894.B1A6C9E0@madeira.mpr.ca>; Tue, 4 Mar 1997 12:07:56 -0800 Message-Id: <01BC2894.B1A6C9E0@madeira.mpr.ca> From: Rhona Charron To: "'Spool'" , "'andriessen@wtm.tudelft.nl'" , "'bam@a.gp.cs.cmu.edu'" , "'beyer@acm.org'" , "'bjfogg@stanford.edu'" , "'byrne+@andrew.cmu.edu'" To: "'Maxine Cohen'" , "'David G. Novick'" , "'elizabeth.muncher@daytonoh.ncr.com'" , "'Gillian Crampton Smith'" , "'graefe@tnpubs.enet.dec.com'" , "'Gary Strong'" To: "'hgoodell@tem.teainet.com'" , "'jgeyerman@techdev.compuserve.com'" , "'karen@acm.org'" , "'laurie@scis.nova.edu'" , "'lorettas'" , "'perlman@acm.org'" To: "'Shannon Kathleen Ford'" , "'Stephanie Rosenbaum'" , "'tom.graefe@lkg.mts.dec.com'" Subject: CHI97 SIGS in SIGCHI Bulletin October issue Date: Tue, 4 Mar 1997 12:07:54 -0800 Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Status: RO Hi all, This is just a reminder (or a first notice, depending on how closely everyone read the fine print a few months ago... :) from the Editor of the upcoming deadline for the CHI Bulletin. Any questions or comments, please feel free to call/email. Cheers, Rhona and Mark Workshops, SIGs Co-chairs <> Dear CHI 97 event organisers. The October SIGCHI Bulletin each year is dedicated to the CHI conference, covering all the events that otherwise don't get widely published. Check out http://www.acm.org/sigchi/bulletin/1996.4 for last year's. This year will be just the same, so I'd like to communicate the details to you. We normally carry a number of articles: * SIGs: SIGs are more than welcome to submit a report. It would be good if you would pass the invitation on. The deadline is 1st July. While plain text is fine, we *love* diagrams and photos. Full details of how to submit can be found at the Bulletin web site (http://www.acm.org/sigchi/bulletin). Many thanks! Steven Pemberton From perlman@turing.acm.org Wed Mar 19 11:04:18 1997 Received: from turing.acm.org (turing.acm.org [199.222.69.20]) by mail.acm.org (8.8.4/8.7.5) with SMTP id LAA18584 for ; Wed, 19 Mar 1997 11:04:15 -0500 Received: by turing.acm.org; id AA23184; Wed, 19 Mar 1997 11:04:11 -0500 From: Gary PERLMAN Message-Id: <9703191604.AA23184@turing.acm.org> Subject: Re: CHI97 SIG INFORMATION To: RCHARRON@mpr.ca (Rhona Charron) Date: Wed, 19 Mar 1997 11:04:11 -0500 (EST) Cc: perlman@acm.org, rcharron@mprgate.mpr.ca, jspool@uie.com, instone@turing.acm.org In-Reply-To: <199702012134-23151@mpr.ca> from "Rhona Charron" at Feb 1, 97 01:34:04 pm X-Mailer: ELM [version 2.4 PL25] Content-Type: text Status: RO > Thank you very much for submitting an application for the CHI 97 Special > Interest Group (SIG) ACM SIGCHI Information Infrastructure. > We are very happy to inform you that your submission has been accepted. > > The date, time and location for your SIG is:Wed, March26: 9:00-10:30 > Cairo/Vancouver Room I've had a couple of complaints about the conflict between our Webish SIG and Jared's. I am wondering if I can find out the time we actually get the room so that we can schedule our SIG a little early (maybe 8:00, but probably 8:30) and run a little fast so that we can jump over for Jared's, who might coordinate with us by delaying or repeating some parts (at least handouts). So, when do we get the rooms? And is this idea feasible? Gary From epfran1@PacBell.COM Thu Apr 10 13:59:03 1997 Received: from gw3.pacbell.com (gw3.PacBell.COM [129.245.2.24]) by mail.acm.org (8.8.4/8.7.5) with SMTP id NAA33418; Thu, 10 Apr 1997 13:58:40 -0400 Received: from srv.PacBell.COM (mother.srv.PacBell.COM) by gw3.pacbell.com (5.x/PacBell-10/18/95) id AA17742; Thu, 10 Apr 1997 10:52:18 -0700 Received: from zeus.srv.PacBell.COM by srv.PacBell.COM (4.1/Mother-7/26/95) id AA27965; Thu, 10 Apr 97 10:49:17 PDT Received: from [129.245.90.15] (epfran1.srv.PacBell.COM) by zeus.srv.PacBell.COM (4.1/SMI-4.1) id AA07021; Thu, 10 Apr 97 10:49:15 PDT X-Sender: epfran1@zeus.srv.pacbell.com Message-Id: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Date: Thu, 10 Apr 1997 10:46:48 -0700 To: perlman@acm.org, debra_herschmann@usccmail.lehman.com, dray.chi@xerox.com, nardi@apple.com, smetros@utk.edu, Dave_Roberts@aussmtp.austin.ibm.com, ellen@communities.com, pete@dcs.qmw.ac.uk, csnyder@uie.com, karen@acm.org, sarahb@werple.mira.net.au, nass@leland.stanford.edu, poynton@poynton.com, gershon@mitre.org, jakob@eng.sun.com, jacob@cs.tufts.edu, hewett@duvm.ocs.drexel.edu, alee@nynexst.com, william@horton.com, thom@copper.ucs.indiana.edu, luperfoy@mitre.org, kmcgraw@clark.net, tony_salvador@ccm.jf.intel.com, neale@kodak.com, aaron@amanda.com, bdamer@ccon.org, drdeb@vineyard.net, mmillier@ideal.jf.intel.com, tandyt@microsoft.com, iansmith@cc.gatech.edu From: epfran1@PacBell.COM (Ellen Francik) Subject: CHI 97/Thanks, and further contact info Cc: Jim_A_Larson@ccm.jf.intel.com, Andrew Sears , jackoj@fiu.edu Status: RO Dear CHI 97 Tutorial instructors, Jim and I would like to thank all of you for your contributions to a very successful CHI 97! A few brief notes: - Your evaluations are being handled by the CHI 98 Tutorials cochairs, Andrew Sears and Julie Jacko. I know the evaluations are very close to being sent to you. If you have any questions, Andrew can be reached at sears@cs.depaul.edu. - If you have questions about honorarium payment, you can contact Borysa Struk in the CHI 97 Conference Office: bos@ea.net. - Finally, I am about to leave on a 6-month leave of absence (beginning Wednesday the 16th). I won't be checking email at work. So, in the unlikely event there are any lingering questions about CHI 97, you can contact Jim, Jim_A_Larson@ccm.jf.intel.com. Best, Ellen Francik Pacific Bell CHI 97 Tutorials Co-Chair 2600 Camino Ramon, Rm. 3E050H chi97-tutorials@acm.org San Ramon, CA 94583 USA +1 510-867-8662 From utest@hubcap.clemson.edu Wed Mar 26 18:03:21 1997 Received: from hubcap (localhost [127.0.0.1]) by hubcap.clemson.edu (8.8.5/8.8.5) with SMTP id SAA18050; Wed, 26 Mar 1997 18:02:25 -0500 (EST) Date: Wed, 26 Mar 1997 18:02:25 -0500 (EST) Message-Id: <5040100002324096000002L062*@MHS> Errors-To: tharon@hubcap.clemson.edu Reply-To: isensee@us.ibm.com Originator: utest@hubcap.clemson.edu Sender: utest@hubcap.clemson.edu Precedence: bulk From: Scott Isensee To: Multiple recipients of list Subject: Re: ? Anything from the "Web Interfaces Live" panel session X-Listprocessor-Version: 6.0c -- ListProcessor by Anastasios Kotsikonas Status: RO Classification: The panel didn't get to the whitepages site or most of the others they were going to review. The panel was supposed to be reviewing sites they were seeing for the first time. In that situation, they could only make quick and superficial comments about each site. Some members of the audience went to the microphones to make critical comments and it became rather tense and antagonistic for a few minutes. The session then morphed into a question and answer session between members of the audience and the panel. That turned out to be quite interesting. Some of the points that came up were: - There will be 3 million pages added to the web next year. Divide that by the number of CHI members and it is obvious that most web pages will be developed with no professional UI design input. - It is quite difficult to design a good application for the web. There is much user confusion between the controls provided by the browser and the controls owned by an application. - Web UIs vary greatly. It is much like the days before GUIs came along. The major GUIs provided toolkits that encouraged consistency by making it easier to use a control as specified by the style guide than to use it another way. On the web it is as easy to do something wrong as it is to do it right. - The web is just a set of protocols. It does not require use of a browser. Applications can created with an OOUI and use the web without using a browser. - There is a scarcity of published research on UI issues for the web. Many companies are designing web sites without doing research or testing and even those who do test are often not publishing the results. - There was also a long discussion of whether the 7 +/- 2 short term memory limit is relevant to recommendations for menu length or number of links. Scott Isensee IBM tie line: 678-1195 Voice: (512) 838-1195 Fax: (512) 838-5989 Zip 9446 Austin, TX Internet: isensee@austin.ibm.com User Interface Architecture and Design From perlman Mon Jul 7 11:02:52 1997 Received: by turing.acm.org; id AA20474; Mon, 7 Jul 1997 11:02:48 -0400 From: Gary PERLMAN Message-Id: <9707071502.AA20474@turing.acm.org> Subject: CHI-ANNOUNCEMENTS: approval required (3B6059) (fwd) To: Steven.Pemberton@cwi.nl Date: Mon, 7 Jul 1997 11:02:47 -0400 (EDT) Cc: perlman X-Mailer: ELM [version 2.4 PL25] Content-Type: text Content-Length: 1148 Status: RO Forwarded message: > The CHI 97 Electronic Proceedings are going online in four phases: > > 1. Proceedings proper (papers, design briefings, tech-notes), > including introductory matter (Society Welcome, Chairs' Welcome, > Special Thanks, Committee Listings, and Technical Program > Overview) > 2. Extended abstracts (early): all extended abstracts except > late-breaking > 3. Late breaking and informal presentations > 4. Author and keyword indexes. > > Phases 1 and 2 are now online, and can be found at: > > http://www.acm.org/sigchi/chi97/proceedings/ The links to sections inside: http://www.acm.org/sigchi/chi97/proceedings/tutorial/gp.htm and in: http://www.acm.org/sigchi/chi97/proceedings/sig/gp1.htm are still not working. The scripts to go through them and add links are apparently still confused by existing tables of contents, unquoted URLs, or something else. I'd appreciate having them fixed so I don't look like I don't know how to put links ina document. I just don't know how to put links in a document that will be processed by a filter that doesn't know how to edit links in a document. :-) Gary