From MAILER-DAEMON Tue Jan 28 10:15:18 2003 Date: 28 Jan 2003 10:15:18 -0500 From: Mail System Internal Data Subject: DON'T DELETE THIS MESSAGE -- FOLDER INTERNAL DATA X-IMAP: 1043766918 0000000000 Status: RO This text is part of the internal format of your mail folder, and is not a real message. It is created automatically by the mail system software. If deleted, important folder data will be lost, and it will be re-created with the data reset to initial values. From robin.jeffries@sun.com Mon Jan 20 13:35:04 2003 Return-Path: Received: from alias.acm.org (alias.acm.org [199.222.69.90]) by turing.acm.org (8.11.6/8.11.6) with ESMTP id h0KIZ4a15762 for ; Mon, 20 Jan 2003 13:35:04 -0500 Received: from patan.sun.com ([192.18.98.43]) by alias.acm.org (ACM Email Forwarding Service) with ESMTP id COB73880 for ; Mon, 20 Jan 2003 13:46:21 -0500 Received: from jurassic.eng.sun.com ([129.146.17.55]) by patan.sun.com (8.9.3+Sun/8.9.3) with ESMTP id LAA16752 for ; Mon, 20 Jan 2003 11:46:18 -0700 (MST) Received: from sun.com (vpn-129-152-200-67.East.Sun.COM [129.152.200.67]) by jurassic.eng.sun.com (8.12.7+Sun/8.12.7) with ESMTP id h0KIkFL5747184; Mon, 20 Jan 2003 10:46:16 -0800 (PST) Message-ID: <3E2C43F9.C06AE0B9@sun.com> Date: Mon, 20 Jan 2003 10:46:17 -0800 From: Robin Jeffries X-Mailer: Mozilla 4.74 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: perlman@acm.org Subject: Review TOCHI paper? Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Status: RO X-Status: A X-Keywords: I am an associate editor for TOCHI. I have a paper on an alternative approach to localization, and I'm looking for reviewers. Would you be willing to review this? Here is the title and abstract of the paper: Subtitled Interaction: Complementary Support as an Alternative to Localization Abstract Many computer users face problems in their interaction as a result of the native language employed by the application. The language of the application is often at variance with the native language of its users. This issue is frequently addressed through localization. In turn, localization generates a range of new problems. We propose an alternative to localization that is analogous to cinematic subtitles. This has the potential to reduce the user interaction defects that otherwise arise with localization whilst benefiting users through an additional channel of information in their own language. This paper outlines a prototype implementation and describes our initial evaluation of this approach. We suggest that our complementary 'subtitles' promise consistent support for all applications in the user's computing environment and yield a system that is expandable and much easier to maintain than pre-localized software. Thanks, Robin From robin.jeffries@sun.com Mon Jan 20 14:32:04 2003 Return-Path: Received: from kathmandu.sun.com (kathmandu.sun.com [192.18.98.36]) by turing.acm.org (8.11.6/8.11.6) with ESMTP id h0KJW3a16504 for ; Mon, 20 Jan 2003 14:32:03 -0500 Received: from jurassic.eng.sun.com ([129.146.17.55]) by kathmandu.sun.com (8.9.3+Sun/8.9.3) with ESMTP id MAA22180 for ; Mon, 20 Jan 2003 12:43:16 -0700 (MST) Received: from sun.com (vpn-129-152-200-67.East.Sun.COM [129.152.200.67]) by jurassic.eng.sun.com (8.12.7+Sun/8.12.7) with ESMTP id h0KJhDL5755874; Mon, 20 Jan 2003 11:43:14 -0800 (PST) Message-ID: <3E2C5151.10908CA8@sun.com> Date: Mon, 20 Jan 2003 11:43:14 -0800 From: Robin Jeffries X-Mailer: Mozilla 4.74 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: Gary PERLMAN Subject: Re: Review TOCHI paper? References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Status: RO X-Status: X-Keywords: Next reply will have all the gory details. I'm not convinced that there is research going on in localization. When I looked it up in the hcibib, yours was one of the few names I encountered (except this author). And the references in the document weren't of much help. If, as you read the paper, you can think of other names for reviewers who would know an area close to this, I'd appreciate it. Robin Gary PERLMAN wrote: > > Sounds interesting. I could take on the task. > > Caveat Editor: I'm pretty well versed on the practical aspects of > localization and research methodology, but I know nothing about any > research going on on localization. > > Gary > > On Mon, 20 Jan 2003, Robin Jeffries wrote: > > > I am an associate editor for TOCHI. I have a paper on an alternative > > approach to localization, and I'm looking for reviewers. Would you be > > willing to review this? > > > > Here is the title and abstract of the paper: > > > > Subtitled Interaction: Complementary Support as an Alternative to > > Localization > > > > Abstract > > > > Many computer users face problems in their interaction as a result of > > the native language employed by the application. The language of the > > application is often at variance with the native language of its users. > > This issue is frequently addressed through localization. In turn, > > localization generates a range of new problems. We propose an > > alternative to localization that is analogous to cinematic subtitles. > > This has the potential to reduce the user interaction defects that > > otherwise arise with localization whilst benefiting users through an > > additional channel of information in their own language. This paper > > outlines a prototype implementation and describes our initial evaluation > > of this approach. We suggest that our complementary 'subtitles' promise > > consistent support for all applications in the user's computing > > environment and yield a system that is expandable and much easier to > > maintain than pre-localized software. > > > > > > Thanks, > > > > Robin > > From robin.jeffries@sun.com Mon Jan 20 14:34:09 2003 Return-Path: Received: from nwkea-mail-2.sun.com (nwkea-mail-2.sun.com [192.18.42.14]) by turing.acm.org (8.11.6/8.11.6) with ESMTP id h0KJY9a16518 for ; Mon, 20 Jan 2003 14:34:09 -0500 Received: from jurassic.eng.sun.com ([129.146.17.55]) by nwkea-mail-2.sun.com (8.9.3+Sun/8.9.3) with ESMTP id LAA15324 for ; Mon, 20 Jan 2003 11:45:16 -0800 (PST) Received: from sun.com (vpn-129-152-200-67.East.Sun.COM [129.152.200.67]) by jurassic.eng.sun.com (8.12.7+Sun/8.12.7) with ESMTP id h0KJjDL5756063; Mon, 20 Jan 2003 11:45:14 -0800 (PST) Message-ID: <3E2C51C9.99D72062@sun.com> Date: Mon, 20 Jan 2003 11:45:13 -0800 From: Robin Jeffries X-Mailer: Mozilla 4.74 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: Gary PERLMAN Subject: ToCHI paper 02-25-0 References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Status: RO X-Status: A X-Keywords: Thank you for agreeing to review the following ToCHI submission: 02-25-0 Subtitled Interaction: Complementary Support as an Alternative to Localization Giorgos Lepouras George R.S. Weir The paper is available on the ACM TOCHI Web server in Adobe Acrobat PDF format. The URL for the submission is: Your browser will prompt for a User name and a Password. Enter the following information: User: 02-25 Password: redFlash The ACM TOCHI review form is located at . Also there is a page containing information for TOCHI reviewers at . Thanks, Robin From robin.jeffries@Sun.COM Mon Mar 3 05:59:39 2003 Return-Path: Received: from kathmandu.sun.com (kathmandu.sun.com [192.18.98.36]) by turing.acm.org (8.11.6/8.11.6) with ESMTP id h23AxcP13812 for ; Mon, 3 Mar 2003 05:59:38 -0500 Received: from jurassic.eng.sun.com ([129.146.17.55]) by kathmandu.sun.com (8.9.3+Sun/8.9.3) with ESMTP id RAA16304; Sat, 1 Mar 2003 17:09:35 -0700 (MST) Received: from sun.com (vpn-129-152-201-7.East.Sun.COM [129.152.201.7]) by jurassic.eng.sun.com (8.12.8+Sun/8.12.8) with ESMTP id h2209Uil821378; Sat, 1 Mar 2003 16:09:31 -0800 (PST) Message-ID: <3E614B5C.BFBE812F@sun.com> Date: Sat, 01 Mar 2003 16:07:56 -0800 From: Robin Jeffries X-Mailer: Mozilla 4.74 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 Subject: [Fwd: TOCHI submission 02-25-0] Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Status: RO X-Status: X-Keywords: Here is what I sent to the author. Each of you found something different in the paper, and the end result was very interesting. Thanks for your hard work. Robin Robin Jeffries wrote: > > Dear Dr. Lepouras, > > I've received reviews from 3 reviewers for your paper, "Subtitled > Interaction: Complementary Support as an Alternative to Localization". > The reviewers all saw very different things in your paper, but each > found a significant issue that requires a substantial change to the > paper in order for it to be acceptable to TOCHI. > > Before addressing the concerns about the paper, I want to point out that > all the reviewers considered this to be interesting work, and they were > all pleased to find research in the area of globalization. > Granted, I drew reviewers from a subset knowledgeable about > globalization issues, but I think you should take this enthusiasm for > your overall direction seriously, and revise the paper for publication > in TOCHI or elsewhere. > > Reviewer A raises the issue that concerns me the most. He points out > that the technology being evaluated in this paper is not novel, as is > claimed by the authors, but exists in several commercial systems. While > the commercial systems he cites do not do exactly what the system you > evaluated does, their capabilities are similar enough that you need to > take them into account. Best would be to evaluate one of these systems > as your complementary support condition; if that doesn't make sense > (they seem only to be available for Asian languages, and you may not > have access to native speakers of those languages), you should at least > compare your system to these commercial systems. > > The fact that the approach being proposed is not novel may make the > revised paper inappropriate for TOCHI. While we certainly encourage the > evaluation of existing approaches, showing that a commercially available > system has value isn't the kind of paper that we would encourage for > TOCHI. If your reevaluation shows something more surprising, (e.g., > that such systems either do not work or work in particular limited > ways), and/or you can explain the results via a theoretical approach > that would enable you to predict whether similar systems would be > effective, that would be a wonderful submission to TOCHI. However, if > your results simply show that these existing systems work, you may want > to submit the revised paper to some other journal. > > Reviewer B takes issue with the experiment you have done, or at least > how you have reported it. I agree with him that the report does not > follow a standard style, and this makes interpreting the results > challenging. I would attend carefully to his concerns, especially that > the numbers in your tables do not add up consistently, and that using an > application in the participants' native tongue was the worst of the 3 > conditions (although not significantly). This concerned me as well. I > would need substantially more detail in the description of the training > procedure and the actual tasks given so as to interpret the difference > as something other than a flaw in your procedure. > > One area that I think reviewer B could have emphasized more was the > reporting of statistics. You frequently use the phrase "considered > significant", which is misleading. Significance level is something that > you, the researcher, should set as a criterion of your experiment and > report at the beginning of the results section. Your results are > significant or not because of the significance criterion you have chosen > for the experiment, not due to some vaguely addressed conventional > standard. Also, while you often temporize your results as possibly > being due to low sample size, this is an issue for so many of your > non-significant results that I would strongly encourage you to redo the > experiment with at least 12 participants per condition. While one can > never prove the null hypothesis, having more confidence that a > significant effect would have occurred if there were substantial > differences between groups would make this a much stronger paper. > > Reviewer C focuses on the related literature and points out several > papers that would be useful for you to cite. He is also concerned about > the small sample size and whether this makes the conclusions too weak > for publication in TOCHI. I agree with his concerns, but they are > dwarfed by the issues raised by Reviewer B. > > Because of all these concerns, I am recommending to the Editor in Chief > that we reject this paper for TOCHI. If you can address all the > concerns of the reviewers, I welcome you to resubmit, but the resulting > paper would be different enough to be considered a separate submission. > > Robin Jeffries > Action Editor > > ============================================================================= > > Review A > > ACM Transactions on Computer-Human Interaction > > Report of Referee: A > Reference Number: 02-25-0 > > Title: Subtitled Interaction: Complementary Support as an Alternative to > Localization > > Authors: Giorgos Lepouras, George R.S. Weir > > I. General Evaluation. Please explain any "No" answers. > > A. Is the title appropriate? Yes > > B. Is the abstract satisfactory? Yes (however see notes below on > content). > > C. Are the keywords well chosen? Yes > > D. Content > > 1. Does the paper have enough originality and importance to merit > publication? > > While the technology application which was presented is valid and > important, the authors seem to be unaware that the ideas are not > original, and many products exist and are widely used in Asia which > implement very similar ideas. > > 2. Are the ideas, algorithms, results or studies technologically and > methodologically sound? > > I believe so. > > 3. Are the references to earlier work adequate? If not, please indicate > additional references that would be appropriate. > > Numerous products exist for the Asian market which work like the > technology described in this paper, but were not mentioned. Here are > links > to a few of these: > http://www.china-guide.com/chinesedictionary.html > http://www.a2001.com/english/shop/roboword/v5.html > > http://www.chinesesoftware.com/cgi-bin/shop.pl/page=translators_main.htm > > E. Presentation > > 1. Is the paper well organized? Yes > > 2. Is too much, too little or appropriate material included? Too little. > > In addition to the material related to existing technologies in > Asia, the authors did not touch on another important point, > specifically that many computer users consciously choose to work in > a second language, particularly English, not because localized > versions of their software are not available, but because they want > or need to work in English, and are reasonable fluent. Technologies > such as that presented in this paper are ideal for this class of > user, and the test results bear this out. I felt the test results > were skewed, partly because the number of subjects was too small, > and also because they were quite proficient in English. Test > results with subjects with little or no English knowledge would > have been interesting to see. > > 3. Is the presentation clear? Yes > > 4. Is the English satisfactory? Yes, with these comments: > > The phrase "knock-on effect" is used twice, and the word "outwith". > This are quite obscure (I can find no reference for "outwith") and > such things should be avoided. Also, in the first two sentences of > the introduction, there seems to be a disconnect, as if, perhaps, a > sentence was missing between the two which made the connection > between historical use of English and "researchers and scientists". > As a nit pick, the expression "X-Windows" (4th paragraph of section > 2) was used, which is invalid and should be "the X Windows System". > > II. Recommendation > __ Accept as is > __ Accept with minor revision (no additional review necessary) > _X_ Reconsider after changes (additional review necessary) > __ Reject > __ Submit elsewhere > > III. Detailed Review > > (It is very important to the editors and the authors that you provide a > full account of your evaluation and recommendation. These reviews are a > significant and often overlooked contribution to the scientific > process.) > > Most of my thoughts are captured above, but I'd just like to add that > I was very excited to see a paper appear on this subject. In our > globalization team, we have recognized, for some years, the potential of > this sort of alternative globalization tool. I was obviously > disappointed, however, that the authors seemed not to have done their > homework when it came to understanding what similar technologies exist > today, and would like to see them resubmit the paper after surveying > these technologies, and perhaps doing a bit more thorough testing of > their ideas. > > ============================================= > Review B > > ACM Transactions on Computer-Human Interaction > > Report of Referee: B > Reference Number: 02-25-0 > > Title: Subtitled Interaction: Complementary Support as an Alternative > to Localization > > Authors: > Giorgos Lepouras > George R.S. Weir > > I. General Evaluation. Please explain any "No" answers. > > A. Is the title appropriate? > > okay > > B. Is the abstract satisfactory? > > yes > > C. Are the keywords well chosen? > > no. I'd add > globalization, internationalization, i18n, l10n, translation, > multilingual interfaces > > D. Content > > 1. Does the paper have enough originality and importance to merit > publication? > > I think this aspect is okay. The idea of "subtitles" is original > and may be a practical alternative to localization. > > 2. Are the ideas, algorithms, results or studies technologically and > methodologically sound? > > I think there are some presentation problems in the paper. > I could not tell what was done in several parts of the study. > I suspect there are some confounding factors in the study > because of some of the results, so I would want to see a new > experimental evaluation before concluding that there are > real benefits of the approach. This is explained below. > > 3. Are the references to earlier work adequate? If not, please indicate > additional references that would be appropriate. > > I do not feel competent to evaluate this. > > E. Presentation > > 1. Is the paper well organized? > > The main contribution is the experimental evaluation of a technique, > but the authors do not follow standard methods of presentation > (e.g., APA style) nor do they follow an organized approach. > I'd prefer to see section headings used consistently to identify > different experiments and phases of experiments. > > 2. Is too much, too little or appropriate material included? > > Much of the introduction is nicely written, but not relevant > to the evaluation of subtitled interaction. I thought that sections > 1 and 2 (pages 1-6) could be replaced by a paragraph or two. > Section 3 (which has subsections of 2.x) had some interesting > coverage of issues of culture and icons, but it was barely > relevant to the novel work presented in the paper. I think > that the first 13 pages could be replaced by two pages of > introduction. > > 3. Is the presentation clear? > > Many aspects are not described in enough detail, > See the detailed review. > > 4. Is the English satisfactory? > > Yes. > > II. Recommendation > __ Accept as is > __ Accept with minor revision (no additional review necessary) > X_ Reconsider after changes (additional review necessary) > __ Reject > __ Submit elsewhere > > III. Detailed Review > > (It is very important to the editors and the authors that you provide a > full account of your evaluation and recommendation. These reviews are a > significant and often overlooked contribution to the scientific > process.) > > This paper presents some startling results about a method for > making software more easily used in multiple languages. > Unfortunately, the methodology is not described well, > and some of the results are so unintuitive that they suggest > confounding in the evaluation. For the Greek interface to > fare no better than the English interface for Greek users, > I must conclude that something is not right, and without > enough detail in the paper, I can not tell what is not right. > > I thought the recoding of conditions as letters made the paper > harder to read. I suggest using names for conditions like: > Subtitled or Complementary > English or Non-Native or Foreign > Greek or Native or Localized > > Group A = complementary support (Greek subtitles on English interface) > Group B = English interface for Word 95 > Group C = Greek interface for Word 95 > > Subjects had three 90 minute training sessions over one month, > which is a lot of training with a system. It is a concern that > the training may have been by the experimenters, who over 270 > minutes (4.5 hours) might have conveyed some biases. Additionally, > the training included "the correspondence between original and local > language terms", which would have been useful for Group A, > and critical for Group B, but perhaps a source of confusion for Group C > as it was not useful to realistic use of the software. > > Phase 1 = unstructured formatting task > > The results in Table 4 show better performance for Group A than > for the other groups, although this was not significant. > It is startling that users would perform equally well, > even slightly worse, in Group C than Group B. Replacing the > group codes, Greek students performed slightly worse on the > task with the Greek system than the English system. This > could be attributed to training, or perhaps to the subjective > evaluation of the correctness of the tasks, but the result > is startling and suggests problems in the experimental procedure, > or perhaps just bad luck in random sampling. The results were > not significant, so perhaps this is just a sampling artifact. > Still, I'd expect Greek students to do better with a Greek interface. > There was some discussion of how the tasks were rated, but not by whom, > nor whether there was adequate inter-rater reliability. > > Phase 2 = logged tasks > > I was unable to follow the explanation of how errors were judged. > I wondered if different criteria for judging errors would have > produced different results. It was not clear from the discussion > who was rating nor whether there was any concern for inter-rater > reliability. The results did not add up, literally. In Table 5, > the total times were reported as: > Group A 799.44 > Group B 960.36 > Group C 954.07 > but the sum of the mean times were: > Group A 79.18 > Group B 201.00 > Group C 211.58 > So I can not tell what was done with the data. If the mean times > are correct, then again, the Greek version fared worse than the > English version (although probably non-significantly) and the > times for the Subtitled version were less than half the times > of the others. These results are not credible. Something is wrong. > > Phase 3 = demographic ratings and memory tests > > The demographic ratings in Phase 3 perhaps should have been done > before Phase 1 and used to assign subjects to conditions, but the > ratings were similar across conditions, so that aspect of experimental > control seems to have presented no problems. > > General Conclusions from the Review > > >From the description of the experiments, I would be unable to replicate > what was done in this study, and I am unable to evaluate if the > procedure was reasonable. Of particular concern are the training > procedures and the possible bias introduced to subjects by > experimenters; > subjects in Group A might have been able to guess that the experimenters > had > a vested interest in the subtitled interface and wanted to help them. > The training in English for the Greek interface users must have seemed > irrelevant to realistic use, and I wonder if the memory questions in > Phase 3 were relevant to realistic use. > > The results are so contrary to intuition that I think a higher standard > of evidence is appropriate. Ignoring the subtitled interface, in Phase > 1 and 2, > subjects narrowly and non-significantly performed worse with the Greek > interface > than the English. One could argue that the language of the interface > does not significantly affect performance, which for some types of > tasks, > might be possible, but is very counterintuitive. Subjects rated their > own > knowledge of English as generally good, but I have no way of knowing how > what that means. Perhaps it means they are fluent by some standards, > and if that is the case, then the differences between conditions for > these users are unimportant. Including the subtitled interface, it > appears > that it is better than the English interface, which is intuitive, but it > is also better than the Greek interface, which is intriguing. If the > English interface had fared worse than the Greek interface, I might > be inclined to conclude that there was some synergy in the subtitled > interface, > but when Greek users do no better on a Greek interface than an English > interface, > the results do not make sense to me. > > Some suggestions: > > * Use standardized tests (e.g., TOEFL) to measure knowledge of English. > * Use these test results to assign subjects to different conditions. > * Describe the training in more detail. > * Avoid potential bias in training by using disinterested parties. > * Consider offering little or no training to avoid that possible > confounding > and to be more realistic. > * Describe the tasks in more detail. > * Describe the ratings of correctness in more detail. > * If ratings of correctness are made, then use unbiased judges > and report the inter-rater reliability of those judges. > * Consider using time as the main dependent measure with some cutoff > time > after which a task is considered done. > * To increase statistical power, consider using more subjects or a > within-subjects design in which subjects use: > - Greek only and then Greek with English subtitles > - English only and then English with Greek subtitles > Then, some comparisons can be made within subjects, > although care must be taken to counterbalance the tasks > done first and second by different groups. > > ============================================== > > Review C > > ACM Transactions on Computer-Human Interaction > > Report of Referee: C > Reference Number: 02-25-0 > > Title: Subtitled interaction: Complementary support as an alternative > to localization > > Authors: Giorgos Lepouras and George R.S. Weir > > I. General Evaluation. Please explain any "No" answers. > > A. Is the title appropriate? > > Yes. > > However, I do have one issue: In my view, the "complementary support" > the authors are advocating is a localization technique, in that it tries > to accommodate the linguistic needs of users (one element of > localization). Therefore, it's not quite right to describe it as an > (exclusive) alternative to localization - even though late in the paper > the authors maintain that using complementary support makes localization > unnecessary. > > B. Is the abstract satisfactory? > > Yes. > > However, there's a strong indication that translation and language > issues are the core concern of localization, even though later in the > paper (Section 2.3) the authors' mention that cultural/behavioral issues > also are of importance. The localization community is sensitive to the > widely held misperception that localization only means translation > (which is the easiest of its problems). I would like to see something in > the abstract acknowledging the importance of cultural/behavioral factors > as well as translation. > > C. Are the keywords well chosen? > > Yes. > > D. Content > > 1. Does the paper have enough originality and importance to merit > publication? > > No. > > I don't believe the weight of this work (at present) is sufficient to > merit publication in a journal like TOCHI. It is more a report of > research in progress, with many permutations still to be examined. (It > may be more appropriate for conference presentation, until more in-depth > data are available, from further experiments.) The idea of using tool > tips to enhance the meaning of onscreen labels during mouse-overs is not > new, though a test of their use specifically for localization is novel. > > 2. Are the ideas, algorithms, results or studies technologically and > methodologically sound? > > Yes. > > The experiment is correctly framed and conducted, but as the authors > remark repeatedly, the small cell sizes cast doubt on the findings. Most > of the tests were statistically insignificant (not surprising with small > cells). Also, as I read through the results report I found myself > looking actively for a clear statement of what it seemed the data were > saying - which is not exactly what the authors were suggesting. They did > finally offer the "obvious" (to me) interpretation, but not forcefully: > The use of two languages in a task is reinforcing, providing a richer > semantic experience, regardless of localization issues. That is why, it > seems to me, the complementary support version did better in some ways > than either the English/foreign or Greek/native versions. The point is > that the authors' presentation of findings and later discussion doesn't > focus on what at least I see as the main contribution of this work. > > 3. Are the references to earlier work adequate? If not, please indicate > additional references that would be appropriate. > > No. > > Given the translation focus of the paper, it certainly seems that the > authors should refer to very relevant material in the bi-monthly trade > journal, "Multilingual Computing & Technology" (www.multilingual.com). > Also, several of their references are a bit dated; much has been > published in localization during the past five years. Some of this is > touched upon, but lightly. For example, in the authors' discussion of > globalization methodology (Section 2.3; 2.6), they might profit from a > paper by Carrie Livermore and Jose Coronado, "Growing Pains of Designing > Global Products", in the proceedings of the 2001 IWIPS workshop > (contact: jose_coronado@hyperion.com). This paper describes a software > engineering process that avoids the need to totally re-engineer > localized versions of products as expensive one-off efforts (a point > discussed by the current paper's authors). > > Possibly a nit: The reference style within the text is inconsistent. In > several instances, the authors do not cite the names of all authors (up > to three), whereas elsewhere they do. They've also left "????" embedded > in the References section entry for Tognazzini, B. 1989. > > E. Presentation > > 1. Is the paper well organized? > > Yes. > > However, as noted below, it is slow to get to the specialized content > that is the core of the study. > > 2. Is too much, too little or appropriate material included? > > The paper is slow to develop, in part because of a generalized focus. > Much of the work cited early is not specific to cross-cultural interface > design, but instead is widely accepted, general UI design precepts. The > authors are slow to focus specifically on the specialized area of > localization, separate from such general concepts. > > 3. Is the presentation clear? > > Yes. > > The general structure is good, but the arguments presented seem to be a > bit too verbose. More punchy, even bulleted, style would make the work > "move along" to the relevant issues more briskly. > > 4. Is the English satisfactory? > > Yes. > > Except, I wonder how many readers who are not from the Commonwealth will > know what "knock-off effect" means (e.g., Section 2.5, Paragraph 1, and > elsewhere). > > > II. Recommendation > __ Accept as is > __ Accept with minor revision (no additional review necessary) > __ Reconsider after changes (additional review necessary) > __ Reject > x_ Submit elsewhere > > Conferences where this work might make a contribution include the > International Workshop for Internationalization of Products and Systems > (www.iwips2003.org) or the International Conference on > Cultural Attitudes towards Technology and Communication > (http://www.it.murdoch.edu.au/~sudweeks/catac02/). > > III. Detailed Review > > (It is very important to the editors and the authors that you provide a > full account of your evaluation and recommendation. These reviews are a > significant and often overlooked contribution to the scientific > process.) > > I hope that my detailed remarks above are sufficient to explain my > recommendation that this paper be submitted elsewhere. Since > presumptions about language are so key to this work, I'd recommend that > a trained linguist and a cognitive psychologist also review it to help > the authors focus their arguments. I'd also suggest that the authors > employ a panel of localization experts to assess the L10N quality of any > product that they plan to use in comparison with a non-localized > product. (If the localization applied was not state of the art, > industrial quality, the contrasts with a non-localized product will > suffer. The critique of the localized site then would tend toward a > usability test rather than a localization effects test.)