CS295J/Proposal reviews from class 8

These are reviews and intellectual contributions for the research proposal, as specified in assignment 7, part A.

Andrew Bragdon

Intellectual merit

(OK other people didn't really seem to write reviews of the proposal?? My understanding is we are supposed to write a critical review of the proposal, simulating a third party reviewer like one might see on an NSF/NIH panel reviewing the proposal.)

Overall, I would say as a reviewer that the proposal is poor because it lacks focus, and should not be funded, being far below the pay line. It proposes many interrelated things which are similar really only in that they all pertain to HCI, many of them vaguely described. As an NIH reviewer I expect a research proposal to be a cohesive whole with a specific set of goals which make sense together. Just looking at the specific contributions, one wonders what is the relationship between a unified set of design guidelines and "a low-overhead mechanism for capturing event-based interactions between a user and a computer"? Both could be valuable contributions in their own right, but are distinct and unrelated. The PIs should have explained the relationships between the separate contributions.

Similarly, the background research provided does not match the specific contributions well, and the connection is not explained. At a minimum, I would expect the authors to tell a cohesive narrative in which background, specific contributions, preliminary work, and so on fit into a larger, coherent vision - rather than the jumbled list as the proposal stands. I would expect, at minimum, that the authors do the following (see gaps below).

Gaps

• Proposal needs to make sense as a whole
• Proposal reads like a laundry list of unrelated components; we need a unified narrative
• Connection between background and contributions is not made clear enough
• Project summary is out of date
• Should have supporting diagrams and figures to clearly communicate the ideas in the project

Suggestion on Overall Theme

I think that rather than a "grand unified theory of HCI" a more realistic, and frankly plausible, goal for the grant would be to propose a theory or empirical study of one specific area or two or more closely related areas. This will make the goal seem doable within 3 years and $1.5 million. It will also, hopefully, bring a sense of purpose to the proposal - a set of core goals.

If we did scope down the proposal to a smaller subset of the current goals, then we should consider also making sure that we are "proposing enough"; that the money could not be spent more efficiently on a different project as well.

Gideon

Intellectual merit

There seems to be a critical lack of focus in your proposal. Are you trying to develop a unified theory of human-computer interaction?

In your contributions section, you speak of developing a comprehensive model of user-interaction in order to predict performance. You also speak of collecting data, and even a new low-cost eye-tracking apparatus. If your goal is to develop a model by taking insights from research in other fields, where do you plan to incorporate your own data collection?

Why do you think there is no unified theory of HCI? This is a rather new field of science, and it is generally agreed that much research needs to be done. A unified theory of human-computer interaction seems a bit ambitious at this time. After all, there is no such thing as a unified theory of cognitive science, psychology, computer science, sociology, or any major discipline really. If the fields that form the basis for your research program are not unified, how can it be possible to make something unified out of them.

What is meant by "predict user performance?" This is very vague. Once again, phrased this way it appears overly ambitious. You do outline 7 specific contributions. These are unrelated and together form a very strange mixture of fields to look at when considering your overall goal.

In conclusion, please resubmit this after developing a more concise and coherent proposal. Perhaps you would benefit from focusing on one specific area to look at. It seems to us that you're trying to develop a revolutionary theory for HCI without having the necessary and extremely comprehensive background to do so.

However, your individual ideas are very interesting, and we think some of them have potential for publication, perhaps as independent journal articles.

Thank you.

Gaps

• There are many gaps in your proposal, such as "Build X".
• Overall lack of coherence
• Not a comprehensive background section

EJ

Intellectual merit

While attempts have been made in the past to apply cognitive theory to the task of developing human-computer interfaces, there remains much work to be done. No standard and widespread model for the cognitive interaction with a computer exists. The roles of perception and cognition, while examined and studied independently, are often at odds with empirical and successful design guidelines in practice. Methods of study and evaluation, such as eye-tracking and workflow analysis, are still governed primarily by the needs at the end of the development process, with no quantitative model capable of influencing efficiency and consistency in the field.

We demonstrate in wide-ranging preliminary work that cognitive theory has a tangible and valuable role in all the stages of interface design and evaluation: models of distributed cognition can exert useful influence on the design of interfaces and the guidelines that govern it; algorithmic workflow analysis can lead to new interaction methods, including predictive options; a model of human perception can greatly enhance the usefulness of multimodal user study techniques; a better understanding of why classical strategies work will bring us closer to the "holy grail" of automated interface evaluation and recommendation. We can bring the field further down many of the only partially-explored avenues of the field in the years ahead.

Gaps

• Project summary needs expanding and context, but this may be impossible before we solidify other sections
• Background is strong, and does not have an excess of context, but that may be desirable here. Some notes/outlines/questions need to be answered/expanded/removed.
• Is "Significance" supposed to be significantly (ha) different from a projection of the influence and effects of the study? Perhaps these estimates should be parsed out of contributions and placed here.
• Need stronger distinction/clarification between "Aims" and "Contributions."

Eric

Intellectual merit

With the emergence of documented interaction histories in scientific visualization comes a new source of data for predicting user interactions. Correct prediction and corresponding UI modifications allow for a more personalized interface that can improve the user's efficiency in data exploration and enables groups of researchers working on the same type of task to more efficiently learn from one another.

In a 30-hour preliminary study, we have implemented a basic interaction prediction module using a relational markov model and shown through a series of user studies that it predicts on average 35% better than chance. We have also created a module that provides basic recommendations to the user based on these interaction predictions, and have shown that the user clicks on a recommended action 20% of the time, leading to an average task speedup of 8%.

The tasks for future work are twofold: first, we will improve and generalize our prediction module to allow for more accurate predictions in a wide variety of interfaces, including those with a larger number of possible actions and states. Second, we will further study the question of, given predictions of future interactions, how to modify the interface beyond giving basic recommendations. Ultimately, research in both of these directions will allow researchers to more efficiently glean information from complex data, enabling them to more quickly and easily contribute to their respective fields.

Gaps

• A few places in the proposal are cut off mid-sentence or are otherwise incomplete, (e.g. some unfinished lists with "???" bullet points)
• The project summary makes it sound like our projects are more integrated than perhaps they are. Maybe we need to either re-evaluate our project summary, or better unify our individual projects somehow.

Adam

Note: EJ and I have generated joint specific aims and contributions and plan to generate unified preliminary results.

Second Note: According to other examples I misunderstood when I construed this section as a review of intellectual merit instead of writing an intellectual merit section. I'll add one soon.

Intellectual merit

The intellectual merit of the program is currently hard to determine because it is so fractionated. It seems to me that we have two or three goals:

1. Using cognitive principles to develop new design guidelines, especially ones that are above the level of the individual task. The content related to working spheres and distributed cognition mostly belongs here.
2. Using cognitive principles to make existing design principles more quantifiable. This could involve building an integrated cognitive model, but I think it's more plausible to generate separate measures for different cognitive/design principles.
3. Improving methods for assessing interfaces. To the degree that this involves using cognitive simulations, it overlaps with goal number 2. Development of an eye-tracking method or an information integration framework would be unrelated to goal number 2. I don't think anyone has been pursuing either of these ideas and I would recommend dropping them.

I think reorganizing our efforts so that they all fit together, either along these lines or some other way, is essential at this point.

Gaps

• Currently I perceive more gaps than content. See above.

Steven

Intellectual merit

As a proposer:

Cognitive theories of human-computer interaction are neither few nor far between, but appear to have received minimal attention in the HCI community and less, if not absolutely no, implementation in the design of digital interfaces. There is not one weak link in the chain from cognitive HCI theories, models, guidelines, etc., but a general dearth of scholarly agreement at every stage of this progression. Scholars maintain antiquated, esoteric models and have not used their combined knowledge of proven cognitive processes and relevant current design practices to improve the human-computer relationship.

There exist multiple gaps in the spectrum of cognitive HCI theory and design, the most pressing of which we have demonstrated a capacity to fill. We have demonstrated in several preliminary studies and experiments the tangible prospect of not only developing a knowledge base for the advancement of cognitive HCI theory and design but for the development of sets of empirically verified models and guidelines which may aid program design at every level of the theoretical/technical spectrum. Our project thus promises not only to bring the field of cognitive HCI up to date but to provide a foundation for myriad related research topics.

Gaps

• Specific aims ought to be revised in light of the further development and specification of our individual goals
• Perhaps we should have a background section on prominent HCI concepts – the fact that GOMS isn’t mentioned once in the proposal seems like an oversight
• Constrain specific aims to preliminary results - It seems they ought to be a bit more congruent

Jon Ericson

Intellectual merit

Overall, the proposal is strong in terms of its potential intellectual contribution to HCI; however, considered as a whole, it lacks the necessary coherence to be fundable. Many of the specific contributions are interesting and innovative, however, the relationship between them is unclear. How will the proposed model of user workflow be integrated with the model of human cognition? Are these intended to be subcomponents of a larger, more comprehensive HCI model or will they remain separate modules? Developing a reliable method for collecting comprehensive data on user performance would be useful in collecting the user data necessary to inform the cognitive model but might constitute its own research program -- an entire grant proposal could be devoted to this enterprise. The specific aims lack coherence; it is not clear how each of the specific aims contribute to the overarching goal of the project.

Despite these shortcomings, the proposal does have several unique strengths. Most current models of HCI do not explicitly incorporate many relevant facts of human ecology, and some of the utility of the original notion of "affordances" has been somewhat lost in translation; efforts to incorporate Gibsonian principles into the cognitive model would be fruitful if those principles could be sufficiently formalized. Reconciling this approach with information-processing approaches to perception would represent a significant advance in applying perceptual theory to HCI, however, the proposal does address how this reconciliation will be accomplished. The promise of a model component that incorporates the notion of working spheres, and the possiblity of automated user performance data capture methods are innovative, but their relationship to the overall goal of the project needs to be clarified. The proposal would benefit from explicit connections between the unifying theme and the steps necessary to realize the larger goal of the project.

Gaps

• The section on Gibsonianism could be improved/expanded considerably.
• Specific aims are not coherent.
• There is a certain amount of overlap between some of the specific contributions and our user studies; perhaps some of them could be merged.
• We need to run an experiment in which we use a rudimentary model of human cognitive abilities to predict some aspect of user interaction and compare those predictions to actual performance.

Ian Spector

Intellectual merit

While the intent of the project is noble, the proposal lacks enough specificity of direction. There has clearly been substantial effort put towards background theory, but the project is so wide-reaching in its aims at the expense of details on the actual integration of the disciplines of cognition, design, and human-computer interaction. Furthermore, what is being proposed has the appearance of a natural evolution of interaction design rather than the formal creation of a new multidisciplinary look at the intersection between cognition and HCI. In this respect it may be worthwhile to investigate the effectiveness of this approach through funding.

As far as the novelty of the proposal is concerned, there currently exists a small but significant body of knowledge regarding cognitively "enhanced" HCI that continues to grow. This body of work incorporates various aspects of the proposal with each study comprising a very narrow focus. There may be some merit to the proposal if its aim is to tie each of these smaller studies together and once completed, determine a unified theory. However, there is still an abundant amount of work yet to be completed.

Gaps

• The link between cognition and HCI, and how the proposal intends to create that link, needs to be strengthened
• It would be good to perhaps begin listing specific experiments that may be useful in bolstering the validity of the proposal

Trevor

Summary

This research proposal aims to bridge a series of gaps between research in cognitive/perceptual science and human-computer interaction. The proposed work plan, however, lacks specificity as to how this integration will be achieved and/or validated. Further, the proposal does little to truly distinguish itself from previous work. While the background section is extensive, the contributions section does not offer compelling evidence as to why this work would make novel contributions to the fields of cognition or HCI.

In general, the proposal lacks realistic, tangible milestones. The main deliverables seem to be ideas and theories, and we at the Institute would prefer to see more quantifiable evidence.

Intellectual Merit

There is certainly intellectual merit in achieving the goals outlined in the project summary; however, the scattershot nature of the proposal and work plan make it very difficult to determine whether or not those goals are achievable.

Broader Impact

While the broader impact of this work is certainly implicit, there is no mention of how the contributions of this work extend to other areas. The proposal would greatly benefit from explicitly stating concrete applications of its results, and speaking to its overall impact.

Gaps

• Direct mapping between results and contributions.
• Mapping between individual contributions and centralized theme of the proposal.