CS295J/Research proposal before.11/NSF Expedition Reviews
Proposal Information
Proposal Number: 1064261
Proposal Title: Collaborative Research: Cognitive Optimization of Brain-Science Visual-Analysis Tools
Received by NSF: 09/10/10
Principal Investigator: David Laidlaw
Co-PI(s): David Badre Steven Sloman
This Proposal has been Electronically Signed by the Authorized Organizational Representative (AOR).
NSF Program Information
NSF Division: Division of Computer and Communication Foundations
NSF Program: Experimental Expeditions
Program Officer: Mitra Basu
PO Telephone: (703) 292-8649
PO Email: mbasu@nsf.gov
Panel Summary #1
Proposal Number: 1064261 Panel Summary: Panel Summary
Panel Summary for Expeditions Preliminary Proposals
Preliminary Proposal Summary (Vision/Goals of the Expedition) This proposal is focused on blending the areas of cognitive science, neuroscience, and HCI to develop new tools that would help in understanding the interrelationships in complex interconnected data sets. The work would focus on brain science activities but would likely be applicable to many other areas that have complex interconnected data such as crime and terrorism analysis.
The strength of this proposal is in the benefits it would bring to the intersections of cognitive science, neuroscience, and HCI.
Intellectual Merit
The authors of this proposal are planning to build cognitive models of brain scientists' perception and reasoning in performing their research. They then intend to use these models to develop new and improved interaction and visualization techniques for tracing neural pathways, with the expectation that use of the cognitive models will reduce the trial and error required to produce effective tools. Additionally, the cognitive models may even result in the invention of new visualizations through a more systematic exploration of the design space. One novel aspect of this proposal is the inclusion of heuristic knowledge of artists and visual designers related to cognition and perception. However the proposal does not elaborate on how the PIs would use heuristic knowledge of artists and visual designers; this is only mentioned in the introduction but never developed in the proposal.
Overall the panel had difficulty coming to a common understanding of the proposal contents. The range of reviews mirrored the range of what people had read into the proposal and their enthusiasm for the proposal topic.
That said, the proposal failed to convince the reviewers along a number of dimensions.
First, the proposal fails to articulate a clear research plan and a clear set of outcomes. As an example, the proposal mentions a number of cognitive concepts that will be incorporated in the interface, such as causal reasoning and dual systems theory. Utilizing cognitive principles to inform this research is applauded, but we expected to see some indication of how they would be put into practice. The proposal is even less clear when it comes to goal maintenance.
A more ambitious undertaking is that of predicting user performance. This aspect of the work is motivated by previous studies showing that student performance on algebra problems can be predicted based on eye movements. This is a very interesting result, but it is not clear how it would apply to an entirely different domain that requires a different and more taxing set of cognitive skills. The proposal does not describe how user behavior will be measured (other than through eye trackers) or even how performance is going to be measured. Algebra problems are generally closed ended, with a well-defined solution, whereas exploratory data analysis of neuroinformatics data is an open-ended problem.
The proposal needs to be much more explicit about the techniques used to derive predictions including the track records of these techniques and the ways in which these techniques may need to be enhanced to be used in particular application domains.
Another concern is methodological. We suggest that the team identify benchmark tasks that would be representative of the cognitive skills that the interface attempts to capture, and would also vary in their degree of complexity.
Broader Impacts
The proposed work would provide research opportunities for faculty, postdocs and students working in the project. A tool with the capabilities described in the proposal would mostly benefit the brain science research community. The tool will be used in two computer science courses at Brown.
The panel thought that the proposal team could do more work in considering possibilities for broader societal impact and for designing more impactful outreach and education activities that would extend beyond the Brown community.
Summary Rationale for the Recommendation
Despite enthusiasm for this topic and the potential for significant impact if successful, the panel could not support the pre-proposal at this time due to a lack of coherent vision and a realizable plan of implementation.
This is a very ambitious proposal that seeks to develop a new generation of visualization tools for the analysis of neuroinformatics data. While this is the kind of big-picture, high-risk project that the Expeditions in Computing program is designed to support, the proposal itself fails to provide a plan for achieving its high-level, abstract objectives.
Put an X next to the appropriate category
Invite
Invite-if-possible
Do-Not-Invite X
This summary was read by/to the panel, and the panel concurred that the summary accurately reflects the panel discussion.
Panel Recommendation: Do Not Invite
Review #1
Proposal Number: 1064261 NSF Program: Experimental Expeditions
Principal Investigator: Laidlaw, David H
Proposal Title: Collaborative Research: Cognitive Optimization of Brain-Science Visual-Analysis Tools
Rating: Very Good
REVIEW:
What is the intellectual merit of the proposed activity?
This proposal is focused on blending the areas of cognitive science, neuroscience, and HCI to develop new tools that would help in understanding the interrelationships in complex interconnected data sets. One novel aspect of this proposal is the inclusion of heuristic knowledge of artists and visual designers related to cognition and perception. The work would focus on brain science activities but would likely be applicable to many other areas that have complex interconnect data such as crime and terrorism analysis. The strength of this proposal is in the benefits it would bring to the intersections of cognitive science, neuroscience, and HCI. The leadership team of distinguished faculty and researchers is well qualified to conduct this research. The entire team is an appropriate blend of researchers representing all of the subordinate areas of research. The quality of prior work from all participating members is uniformly outstanding and appropriate for this endeavor. While there have been much work of a similar nature done in the past, this project would move our knowledge forward on a number of new fronts. In addition, the inclusion of heuristic knowledge of artists and visual designers related to cognition and perception is novel. The proposal is well conceived and organized and clearly presented. With the pupil tracking device requested in the budget, there appears to be sufficient access to all the required resources necessary for this undertaking. There seems to be a wealth of available experimental facilities in the associated institutions. Some of the relevant equipment includes tiled display walls, stereo-enabled desktop displays, ultra-high-resolution Wheatstone stereoscope, haptic devices, and a virtual-reality cave to come online later this year. The leadership plan appears to be appropriate for the small size of the project personnel. The team members have a good history of interaction on related academic activities. Other than the nominal support for traditional computing need by the computer science department at Brown, there did not appear to be any specific institutional support for this proposal. The budget is well thought out, clearly described, and justified. The collaboration amongst the faculty of Brown, Stanford, and the Rhode Island School of Design appears to be very appropriate for this project. The project would bring together cognitive scientists, visualization experts, and other domain specialists to bridge the gap between theory and practice in this area of brain research. Clearly the synergy in this group would help to insure the success of this work.
What are the broader impacts of the proposed activity?
The value of the proposed work appears to be very important in this specific area of research. While this is not my specific area of expertise, I have some concern whether this project meets the over arching goals of the Expeditions in Computing Initiative. In particular, it would seem to have the potential to stimulate interest in this area for graduate students in related disciplines, but may not be very successful in drawing attention to STEM studies amongst the K-12 age group. A major focus of this proposal is the conceptualization, design, and development of a software framework for predicting user performance. It would gather information on specific models, user interfaces, and user goals and endeavor to produce probabilistic estimates of the state of users over time as predicted by the models. I did not see anything in the proposal that indicated that it would be of particular interest to youth and underrepresented groups. The single sentence on stimulating effective knowledge transfer did not seem convincing.
Summary Statement
This is a good general research proposal in the area of brain research and the understanding of interconnected relationships of complex data sets. There are some novel research concepts, including the heuristic knowledge of artists and visual designers related to cognition and perception. The leadership team is well qualified to lead this endeavor and the outlook for good research results look promising. The proposed budget is in line with the proposed activities and personnel commitments. This proposal should fair well as a general unsolicited proposal for NSF. I would rank this proposal to be better than many of the other proposals of the Expeditions in Computing Initiative.
Review #2
Proposal Number: 1064261
NSF Program: Experimental Expeditions
Principal Investigator: Laidlaw, David H
Proposal Title: Collaborative Research: Cognitive Optimization of Brain-Science Visual-Analysis Tools
Rating: Good
REVIEW:
What is the intellectual merit of the proposed activity?
Intellectual Merit: The main goal is to push the frontiers of data visualization, with secondary thrusts on improved learning in cognition about how we look at data. The strengths of the proposal are that the proposed tool is, as far as I know, ground-breaking in that it will actively change based on the user. Further, the researchers are well qualified, with expertise in the psychology as well as in the computer science. The major weakness was that I was not really clear how the software tool would eventually work. For instance, they talked a little about following pupils of the user - but I was not clear how they would capitalize off of that knowledge.
Value added: I think the proposed research could be complex and important enough to warrant this investment - however I could not find a clearly defined path of attack in the proposal. It fits well within the 3 program goals, with probably the greatest emphasis on the first. Intelligent data visualization tools that can react to the user will open many new doors in understanding science. It will impact and inspire future computer scientists, although I do no see a preference for underrepresented groups. Finally, it has the potential to stipulate new significant findings in science and in education.
Leadership plan: The leadership plan seemed well thought out. They have a diverse set of researchers, each with their unique skill set.
What are the broader impacts of the proposed activity?
Broader Impact: The work will be distributed to all who want to use it and will be used in classes at Brown, affecting students at all levels (either as developers or clients). I found their vision here a little short-sighted.
Summary Statement
In their proposal entitled 'Collaborative Research: Cognitive Optimization of Brain-Science Visual-Analysis Tools,' the authors propose to use understandings from cognitive science, neuroscience and human - computer interaction to develop better tools for examining data. In particular, they will develop software to that will visualize neural connections in the brain. At the same time they will actively measure the client and use these data to predict what the client will want to see next. They present a compelling case that we need better visualization tools for understanding the brain.
Review #3
Proposal Number: 1064261
NSF Program: Experimental Expeditions
Principal Investigator: Laidlaw, David H
Proposal Title: Collaborative Research: Cognitive Optimization of Brain-Science Visual-Analysis Tools
Rating: Excellent
REVIEW:
What is the intellectual merit of the proposed activity?
The dominant approach to developing interactive systems is for the developers to interact with the envisioned users to gather the general requirements for the application and to construct software based on the developer's intuitions as to how the users will actually interact with the software. Depending on the sophistication of the organization, cycles of usability testing and re-design are used to refine the interface; alternately, they may simply release the software to the users and wait for the complaints or lack of sales. An alternative to this expensive process is to construct explicit models of the perceptual and decision making processes of the users and then use these models to inform the design process. Work on cognitive models such as GOMS and ACT began about three decades ago and has progressed slowly but steadily throughout the time period and there have been a number of small demonstrations that such models can, in fact, eliminate most or all of the iteration previously required. The authors of this are planning to build cognitive models of brain scientists' perception and reasoning in performing their research. They then intend to use these models to develop new and improved interaction and visualization techniques for tracing of neural pathways, with the expectation that use of the cognitive models will reduce the trial and error required to produce effective tools. Additionally, the cognitive models may even result in the invention of new visualizations through a more systematic exploration of the design space. Although the focus of the recent work in cognitive models has been to develop engineering models which are capable of being used outside of the research setting, use of such models in the design of interactive systems has been slow to catch on. If nothing else, construction of the models requires a large amount of intellectual labor and, to date, impressive examples of the use of these models to justify that labor investment have been rare. This work has the potential for providing such a critical example and could be the impetus to finally move cognitive models into widespread use. Additionally, work in as complex an area as brain science will ensure that the cognitive modeling tools can handle nearly any application. Finally, if the models do result in improved tools, the research may result in new findings in the brain science field. The primary risk in this proposal is that the cognitive models which can be created are too weak to support the design process. There is no guarantee that the computer scientists and psychologists doing this research will be able understand and model the cognitive processes of a brain scientist.
Value-added of funding the activity as an Expedition This work requires substantial commitment on the part of the computer scientists and psychologists to learn the brain science domain and on the part of the brain scientists for their interaction with the cognitive scientists. Such a commitment is unlikely to be obtained with smaller, more fragmented funding. Industry or venture capital are unlikely to fund this kind of research. The main knowledge transfer methods will be the mentoring of graduate students and the addition of formal courses intended to teach about interdisciplinary collaboration. In addition, the software they develop will be made available for distribution. Except through the rather limited vehicle of scholarly publication, it is not clear how the cognitive models themselves are to be made available. The authors may want to consider using their own visualization capabilities to explain the models.
Leadership and Collaboration Plan Only two institutions are involved in this work and the senior researchers are all located at one of the two institutions. This should minimize coordination problems. Funding for the primary brain scientist in this research is at the 50% level and his supervisor has a nominal level of funding. This is a cause for concern, given the level of commitment required to support what is, essentially, someone else's area of research. A higher level of funding would be desirable, even if a substantial amount of the funded time is spent on pure brain science research.
What are the broader impacts of the proposed activity?
The proposal includes training a number of graduate and post doctoral students; in fact, most of the funding requested is for student support. As mentioned earlier, to the extent this work results in wider acceptance and usage of cognitive models, particularly in the development of scientific software, it will accelerate the construction of interactive systems which can be used efficiently.
Summary Statement
This work should lead signficantly wider use of cognitive modeling in interactive systems design as well as provide researchers in brain science with superior tools.
Review #4
Proposal Number: 1064261
NSF Program: Experimental Expeditions
Principal Investigator: Laidlaw, David H
Proposal Title: Collaborative Research: Cognitive Optimization of Brain-Science Visual-Analysis Tools
Rating: Fair
REVIEW:
What is the intellectual merit of the proposed activity?
Proposal Summary
This proposal seeks to develop new visualization techniques that will assist brain scientists with the interpretation of high-dimensional data. For this purpose, the PI will incorporate design principles and knowledge from cognitive science, neuroscience and human computer interaction. the visualization system will also capture data of scientists as they use the tool, and compare it with computational models from cognition, perception and art. The tool will also be able to predict user performance and user state over time. The tool will be released through an open-source license, and will be incorporated into two courses. The team has worked together for a number of years.
Criterion 1. What is the intellectual merit of the proposed activity?
Strengths
The kind of tool envisioned in this proposal would be invaluable, not only in neuroinformatics but also on other disciplines that deal with high-dimensional, multi-scale data, from social networks to geospatial information.
Weaknesses
Despite its laudable objective, this work is not ready for further scrutiny as a full proposal.
First, the proposal fails to articulate a clear research plan and a clear set of outcomes. As an example, the proposal mentions a number of cognitive concepts that will be incorporated in the interface, such as causal reasoning and dual systems theory. How are these principles going to be used to design a better visualization, and how are they going to be tested? I very much like the idea of using cognitive principles, but would have expected to see some indication of how they would be put into practice. The proposal is even less clear when it comes to goal maintenance: "We will use these principles to determine which tasks to make easily accessible to users and which to put in the background." This is a general problem for interface design, not a solution to the problem.
A more ambitious undertaking is that of predicting user performance. This aspect of the work is motivated by previous studies showing that student performance on algebra problems can be predicted based on eye movements. This is a very interesting result, but it is not clear how it would apply to an entirely different domain that requires a different and more taxing set of cognitive skills. This is a wild extrapolation. The proposal does not describe how user behavior will be measured (other than through eye trackers) or even how is performance going to be measured. Algebra problems are generally closed ended, with a well-defined solution, whereas exploratory data analysis of neuroinformatics data is an open-ended problem.
Another concern is methodological. Say that the proposal had articulated a clear plan and a reasonable set of deliverables for a new generation of visualization interfaces. Wouldn't it be better to test this interface on some benchmark problems, and see how it facilitates performance relative to a standard interface? These benchmark problems would be representative of the cognitive skills that the interface attempts to capture, and would also vary in their degree of complexity.
To what extent does the proposed activity suggest and explore creative, original, or potentially transformative concepts?
The proposal is very ambitious in its overall objectives. A visualization tool having the characteristics suggested in the proposal would be invaluable to brain science as well as to other scientific disciplines dealing with high-dimensional complex data, such as genomics/proteomics, geospatial analysis, network analysis, etc. However, the proposal fails to turn a high-level concept into a realizable implementation.
Is the work of sufficient import, scale, and/or complexity to justify this type of investment?
The brain is one of the scientific frontiers for the 21st century. The proposal has the complexity and scale worthy of this type of investment, but the proposal fails to deliver a realistic plan (if any plan at all) or even specifications.
Will the work contribute to realization of the EIC program goals and is it likely to demonstrate completion of these goals?
Understanding the brain is one of our greatest scientific challenges. Unfortunately, without a clear research plan it is difficult to asses the likelihood that the proposal will be able to demonstrate completion of its overall goals.
Value of the experimental systems or shared experimental facilities proposed
The investigators will utilize some shared facilities in their research and will share software and data that they produce to allow further research by others. The proposed software testbed will be used across the collaborators to test models of cognition and perception in the context of HCI.
Leadership and Collaboration Plan
The investigators have worked together for a number of years, have taught classes together, and their students have attended classes from each other. No leadership or collaboration plan is discussed beyond this.
What are the broader impacts of the proposed activity?
Criterion 2. What are the broader impacts of the proposed activity?
Strengths
The proposed work would provide research opportunities for faculty, postdocs and students working in the project. A tool with the capabilities described in the proposal would mostly benefit the brain science research community. The tool will be used in two computer science courses at Brown.
Weaknesses
Societal benefits of this tool would derive from its scientific merit to the extent that it would help understand the brain. Given the characteristics of this project, I wonder if other NSF funding opportunities would be more suitable, such as the FODAVA program or the interdisciplinary program in neuroscience at CISE. I also wonder whether this work should be funded instead by NIH (NIBIB, NIMH). The budget contains a request for $3,000 to cover costs of animal (mouse) care; why is this needed given that the proposal is for software development?
Summary Statement
This is a very ambitious proposal that seeks to develop a new generation of visualization tools for the analysis of neuroinformatics data. The tools would allow brain scientists to explore high-dimensional data, and the tool would also predict user performance and state. The proposal is inspired by principles from cognitive science, neuroscience and HCI. While this is the kind of big-picture, high-risk project that the Expeditions in Computing program is designed to support, the proposal itself fails to provide a plan for achieving its high-level, abstract objectives. The proposal does not provide a leadership or collaboration plan.
Review #5
Proposal Number: 1064261
NSF Program: Experimental Expeditions
Principal Investigator: Laidlaw, David H
Proposal Title: Collaborative Research: Cognitive Optimization of Brain-Science Visual-Analysis Tools
Rating: Fair
REVIEW:
What is the intellectual merit of the proposed activity?
PROPOSAL OBJECTIVES AND APPROACH The proposal develops a variety of tools for interactive analysis and reasoning for brain scientists.
INTELLECTUAL MERIT The project addresses research in three areas: human-computer interaction, cognitive modeling and the connectivity in the brain. It lists11 items that are to be developed.
The proposal is vague and has a lot of repeatability. It is not well written. It is not clear what research experiments are performed.
The team is fine.
What are the broader impacts of the proposed activity?
There is a great need for the tools by the computational neuroscience and cognitive science community. This project will develop some of these tools.
The details of the educational plan are not given.
Summary Statement
The proposal can be strengthened by focusing and making the challenges and ideas more clear.
Review #6
Proposal Number: 1064261
NSF Program: Experimental Expeditions
Principal Investigator: Laidlaw, David H
Proposal Title: Collaborative Research: Cognitive Optimization of Brain-Science Visual-Analysis Tools
Rating: Poor
REVIEW:
What is the intellectual merit of the proposed activity?
The proposed activity of developing and improving neuroscience visual analysis tools is very important. Currently access to the genre of systems described in this proposal, especially in areas with complex sub-structure such as neuroscience, is lacking and the proposed activity could have a profound effect on the state-of-the-art in the field. The possible interplay between the user-interface experts and biomedical informatics developers is a possible strength. The available team and resources are very strong and capable with an excellent track record in the field.
However, the proposed activities within the proposed are completely underspecified. Given the scale of the project and the challenging nature of the problems to be faced, the level of technical detail and planning presented in the proposal is insufficient and unconvincing. A great many claims are made in the proposal with no clear measurable end-points to determine how success within the project could be evaluated. Specifically, the authors claims to employ developments in concepts of cognition and perception to assist scientists reasoning. What aspects of reasoning? For which tasks? Within which discipline? If this is only related to tractography, what sort of scientific hypotheses to the applicants expect to address? Are they relating these representations of neural connectivity to studies in animals? Are they relating these analyses to other modalities of imaging data? How do they intend to reason over the complex semantics of these other experimental types? These are all glaring omissions from the proposal. The description of the cognitive science aspects of the project were marginally better specified, but I still found the details lacking. For example, the claim was made that 'our system will tune itself to individual work styles. How? What technical elements will the system exploit to accomplish this? In particular, the applicants must spend more time specifying the precise tasks that the system is designed to tackle before it is possible to improve or optimize performance at that task.
In particular, the statement "We expect the core element to evolve significantly through the five years of the project. It cannot be meaningfully defined without the data we will acquire from users, so details beyond this overview are not possible yet" is incredibly revealing and suggests that the applicants themselves do not have a clear idea of how they intend to solve these problems.
The Figures presented in the proposal looked confusing and uninformative, adding nothing to the argument that these systems would actually help a scientist understand the underlying data.
I very much liked the proposed idea of using the system directly in courses taught at Brown and other collaborating institutions. I think that this is a sizable innovation that would be very welcome in the field and might even form the basis of evaluation metrics for the success of the system (which could address one of my previous criticisms of the project).
What are the broader impacts of the proposed activity?
The activity is well integrated with training and learning. There are a number of students in the group, all of whom would have the opportunity to work with professionals from a very different focus. The interdisciplinary nature of the work coupled with the need for analysis tools in biology would be an excellent synergy to cultivate. The presence of high-end graphics equipment (such as a virtual-reality cave and haptic displays, etc.) is a plus for the project but also is a hindrance to enable the developers to release their work to a broader audience. If the system is only available to the small number of people who have access to such facilities, then the impact of the work would be lessened.
The proposed activity makes no specific claims to target or support underrepresented groups explicitly.
The underspecification of the technical aspects of the project undermine the open-source distribution of the code. It is technically demanding to generate usable open source products for other people to use. Notably, browsing the co-PIs webpages, there were no easily accessible open-source software products noticeably available.
Summary Statement
Although the high-level conceptualization of this project is exciting, the way that the project was described in the proposal was massively underspecified. Technical details were lacking and some fundamental aspects of the project's conceptualization in terms of the scientific domain under study were missing. There was no timetable, and no evaluation proposed to see how progress would be measured. The authors should be careful about making high-level claims concerning the possible impact of the proposed without a more carefully constructed argument to back up the claims.