CS295J/Review Response

Pre-proposal (Expeditions) Reviews and Response

Proposal Number: 1064261

Panel Summary for Expeditions Preliminary Proposals

Preliminary Proposal Summary (Vision/Goals of the Expedition)

1. However the proposal does not elaborate on how the PIs would use heuristic knowledge of artists and visual designers; this is only mentioned in the introduction but never developed in the proposal.

We have added a section called "Evaluating Predictive Models" that outlines performance metrics the system will predict, e.g., task completion time, number of insights (see Saraiya et al., 2005), and methods for predicting user state and goals from observed user interactions and eye-tracking. We plan to collect and compare heuristic design knowledge with predictive models of visual-search behavior using eye-tracking; one expected contribution will be validating or revising existing design guidelines. [ Steven Gomez ]

Overall the panel had difficulty coming to a common understanding of the proposal contents. The range of reviews mirrored the range of what people had read into the proposal and their enthusiasm for the proposal topic.

That said, the proposal failed to convince the reviewers along a number of dimensions.

2. First, the proposal fails to articulate a clear research plan and a clear set of outcomes. As an example, the proposal mentions a number of cognitive concepts that will be incorporated in the interface, such as causal reasoning and dual systems theory. Utilizing cognitive principles to inform this research is applauded, but we expected to see some indication of how they would be put into practice. The proposal is even less clear when it comes to goal maintenance.

We have added a section called "Task Analysis" that outlines how we will analyze and encode tasks and sub-tasks that prime or conflict them. In a subsection titled "Applications", we outline the design of a software module that facilitates analyst 'goal maintenance' by collecting user input, predicting goals from this, and displaying suggestions for similar or priming sub-goals. [ Steven Gomez ]

"We have included a plan for implementing cognitive principles. Strategies (through eye-tracking and other means outlined above) will include monitoring users' preferences and cognitive strategies through reaction time data and verbal reports from users. Historically, such strategies have proved successful when gathering user information, input, and comprehension; we expect the same in our implementation. We also intend to use eye-tracking and reaction time procedures in order to best gauge cognitive load to the extent that it affects user experience, as we strive to understand user adaptation and navigation strategies within our applications." [ Clara Kliman-Silver ]

3. A more ambitious undertaking is that of predicting user performance. This aspect of the work is motivated by previous studies showing that student performance on algebra problems can be predicted based on eye movements. This is a very interesting result, but it is not clear how it would apply to an entirely different domain that requires a different and more taxing set of cognitive skills. The proposal does not describe how user behavior will be measured (other than through eye trackers) or even how performance is going to be measured. Algebra problems are generally closed ended, with a well-defined solution, whereas exploratory data analysis of neuroinformatics data is an open-ended problem.

Addressed by response to comment 1. [ Steven Gomez ]

4. The proposal needs to be much more explicit about the techniques used to derive predictions including the track records of these techniques and the ways in which these techniques may need to be enhanced to be used in particular application domains.

Addressed by response to comment 1. [ Steven Gomez ]

5. Another concern is methodological. We suggest that the team identify benchmark tasks that would be representative of the cognitive skills that the interface attempts to capture, and would also vary in their degree of complexity.

We will break down the brain circuitry exploration process into a set of simpler cognitive tasks through cognitive task analysis. Benchmark tests that involve solving simple problems related to brain circuitry analysis will then be designed based on the identified cognitive tasks. These benchmark tests can then be used to evaluate the users' cognitive performance when working with our system in comparison with existing systems. [ Hua, Sep.27, 2011 ]

As the brain exploration process may involve essential complex tasks that are not able to break down into simpler tasks, we choose to evaluate the tool qualitatively over a long-term period: observe and interview users regarding their experience with the tools in order to discover critical features that the existing tool is missing. The researcher will also provide technical support if necessary. (Diem Tran 21:22, 28 September 2011 (EDT))

6. Broader Impacts

The proposed work would provide research opportunities for faculty, postdocs and students working in the project. A tool with the capabilities described in the proposal would mostly benefit the brain science research community. The tool will be used in two computer science courses at Brown. The panel thought that the proposal team could do more work in considering possibilities for broader societal impact and for designing more impactful outreach and education activities that would extend beyond the Brown community.

We have highlighted some of the broader implications for our project, such as making tools publicly available for other communities of brain scientists, medical researchers, and even instructors to learn about brain connectivity through our visualization initiatives. We have also outlined plans to develop tools for elementary, middle, and high school students as they explore the brain, its anatomy, and its capabilities. [ Clara Kliman-Silver ]

Review #1

7. Other than the nominal support for traditional computing need by the computer science department at Brown, there did not appear to be any specific institutional support for this proposal.

What are the broader impacts of the proposed activity?

8. The value of the proposed work appears to be very important in this specific area of research. While this is not my specific area of expertise, I have some concern whether this project meets the over arching goals of the Expeditions in Computing Initiative. In particular, it would seem to have the potential to stimulate interest in this area for graduate students in related disciplines, but may not be very successful in drawing attention to STEM studies amongst the K-12 age group.

Addressed by response to comment 6. [ Clara Kliman-Silver ]

9. I did not see anything in the proposal that indicated that it would be of particular interest to youth and underrepresented groups.

10. The single sentence on stimulating effective knowledge transfer did not seem convincing.

Review #2

11. The major weakness was that I was not really clear how the software tool would eventually work. For instance, they talked a little about following pupils of the user - but I was not clear how they would capitalize off of that knowledge.

As mentioned in the response to the introduction of the pre-proposal, we have expanded out plan about how we will incorporate pupil data from eye-tracking into creating an interface that was best optimized for data analysis (Jenna Zeigen)

Review #3

12. The primary risk in this proposal is that the cognitive models which can be created are too weak to support the design process. There is no guarantee that the computer scientists and psychologists doing this research will be able understand and model the cognitive processes of a brain scientist.

We have included our experience of developing visualization tools for brain study at Brown, and also we have a detailed timeline, in each phase, brain scientists will be involved in the process of model evaluation, testing, and providing feedback.( --- Chen Xu)

13. Except through the rather limited vehicle of scholarly publication, it is not clear how the cognitive models themselves are to be made available. The authors may want to consider using their own visualization capabilities to explain the models.

On page x, section y, we have included preliminary visualizations, which are examples of the visualizations we will implement to explain the cognitive models we intend to incorporate in our research. [ Clara ]

Review #4

Weaknesses

Despite its laudable objective, this work is not ready for further scrutiny as a full proposal.

14. First, the proposal fails to articulate a clear research plan and a clear set of outcomes. As an example, the proposal mentions a number of cognitive concepts that will be incorporated in the interface, such as causal reasoning and dual systems theory. How are these principles going to be used to design a better visualization, and how are they going to be tested? I very much like the idea of using cognitive principles, but would have expected to see some indication of how they would be put into practice. The proposal is even less clear when it comes to goal maintenance: "We will use these principles to determine which tasks to make easily accessible to users and which to put in the background." This is a general problem for interface design, not a solution to the problem.

Reviewer 4 indicated that the proposal was not explicit about how cognitive principles would be implemented in the project. In section ___ on page __, we have elucidated how the principles listed will concretely lead to better, more optimized visualizations, as well as the studies we plan to perform to show that the visualizations are indeed efficient. (Jenna Zeigen)

15. A more ambitious undertaking is that of predicting user performance. This aspect of the work is motivated by previous studies showing that student performance on algebra problems can be predicted based on eye movements. This is a very interesting result, but it is not clear how it would apply to an entirely different domain that requires a different and more taxing set of cognitive skills. This is a wild extrapolation. The proposal does not describe how user behavior will be measured (other than through eye trackers) or even how is performance going to be measured. Algebra problems are generally closed ended, with a well-defined solution, whereas exploratory data analysis of neuroinformatics data is an open-ended problem.

We have made a detailed timeline, and in phase/year 3, we have included the way to measure user performance and evaluate the cognitive models. We evaluate the models prediction against actual user data, in addition to eye-tracking, computer interaction logging, video logging and skin conductance response will be adopted, and the results and help refine the models.( --- Chen Xu)

16. Another concern is methodological. Say that the proposal had articulated a clear plan and a reasonable set of deliverables for a new generation of visualization interfaces. Wouldn't it be better to test this interface on some benchmark problems, and see how it facilitates performance relative to a standard interface? These benchmark problems would be representative of the cognitive skills that the interface attempts to capture, and would also vary in their degree of complexity.

We mainly test the user interface performance by empirical ways. But they are based on the principles of perception and goal selection . We also have included some standards like performance time of an average user completing a unit task . We will include a comparison between the past and present interface in one unit task performance test. ( --- Wenjun Wang)

To what extent does the proposed activity suggest and explore creative, original, or potentially transformative concepts?

The proposal is very ambitious in its overall objectives. A visualization tool having the characteristics suggested in the proposal would be invaluable to brain science as well as to other scientific disciplines dealing with high-dimensional complex data, such as genomics/proteomics, geospatial analysis, network analysis, etc. However, the proposal fails to turn a high-level concept into a realizable implementation.

Our framework allows more modules to be involved in as the system evolves in future. The interface facilitating viewing data and the model to make prediction can be both applied to other disciplines .( --- Wenjun Wang)

Will the work contribute to realization of the EIC program goals and is it likely to demonstrate completion of these goals?

Understanding the brain is one of our greatest scientific challenges. Unfortunately, without a clear research plan it is difficult to asses the likelihood that the proposal will be able to demonstrate completion of its overall goals.

We have added a new section named "Detailed Plan" which describes in details our proposed plan throughout the project duration. We divide the plan into milestones, allocate manpower and budget, and provide tentative schedule for each milestone. [ Diem Tran ]

Weaknesses

17. Societal benefits of this tool would derive from its scientific merit to the extent that it would help understand the brain. Given the characteristics of this project, I wonder if other NSF funding opportunities would be more suitable, such as the FODAVA program or the interdisciplinary program in neuroscience at CISE. I also wonder whether this work should be funded instead by NIH (NIBIB, NIMH). The budget contains a request for $3,000 to cover costs of animal (mouse) care; why is this needed given that the proposal is for software development?

While the scientific merit of this project is most directly applicable to understanding the brain, it should be noted that the goal to “convincingly demonstrate that the employed techniques facilitate better analysis” (described in section c.1) is unique to this proposal, and would have impacts that reached into any other discipline in which vast amounts of data needs to be interpreted. [ Michael Spector ] In addition, it is our hope that much of the development that we do in creating an interface and visualizations that adapt to the user needs will be easily implemented into a much wider range of programs not only across disciplines, but also educational levels, allowing students to find a way to learn and explore that best fits their learning style and needs. [ Jenna Zeigen ]

Review #5

18. The proposal is vague and has a lot of repeatability. It is not well written. It is not clear what research experiments are performed.

As described in the proposal, we apply cognition and visualization principles to develop a brain circuits and connectivity visualization tool. We evaluate our tool using methodologies established in cognitive science, psychology and human computer interaction research. Detailed explanations of our approach are addressed in comments to review 1, 2, 5, 14, 16. [ Diem Tran ]

19. The details of the educational plan are not given.

As noted on page 26 (section g, "Outreach, Education, and Sustainability Plan"), we plan to educate users on the use of our developed tool through the same channels as our outreach efforts; namely, to the visualization, interaction, and brain science communities. We also note that we will deploy our tool in academic settings, such as labs and classrooms in the brain sciences, which will benefit those groups from an educational point of view, as well as provide another avenue through which we can receive feedback on our software. [ Michael Spector ]

Review #6

20. However, the proposed activities within the proposed are completely underspecified. Given the scale of the project and the challenging nature of the problems to be faced, the level of technical detail and planning presented in the proposal is insufficient and unconvincing. A great many claims are made in the proposal with no clear measurable end-points to determine how success within the project could be evaluated. Specifically, the authors claims to employ developments in concepts of cognition and perception to assist scientists reasoning. What aspects of reasoning? For which tasks? Within which discipline? If this is only related to tractography, what sort of scientific hypotheses to the applicants expect to address? Are they relating these representations of neural connectivity to studies in animals? Are they relating these analyses to other modalities of imaging data? How do they intend to reason over the complex semantics of these other experimental types? These are all glaring omissions from the proposal. The description of the cognitive science aspects of the project were marginally better specified, but I still found the details lacking. For example, the claim was made that 'our system will tune itself to individual work styles. How? What technical elements will the system exploit to accomplish this? In particular, the applicants must spend more time specifying the precise tasks that the system is designed to tackle before it is possible to improve or optimize performance at that task.

Response: We have added a section called "Task Analysis," in which we outline a plan for specifying visual analysis tasks through user observation. We will begin with loosely structured observations to generate hypotheses about primary tasks and user strategies in this domain, and continue with cognitive task analysis methods to focus and test these hypotheses. We present preliminary results that show the viability of this plan. (Caroline Ziemkiewicz 10:48, 22 September 2011 (EDT))

Concrete examples and specifications for tasks have been added to section ____. Through pilot studies and interviews, we have identified the most important tasks performed by brain scientists. The proposal now includes a preliminary analysis of cognitive principles that can be used to assist in these tasks, together with hypotheses that we plan to address.
[ Nathan ]

21. In particular, the statement "We expect the core element to evolve significantly through the five years of the project. It cannot be meaningfully defined without the data we will acquire from users, so details beyond this overview are not possible yet" is incredibly revealing and suggests that the applicants themselves do not have a clear idea of how they intend to solve these problems.

We have included a diagram to clarify the software architecture described in the proposal. We implemented a simple version of the architecture as a proof of concept, as described in the section "Preliminary Work". Individual modules will be developed and adjusted with an iterative testing schedule. [It would also be good to add, "This is similar to another problem/tool that worked successfully in this cited paper", but I haven't found a good example yet.] [ Steven Gomez ]

22. The Figures presented in the proposal looked confusing and uninformative, adding nothing to the argument that these systems would actually help a scientist understand the underlying data.

We have incorporated a new set of figures into the proposal, which are more focused on the concept and design of the proposed system; in particular, we have included some figures to demonstrate the prototype design for different views of the system. [ Hua ]

23. The presence of high-end graphics equipment (such as a virtual-reality cave and haptic displays, etc.) is a plus for the project but also is a hindrance to enable the developers to release their work to a broader audience. If the system is only available to the small number of people who have access to such facilities, then the impact of the work would be lessened.

We have expanded our dissemination plan to include multiple distributions for different end-user workstations. Modules that require high-end hardware for online data capture (e.g., eye-tracking) or display will be disabled in the basic distributions for Windows and Mac. Our research agenda includes experiments with this equipment -- for instance, to validate cognitive models or design guidelines -- but it is not necessary for most end users who download releases of the tool. [ Steven Gomez ]

24. The proposed activity makes no specific claims to target or support underrepresented groups explicitly.

25. The underspecification of the technical aspects of the project undermine the open-source distribution of the code. It is technically demanding to generate usable open source products for other people to use. Notably, browsing the co-PIs webpages, there were no easily accessible open-source software products noticeably available.

- Since the review, the PI's lab has published a large selection of source code from previous projects, available from its website (vis.cs.brown.edu), with provisions for multiple platforms.
- We are undertaking a project to make anonymized images of healthy and abnormal brains (and related data), from our projects and those of a number of collaborators worldwide, freely available to the scientific community.
- Development will be done using a publicly visible source code repository (e.g., Github), so that other scientists and the public may be able to track progress, comment on changes, provide feedback, and even contribute pieces of code to the software.
[ Nathan ]

Summary Statement

Although the high-level conceptualization of this project is exciting, the way that the project was described in the proposal was massively underspecified. Technical details were lacking and some fundamental aspects of the project's conceptualization in terms of the scientific domain under study were missing. There was no timetable, and no evaluation proposed to see how progress would be measured. The authors should be careful about making high-level claims concerning the possible impact of the proposed without a more carefully constructed argument to back up the claims.

Full Proposal (SI2-SSI) Review and Response

Dear reviewers

Thank you for your insightful, constructive, and helpful comments. We particularly appreciated the positive evaluation of the proposal and the potential of our methods for enhancing visual data analysis with regards to brain functioning.

We are overwhelmingly grateful to have received a 'competitive' summary evalution.

We have noted and addressed the criticisms below in the context of the reviews received. Our responses and details concerning our revisions are in bold.

Regards,

David Laidlaw et al.

1. The panel's discussion focused on the sustainability issue beyond the end of the project support timeframe. While a section of the proposal talks about the Outreach, Education, and Sustainability Plan, the community outreach and sustainability aspects are treated somewhat cursorily. These two issues are closely related: without community support, the software is unlikely to be sustainable in the long run. On the other hand, the proposers appear to be well known in their field, which may enhance community uptake.

As added in section G, the plan to create long-term sustainability now includes more detail. Specifically we will focus on creating a lasting open source community that provides frequent updates to the code centralized by the core research team following a Macro R&D development infrastructure. ( Stephen Brawner)

2. The proposal was perhaps over-ambitious; aspects of the evaluation (for example, eye tracking) will create large amounts of data that will require correspondingly intensive data analysis. However, the panel felt that even if the project did not accomplish every detail of the proposal, it would still be highly worthwhile. Similarly, while some aspects of the project might be seen as risky, a certain amount of risk is acceptable in NSF proposals, or even expected. Moreover, any risk is mitigated by the qualifications of the PIs, as exemplified by their excellent track record.

3. The proposers have laid out a detailed five year plan. One panelist questioned whether sufficient attention had been paid to issues of sustainability, in particular there was no mention made of plans for software support beyond the end of the five year plan.

Addressed by response to comment 1 (Stephen)

4. The proposers see this as sort of a prototype of the way software could be developed to support other scientific endeavors; this effect appears to constitute most of the Broader Impact (I wouldn't count benefit to "the entire brain science research community", mentioned in the BI statement, as broader impact).

In the Broader Impact section, we have described a plan for additional outreach activities, including launching a new website for uploading and sharing brain circuit diagrams created with the visualization tool. This site will be designed for high school-aged viewers, and will be a portal for sharing current neuroscience and cognitive science researchers with youth. (Steven Gomez)

5. One quibble I have in the Management and Coordination Plan is the statement that "The Stanford researchers will visit Brown if face-to-face interactions become necessary." While electronic communication allows collaboration in ways that would not have been possible before, I think it's not a question of whether face-to-face will be necessary, but how often. Fortunately, this appears to have been built into the travel budget.

As noted by the reviewer, face to face communication will be necessary and not an 'if' but a 'when'. Researchers at Brown will plan to coordinate visits in California and vice-versa. [I would add more detail in this response: "We have included a preliminary schedule of visits between researchers at Stanford and Brown corresponding to major milestones in the Research Plan. (Steven Gomez)]

6. As the proposal is for SI2-SSI, I would like to know more on what the team plans to do to ensure the sustainability of the software and develop open-source community support.

Addressed by response to comment 1

7. It is also unclear what the team will do to integrate diversity into the proposed activity.

Review #1

Rating: Excellent

REVIEW:

What are the broader impacts of the proposed activity?

What is the intellectual merit of the proposed activity?

SSI proposal.

A five year project to develop software for visualization and analysis of brain circuitry by working with brain researchers to analyze their cognitive processes that need to be supported. The software is to help link the visualization workflow to a "decisional" workflow, supporting "reasoning and analysis at a high level, rather than just displaying data."

The development phase will include studies of user interaction at both low level (eye tracking, mouse click logs) and high level (decision making).

What is the intellectual merit of the proposed activity?

The PI's of this project are proposing to develop, test, and deploy software tools for scientific study of brain circuits. The project will focus on building a cyber infrastructure software system that is intended to improve the speed at which those doing brain research are able to complete their data analysis and it will advance the understanding of human cognition. This project has potential to have impact in the way researchers in the field collect and analyze data by providing an a rich set of cyber infrastructure tools for use in studying and modeling brain circuits. The intellectual merit of the project is very high as it has the potential to greatly reduce the time required to collect and analyze data.

What are the broader impacts of the proposed activity?

This project will provide an enhanced set of software tools to researchers in several areas that are conducting research that is related to the human brain. Areas of research that the cyber infrastructure software can be used in included: gene regulation, protein signaling and even crime and terrorism analysis and all have the potential to benefit.

Summary Statement

While this work is mostly outside my domain of expertise, it looks like a well chosen problem and an interdisciplinary effort. In particular, the team appears to have the broad range of people they would need to pull this off, from experts from the target audience (who suggested the project, and is listed as a co-PI), to cognitive scientists and computer scientists. In addition to the Intellectual and Broader Impact criteria, the specific "additional criteria" listed in the Program Solicitation have been explicitly and (to the extent that I can tell) well addressed. The required supplemental documents address the required points as well.

REVIEW:

What is the intellectual merit of the proposed activity?

The proposed activity will allow brain scientists to visualize brain functions more easily and in much more detail than in the past.Network models, coupled with sophisticated methods for dimensionality reduction, promise to offer unique insights into the workings of the human brain. Moreover, the proposed visualization tool will go through rigorous evaluation that will allow its constant improvement. The proposers form a very strong group of well-established researchers in brain science and data visualization, offering a unique collaboration.

What are the broader impacts of the proposed activity?

Researchers from other disciplines, e.g., who study gene regulation, protein signaling, or perform crime and terrorism analysis, etc., have the potential to be benefited by the proposed software.

What is the intellectual merit of the proposed activity?

The goal of this proposal is to develop, test, and deploy interactive visualization tools for scientific study of brain circuits. The tools will help brain researchers view brain circuits at multiple scales and perform sophisticated analysis of research hypotheses. The team members have a decade of experience developing scientific visualization tools for scientific users and consist of experts in cognitive science, neuroscience, computer science, and visual design. They are well qualified to conduct the project.

Review #2

Rating: Excellent

REVIEW:

Summary Statement

This project is focused on an area of research that spans several disciplines that will be able to utilize the cyber infrastructure software that will be developed. The PI's have a proven record of accomplishment in prior research projects. The project has intellectual merit and will have a broad impact by providing an enhanced set of software tools that will facilitate the efforts of researchers doing work related to studying and modeling the brain.

Review #3

Rating: Very Good

Summary Statement

This unique collaboration between brain scientists and data visualization scientists promises to offer tremendous benefits to the scientific community. The software that will be developed, will allow researchers to understand the signal pathways in the human brain in more detail than ever before. The software will be analyzed through a rigorous process, using models of cognition.

Review #4

Rating: Good

REVIEW:

What are the broader impacts of the proposed activity?

This is an interdisciplinary project and the target user community is brain scientists. The tools will be made available to the public and are expected to benefit the entire brain science research community as well as other disciplines studying linked types of data. The tools can also be used in classes to help students understand connectivity.

Summary Statement

I can see the proposed work will be valuable for brain scientists to study connectivity and dynamics of neural circuits in intact brain as existing systems all have limitations and cannot satisfy the needs of brain scientists as discussed in the proposal. Other scientific domains may need similar tools.

This proposal focuses on an interesting problem for which it also provides a novel solution. Therefore, I think this is a quality proposal and worthy of support.

Removed Chunks of the Proposal (not included in the response letter)

Proposal Information

Proposal Number:

1064261

Proposal Title:

Collaborative Research: Cognitive Optimization of Brain-Science Visual-Analysis Tools

Received by NSF:

09/10/10

Principal Investigator:

David Laidlaw

Co-PI(s):

David Badre

Steven Sloman

This Proposal has been Electronically Signed by the Authorized Organizational Representative (AOR).

NSF Program Information

NSF Division:

Division of Computer and Communication Foundations

NSF Program:

Experimental Expeditions

Program Officer:

Mitra Basu

PO Telephone:

(703) 292-8649

PO Email:

mbasu@nsf.gov

Proposal Number:

1064261

NSF Program:

Experimental Expeditions

Principal Investigator:

Laidlaw, David H

Proposal Title:

Collaborative Research: Cognitive Optimization of Brain-Science Visual-Analysis Tools

Proposal Summary

This proposal is focused on blending the areas of cognitive science, neuroscience, and HCI to develop new tools that would help in understanding the interrelationships in complex interconnected data sets. The work would focus on brain science activities but would likely be applicable to many other areas that have complex interconnected data such as crime and terrorism analysis.

The strength of this proposal is in the benefits it would bring to the intersections of cognitive science, neuroscience, and HCI.

Intellectual Merit

The authors of this proposal are planning to build cognitive models of brain scientists' perception and reasoning in performing their research. They then intend to use these models to develop new and improved interaction and visualization techniques for tracing neural pathways, with the expectation that use of the cognitive models will reduce the trial and error required to produce effective tools. Additionally, the cognitive models may even result in the invention of new visualizations through a more systematic exploration of the design space. One novel aspect of this proposal is the inclusion of heuristic knowledge of artists and visual designers related to cognition and perception.

Broader Impacts

The proposed work would provide research opportunities for faculty, postdocs and students working in the project. A tool with the capabilities described in the proposal would mostly benefit the brain science research community. The tool will be used in two computer science courses at Brown.

Summary Rationale for the Recommendation

Despite enthusiasm for this topic and the potential for significant impact if successful, the panel could not support the pre-proposal at this time due to a lack of coherent vision and a realizable plan of implementation.

This is a very ambitious proposal that seeks to develop a new generation of visualization tools for the analysis of neuroinformatics data. While this is the kind of big-picture, high-risk project that the Expeditions in Computing program is designed to support, the proposal itself fails to provide a plan for achieving its high-level, abstract objectives.

Put an X next to the appropriate category

Invite

Invite-if-possible

Do-Not-Invite X

This summary was read by/to the panel, and the panel concurred that the summary accurately reflects the panel discussion.

Panel Recommendation: Do Not Invite

Review #1

Rating: 'Very Good

What is the intellectual merit of the proposed activity?

This proposal is focused on blending the areas of cognitive science, neuroscience, and HCI to develop new tools that would help in understanding the interrelationships in complex interconnected data sets. One novel aspect of this proposal is the inclusion of heuristic knowledge of artists and visual designers related to cognition and perception. The work would focus on brain science activities but would likely be applicable to many other areas that have complex interconnect data such as crime and terrorism analysis. The strength of this proposal is in the benefits it would bring to the intersections of cognitive science, neuroscience, and HCI.

The leadership team of distinguished faculty and researchers is well qualified to conduct this research. The entire team is an appropriate blend of researchers representing all of the subordinate areas of research.

The quality of prior work from all participating members is uniformly outstanding and appropriate for this endeavor.

While there have been much work of a similar nature done in the past, this project would move our knowledge forward on a number of new fronts. In addition, the inclusion of heuristic knowledge of artists and visual designers related to cognition and perception is novel.

The proposal is well conceived and organized and clearly presented.

With the pupil tracking device requested in the budget, there appears to be sufficient access to all the required resources necessary for this undertaking.

There seems to be a wealth of available experimental facilities in the associated institutions. Some of the relevant equipment includes tiled display walls, stereo-enabled desktop displays, ultra-high-resolution Wheatstone stereoscope, haptic devices, and a virtual-reality cave to come online later this year.

The leadership plan appears to be appropriate for the small size of the project personnel. The team members have a good history of interaction on related academic activities.

The budget is well thought out, clearly described, and justified.

The collaboration amongst the faculty of Brown, Stanford, and the Rhode Island School of Design appears to be very appropriate for this project. The project would bring together cognitive scientists, visualization experts, and other domain specialists to bridge the gap between theory and practice in this area of brain research. Clearly the synergy in this group would help to insure the success of this work.

A major focus of this proposal is the conceptualization, design, and development of a software framework for predicting user performance. It would gather information on specific models, user interfaces, and user goals and endeavor to produce probabilistic estimates of the state of users over time as predicted by the models.

Summary Statement

This is a good general research proposal in the area of brain research and the understanding of interconnected relationships of complex data sets. There are some novel research concepts, including the heuristic knowledge of artists and visual designers related to cognition and perception. The leadership team is well qualified to lead this endeavor and the outlook for good research results look promising. The proposed budget is in line with the proposed activities and personnel commitments. This proposal should fair well as a general unsolicited proposal for NSF. I would rank this proposal to be better than many of the other proposals of the Expeditions in Computing Initiative.

Review #2

Rating: Good

What is the intellectual merit of the proposed activity?

Intellectual Merit: The main goal is to push the frontiers of data visualization, with secondary thrusts on improved learning in cognition about how we look at data. The strengths of the proposal are that the proposed tool is, as far as I know, ground-breaking in that it will actively change based on the user. Further, the researchers are well qualified, with expertise in the psychology as well as in the computer science.

Value added:

I think the proposed research could be complex and important enough to warrant this investment - however I could not find a clearly defined path of attack in the proposal. It fits well within the 3 program goals, with probably the greatest emphasis on the first. Intelligent data visualization tools that can react to the user will open many new doors in understanding science. It will impact and inspire future computer scientists, although I do no see a preference for underrepresented groups. Finally, it has the potential to stipulate new significant findings in science and in education.

Leadership plan:

The leadership plan seemed well thought out. They have a diverse set of researchers, each with their unique skill set.

What are the broader impacts of the proposed activity?

Broader Impact: The work will be distributed to all who want to use it and will be used in classes at Brown, affecting students at all levels (either as developers or clients). I found their vision here a little short-sighted.

Summary Statement

In their proposal entitled 'Collaborative Research: Cognitive Optimization of Brain-Science Visual-Analysis Tools,' the authors propose to use understandings from cognitive science, neuroscience and human - computer interaction to develop better tools for examining data. In particular, they will develop software to that will visualize neural connections in the brain. At the same time they will actively measure the client and use these data to predict what the client will want to see next. They present a compelling case that we need better visualization tools for understanding the brain.

Review #3

Rating: 'Excellent

What is the intellectual merit of the proposed activity?

The dominant approach to developing interactive systems is for the developers to interact with the envisioned users to gather the general requirements for the application and to construct software based on the developer's intuitions as to how the users will actually interact with the software. Depending on the sophistication of the organization, cycles of usability testing and re-design are used to refine the interface; alternately, they may simply release the software to the users and wait for the complaints or lack of sales.

An alternative to this expensive process is to construct explicit models of the perceptual and decision making processes of the users and then use these models to inform the design process. Work on cognitive models such as GOMS and ACT began about three decades ago and has progressed slowly but steadily throughout the time period and there have been a number of small demonstrations that such models can, in fact, eliminate most or all of the iteration previously required.

The authors of this are planning to build cognitive models of brain scientists' perception and reasoning in performing their research. They then intend to use these models to develop new and improved interaction and visualization techniques for tracing of neural pathways, with the expectation that use of the cognitive models will reduce the trial and error required to produce effective tools. Additionally, the cognitive models may even result in the invention of new visualizations through a more systematic exploration of the design space.

Although the focus of the recent work in cognitive models has been to develop engineering models which are capable of being used outside of the research setting, use of such models in the design of interactive systems has been slow to catch on. If nothing else, construction of the models requires a large amount of intellectual labor and, to date, impressive examples of the use of these models to justify that labor investment have been rare. This work has the potential for providing such a critical example and could be the impetus to finally move cognitive models into widespread use.

Additionally, work in as complex an area as brain science will ensure that the cognitive modeling tools can handle nearly any application.

Finally, if the models do result in improved tools, the research may result in new findings in the brain science field.

Value-added of funding the activity as an Expedition

This work requires substantial commitment on the part of the computer scientists and psychologists to learn the brain science domain and on the part of the brain scientists for their interaction with the cognitive scientists. Such a commitment is unlikely to be obtained with smaller, more fragmented funding. Industry or venture capital are unlikely to fund this kind of research.

The main knowledge transfer methods will be the mentoring of graduate students and the addition of formal courses intended to teach about interdisciplinary collaboration. In addition, the software they develop will be made available for distribution.

Leadership and Collaboration Plan

Only two institutions are involved in this work and the senior researchers are all located at one of the two institutions. This should minimize coordination problems.

Funding for the primary brain scientist in this research is at the 50% level and his supervisor has a nominal level of funding. This is a cause for concern, given the level of commitment required to support what is, essentially, someone else's area of research. A higher level of funding would be desirable, even if a substantial amount of the funded time is spent on pure brain science research.

What are the broader impacts of the proposed activity?

The proposal includes training a number of graduate and post doctoral students; in fact, most of the funding requested is for student support.

As mentioned earlier, to the extent this work results in wider acceptance and usage of cognitive models, particularly in the development of scientific software, it will accelerate the construction of interactive systems which can be used efficiently.

Summary Statement

This work should lead signficantly wider use of cognitive modeling in interactive systems design as well as provide researchers in brain science with superior tools.

Review #4

Rating: 'Fair

What is the intellectual merit of the proposed activity?

Proposal Summary

This proposal seeks to develop new visualization techniques that will assist brain scientists with the interpretation of high-dimensional data. For this purpose, the PI will incorporate design principles and knowledge from cognitive science, neuroscience and human computer interaction. the visualization system will also capture data of scientists as they use the tool, and compare it with computational models from cognition, perception and art. The tool will also be able to predict user performance and user state over time. The tool will be released through an open-source license, and will be incorporated into two courses. The team has worked together for a number of years.

Criterion 1. What is the intellectual merit of the proposed activity?

Strengths

The kind of tool envisioned in this proposal would be invaluable, not only in neuroinformatics but also on other disciplines that deal with high-dimensional, multi-scale data, from social networks to geospatial information.

Is the work of sufficient import, scale, and/or complexity to justify this type of investment?

The brain is one of the scientific frontiers for the 21st century. The proposal has the complexity and scale worthy of this type of investment, but the proposal fails to deliver a realistic plan (if any plan at all) or even specifications.

Value of the experimental systems or shared experimental facilities proposed

The investigators will utilize some shared facilities in their research and will share software and data that they produce to allow further research by others. The proposed software testbed will be used across the collaborators to test models of cognition and perception in the context of HCI.

Leadership and Collaboration Plan

The investigators have worked together for a number of years, have taught classes together, and their students have attended classes from each other. No leadership or collaboration plan is discussed beyond this.

What are the broader impacts of the proposed activity?

Criterion 2. What are the broader impacts of the proposed activity?

Strengths

The proposed work would provide research opportunities for faculty, postdocs and students working in the project. A tool with the capabilities described in the proposal would mostly benefit the brain science research community. The tool will be used in two computer science courses at Brown.

Summary Statement

This is a very ambitious proposal that seeks to develop a new generation of visualization tools for the analysis of neuroinformatics data. The tools would allow brain scientists to explore high-dimensional data, and the tool would also predict user performance and state. The proposal is inspired by principles from cognitive science, neuroscience and HCI. While this is the kind of big-picture, high-risk project that the Expeditions in Computing program is designed to support, the proposal itself fails to provide a plan for achieving its high-level, abstract objectives. The proposal does not provide a leadership or collaboration plan.

Review #5

Rating: Fair

What is the intellectual merit of the proposed activity?

PROPOSAL OBJECTIVES AND APPROACH

The proposal develops a variety of tools for interactive analysis and reasoning for brain scientists.

INTELLECTUAL MERIT

The project addresses research in three areas: human-computer interaction, cognitive modeling and the connectivity in the brain. It lists11 items that are to be developed.

The team is fine.

What are the broader impacts of the proposed activity?

There is a great need for the tools by the computational neuroscience and cognitive science community. This project will develop some of these tools.

Summary Statement

The proposal can be strengthened by focusing and making the challenges and ideas more clear.

Review #6

Rating: Poor

What is the intellectual merit of the proposed activity?

The proposed activity of developing and improving neuroscience visual analysis tools is very important. Currently access to the genre of systems described in this proposal, especially in areas with complex sub-structure such as neuroscience, is lacking and the proposed activity could have a profound effect on the state-of-the-art in the field. The possible interplay between the user-interface experts and biomedical informatics developers is a possible strength. The available team and resources are very strong and capable with an excellent track record in the field.

I very much liked the proposed idea of using the system directly in courses taught at Brown and other collaborating institutions. I think that this is a sizable innovation that would be very welcome in the field and might even form the basis of evaluation metrics for the success of the system (which could address one of my previous criticisms of the project).

What are the broader impacts of the proposed activity?

The activity is well integrated with training and learning. There are a number of students in the group, all of whom would have the opportunity to work with professionals from a very different focus. The interdisciplinary nature of the work coupled with the need for analysis tools in biology would be an excellent synergy to cultivate.

Summary Statement

Although the high-level conceptualization of this project is exciting, the way that the project was described in the proposal was massively underspecified. Technical details were lacking and some fundamental aspects of the project's conceptualization in terms of the scientific domain under study were missing. There was no timetable, and no evaluation proposed to see how progress would be measured. The authors should be careful about making high-level claims concerning the possible impact of the proposed without a more carefully constructed argument to back up the claims.

Unused Parts from Proposal

INTELLECTUAL MERIT (INCLUDING POTENTIAL TRANSFORMATIVE ASPECTS):

POSITIVE ASPECTS OF THE PROPOSAL AND PROPOSED RESEARCH:

The panel noted that the proposal was clear and comprehensive, and addressed the criteria for SI2 proposals as well as the general NSF criteria. The proposed software would enhance both visualization of data on brain function and the knowledge discovery process of researchers in this area.

==========================

BROADER IMPACTS:

POSITIVE ASPECTS OF THE PROPOSAL AND PROPOSED RESEARCH:

The panel believes that the framework of this project would be portable to such other fields as gene regulation and the analysis of other complex networks.

SHORTCOMINGS AND WEAKNESSES OF THE PROPOSAL AND PROPOSED RESEARCH:

==========================

Panel Summary

SHORTCOMINGS AND WEAKNESSES OF THE PROPOSAL AND PROPOSED RESEARCH:

ADDITIONAL REVIEW CRITERIA:

The proposed software would primarily be of use to brain researchers. Other fields would be impacted indirectly, in the sense that if this way of building software packages combining visualization with support for hypothesis testing and tracking of analyses succeeds, it might provide a pattern for those fields to follow in their own software development processes.

PANEL RECOMMENDATION (CHECK ONE):

[X] Competitive (C)

[ ] Not Competitive (NC)

This panel summary was read by panelists who participated in the discussion of this proposal, and they concurred that the summary accurately reflects the panel discussion.

Panel Recommendation: Competitive

SYNTHESIS COMMENTS:

The panel agreed that this was a highly competitive proposal. In particular, the proposal attacks a large but tractable problem, and the team's wide and deep expertise gives the project a high probability of success.