Exp 12 reviews
Panel Summary #1
Proposal Number: 1317515
Panel Summary: Panel Summary
PROPOSAL SUMMARY (VISION/GOALS OF THE EXPEDITION)
This proposal seeks to develop a set of instrumented visualization applications for brain and genomics researchers. The instrumentation will allow studies of how the tools are used. Together with knowledge from cognitive science, illustration, and art these studies will lead to the development of predictive cognitive models that will result in (1) better visualization tools and (2) the foundations for a theory of visualization.
INTELLECTUAL MERIT
The panel felt that strengths of this proposal were that core ideas were good: (1) the cognitive models for visualization and (2) the automatic optimization of visualizations. The panel felt that the topic of the project was interesting and the assembled team was a strong one.
The panel identified a number of weaknesses with the proposal. Foremost among these was a lack of detail regarding how the cognitive models would be developed and assessed. Also, there was no time spent in the proposal discussing how the optimization of visualizations would actually occur. The panel felt that, in general, the proposal lacked many details about how precisely the PIs intended to accomplish their larger goals, instead focusing on details of the driving applications. This lack of detail in the proposal's description on the larger science of visualization or field of HCI was considered a significant weakness, that led to a consensus that there was not enough scientific merit to the research agenda for an Expedition-level grant.
BROADER IMPACTS
The panel thought that a strength of the broad impacts was that better visualization tools for brain connectivity and genomics would have impact within domains that use such information, such as neuroscience and medicine.
The panel felt that a weakness of the proposed broader impact was that the proposal did not sufficiently address how visualization tools in the area of brain connectivity and genomics will have relevance to visualization problems outside of those domains, or to issues outside of visualization. The panel agreed that the proposal had ended up being somewhat too focused on the driving applications, to its detriment. The panel also felt that the outreach and dissemination plans were unexceptional.
RECOMMENDATION AND SUMMARY RATIONALE FOR THE RECOMMENDATION
The panel's consensus was "Do Not Invite" (DNI). The intellectual merit of the proposal was not deemed to have the necessary qualifications for transformative impact or ability to catalyze new research to warrant further pursuit.
This summary was read by/to the panel, and the panel concurred that the summary accurately reflects the panel discussion.
Panel Recommendation: Do not invite
Review #1
Proposal Number:
1317515
NSF Program:
Experimental Expeditions
Principal Investigator:
Laidlaw, David H
Proposal Title:
Collaborative Research: Scientific Visual Analytics: Advancing Theory and Practice through Cognitive Modeling
Rating:
Fair
REVIEW:
In the context of the five review elements, please evaluate the strengths and weaknesses of the proposal with respect to intellectual merit.
The proposal aims to develop tools for HCI study, with specific applications in brain connectivity and genomics.
The main focus seems to be collecting data, assemble software for data processing, and provide summary on predictive model outputs. It is difficult to identify a groundbreaking vision and concrete directions that may significantly advances science.
The data collection plan is detailed and appropriate. Though the types of data it will collect, and the manner they are collected, are quite standard in HCI study.
The proposal hinted the long term goal of building infrastructure for automatic optimization of visualization software. This is probably the most interesting aspect in the proposal; it necessarily needs to integrate learning models, user modeling, interface design and software engineering. Unfortunately, little detail is provided to see how the proposal plans to make it happen.
Even for predictive models themselves, the proposal does not contain enough details to convincingly show how ground truth data can be obtained (user interview alone may not be sufficient).
In the context of the five review elements, please
evaluate the strengths and weaknesses of the proposal with respect to broader impacts.
The HCI research may result in better user interactions for multiple science domains. The project will train students, enhance three existing courses, make the software available, and enable junior researchers to get into research earlier.
Please evaluate the strengths and weaknesses of the proposal with respect to any additional solicitation-specific review criteria, if applicable
The PIs have a track record of working together, which I view as a plus. On the other hand, the team is somewhat narrow and small. The two science domains are quite close to each other. While this can be a good synergy, it also raises the question of whether different science domains require significantly different human computer interaction emphasis that may not be immediately obvious from this project.
Summary Statement
A good proposal with strong application domains and established collaboration. Can go deeper in the proposed activities.
Review #2
Proposal Number:
1317515
NSF Program:
Experimental Expeditions
Principal Investigator:
Laidlaw, David H
Proposal Title:
Collaborative Research: Scientific Visual Analytics: Advancing Theory and Practice through Cognitive Modeling
Rating:
Good
REVIEW:
In the context of the five review elements, please evaluate the strengths and weaknesses of the proposal with respect to intellectual merit.
My review focuses on the impact of the research on aspects of brain connectivity -- especially on the prefrontal cortex and the impact of cognitive load on function.
STRENGTHS:
- scientific visual analytics is interesting from a learning perspective. Many individuals learn best from visual and kinesthetic (hands-on) activities. Incorporating both learning styles into the model would presumably facilitate learning and assist in the model development.
- The user data (intentions, traits, interaction histories, states, and analysis outcomes) can potentially provide important information about the user during task performance. What is not clear is how brain function will be measured.
WEAKNESSES:
- Given the proposed activities and deliverables - it seems that Brown University would primarily benefit from the project (see #4, & #13 on Fig.1) and none of the other collaborating institutions are mentioned.
The intellectual merit of the proposal is focued primarily on computer science with some applications to genomics in the development of human computer interaction tools -- specifically scientific visual analytics. Since this area is not in my area of expertise, I cannot make an informed, reasoned evaulation on the impact of this proposal to the field of computer science.
The PI is well-established and has been successful in the past with projects and the deliverables from the project. The collaborators are also known entitities with established labs and contributions to their disciplines.
In the context of the five review elements, please evaluate the strengths and weaknesses of the proposal with respect to broader impacts.
THe goals in contributing to the Expedition program elucidate the broader impacts of the proposal. In this area, knowledge transfer and career development in computer science and engineering are the foci. It appears that there is some focused effort on facilitating diversity. A weakness is the educational component especially with undergraduate and graduate students.
How the tools will benefit the 'entire brain science and genomic research communities' is unclear.
Please evaluate the strengths and weaknesses of the proposal with respect to any additional solicitation-specific review criteria, if applicable
Given that several of the disciplines are not in my area of expertise, I am not sure if the budget is appropriate for the scope and level of the project as I think it is a bit inflated. I will defer this point to the panel regarding the budget for this project.
Summary Statement
The development of a scientific visual analyzer is appealing especially as it taps into the two most common learner modalities (visual and kinesthetic). The work is interesting and could potentially impact the learning and conduct of science. The collaborative team is synergistic and focused.
Review #3
Proposal Number:
1317515
NSF Program:
Experimental Expeditions
Principal Investigator:
Laidlaw, David H
Proposal Title:
Collaborative Research: Scientific Visual Analytics: Advancing Theory and Practice through Cognitive Modeling
Rating:
Fair
REVIEW:
In the context of the five review elements, please evaluate the strengths and weaknesses of the proposal with respect to intellectual merit.
The PIs propose to improve HCI & visual design by building cognitive models that will guide the this process, perhaps even allowing the evaluation of interaction techniques in simulation. The proposal is compelling and well written.
Intellectual merit:
I was not convinced that this research would substantially accelerate the progress of science.
No doubt that advances in science involve gaining insights from large quantities of complex data. In this process there is a balance between (pre-processing) analysis done by statistical modeling and analysis done by human looking at visualization. In some data & pattern analysis tasks is it better to rely relatively more on statistical modeling, and just somewhat on human inspection; in others, human inspection may play a larger role. I would have liked to see more attention on this issue. On what tasks does the balance swing one way or the other? What characterizes tasks in which human inspection should play a larger role? What dimensions / subtasks of large problems are best shown to humans? What statistical modeling pre-processing best lends itself to successful human inspection? What does cognitive modeling say about these questions?
In previous work on genomics, what proportion of discoveries were made possible by HCI & visualization versus the proportion made possible by statistical modeling? When I look at literature in genomics, the great majority seems to be about statistical modeling.
I remain unconvinced that proposed cognitive modeling will really be relevant to improvements in visualization. For example in Section a.7, results show that there exists a relation between visualization and cognition? But this seems obvious. It would be more useful to say how cognitive measurements indicate how to improve creation of visualizations. Does cognitive theory provide just a "better / worse" signal, or does it actually suggest specific improvements to visualization methodology? I would have liked more preliminary evidence that cognitive modeling is a fruitful approach, with more convincing evaluation of benefit.
There is a brief mention of RISD, but I would need more convincing to understand how precisely RISD will contribute to the science.
In the context of the five review elements, please
evaluate the strengths and weaknesses of the proposal with respect to broader impacts.
I am not convinced by this proposal that improvements in cognitive modeling of visualization will have a very broad impact on visualization, or that any related improvements in visualization with have very broad impact on science.
Almost nothing in the proposal about under-represented groups.
Only standard academic dissemination and education plans.
Please evaluate the strengths and
weaknesses of the proposal with respect to any additional solicitation-specific review criteria, if
applicable
Research contribution scale seems a bit too narrow for an Expeditions. It really focusses on visualization, with relatively weak ties to scientific impact. Adding yet more close interactions with scientists, or adding more integration with statistical modeling would help.
The educational plans are extremely weak: Basically just working with existing student bodies. Creating new courses and releasing software is not enough for a grant of this size.
Summary Statement
Strengths: + Interesting and bold to consider the ability to evaluate visualization through simulation within a cognitive model.
Weaknesses: - Unconvinced that cognitive modeling has shown some promise to improve visualization. Would like to see more preliminary work before making an award of this size. - Would like to see more discussion of balance, strengths and weaknesses between (a) analysis done by human inspection of data, (b) analysis done by statistical modeling. - Only vanilla outreach and education plans.
Review #4
Proposal Number:
1317515
NSF Program:
Experimental Expeditions
Principal Investigator:
Laidlaw, David H
Proposal Title:
Collaborative Research: Scientific Visual Analytics: Advancing Theory and Practice through Cognitive Modeling
Rating:
Good
REVIEW:
In the context of the five review elements, please evaluate the strengths and weaknesses of the proposal with respect to intellectual merit.
The main topic of the proposed project is to develop a series of human-computer interaction software that would aid researchers in neuroscience in particular brain science and genomics to analyze their own or existing data. The main focus is on the visualization of large data. It is proposed that the development of the software is made through close monitoring of human reactions on certain types of images, and then use this knowledge in developing software that visualizes data according to human needs. The software is supposed to help researchers to better not just understand the data but also incite new questions and new research developments.
Strengths:
The topic of the project is very interesting and if successful will be useful to a wide range of scientist. The approach to include multi disciplinary team of scientists to consider all aspects of the program is very good.
Weaknesses:
Although the proposed project is very interesting and would promote science in a verity of ways, it is not clear to me what is the novelty in the project. Most of the computer science tools that will be used seem to be those that have already been developed, except now, they will be applied to a much larger data. However, even for this it is not clear *how* it will be developed. For example the proposal says that the team will collect data on user interactions and observations (thorough pupil - following instrumentations) etc. or to collect data about social and work environments, but it is not at all clear *how* will this data be then used. In particular, what will be the initial subjects of the study, what new "computer science" will develop with it? If new algorithms with be developed, what methods will be used. There is a hint in Figure 2 that there sill be graph theoretic analysis, but what type of analysis? Also, the project suggests that a model manager application that receives and parses the system data will be designed, but how this application will be designed, under what premises, it is not clear to me.
Although the members of the team seem strong in their own respective disciplines, I am not sure how well their collaboration is going to develop. From the project description, I could see that the PI communicates with both Stanford and Harvard, but this is not clear for the rest of the team. In particular, there doesn't seem to be a plan how the postdocs and the students would take part in the team and how they will interact between themselves.
Data management plan: it was not clear from the proposal how would the data obtained through experiments with human subjects be (a) collected and (b) disseminated.
In the context of the five review elements, please
evaluate the strengths and weaknesses of the proposal with respect to broader impacts.
The project may have a great impact to the broader society, in particular with the team's plan to use web-based and wiki-based crowd sourcing upon each use of the software and then to use this feedback into strengthening of the software. It seems that there will be a significant number of personnel included in the project. The team plans to teach three new courses in Brown on topics related to the project.
Weaknesses:
Although there are a good number of postdocs and graduate students included in the project, there seems to be no plan for their scientific development. A simple participation in the project would not provide high interdisciplinary training. Also, for a project of this scope, one would expect much larger outreach to the community through other means (media, seminars, public lectures) etc.
Please evaluate the strengths and
weaknesses of the proposal with respect to any additional solicitation-specific review criteria, if
applicable
Value-added of funding the activity as an Expedition
The project is quite interesting and would be valuable to the scientific community, and therefore will provide significant societal benefits. The research is certainly inspired by a quite complex question: how humans visualize data and find patterns in data. From that perspective, the application, if successful, will affect a wider community, not just researchers in brain science and genomics. I am afraid that there are no specifics about the techniques that will be used, what types of cognitive studies and *how* these studies will be translated in algorithms is quite unclear.
Leadership and collaboration plan
The leadership plan seems quite scarce. There are three postdocs, 6-7 graduate students and a similar number of undergraduate students. I could not find a detailed description of the roles of these team members, how would the integrate with the PIs and whether there will be any oversight over the progress of the project portions coming from several different disciplines.
Summary Statement
The proposal focuses on visualization of large data, it is scientifically significant, but it is lacking specifics about the methods that are proposed to be used. The project needs to expand the outreach to the larger community.
Review #5
Proposal Number:
1317515
NSF Program:
Experimental Expeditions
Principal Investigator:
Laidlaw, David H
Proposal Title:
Collaborative Research: Scientific Visual Analytics: Advancing Theory and Practice through Cognitive Modeling
Rating:
Fair
REVIEW:
In the context of the five review elements, please evaluate the strengths and weaknesses of the proposal with respect to intellectual merit.
The primary goal of this proposal is to develop a set of driving software applications for brain and genomics researchers. The main idea is to apply knowledge about human perception and cognition to the interface design.
For many years, visualization researchers have been working on using computer-supported, interactive visual presentations of data to amplify cognition and tons of tools and techniques have been developed. Although the proposed work itself may add some useful tools with functionalities designed specifically for brain and genomics researchers to the existing pool of tools, there is no much novelty in the proposed approach.
In the proposal, the team admits that 'the effort is clearly an ambitious one' and 'this aim is risky'. So why don't they just focus on one application instead of three applications? The proposal doesn't provide enough justification.
The team believes that using higher-level principles of perception and cognition will help improve productivity but fails to explain why or present facts to support this claim. The proposal provides a high-level description of their approach, but it is unclear how they can achieve their goal.
In the context of the five review elements, please
evaluate the strengths and weaknesses of the proposal with respect to broader impacts.
Although some team members have been collaborated on some relevant projects for many years, from their research websites, I don't see any efforts being made to share their research outcomes via collaborative web tools. Without a good platform to disseminate the tools, the potential impact the proposed work can make will be limited and it will only benefit the labs and the institutes that are involved in this project.
Please evaluate the strengths and weaknesses of the proposal with respect to any additional solicitation-specific review criteria, if applicable
Summary Statement
Overall, it is a well-written proposal with a clear goal and the interdisciplinary team has a nice mix of experts in cognitive science, neuroscience, computer science, genomics, computational biology, and visual design. However, I only see a high level picture of the project and, without details, it is unclear how the team is able to achieve their ambitious and risky goal.
Review #6
Proposal Number:
1317515
NSF Program:
Experimental Expeditions
Principal Investigator:
Laidlaw, David H
Proposal Title:
Collaborative Research: Scientific Visual Analytics: Advancing Theory and Practice through Cognitive Modeling
Rating:
Good
REVIEW:
In the context of the five review elements, please evaluate the strengths and weaknesses of the proposal with respect to intellectual merit.
The research will advance knowledge and capability in advanced visualization techniques to help scientists interact with many different modalities, forms and levels of abstraction of biological data. Detailed studies of sensory signals from users interacting with visualizations will be undertaken. The approach builds on and extends methods currently used by the PIs to improve HCI.
The proposal incorporates a plan to evaluate effectiveness of the tools they will build.
The PI and collaborators are top researchers in the field and are highly qualified to carry out the research.
To what extent would the driving application bias the result? Will the tools and finding be applicable mainly to brain science and genomics or more generally applicable to high dimensional data? How will you be able to tell?
Section a.2, Brain Connectivity. How different is the brain connection visualization at different levels of hierarchy from hierarchical views of circuits in CAD?
a.3 analyzing multiple streams of user data and finding data-driven ways of identifying insights - is the assumption that there will be enough users who have an 'aha' moment while being hooked up to sensors? Or will there be user interviews where the user goes back and remembers what they were looking at when they had the insight? It sounds really hard to get enough data.
a.4 section on network visualization - good description of related work, limitations, and what you plan to do about it.
section on knowledge integration - the citation for "existing work" #66 is 13 years old. Hasn't there been progress since then?
Funding experimental work on the pre-frontal cortex of the rat seems out of scope for a CISE grant. Using data generated by others such as the fMRI data is more suitable for the brain circuitry visualization aspect of the proposal. There is substantial funding for domain scientists in biological sciences.
In the context of the five review elements, please
evaluate the strengths and weaknesses of the proposal with respect to broader impacts.
The team expects the work to benefit the application communities targeted by the demonstration projects and the HCI community in general. They expect to inspire their grad students and postdoc and to contribute to course material. The visualization algorithms and frameworks developed will target two demonstration applications - brain connectivity research and connectity analysis in genomics.
Overall, the outreach and knowledge transfer aspect of the proposal is not a topic of focus.
Please evaluate the strengths and weaknesses of the proposal with respect to any additional solicitation-specific review criteria, if applicable
Summary Statement
The purpose of the proposal is to advance scientific visualization by studying cognitive HCI topics with particular emphasis in how users interact with the visualization tools. The proposal includes studies of cognitive processes in order to develop a model that can drive the HCI.
Review #7
Proposal Number:
1317515
NSF Program:
Experimental Expeditions
Principal Investigator:
Laidlaw, David H
Proposal Title:
Collaborative Research: Scientific Visual Analytics: Advancing Theory and Practice through Cognitive Modeling
Rating:
Fair
REVIEW:
In the context of the five review elements, please evaluate the strengths and weaknesses of the proposal with respect to intellectual merit.
This research proposal describes human-computer interaction research that is intended to improve the field by providing a theoretical framework for interactive visualization. The proposal aims to do this by developing a set of tools that allow more powerful development of visualizations in the area of genomics and for brain connectomics. By developing a set of instrumented visualization tools in these domains, the proposal posits that observational analyses of user interactions with the tools coupled with models based upon principles from cognitive science and art will improve visualizations in a transformative way.
A strength of this proposal are that the areas chosen for the driving applications, connectomics and genomics, are important areas in which improved, transformational visualization tools can have significant impact on the underlying science. The collaboration PIs on the proposal have a history of working together, and this fact is significant since the visualization process will be well grounded in the application domain, and communication channels between visualization scientists and domain experts have already been formed. Related to this, another strength of the proposal is that the analysis of the processes for brain circuitry and genomics seems well thought out, and innovative visualization tools would likely result early on in the process.
The proposal has some weaknesses, however. The foremost among these is that it seeks to develop a theoretical foundation for visualization as a sub-area of HCI without acknowledging, borrowing, or seeming to be aware of the large body of theory that already exists for HCI. In particular, from the opening paragraphs the set up of the problem seems to describe field research to develop models and theory without understand how or why a theory could arise. The PIs might try to employ a Grounded Theory Method (e.g., Glaser & Strauss [1967], Corbin & Straus [2007]) or ethnographic techniques (e.g., Crabtree et al. Ethnography considered harmful. CHI 2009), but some coordinated method of conducting the field observations and trials that will lead to a theoretical description should be employed.
Another weakness is scant attention paid to a critical piece of the visualization workflow, that of the cognitive manager and the predictive models it will manage. Little to no description is provided for what these models will be, and other than their basic function, we don't know anything about they are built. An example is cognitive load, which is non-trivial to measure. Assessing it is tricky, and it must be done correctly. Standardized questionnaires exist that attempt to do this (NASA TLX) but they may not be suitable for building the models that the PIs have in mind or measuring the types of cognitive load that users experience. In any event, we don't know, because nothing is said about it. The same goes for the other measures which are used to build these predictive models.
In the context of the five review elements, please
evaluate the strengths and weaknesses of the proposal with respect to broader impacts.
The potential broad impacts of the driving applications are significant, with high potential impact in science and medicine. In particular, a better understanding of brain circuitry can lead to an improved understanding of brain function, and have an impact on diseases such as PTSD, Parkinson's, and learning disorders. Improving genomics has potential significant impacts in all of medicine. Any generalizations from this work to other visualizations would be useful. The dissemination plan is a good one for insuring wide outreach of the results and resulting software.
Please evaluate the strengths and
weaknesses of the proposal with respect to any additional solicitation-specific review criteria, if
applicable
The research and education goals of the project overall, considering the strengths and weaknesses, do not seem sufficient to justify the investment in an Expedition. Leaving aside the weaknesses described above, the framework for human-centered visualization seems more of an incremental set of steps rather than a transformative plan. The questions that the expedition fundamentally asks are good ones in the driving applications area, and the idea of a "theory of visualization" is a good one. However, the core promise of the proposal is not fulfilled in a convincing way.
The leadership team seems sound and strong. The lead PI has the experience and capacity to manage a project of this complexity. The PIs have collaborated successfully before. The post-doctoral mentoring plan is excellent for both Brown and Stanford. The data management plan seems sound. The pupil tracking device is not explicitly mentioned and should be, for a cost above $32k.
Summary Statement
Proposal has some shortcomings that should be fixed.
Review #8
Proposal Number:
1317515
NSF Program:
Experimental Expeditions
Principal Investigator:
Laidlaw, David H
Proposal Title:
Collaborative Research: Scientific Visual Analytics: Advancing Theory and Practice through Cognitive Modeling
Rating:
Fair
REVIEW:
In the context of the five review elements, please evaluate the strengths and weaknesses of the proposal with respect to intellectual merit.
- Goal of developing cognitive models for predicting the effeectiveness of visualizations is well worth pursuing. It would be a fantastic achievement if it could be done. It would both advance knowledge and understanding in Visualization/HCI and benefit society with better visualizations that allow sciewntists to explore data more effectively.
- Collaboration with top researchers in Visualization/HCI as well as scientistis in brain and genomics research. Strong researchers with excellent track records are more likely to produce breakthrough work.
Weaknesses:
- The proposal doesn't really give any explanation of how the researchers will develop cognitive models for assessing visualizations. It simply states that such models will be developed. The lack of any kind of details or examples of the types of models they expect to develop makes it difficult for me to follow how they expect to produce the results the envision.
What is an example of the type of cognitive model the researchers expect to build? How will the model be evaluated?
The proposal does explain what kinds of data will be collected from users and from teh system, but it does not explain how that data will be used to build models. I understand that because this is a proposal it may not be known how the cognitive models will be developed. Nevertheless I would like to have a more detailed sense of the approach the PIs plan to use to build the models.
- My background is in visualization research. and one issue that comes up in this area is that visualization researchers sometimes end up building visualization tools in the service of another area without advancing visualization goals. This proposal gives high level details of the tools that will be built for Brain and Genomics data visualziation, but never explains how exactly building those tools will advance Visualization research. What visualization questions will building those tools serve to answer? Could other domains suffice?
In the context of the five review elements, please
evaluate the strengths and weaknesses of the proposal with respect to broader impacts.
See above
Please evaluate the strengths and weaknesses of the proposal with respect to any additional solicitation-specific review criteria, if applicable
Weakness:
The proposal does not really provide context with respect to important previous work.
- Researchers have previously studied the effectiveness of visualizations at a perceptual and cognitive level. I was surprised to not see more connection with work on graphical perception:
@article{cleveland1984graphical, title={Graphical perception: Theory, experimentation, and application to the development of graphical methods}, author={Cleveland, William S and McGill, Robert}, journal={Journal of the American Statistical Association}, volume={79}, number={387}, pages={531--554}, year={1984}, publisher={Taylor \& Francis} }
and on graph comprehension:
D. Simkin and R. Hastie. An information-processing analysis of graph perception. Journal of the American Statistical Association, 82(398):454û465, 1987.
P. A. Carpenter and P. Shah. A model of the perceptual and conceptual processes in graph comprehension. . Journal of Experimental Psychology: Applied, 4(2):75û100, 1998.
S. M. Kosslyn. Understanding charts and graphs. Applied Cognitive Psychology, 3(3):185û225, 1989
G. L. Lohse. A cognitive model for understanding graphical perception. Human-Computer Interaction, 8(4):353û388, 1993.
S. Pinker. A theory of graph comprehension, pages 73û126. Lawrence Erlbaum Associates, 1990
Summary Statement