CHS reviews 2016
Panel Summary #1
Proposal Number: 1563597
Panel Summary: Panel Summary
A brief statement of what the proposal is about:
The proposal describes a plan to do computational cognitive modeling to inform the design and usage of two scientific visualization systems.
Intellectual merit:
- Strengths
The panel felt that this is a strong research team and is pleased to see a plan with a strong component of HCI theory development and appreciated how well-written the proposal is.
- Weaknesses
The proposal is too vague in several places, particularly in terms of details about the modeling approach to be taken for the higher-level models. The other key area lacking detail is the linkage from data analysis back to the models.
The proposal fails to acknowledge other modeling work on visual search and eye movements in HCI, and could potentially build on some of this existing work.
- To what extent does the proposed activity suggest and explore creative, original, or potentially transformative concepts?
The research is not especially transformative but strong contributions to HCI theory are important and original.
Broader impacts, including enhancing diversity and integrating research and education:
- Strengths
Both the theoretical issues and the applied domains are worthy research areas.
- Weaknesses
The panel expressed some concern about the evaluation processes here in terms of generality.
Results from prior NSF support (if applicable):
No issues.
Soundness of the collaboration plan (if applicable):
The collaboration plan is somewhat weak, saying little about how the collaboration will actually be managed (e.g., will there be regular meetings, or milestones?).
Soundness of the data management plan:
Adequate.
Soundness of the post-doc mentoring plan (if applicable):
n/a
Additional suggestions:
The panel believes there is substantial upside here and that the identified issues should be addressable in a revision, so would thus strongly encourage the PIs to revise and resubmit.
Panel recommendation:
__ Highly Competitive __ Competitive _x_ Low Competitive __ Not Recommended for Funding by the Panel
Justification, including key strengths and critical weaknesses:
While the panel recognizes the critical need for further development of strong theory in HCI and the research team is well-qualified to do so, the panel felt the proposal was too vague in terms of the details of the higher-level modeling to justify a higher ranking.
The summary was read by the panel, and the panel concurred that the summary accurately reflects the panel discussion.
Panel Recommendation: Low Competitive
Review #1
Proposal Number:
1563597
NSF Program:
Cyber-Human Systems (CHS)
Principal Investigator:
Laidlaw, David H
Proposal Title:
CHS: Medium: Improving Visual Analysis Systems for Brain and Genomics Research Using Predictive Models of Sensemaking
Rating:
Very Good
REVIEW:
In the context of the five review elements, please evaluate the strengths and weaknesses of the proposal with respect to intellectual merit.
The PIs propose a human-computer interaction system that will integrate cognition and task performance modeling to improve the performance of humans interacting with exploratory scientific visualization and analysis interfaces. The research is motivated by and applied to two driving applications: brain network analysis and cancer genome analysis.
Intellectual merit: - Strengths The research will yield novel models of visual analysis performance at three levels of granularity: i) task level; ii) reasoning level; and iii) insight generation level. The research aims to utilize and advance two applications: i) brain science; and ii) cancer genome. The research would lead to the development of guidelines for designing visualization layouts for different user tasks. The research will address the issue of individual behavioral differences in users that affect the exploratory behaviors of humans in analysis.
- Weaknesses The proposal describes the process of formative studies, model design and fit, model evaluation and application for each of the three task levels. It is clear that there is a lot of algorithmic development such as data analysis behind all these tasks. The PIs have not identified the challenges from the algorithmic perspective for these analyses.
In the context of the five review elements, please
evaluate the strengths and weaknesses of the proposal with respect to broader impacts.
This research will impact not only the two applications being studied in this research but also others where visual analysis of data is needed. The PIs propose to leverage the lessons learnt from this research to improve the contents of three courses being taught at Brown.
Weakness: Broader impacts could be improved by providing more concrete plans appropriate for a project of this size.
Please evaluate the strengths and weaknesses of the proposal with respect to any additional solicitation-specific review criteria, if applicable
The proposed development of human computer interaction tools that is informed by cognitive and task performance evaluations at three levels of task granularity can be valuable for visual analysis applications particularly the complex applications such as brain science and genome analysis.
Summary Statement
The PIs propose to develop cognitive predictive modeling tools for humans interacting with exploratory scientific visualization and analysis interfaces. The proposal is well written PIs have a good track record. The proposal could describe the challenges from algorithmic perspective to ensure the success of this project to solve the difficult problem proposed. Broader impacts section could be improved by providing more concrete plans appropriate for a project of this size.
Review #2
Proposal Number:
1563597
NSF Program:
Cyber-Human Systems (CHS)
Principal Investigator:
Laidlaw, David H
Proposal Title:
CHS: Medium: Improving Visual Analysis Systems for Brain and Genomics Research Using Predictive Models of Sensemaking
Rating:
Good
REVIEW:
In the context of the five review elements, please evaluate the strengths and weaknesses of the proposal with respect to intellectual merit.
Goal is to perform human-computer interaction (HCI) research to carry out predictive modeling to understand humans interaction with scientific visualization and analysis interfaces. These models will predict measures of task performance and reasoning and provide a quantitative means for evaluating and ultimately improving the design of web applications. Usage data will be used develop and test predictive models of human behavior. Two scientific applications will be used as test case one which support visual analysis of brain connectivity and the other cancer genomics data. The behavioral modeling will be done where individual operations involved in unit visualization tasks will be predicted; understanding of data corresponding to minutes-long, goal-oriented analysis activity will be classified from interactions and a complete user interaction history in a session of tens of minutes will be used to predict the quality and quantity of insights.
Strengths: The intellectual merit is to develop better understanding of how computers and humans interact at different levels, from perceptual and motor to cognitive reasoning. If successful in developing new insight potentially it could help in improving how humans understand and interact with complex visualization systems and help in automating new interfaces in the future.
Weaknesses In formative study each scientist will be asked to think of a series of gene sets they have looked at recently and will be asked interact with MAGI. This seems to bring a wide range of variability between users û since each case will be unique
Another important weakness I see is the use of rather unrelated visualization interface for the proposed study. What some visualization flow which are common to two applications to develop common predictive model? Who will be responsible to develop new prototypes to test the developed hypothesis?
In the context of the five review elements, please
evaluate the strengths and weaknesses of the proposal with respect to broader impacts.
Broader impact of new insights could come from various other scientific and other visualization/task analysis from other completely unrelated domains.
Please evaluate the strengths and weaknesses of the proposal with respect to any additional solicitation-specific review criteria, if applicable
It meets rest of the solicitation requirements
Summary Statement
In summary - while this proposal addresses an important area of making sense of complex visualization data - the predictive modeling lacks in a coherent approach which ties in the two diverse applications that it tries to connect.
Review #3
Proposal Number:
1563597
NSF Program:
Cyber-Human Systems (CHS)
Principal Investigator:
Laidlaw, David H
Proposal Title:
CHS: Medium: Improving Visual Analysis Systems for Brain and Genomics Research Using Predictive Models of Sensemaking
Rating:
Very Good
REVIEW:
In the context of the five review elements, please evaluate the strengths and weaknesses of the proposal with respect to intellectual merit.
The proposed work has many strengths.
The project proposes building theory in HCI.
The project would pursue important modeling work that would build a layered computational model of visual search, reasoning, and sensemaking.
Recent prior work by the PI and his students demonstrate that these researchers have been developing the building blocks necessary for such an integrated model. For example, Guo, Tran, and Laidlaw (2012) articulates the strategic knowledge, in the form of a GOMS model, that would be necessary for the visual analysis of EEG (electroencephalogram, brain map) data.
The project demonstrates a need to develop simulation models that describe and depict how people actually do the tasks, such as by incorporating detailed descriptions of the cognitive strategies that people would use for these processes, and by building GOMS models using CogTool.
The project aims to model individual differences in visual analytic tasks.
The proposed work also has some weaknesses.
Though the proposal demonstrates an appreciation of models that simulate the internal processes and strategies that give rise to the behaviors that can be observed when people do real-world tasks, the models that the project intends to build appear to be just more statistical models that map inputs to outputs. Though the proposal states an appreciation for the work of Card, Moran, and Newell, and of Lohse, the project does not pursue the challenging tasks pursued by these prior researchers but instead the project plan appear to be, like many contemporary research projects: collect sensor data, apply machine learning, map inputs to outputs (which in this case is purported human behavior).
That said, the precise form of the model is not entirely clear.
The modeling of visual processes needs to take eye movements into consideration. There is a decades-old tradition in the study of visual search to ignore the presence and effects of eye movements. This tradition is followed in the PI's prior work (Gramazio, Schloos, and Laidlaw, 2014), which is cited as support of the proposed project, and which cites some of this traditional work, such as that of Wolfe. It is increasingly acknowledged (such as in Findlay and Gilchrist, 2003; Hulleman & Olivers, BBS, forthcoming) that eye movements need to be incorporated into theories and models of visual processes.
There is quite a bit of cognitive modeling of visual search in HCI that the project could build on, but is not cited in the proposal, such as all of the work that appears when typing "visual search in HCI" into Google Scholar.
In the context of the five review elements, please evaluate the strengths and weaknesses of the proposal with respect to broader impacts.
The project aims to build foundational theory in the field of HCI that could be used to predict and improve the usability of a wide range of human-computer systems, greatly benefiting humanity.
Please evaluate the strengths and
weaknesses of the proposal with respect to any additional solicitation-specific review criteria, if
applicable
The Collaboration Plan is adequate but could be a little more specific in how exactly the collaborations and communications would be carried out.
The Data Management Plan is adequate, but the PIs might review the NSF CISE Data Management Guidance at: http://www.nsf.gov/cise/cise_dmp.jsp
Summary Statement
The project aims to build a computational model of human visual analysis that could help to provide foundational theory for the field of HCI. The proposed model does not appear to simulate how people actually do the tasks, which would seem to hinder the ability to evaluate the model with a range of human data (such as eye tracking), and would seem to limit the model's predictive potential.
Review #4
Proposal Number:
1563597
NSF Program:
Cyber-Human Systems (CHS)
Principal Investigator:
Laidlaw, David H
Proposal Title:
CHS: Medium: Improving Visual Analysis Systems for Brain and Genomics Research Using Predictive Models of Sensemaking
Rating:
Very Good
REVIEW:
In the context of the five review elements, please evaluate the strengths and weaknesses of the proposal with respect to intellectual merit.
As a firm believer in the value of computational cognitive modeling in HCI, the proposal has a lot of upside in terms of intellectual merit. GOMS-class analyses of the low-level aspects of user interactions have a long and rich history, and attempting to bridge these with higher-level models of visual reasoning is a laudable goal. Furthermore, the research team seems well-qualified to do so, and has identified a rich testbed.
The only weakness here is that there are a lot of places in the proposal that are light on details. For example, the exact nature of the reasoning model could use more explication; the description needs a little more meat than "it's a two-phase model." In the model evaluation studies, details like simple size and criteria for acceptable goodness of fit would be welcome. (This is OK with the GOMS-class pieces because that's a well-established modeling framework. Your new pieces are not, so they merit more explanation.)
In the context of the five review elements, please evaluate the strengths and weaknesses of the proposal with respect to broader impacts.
Broader impacts are not quite as strong but still good. Impacts for cognitive modeling and HCI are indeed important and could be significant, but not perhaps as broad as one might like.
Further breadth of impact comes only indirectly from hoped-for improvements in the domain applications, but it is not clear how extensively those applications are used and what potential they really have to impact science in their target disciplines. To wit, if MAGI is the dominant tool used by a huge number of researchers in cancer genomics, then improvements there have a better chance of leading to some payoff, but it is not clear whether this is the case.
The proposal says little about diversity and outreach, which is another weakness.
Please evaluate the strengths and weaknesses of the proposal with respect to any additional solicitation-specific review criteria, if applicable
Summary Statement
The proposal describes a research plan to do computational cognitive modeling at multiple levels of abstraction in order to support improvements to the user interfaces of two specific scientific visualization software packages. This is relatively low-risk research in that computational modeling has been successful in this kind of endeavor in the past, but novel in that it has not been applied in this way to visual reasoning domains. Overall the proposal strikes a good balance between tractability and innovation, and consists of a strong research team with appropriate skills.