CHS reviews
Context Statement
In January 2015, the IIS Division received 667 proposals totaling 601 projects for review in the fiscal year 2015 Medium budget class size for the "Information and Intelligent Systems(IIS):Core Programs," solicitation (NSF 14-596) Additional information about the solicitation can be found at http://www.nsf.gov/pubs/2014/nsf14596/nsf14596.htm
Proposals submitted to the IIS Division in response to the "Information and Intelligent Systems (IIS): Core Programs," solicitation are reviewed in panels to permit comparison of related proposals. In some cases, the Division also seeks the advice of several independent ad hoc reviewers for proposals to complement the evaluations provided by the panel review. In rare cases, proposals will only receive ad hoc review, for instance, if the topic doesn't fit within a scheduled panel.
Merit review is a critical component of the National Science Foundation's decision-making process for funding research and education projects. Through the use of rigorous, competitive merit review, NSF maintains high standards of excellence and accountability. It enables investments in projects that couple the best ideas from the most capable researchers and educators, with the advancement of discovery and learning and the enrichment of the science and engineering resources. The merit review criteria are:
1. What is the intellectual merit of the proposed activity? 2. What are the broader impacts of the proposed activity?
Additional information on NSF's merit review criteria can be found at http://www.nsf.gov/bfa/dias/policy/meritreview/; http://www.nsf.gov/pubs/policydocs/pappguide/nsf14001/nsf14_1.pdf (Grant Proposal Guide - Chapter III)
Please refer to the Reviews section above for copies of the reviews and a panel summary, if applicable (on Fastlane at: https://www.fastlane.nsf.gov/jsp/homepage/proposals.jsp). In reading them, please keep in mind that reviewers are addressing their comments primarily to the NSF, not necessarily to you. Remarks are sometimes made without giving detailed references or providing specific suggestions for improvement, although reviewers are encouraged to provide such helpful information.
Decisions about particular proposals are often very difficult and factors other than reviewer comments and ratings enter into the decision making process. Maintaining appropriate balance among subfields, the availability of other funds, and the total amount of funds available to the program for new and renewal proposals, and general Foundation policies are also important decision factors.
We encourage revised and resubmitted proposals that substantially address reviewer comments. Investigators are welcome to seek the advice of the Program Director before resubmissions are prepared. In addition, investigators should be aware that the Foundation will treat the revised proposal as a new proposal that will be subject to the standard review procedures. Information about reconsideration of declined proposals is found in NSF's Grant Proposal Guide (Chapter IV), which should be available at your institution, usually at the office that submitted your proposal or on the Web at: http://www.nsf.gov/pubs/policydocs/pappguide/nsf14001/nsf14_1.pdf
Please note that current IIS program descriptions, proposal submission deadlines, and other information items can be found on the WWW at http://www.nsf.gov/div/index.jsp?org=IIS. We also encourage examination of the NSF information at www.nsf.gov for announcements of new NSF-wide funding opportunities and other items of interest to the research community.
Panel Summary #1
Proposal Number: 1524615
Panel Summary: Panel Summary
A brief statement of what the proposal is about:
This proposal is about developing predictive modeling of humans interacting with exploratory scientific visualization and analysis interfaces by taking 3 types of actions into account: unit task (basic tasks at keystroke level), reasoning tasks (10 min long and deals with hypothesis exploration) and insight formation.
Intellectual merit: - Strengths
--PI has a lot of prior work on this space and builds their proposed work based on existing feasibility studies. --Goes beyond the traditional model of KLM analysis and adds new dimensions which is the right direction for this line of research. --Brain connectivity and cancer genomic (BraiNet and MAGI), the intended domains of analysis are both important and relevant.
- Weaknesses
--While the proposed work is important, the proposal appears broad and vague in some areas. --The levels on sensemaking tasks and insight characterization are less well-defined and should be supplemented with concrete examples and metrics (e.g., cost of cognitive operations). --There are also concerns on how the three levels of modeling would be integrated and be used in conjunction. --More details on the user study are desired. For example, details on the length of a user study, number of experiments should have been discussed. --The higher level tasks and how it will be codified were not clear.
- To what extent does the proposed activity suggest and explore creative, original, or potentially transformative concepts?
Predictive modeling of humans interacting with exploratory scientific visualization and analysis interfaces. It could have potential impact on refined visual parameters, refined layout and interaction designs supporting smoother workflow and new interface components.
Broader impacts, including enhancing diversity and integrating research and education:
- Strengths
--The use of cognitive modeling techniques to examine visual analytics system is novel and could be potentially transformative to HCI and interface designs. -- Analytics on brain and cancer genome analysis are important medical areas.
- Weaknesses
none were identified.
Results from prior NSF support (if applicable):
Has prior funding from NSF and publications.
Soundness of the collaboration plan (if applicable):
N/A
Soundness of the data management plan:
Reasonable.
Soundness of the post-doc mentoring plan (if applicable):
N/A
Additional suggestions:
Panel recommendation:
__ Highly Competitive
__ Competitive
__X__ Low Competitive
__ Not Recommended for Funding by the Panel
Justification, including key strengths and critical weaknesses:
While the overall idea is exciting and we encourage the PI to continue to pursue this area, this proposal lacks specificity and concrete examples to realize the idea proposed.
The summary was read by the panel, and the panel concurred that the summary accurately reflects the panel discussion.
Panel Recommendation: Low Competitive
Review #1
Proposal Number:
1524615
NSF Program:
Cyber-Human Systems (CHS)
Principal Investigator:
Laidlaw, David H
Proposal Title:
CHS: Small: Cognitive and Perceptual Modeling for Exploratory Interactive Visual Analysis
Rating:
Multiple Rating: (Very Good/Good)
REVIEW:
In the context of the five review elements, please evaluate the strengths and weaknesses of the proposal with respect to intellectual merit.
- Strengths The work aims to create novel models that at the very least will be able to predict task completion times for visual analysis tasks. At its most complicated, the authors aim to develop models that will model intermediate reasoning phases and hypothesis generation or insight formation.
The proposer already has a modeling tool for low-level interactions for visual search tasks using KLM model (Tome project). The current work will build on this tool.
The work is leveraging different types of theories to increase the dimensions of the modeling tool. For example, when modeling the sense-making process of a user during the reasoning phase, the authors will investigate multiple frameworks, since different frameworks might capture different aspects and semantics of a users action. The proposal will use the information-seeking framework by Pirollie et al., as well as interaction taxonomy by Gotz and Zhou.
The work is clearly divided into the three levels of tasks/ interaction û the unit task, the reasoning level, and the hypothesis generation phase. For each of these levels, the proposal identifies how data is going to be collected through formative user studies. What aspects of the user actions is going to be used to create a model, and how the model is going to be evaluated through the same initial data, as well as additional studies.
The proposal does a good job acknowledging the risks associated with each phase of the work. It has very briefly identified what remedial actions can be taken if the model does not perform well.
The authors have developed initial software prototypes for the two scientific visual analysis applications û BraiNet and MAGI, which will be the two applications from which user data will be collected. Moreover, the proposer has experience in conducting user studies similar to the ones proposed here.
The results of the work can result in models that other designers of visual analytics can use for their own purpose. It will also allow designers to predict task completion times.
The work also aims to identify how individual differences affect the kind of strategy that is used to understand a task and incorporate that into the model. Therefore, if future UIs can ascertain the behavior type of their user base, they would be able to have different options that provide information/ data in different ways?
- Weaknesses It was not clear until the very end, how data on insight generation is recorded (an earlier explanation of the recording process would have been helpful). However, for the two critical stages of data exploration û the reasoning process û the idea is video record user study sessions and then replay the video to the user and ask them what reasoning where they performing. Similarly, for insight generation, it is upto the user to verbalize when they have had an insight through a think aloud study. When reflecting back at the video, it is possible that the study participant tries to satisfy the interviewer by 'making up' a reason. Similarly, what is meant by insight might be different people, which might affect when they verbalize their insight. It would have been useful for proposal to first define what the proposer means by these two concepts in the context of the types of tasks possible in the BraiNet and MAGI applications. Also, it appears that the proposer has experience in performing these types of studies, which have been published. Therefore, it would have been useful if proposer could explain why the subjective nature of what constitute reasoning or insights is not going to be a problem for the work.
Most of the data collection is going to be through user studies and the qualitative analysis of these studies. The goal is to use statistical analysis and regression modeling on the data collected. Given how different users can be and that the work is likely going to get a variations in the actions performed/ reasoning done, it is possible that quite a few studies need to be performed. Further, all the data from the study has to be qualitatively analyzed û a significant amount of work. It would have been useful to understand the length of a user study that the authors envision, and the number of experiments that they think need to be performed.
The model depends heavily on KLM to identify the operators to be used and thereby the task completion times. The authors do not delve into any details about how the cost of cognitive operations is going to be calculated, and how these operations are going to be incorporated into the KLM model.
Also, KLM assumes sequential task operations. When analyzing visualizations it is possible that some of the subtasks can be performed in parallel, would the model then still work? How would it change?
In the context of the five review elements, please
evaluate the strengths and weaknesses of the proposal with respect to broader impacts.
- Strengths The work is going to produce models that are explicitly geared towards visual analytics tasks. This will further the field and can help future visualization designers in creating their UI.
The work is going to be evaluated in the domain of brain network and cancer genome analysis û significant improvements on how analytics is performed in this domain can help advances in these fields, which are important medical areas.
The PI is going to incorporate concepts and materials into three different courseworks, one of which is an interdisciplinary class.
Please evaluate the strengths and weaknesses of the proposal with respect to any additional solicitation-specific review criteria, if applicable
Summary Statement
The proposal aims to advance the predictive modeling of humans interacting with (large scale) data visualization by taking into account 3 types of action: unit task (basic tasks at keystroke level), reasoning tasks (10 min long and deals with hypothesis exploration) and third û insight formation, which involves a combination of the multiple 2nd level tasks. They will perform formative user studies to collect the data and create a model, which will then be tested on the (formative study) as well as a separate set of studies. MAGI and BraiNet will be used as domains of analysis.
Review #2
Proposal Number:
1524615
NSF Program:
Cyber-Human Systems (CHS)
Principal Investigator:
Laidlaw, David H
Proposal Title:
CHS: Small: Cognitive and Perceptual Modeling for Exploratory Interactive Visual Analysis
Rating:
Fair
REVIEW:
In the context of the five review elements, please evaluate the strengths and weaknesses of the proposal with respect to intellectual merit.
STRENGTHS:
--The proposal has the potential to advance our understanding of how humans and computers interact at different levels.
--Moreover, the understanding could be computationally modeled so that other scientists could expand it.
--Computational model could also provide automation of the development and evaluation of effective techniques potentially getting us closer to the understanding of psychology of human problem solving.
--The idea is novel and certainly has the potential to advance current-state-of-the-art
--Moving beyond analysis keystrokes and going into quantitative reasoning and unit testing is refreshing.
WEAKNESSES:
--I wish the PI would be provide more concrete examples of reasoning and unit testing and how they would be evaluated.
--Not having adequate examples of the reasoning and unit testing cases makes it difficult to understand parts of the proposal.
In the context of the five review elements, please evaluate the strengths and weaknesses of the proposal with respect to broader impacts.
STENGTHS:
--This proposal may have the potential to help scientists from other disciplines to advance their analysis agendas more efficiently with the support of new knowledge generating visual analysis interactive software.
Please evaluate the strengths and weaknesses of the proposal with respect to any additional solicitation-specific review criteria, if applicable
Summary Statement
This proposal is about predictive modeling of humans interacting with exploratory scientific visualization and analysis interfaces. The big picture that the PI painted in this proposal is exciting. However, it was difficult following through the proposal as it became very specific and did not provide specific examples of reasoning and unit testing. It was not clear how the PI would go beyond qualitative and subject testing and provide concrete metrics for evaluations.
Review #3
Proposal Number:
1524615
NSF Program:
Cyber-Human Systems (CHS)
Principal Investigator:
Laidlaw, David H
Proposal Title:
CHS: Small: Cognitive and Perceptual Modeling for Exploratory Interactive Visual Analysis
Rating:
Good
REVIEW:
In the context of the five review elements, please evaluate the strengths and weaknesses of the proposal with respect to intellectual merit.
The proposed work seeks to apply and extend cognitive modeling techniques to better understand how users use interactive visualizations to solve complex tasks in a scientific discovery setting. The PI suggests that instead of using a single-level of modeling (as in the keystroke Level Model, KLM), in visual analysis systems and tasks, there needs to be three different levels of cognitive tasks -- unit level, task level, and insight level. The PI then proposes modeling techniques for these three different levels of tasks. The proposed work is interesting and novel in the following ways:
+ the characterization of the three levels is consistent with the current understanding of visual analytics tasks. By separating the three levels, the proposed work has a higher chance to accurately model the user's behavior (than using the low level KLM). + the PI has conducted feasibility studies for parts of the proposed work, thus demonstrating that the use of a cognitive modeling approach can lead to possible performance prediction by analyzing the user's interactions. + the intended domains of analysis of brain connectivity and cancer genomic are both important and relevant application areas.
While the proposal has many interesting aspects, there are also some possible deficiencies:
- The proposed work is broad and vague in some areas. In the task-level modeling, the PI successfully restricted the task space by focusing on graph visualizations specifically. However, in the other two levels, the definition of sensemaking tasks and insight characterizations are less well defined. Without a similar narrowing of the space, the modeling will be correspondingly more difficult. - Along the same line, the incorporation of individual differences, while relevant, expands the complexity of the proposed work significantly. While the PI is aware some recent findings in the visualization community pertaining to individual differences, the PI has not proposed how these measures will affect the modeling process or be integrated into the models. - Finally, perhaps also due to the broad scope of the proposed work, the PI does not adequately integrate the three levels of modeling into a cohesive process. As it stands, it seems that any of the three can be used to study visual analytics systems, but it appears that the PI had intended for the three to be used in conjunction as opposed to independently.
In the context of the five review elements, please evaluate the strengths and weaknesses of the proposal with respect to broader impacts.
The proposed work has potentially broad impacts to both the scientific domain as well as visual analytics research. In the context of the scientific domains, the proposed research can lead to more efficient tools for the domain scientists. In terms of visual analytics research, the use of cognitive modeling techniques to study visual analytics system is novel and potentially transformative in the same way that GOMS and KLM have been a key component to HCI and interface designs.
Please evaluate the strengths and weaknesses of the proposal with respect to any additional solicitation-specific review criteria, if applicable
Summary Statement
This is am ambitious proposal that seeks to examine multiple aspects of visual analytics interface and system design simultaneously. While the scope and the goal are greatly appreciated, the proposal itself is vague and lacking in detail in some areas. Part of the issue is perhaps due to the fact that there have not been clear taxonomies for either sensemaking level or insight-level modeling the same way that there is a taxonomy for unit-level graph analysis tasks. Without these taxonomies, it is difficult to understand how the models can be concrete and can be used in a general way.
Also relating to the broad scope of the proposal, the PI further suggests the investigation of both individual differences and perceptual modeling into developing quantitative, predictive models for the three levels of cognitive activities. Adding these two dimensions increases the complexity of the proposed research effort significantly which in turn muddles the specific research activities and the potential outcomes.
Finally, as noted earlier, while breaking down visual analytics tasks into three levels of cognitive activities is conceptually reasonable and congruent with current visual analytics research, the proposal lacks a way to integrate them into a cohesive model in the end. Without considering the three models in a holistic way, it is difficult to understand how these models can be used in a real-world setting. For example, it is possible that a particular system design is superior in the unit-level task modeling, but fails at the sensemaking level. Should that be the case, it is unclear how the designer should adjust or update the design.