CS295J/Review Response - Revision history

Steven Gomez: /* Full Proposal (SI2-SSI) Review and Response */

2011-10-13T03:24:25Z

Full Proposal (SI2-SSI) Review and Response

← Older revision		Revision as of 03:24, 13 October 2011
Line 185:		Line 185:


	:'''4.''' ''The proposers see this as sort of a prototype of the way software could be developed to support other scientific endeavors; this effect appears to constitute most of the Broader Impact (I wouldn't count benefit to "the entire brain science research community", mentioned in the BI statement, as broader impact).		:'''4.''' ''The proposers see this as sort of a prototype of the way software could be developed to support other scientific endeavors; this effect appears to constitute most of the Broader Impact (I wouldn't count benefit to "the entire brain science research community", mentioned in the BI statement, as broader impact).

			'''In the Broader Impact section, we have described a plan for additional outreach activities, including launching a new website for uploading and sharing brain circuit diagrams created with the visualization tool. This site will be designed for high school-aged viewers, and will be a portal for sharing current neuroscience and cognitive science researchers with youth. ([[User:Steven Gomez\|Steven Gomez]])'''


	:'''5.''' ''One quibble I have in the Management and Coordination Plan is the statement that "The Stanford researchers will visit Brown if face-to-face interactions become necessary." While electronic communication allows collaboration in ways that would not have been possible before, I think it's not a question of whether face-to-face will be necessary, but how often. Fortunately, this appears to have been built into the travel budget.		:'''5.''' ''One quibble I have in the Management and Coordination Plan is the statement that "The Stanford researchers will visit Brown if face-to-face interactions become necessary." While electronic communication allows collaboration in ways that would not have been possible before, I think it's not a question of whether face-to-face will be necessary, but how often. Fortunately, this appears to have been built into the travel budget.

	'''As noted by the reviewer, face to face communication will be necessary and not an 'if' but a 'when'. Researchers at Brown will plan to coordinate visits in California and vice-versa.'''		'''As noted by the reviewer, face to face communication will be necessary and not an 'if' but a 'when'. Researchers at Brown will plan to coordinate visits in California and vice-versa. [I would add more detail in this response: "We have included a preliminary schedule of visits between researchers at Stanford and Brown corresponding to major milestones in the Research Plan. ([[User:Steven Gomez\|Steven Gomez]])]'''

David Laidlaw: /* Full Proposal (SI2-SSI) Review and Response */

2011-10-06T19:28:02Z

Full Proposal (SI2-SSI) Review and Response

← Older revision		Revision as of 19:28, 6 October 2011
Line 199:		Line 199:

	:'''7.''' ''It is also unclear what the team will do to integrate diversity into the proposed activity.		:'''7.''' ''It is also unclear what the team will do to integrate diversity into the proposed activity.


			===Review #1===
			:''
			:''Rating: Excellent
			:''
			:''
			:''REVIEW:
			:''
			:''
			:''What are the broader impacts of the proposed activity?

			:''What is the intellectual merit of the proposed activity?
			:''
			:''SSI proposal.
			:''
			:''A five year project to develop software for visualization and analysis of brain circuitry by working with brain researchers to analyze their cognitive processes that need to be supported. The software is to help link the visualization workflow to a "decisional" workflow, supporting "reasoning and analysis at a high level, rather than just displaying data."
			:''
			:''The development phase will include studies of user interaction at both low level (eye tracking, mouse click logs) and high level (decision making).

			:''What is the intellectual merit of the proposed activity?
			:''
			:''The PI's of this project are proposing to develop, test, and deploy software tools for scientific study of brain circuits. The project will focus on building a cyber infrastructure software system that is intended to improve the speed at which those doing brain research are able to complete their data analysis and it will advance the understanding of human cognition. This project has potential to have impact in the way researchers in the field collect and analyze data by providing an a rich set of cyber infrastructure tools for use in studying and modeling brain circuits. The intellectual merit of the project is very high as it has the potential to greatly reduce the time required to collect and analyze data.
			:''

			:''What are the broader impacts of the proposed activity?
			:''
			:''This project will provide an enhanced set of software tools to researchers in several areas that are conducting research that is related to the human brain. Areas of research that the cyber infrastructure software can be used in included: gene regulation, protein signaling and even crime and terrorism analysis and all have the potential to benefit.

			:''Summary Statement
			:''
			:''While this work is mostly outside my domain of expertise, it looks like a well chosen problem and an interdisciplinary effort. In particular, the team appears to have the broad range of people they would need to pull this off, from experts from the target audience (who suggested the project, and is listed as a co-PI), to cognitive scientists and computer scientists. In addition to the Intellectual and Broader Impact criteria, the specific "additional criteria" listed in the Program Solicitation have been explicitly and (to the extent that I can tell) well addressed. The required supplemental documents address the required points as well.


			''REVIEW:
			:''
			:''What is the intellectual merit of the proposed activity?
			:''
			:''The proposed activity will allow brain scientists to visualize brain functions more easily and in much more detail than in the past.Network models, coupled with sophisticated methods for dimensionality reduction, promise to offer unique insights into the workings of the human brain. Moreover, the proposed visualization tool will go through rigorous evaluation that will allow its constant improvement. The proposers form a very strong group of well-established researchers in brain science and data visualization, offering a unique collaboration.
			:''
			:''What are the broader impacts of the proposed activity?
			:''
			:''Researchers from other disciplines, e.g., who study gene regulation, protein signaling, or perform crime and terrorism analysis, etc., have the potential to be benefited by the proposed software.
			:''
			:''What is the intellectual merit of the proposed activity?
			:''
			:''The goal of this proposal is to develop, test, and deploy interactive visualization tools for scientific study of brain circuits. The tools will help brain researchers view brain circuits at multiple scales and perform sophisticated analysis of research hypotheses. The team members have a decade of experience developing scientific visualization tools for scientific users and consist of experts in cognitive science, neuroscience, computer science, and visual design. They are well qualified to conduct the project.



			===Review #2===
			:''
			:''Rating: Excellent
			:''
			:''
			:''REVIEW:
			:''
			:''
			:''Summary Statement
			:''
			:''This project is focused on an area of research that spans several disciplines that will be able to utilize the cyber infrastructure software that will be developed. The PI's have a proven record of accomplishment in prior research projects. The project has intellectual merit and will have a broad impact by providing an enhanced set of software tools that will facilitate the efforts of researchers doing work related to studying and modeling the brain.



			===Review #3===
			:''
			:''Rating: Very Good
			:''
			:''
			:''Summary Statement
			:''
			:''This unique collaboration between brain scientists and data visualization scientists promises to offer tremendous benefits to the scientific community. The software that will be developed, will allow researchers to understand the signal pathways in the human brain in more detail than ever before. The software will be analyzed through a rigorous process, using models of cognition.



			===Review #4===
			:''
			:''Rating: Good
			:''
			:''
			:''REVIEW:
			:''
			:''
			:''What are the broader impacts of the proposed activity?
			:''
			:''This is an interdisciplinary project and the target user community is brain scientists. The tools will be made available to the public and are expected to benefit the entire brain science research community as well as other disciplines studying linked types of data. The tools can also be used in classes to help students understand connectivity.
			:''
			:''
			:''Summary Statement
			:''
			:''I can see the proposed work will be valuable for brain scientists to study connectivity and dynamics of neural circuits in intact brain as existing systems all have limitations and cannot satisfy the needs of brain scientists as discussed in the proposal. Other scientific domains may need similar tools.
			:''
			:''This proposal focuses on an interesting problem for which it also provides a novel solution. Therefore, I think this is a quality proposal and worthy of support.

	==Removed Chunks of the Proposal (not included in the response letter)==		==Removed Chunks of the Proposal (not included in the response letter)==

David Laidlaw: /* Unused Parts from Proposal */

2011-10-06T19:27:40Z

Unused Parts from Proposal

@@ Line 572: / Line 572: @@
 :''SYNTHESIS COMMENTS:
 :''
 :''The panel agreed that this was a highly competitive proposal. In particular, the proposal attacks a large but tractable problem, and the team's wide and deep expertise gives the project a high probability of success.

David Laidlaw: /* Removed Chunks of the Proposal (not included in the response letter) */

2011-10-06T19:26:34Z

Removed Chunks of the Proposal (not included in the response letter)

@@ Line 1: / Line 1: @@
 == Pre-proposal (Expeditions) Reviews and Response ==
 :''Proposal Number: 1064261
@@ Line 157: / Line 156: @@
 ==Removed Chunks of the Proposal (not included in the response letter)==

David Laidlaw at 19:25, 6 October 2011

2011-10-06T19:25:09Z

← Older revision		Revision as of 19:25, 6 October 2011
Line 1:		Line 1:
	~~== Full Proposal (SI2-SSI) Review and Response ==~~

	~~Dear reviewers~~

	Thank you for your insightful, constructive, and helpful comments. We particularly appreciated the positive evaluation of the proposal and the potential of our methods for enhancing visual data analysis with regards to brain functioning.

	~~We are overwhelmingly grateful to have received a 'competitive' summary evalution.~~

	~~We have noted and addressed the criticisms below in the context of the reviews received. Our responses and details concerning our revisions are in bold.~~

	~~Regards,~~

	~~David Laidlaw et al.~~


	:'''1.''' ''The panel's discussion focused on the sustainability issue beyond the end of the project support timeframe. While a section of the proposal talks about the Outreach, Education, and Sustainability Plan, the community outreach and sustainability aspects are treated somewhat cursorily. These two issues are closely related: without community support, the software is unlikely to be sustainable in the long run. On the other hand, the proposers appear to be well known in their field, which may enhance community uptake.

	'''As added in section G, the plan to create long-term sustainability now includes more detail. Specifically we will focus on creating a lasting open source community that provides frequent updates to the code centralized by the core research team following a Macro R&D development infrastructure.''' ([[User: Stephen Brawner \| Stephen Brawner]])'''


	:'''2.''' ''The proposal was perhaps over-ambitious; aspects of the evaluation (for example, eye tracking) will create large amounts of data that will require correspondingly intensive data analysis. However, the panel felt that even if the project did not accomplish every detail of the proposal, it would still be highly worthwhile. Similarly, while some aspects of the project might be seen as risky, a certain amount of risk is acceptable in NSF proposals, or even expected. Moreover, any risk is mitigated by the qualifications of the PIs, as exemplified by their excellent track record.


	:'''3.''' ''The proposers have laid out a detailed five year plan. One panelist questioned whether sufficient attention had been paid to issues of sustainability, in particular there was no mention made of plans for software support beyond the end of the five year plan.

	~~'''Addressed by response to comment 1''' ([[User: Stephen Brawner\|Stephen]])~~


	:'''4.''' ''The proposers see this as sort of a prototype of the way software could be developed to support other scientific endeavors; this effect appears to constitute most of the Broader Impact (I wouldn't count benefit to "the entire brain science research community", mentioned in the BI statement, as broader impact).


	:'''5.''' ''One quibble I have in the Management and Coordination Plan is the statement that "The Stanford researchers will visit Brown if face-to-face interactions become necessary." While electronic communication allows collaboration in ways that would not have been possible before, I think it's not a question of whether face-to-face will be necessary, but how often. Fortunately, this appears to have been built into the travel budget.

	~~'''As noted by the reviewer, face to face communication will be necessary and not an 'if' but a 'when'. Researchers at Brown will plan to coordinate visits in California and vice-versa.'''~~


	~~:'''6.''' ''As the proposal is for SI2-SSI, I would like to know more on what the team plans to do to ensure the sustainability of the software and develop open-source community support.~~

	~~'''Addressed by response to comment 1'''~~


	~~:'''7.''' ''It is also unclear what the team will do to integrate diversity into the proposed activity.~~

	== Pre-proposal (Expeditions) Reviews and Response ==		== Pre-proposal (Expeditions) Reviews and Response ==

David Laidlaw: /* Full Proposal (SI2-SSI) Review and Response */

2011-10-06T19:24:35Z

Full Proposal (SI2-SSI) Review and Response

← Older revision		Revision as of 19:24, 6 October 2011
Line 3:		Line 3:
	Dear reviewers		Dear reviewers

	Thank you ~~so much~~ for your insightful, constructive, and helpful comments. We particularly appreciated the positive evaluation of the proposal and the potential of our methods for enhancing visual data analysis with regards to brain functioning.		Thank you for your insightful, constructive, and helpful comments. We particularly appreciated the positive evaluation of the proposal and the potential of our methods for enhancing visual data analysis with regards to brain functioning.

	We are overwhelmingly grateful to have received a 'competitive' summary evalution.		We are overwhelmingly grateful to have received a 'competitive' summary evalution.
Line 40:		Line 40:


	:'''7.''' ''It is also unclear what the team will do to integrate diversity into the proposed activity.		:'''7.''' ''It is also unclear what the team will do to integrate diversity into the proposed activity.


	== Pre-proposal (Expeditions) Reviews and Response ==		== Pre-proposal (Expeditions) Reviews and Response ==

David Laidlaw: /* Removed Chunks of the Proposal (not included in the response letter) */

2011-10-06T19:24:06Z

Removed Chunks of the Proposal (not included in the response letter)

← Older revision		Revision as of 19:24, 6 October 2011
Line 201:		Line 201:


	===Removed Chunks of the Proposal (not included in the response letter)===		==Removed Chunks of the Proposal (not included in the response letter)==

	:''Proposal Information		:''Proposal Information

David Laidlaw: /* Removed Chunks of the Proposal (not included in the response letter) */

2011-10-06T19:23:41Z

Removed Chunks of the Proposal (not included in the response letter)

← Older revision		Revision as of 19:23, 6 October 2011
Line 193:		Line 193:
	'''- Development will be done using a publicly visible source code repository (e.g., Github), so that other scientists and the public may be able to track progress, comment on changes, provide feedback, and even contribute pieces of code to the software. <br />		'''- Development will be done using a publicly visible source code repository (e.g., Github), so that other scientists and the public may be able to track progress, comment on changes, provide feedback, and even contribute pieces of code to the software. <br />
	[ [[User:Nathan Malkin\|Nathan]] ]		[ [[User:Nathan Malkin\|Nathan]] ]

			Summary Statement

			Although the high-level conceptualization of this project is exciting, the way that the project was described in the proposal was massively underspecified. Technical details were lacking and some fundamental aspects of the project's conceptualization in terms of the scientific domain under study were missing. There was no timetable, and no evaluation proposed to see how progress would be measured. The authors should be careful about making high-level claims concerning the possible impact of the proposed without a more carefully constructed argument to back up the claims.




	===Removed Chunks of the Proposal (not included in the response letter)===		===Removed Chunks of the Proposal (not included in the response letter)===

David Laidlaw: /* Unused Parts from Proposal */

2011-10-06T19:21:44Z

Unused Parts from Proposal

← Older revision		Revision as of 19:21, 6 October 2011
Line 516:		Line 516:
	:''		:''

	==Unused Parts from Proposal==		=Unused Parts from Proposal=

David Laidlaw at 19:19, 6 October 2011

2011-10-06T19:19:17Z

← Older revision		Revision as of 19:19, 6 October 2011
Line 80:		Line 80:

	:''		:''
	:'''''6. '''The panel thought that the proposal team could do more work in considering possibilities for broader societal impact and for designing more impactful outreach and education activities that would extend beyond the Brown community.		:'''''6. '''Broader Impacts
			The proposed work would provide research opportunities for faculty, postdocs and students working in the project. A tool with the capabilities described in the proposal would mostly benefit the brain science research community. The tool will be used in two computer science courses at Brown.
			The panel thought that the proposal team could do more work in considering possibilities for broader societal impact and for designing more impactful outreach and education activities that would extend beyond the Brown community.
	:''		:''
	'''We have highlighted some of the broader implications for our project, such as making tools publicly available for other communities of brain scientists, medical researchers, and even instructors to learn about brain connectivity through our visualization initiatives. We have also outlined plans to develop tools for elementary, middle, and high school students as they explore the brain, its anatomy, and its capabilities.''' [ [[User:Clara Kliman-Silver\|Clara Kliman-Silver]] ]		'''We have highlighted some of the broader implications for our project, such as making tools publicly available for other communities of brain scientists, medical researchers, and even instructors to learn about brain connectivity through our visualization initiatives. We have also outlined plans to develop tools for elementary, middle, and high school students as they explore the brain, its anatomy, and its capabilities.''' [ [[User:Clara Kliman-Silver\|Clara Kliman-Silver]] ]