VrlWiki - User contributions [en]

Caroline Ziemkiewicz, Postdoc 2010-12

2012-05-30T17:52:23Z

Caroline Ziemkiewicz: /* Interview Notes */

=Legacy Page=

Postdoc from Fall 2010 - Spring 2012. Worked on cognitive modeling and task analysis projects in scientific visualization. Now working at Aptima in Boston.

==Publications==

Caroline Ziemkiewicz, Steven R. Gomez, and David H. Laidlaw. Analysis Within and Between Graphs: Observed User Strategies in Immunobiology Visualization. In ACM CHI Notes, 2012. ([http://vis.cs.brown.edu/docs/pdf/g/Cziemkiewicz-2012-AGO.pdf.html pdf]) ([http://vis.cs.brown.edu/results/bibtex/Cziemkiewicz-2012-AGO.bib bibtex])

Steven R. Gomez, Radu Jianu, Caroline Ziemkiewicz, Hua Guo, and David H. Laidlaw. Different Strokes for Different Folks: Visual Presentation Design Between Disciplines. In Proceedings of IEEE InfoVis, 2012. ([http://vis.cs.brown.edu/docs/pdf/bib/g/Gomez-2012-DSD.pdf.html pdf]) ([http://vis.cs.brown.edu/results/bibtex/Gomez-2012-DSD.bib bibtex]).

==Files and Data==

Videos and annotations of the observation sessions I ran for this project are under /map/gfx0/data/CarolineZ.

I used two annotation tools for these observations: [http://www.anvil-software.de/ Anvil] (files named *.anvil) and [http://www.lat-mpi.eu/tools/elan/ ELAN] (files named *.eaf). They're both free to download and cross-platform. Of the two, I quite strongly recommend ELAN.

In addition to the digital versions, there are hard copies of all the videos on tape with the lab camera.

===Interview Notes===

Here are the notes I took during the observation sessions or pre/post observation interviews. Warning, they're pretty rough.

* [[Jaime Darce]], Harvard CBDM
* [[Ei Wakamatsu]], Harvard CBDM
* [[Daniela Cipolletta]], Harvard CBDM
* [[Dalia Burzyn]], Harvard CBDM
* [[Eric Fields]], Tufts Neuroscience
** [[Fields Insight Timeline]]

==Contact==

My permanent email is [mailto:cziemki@gmail.com here]. Please feel free to contact me with any questions!

Eric Fields

2012-05-30T17:51:15Z

Caroline Ziemkiewicz: Created page with "These are some notes taken during the Fields observation. I took these by hand on my tablet and used automated handwriting recognition to transcribe them, thus "come up with a..."

These are some notes taken during the Fields observation. I took these by hand on my tablet and used automated handwriting recognition to transcribe them, thus "come up with all sneakers" and so on.

---

Self relevance and emotional valence

People think now positively score themselves than other people

So something about you might be more associated with positive information

At what point in tune does that become relevant

3 313

Something I've done a million tines before

Problems

317 little difference

318 5

Look at stars to back up intuitive stuff from eeg waves

Generating as bunch of spss charts

Train

Train

M400 expected to be stronger in the back, surprising

23 6/7

Higher ask think about what these results mean

Rubbing stats again

Excel being slow... something disappointing?

Ttest with wave lines

Has to redo cute to bug

28 6/7

Rainbow color map ruining this scientist's life

Orioles with computer slowness

I think I'll want to code his loquacious n ss.. interesting

337/8

Silence as precursor to insight... probably an individual thing though.

Or silence Plus interactions

More excel problems

Goes to earlier saved version... didn't look at it closely before

More silence

38 7/8

Thinking about what late positivity means

Next step is to come up with all sneakers in multiple ways

Any of these vary with individual differences, depression scales

All analyses, come up with a narrative that makes sense of it all

Close to what I expected

Interesting insight

Particularly with later positivity, not just a magnifying... but pattern different between the two, mostly for positive

Fields Insight Timeline

2012-05-30T17:48:03Z

Caroline Ziemkiewicz: Created page with "This is a compiled timeline of insights, rated mental effort, and interactions from the Eric Fields observation. The timestamps are odd because the face video was running slig..."

This is a compiled timeline of insights, rated mental effort, and interactions from the Eric Fields observation. The timestamps are odd because the face video was running slightly faster than real time (due to me trying to do extended play on the camera). I had to manually sync the two videos up by comparing the screen visible in the face video with the screen capture. So when you see two timestamps, the first is the screen capture video and the second is the face video. After each five-minute timestamp is a self-reported mental effort rating from 0 to 9. Insights are timestamped using the face video.

'''13:00 / 00:48 - 3'''

--- / 3:26/3:50 - break in video

expview

changeparam

changeparam

changeparam

changeparam

INSIGHT 03:57 - little smaller than predicted, not sure if stat sig

INSIGHT 04:32 - main finding of the study

'''18:00 / 04:47 - 5'''

changeparam

changeparam

spss

notepad

spss

excel

spss

excel

INSIGHT 4:57 - no difference for the neutral phrases, not surprising.

INSIGHT 5:00 - no difference for negative, which is a little surprising.

INSIGHT 8:20 - something sig, what we were hoping for; confirm visual findings

INSIGHT 8:39 - results are good because not a million different interactions being significant. (clean)

INSIGHT 8:47 - effect is stronger in front than back... that's unexpected

'''23:00 / 9:00 - 6.5'''

spss

excel

expview

changeparam

changeparam

INSIGHT 10:45 - something made the effect weaker. hypothesis as to why.

'''28:00 / 12:58 - 6.5'''

expview

change param

change param

change param

excel

spss

excel

spss

expview

change param

change param

change param

change param

change param

change param

change param

change param

change param

INSIGHT 13:15 same basic thing... front/back confusion.

INSIGHT 14:55 something about late positivity.

INSIGHT 15:53 difference in late positivity in emotions conditions (?) has a hypothesis, finds this interesting. sounds like a new interpretation of study

'''33:00 / 16:49 - 7.5'''

change param

excel

spss

excel

excel

expview

change param

excel

INSIGHT 20:25 - effect is complicated because it differs across the scalp.

'''38:00 / 20:52 - 7.5'''

he's not actually staring at all, he's constantly changing parameters... even in the excel sheet he tends to scroll or click around when he's thinking hard.

okay, so ~4 minutes EP to 5 minutes real world.

mental effort as self rated steadily increases... even when he gets mired in SPSS errors

lessons learned: need finer grained effort rating. post-hoc might help.

stops interaction while thinking hard. what does that mean for this kind of study?

meaningless interactions are common even tho his system isn't interactive. scroll, stare.

got used to the mental effort rating after 3 iterations

switching back and forth to answer questions... reminds me of jaime and dalia.

swtiches views when he isn't sure about something.

does rate higher in times of frustration... five-6 looked like a sweet spot.

however, 15:53 ( at 7.5 or whatev) sounds like the meatiest insight so far. sounds like a research question for his followup study...

how do i square stops interaction vs. switching intuition.. gotta check the screen cap.

making the assumption that mental effort is pretty continuous, which is not a good assumption. ultimately fnirs might be a better idea...

actually, here's a thing: whatever the absolute values are, the times of most insights are clearly associated with eric upping his mental effort rating. 1 insight, no change. 3+, it increases.

Dalia Burzyn

2012-05-30T17:45:43Z

Caroline Ziemkiewicz: Created page with "'''Intro''' New data from someone's experiment comparing 2 different mice types with t-reg and depleted t-cells induce a muscle injury to see t-cell effect on muscle regene..."

'''Intro'''

New data from someone's experiment

comparing 2 different mice types with t-reg and depleted t-cells

induce a muscle injury to see t-cell effect on muscle regeneration

gene expression in the muscle

not used to analyzing this sample - unfamiliar domain

did some pre-processing that i skipped recording due to low videotape (similar to jaime)

'''Analysis'''

just exploring, making comparisons

* generates a plot
* selects outliers, looks at table
* highlighting
* looking a lot at the table

? looking for immune-related genes

hypothesis is that they'll be up-regulated

also need to look up gene names

* looking at some more comparison
* leaves genepattern
* finds some older graphs from an excel sheet
* working with excel

? file in excel is the one used to generate multiplot graphs... excel file with all genes named and with expression value for each sample.

multiplot doesn't show gene names, just probe number

to find something specific, she looks at excel

* looking thoughtfully at excel sheet
* searches for another gene
* startled when she sees a weird shape!
* investigates further by looking at genes in the sidebar table
* replots, replots again... trying several different plots
* looks at the email where she received the data. produces some kind of quality chart, but she doesn't know how to read it.
* finds another weird shape, scans genes in the table.

this file has old and new samples together

some old samples were bad quality

the weird shape is bad data

* wants to split up old and new samples instead of averaging them

.. i run out of tape, there's a gap while i switch it out

more data problems! the + and - in a sample name are getting cut out in a pull-down menu, and that's the most important information in the sample name

* asks daniela for help, they go back and forth
* tries again

at this point, would ask scott or the student who generated the data for help

? (what does bad-quality data mean?)

have tissue or cells, extract RNA, submit it to a company that does the microarray assay, and they send back data

when data is sent outside, who knows what happens to it?

problems can be contamination, broken molecules, RNA destruction

or processing issue at outside company

can mean issues with a whole batch of data

Daniela Cipolletta

2012-05-30T17:44:12Z

Caroline Ziemkiewicz:

'''Intro'''

This is new data that she hasn't looked at before, part of her current research project

Comparing t-cells from different tissue (adipose tissue)

Comparing in a different mouse strain (from earlier experiments in this project

'''Analysis'''

* loading data into multiplot
* opening a blank ppt presentation
* names the slide
* waiting on upload
* starts setting up axes of a graph (on ppt), makes two graph setups, draws more labels
* getting her slide set up while she waits on upload
* has a side-by-side multiplot view
* loads up each graph
* turns graphs gray (familiar with graph design?)
* selects some outliers, looks at sidebar table
* changes graph 1, selects outliers again
* saves the result
* seems frustrated?
* generating and saving graphs frequently

? she's highlighting with a previously discovered signature that she doesn't expect to be present here

* adding lots of highlights
* propagates highlights to view 2
* looking at outliers
* removing genes based on some criteria
* highlighting several specific genes

? highlighting the transcription factors - hallmark genes of this (?)

variance between knockout and wild type mice

* generates a volcano plot (p-values)
* propagates highlight, looks intently back and forth between the two
* filters
* looks at significant genes

keeps needing to resize the window to change the settings and get back to a big view

doesn't always use the multiple views in a linked fashion

* nods to herself

? comparing FAT vs. something else

wild type model significantly more represented

strange bias toward wild type

signatures that are either over-represented (red) or under-represented (blue)

daniela reaches conclusion: knockout can move signature in this direction

new information, but in line with previous observations

* nods again

looking at where stuff is... fitting expectations, etc.

* propagates highlight again

mostly seems to be doing manual linking

* goes to get a colleague for discussion
(videos break, sound probably goes slightly off-sync...)
* returns
* clears some highlights

looking for a signature given by jaime. from another platform, so it needs to be translated.

* finally finds the signature and puts it in her graphs
* has trouble seeing what highlights are on due to screen space issues
* puts highlight in view 2 and looks back and forth.

'''Post-analysis discussion'''

mice with transcription factor

fusion with protein allowed genes to behave properly.

looking if cells with these hallmark factors are different

checked first replicate a month ago

this is a follow-up experiment

process steps:

what is the variability?

filter for most solid data

looking from genes to see how they change

previously, some genes don't belong to this, appearing in knockout mice

check if there are still weird genes and find them

next, to make a gene list of differnt experiments between animal models, study to see if they follow in a particular pattern.

explain phenotype in disease?

likes to have two plots to see two different comparisions for same population

see significance as well as behavior

if I don't have 2 I have to go back and forth

4 is even better

sometimes you do multiple comparison - many populations

going back and forth you can forget and lose time

big files, 100 samples, 30 populations

wants to grab multiple highlight files

Daniela Cipolletta

2012-05-30T17:41:25Z

Caroline Ziemkiewicz: Created page with "Intro This is new data that she hasn't looked at before, part of her current research project Comparing t-cells from different tissue (adipose tissue) Comparing in a differen..."

Intro

This is new data that she hasn't looked at before, part of her current research project
Comparing t-cells from different tissue (adipose tissue)
Comparing in a different mouse strain (from earlier experiments in this project

Analysis

- loading data into multiplot
- opening a blank ppt presentation
- names the slide
- waiting on upload
- starts setting up axes of a graph (on ppt), makes two graph setups, draws more labels
- getting her slide set up while she waits on upload
- has a side-by-side multiplot view
- loads up each graph
- turns graphs gray (familiar with graph design?)
- selects some outliers, looks at sidebar table
- changes graph 1, selects outliers again
- saves the result
- seems frustrated?
- generating and saving graphs frequently

? she's highlighting with a previously discovered signature that she doesn't expect to be present here

- adding lots of highlights
- propagates highlights to view 2
- looking at outliers
- removing genes based on some criteria
- highlighting several specific genes

? highlighting the transcription factors - hallmark genes of this (?)
variance between knockout and wild type mice

- generates a volcano plot (p-values)
- propagates highlight, looks intently back and forth between the two
- filters
- looks at significant genes

keeps needing to resize the window to change the settings and get back to a big view
doesn't always use the multiple views in a linked fashion

- nods to herself
? comparing FAT vs. something else
wild type model significantly more represented
strange bias toward wild type
signatures that are either over-represented (red) or under-represented (blue)
daniela reaches conclusion: knockout can move signature in this direction
new information, but in line with previous observations

- nods again
looking at where stuff is... fitting expectations, etc.

- propagates highlight again
mostly seems to be doing manual linking

- goes to get a colleague for discussion
(videos break, sound probably goes slightly off-sync...)
- returns
- clears some highlights

looking for a signature given by jaime. from another platform, so it needs to be translated.

- finally finds the signature and puts it in her graphs
- has trouble seeing what highlights are on due to screen space issues
- puts highlight in view 2 and looks back and forth.

Post-analysis discussion

mice with transcription factor
fusion with protein allowed genes to behave properly.
looking if cells with these hallmark factors are different
checked first replicate a month ago
this is a follow-up experiment

process steps:
what is the variability?
filter for most solid data
looking from genes to see how they change

previously, some genes don't belong to this, appearing in knockout mice
check if there are still weird genes and find them

next, to make a gene list of differnt experiments between animal models, study to see if they follow in a particular pattern.
explain phenotype in disease?

likes to have two plots to see two different comparisions for same population
see significance as well as behavior
if I don't have 2 I have to go back and forth
4 is even better
sometimes you do multiple comparison - many populations
going back and forth you can forget and lose time

big files, 100 samples, 30 populations
wants to grab multiple highlight files

Ei Wakamatsu

2012-05-30T17:40:54Z

Caroline Ziemkiewicz: Created page with "Intro studying co-stimulatory (costim) molecules for t-cells also looking at foxb3 which expresses GFP sort the cells into positive or negative (expression?), analyze gene pa..."

Intro

studying co-stimulatory (costim) molecules for t-cells
also looking at foxb3 which expresses GFP
sort the cells into positive or negative (expression?), analyze gene patterns
5 conditions, 2 time steps, 2 cell populations - lots of groups to compare!
costim vs. cd3

Analysis

ei often refers to previously-generated graphs from a powerpoint presentation on his laptop - sometimes to help explain something to me and sometimes as a guide. i think he's trying to recreate the analysis that led to that paper / presentation... so he is more direct and less exploratory than jaime

- refers to a kind of matrix on the laptop
first need to check data quality using express matrix
has about 30 datasets he's comparing
- uploads data to genepattern, launches express matrix
selects a subset of data to compare
significant loading time
- EM brings up a scatterplot matrix with correlation values (a la iPCA)
bad data (?) shows an s-shaped pattern in the scatterplot
- looks for any sign of the s-shaped pattern
- clicks on several individual scatterplots to see in more detail
shapes don't look bad, so i can progress to the next step
if you do find an s-shape, you need to remove the bad data and normalize again
- refers to his laptop again
there are two time steps in the study; want to see whether these time steps are shared or not
shared means that the same genes behave similarly across the two time points
- looks at scatterplots in multiplot
- determines that genes are not shared between the time conditions
- selects some genes and looks at them in the sidebar table

so that's how analysis with genepattern works, but recently has been using s+ and r instead

- opens S+, looks at a data table to point out his different conditions
much faster than genepattern
- brings up a command line window
- types in commands a bit tentatively - thinking about it
- generates a filtered dataset
is really pleased about how fast it is to remove some unwanted data
- typing more commands
removing genes where max value is below 100

why? to remove noise or non-reproducable effects (seems like same thing jaime was concerned with, re. variance)

- refers to laptop again
to extract genes from several conditions
- looking at a scatterplot on his laptop
cd3 vs. something??
focus on "active" regions (center top and center bottom)
compare just one gene with genepattern is easy... but lots of conditions takes too much time
(somewhere hadley wickham is feeling really vindicated and doesn't know why)
- looking at some saved S+ code
- copies code into the command window
execution is pretty slow
- generates a smaller table
- checks the code again

? now he has two datasets and doesn't know which is the right one
seems to be looking for the line of code that named the new data

- finds the table he's looking for
looking at number of shared genes between each pair of conditions; given time steps, cell type, etc.
which conditions are similar and which have no similarity

after this, i want to determine each co-stimulatory molecule's signature

quite enthusiastic about S+ & R - less visual than GP, but much faster
ed: certainly the fact that gp makes all comparisons serial would make his particular questions pretty unbearable to answer! but would this be helped with a multiview system? hard to say. it's also too many comparisons for useful multiviewing...

Jaime Darce

2012-05-30T17:40:06Z

Caroline Ziemkiewicz: Created page with "Intro using data from elsewhere compares two sets of cells - focus on regulatory t-cells wild type vs. missing one gene (PRDM-1) what's the difference between normal T-regs ..."

Intro

using data from elsewhere
compares two sets of cells
- focus on regulatory t-cells
wild type vs. missing one gene (PRDM-1)

what's the difference between normal T-regs and those that lack the gene?

in field, it's thought that reg cells behave differently based on particular transcription factors

because data is from another lab, it's based on a different platform and needs to be translated somewhat

Analysis 1

- Trying to upload data to genepattern, runs into errors right away
attributed to the difference between the two platforms
- editing a config file by hand
- goes to IT guy for help
some dead air on the recordings
- returns with solution
- goes to GP website
- uploads data to multiplot, chooses to launch visualizer
- Picks some settings and generates a scatterplot

? trying to get the genes that are significantly expressed by a fold change vs. coefficient of variance
3 replicates in dataset
expression values with a lot of variance
looking for consistent and plausible patterns (not noise or unreproducable effects)
...thus, why variance is so important

- highlighting a set of points by selecting some criteria
needs to go to a rather large dialog box to do so
when criteria are changed it doesn't show up immediately; must hit a "Plot" button
jaime sometimes plots intermediate steps anyway
- uses two different colors (red and blue) to highlight different things
- looking for other files
gets frustrated

? not very familiar with this data
highlighted genes are overexpressed in one condition vs. another
different naming conventions between the two datasets

gives up on this analysis and starts a different one.

Analysis 2

- uploading new file
< i like to turn the data upside down, sideways
looking for realness
if you try different plots, different views, and still see something, can be more reassured that these genes are differentially expressed.
do these genes fall within this signature or not?
in the end it's always the same, right, you want to see the diff b/w one population and the other
regular cell vs. modified foxb3
already observed the phenotype. what's the difference b/w cells that are normal vs. modified ones
two different mouse lines as well: NOD vs. pb6 (?)
- brings up a scatterplot with a correlation line
- highlighting red and blue groups
- changing plot to see if outliers are still outliers

? looking to see if outlier-ness is unique to regulatory cells. it is for blue genes, but not for red ones. so blue are more interesting (i wrote "red are more" in my notes... but that makes no sense. check video.)

- looks at volcano plot, which focuses on p-values
- zooming in on the blue points?
- filtering signficantly over-expressed genes

getting a signature for blues
are they also in cells infected by this factor?
another set of data is included now; look at cells infected w/ mt vector vs. a vector containing foxb3.
taking earlier group and looking at it in another dataset.
the differentiation requires more than just one factor.
interactions of factors and association w/ mutation
now i can go back and see what's missing w/o PRDM-1
(which was a factor in analysis 2)
different between populations - how to get to the mutant
you can do this sort of thing for hours!

output? a signature
seeing genes uniquely expressed by T-regs that have foxb3
learning how reg cells function
followup with in-depth biochemical analysis
where can i go next?
association of foxb3 with transcription factors? what level is the effect at?
now do more specific experiments
then do this same analysis again

Caroline Ziemkiewicz, Postdoc 2010-12

2012-05-30T17:39:20Z

Caroline Ziemkiewicz:

=Legacy Page=

Postdoc from Fall 2010 - Spring 2012. Worked on cognitive modeling and task analysis projects in scientific visualization. Now working at Aptima in Boston.

==Publications==

Caroline Ziemkiewicz, Steven R. Gomez, and David H. Laidlaw. Analysis Within and Between Graphs: Observed User Strategies in Immunobiology Visualization. In ACM CHI Notes, 2012. ([http://vis.cs.brown.edu/docs/pdf/g/Cziemkiewicz-2012-AGO.pdf.html pdf]) ([http://vis.cs.brown.edu/results/bibtex/Cziemkiewicz-2012-AGO.bib bibtex])

Steven R. Gomez, Radu Jianu, Caroline Ziemkiewicz, Hua Guo, and David H. Laidlaw. Different Strokes for Different Folks: Visual Presentation Design Between Disciplines. In Proceedings of IEEE InfoVis, 2012. ([http://vis.cs.brown.edu/docs/pdf/bib/g/Gomez-2012-DSD.pdf.html pdf]) ([http://vis.cs.brown.edu/results/bibtex/Gomez-2012-DSD.bib bibtex]).

==Files and Data==

Videos and annotations of the observation sessions I ran for this project are under /map/gfx0/data/CarolineZ.

I used two annotation tools for these observations: [http://www.anvil-software.de/ Anvil] (files named *.anvil) and [http://www.lat-mpi.eu/tools/elan/ ELAN] (files named *.eaf). They're both free to download and cross-platform. Of the two, I quite strongly recommend ELAN.

In addition to the digital versions, there are hard copies of all the videos on tape with the lab camera.

===Interview Notes===

Here are the notes I took during the observation sessions or pre/post observation interviews.

* [[Jaime Darce]], Harvard CBDM
* [[Ei Wakamatsu]], Harvard CBDM
* [[Daniela Cipolletta]], Harvard CBDM
* [[Dalia Burzyn]], Harvard CBDM
* [[Eric Fields]], Tufts Neuroscience
** [[Fields Insight Timeline]]

==Contact==

My permanent email is [mailto:cziemki@gmail.com here]. Please feel free to contact me with any questions!

User:Caroline Ziemkiewicz

2012-05-29T18:25:58Z

Caroline Ziemkiewicz: Replaced content with "Postdoc 2010-12. For more information, see my legacy page."

Postdoc 2010-12. For more information, see my [[Caroline Ziemkiewicz, Postdoc 2010-12|legacy page]].

Caroline Ziemkiewicz, Postdoc 2010-12

2012-05-29T18:25:11Z

Caroline Ziemkiewicz: Created page with "=Legacy Page= Postdoc from Fall 2010 - Spring 2012. Worked on cognitive modeling and task analysis projects in scientific visualization. Now working at Aptima in Boston. ==..."

=Legacy Page=

Postdoc from Fall 2010 - Spring 2012. Worked on cognitive modeling and task analysis projects in scientific visualization. Now working at Aptima in Boston.

==Publications==

Caroline Ziemkiewicz, Steven R. Gomez, and David H. Laidlaw. Analysis Within and Between Graphs: Observed User Strategies in Immunobiology Visualization. In ACM CHI Notes, 2012. ([http://vis.cs.brown.edu/docs/pdf/g/Cziemkiewicz-2012-AGO.pdf.html pdf]) ([http://vis.cs.brown.edu/results/bibtex/Cziemkiewicz-2012-AGO.bib bibtex])

Steven R. Gomez, Radu Jianu, Caroline Ziemkiewicz, Hua Guo, and David H. Laidlaw. Different Strokes for Different Folks: Visual Presentation Design Between Disciplines. In Proceedings of IEEE InfoVis, 2012. ([http://vis.cs.brown.edu/docs/pdf/bib/g/Gomez-2012-DSD.pdf.html pdf]) ([http://vis.cs.brown.edu/results/bibtex/Gomez-2012-DSD.bib bibtex]).

==Files and Data==

Videos and annotations of the observation sessions I ran for this project are under /map/gfx0/data/CarolineZ.

I used two annotation tools for these observations: [http://www.anvil-software.de/ Anvil] (files named *.anvil) and [http://www.lat-mpi.eu/tools/elan/ ELAN] (files named *.eaf). They're both free to download and cross-platform. Of the two, I quite strongly recommend ELAN.

In addition to the digital versions, there are hard copies of all the videos on tape with the lab camera.

==Contact==

My permanent email is [mailto:cziemki@gmail.com here]. Please feel free to contact me with any questions!

Legacy Research

2012-05-29T18:24:25Z

Caroline Ziemkiewicz: Created page with "This page has information on departed researchers and the projects they worked on. If you're leaving, make a page for yourself! ==People== * [[Caroline Ziemkiewicz, Postdoc ..."

This page has information on departed researchers and the projects they worked on. If you're leaving, make a page for yourself!

==People==

* [[Caroline Ziemkiewicz, Postdoc 2010-12]]

User:Caroline Ziemkiewicz

2012-05-29T18:18:08Z

Caroline Ziemkiewicz:

=Legacy Page=

Postdoc from Fall 2010 - Spring 2012. Worked on cognitive modeling and task analysis projects in scientific visualization. Now working at Aptima in Boston.

==Publications==

Caroline Ziemkiewicz, Steven R. Gomez, and David H. Laidlaw. Analysis Within and Between Graphs: Observed User Strategies in Immunobiology Visualization. In ACM CHI Notes, 2012. ([http://vis.cs.brown.edu/docs/pdf/g/Cziemkiewicz-2012-AGO.pdf.html pdf]) ([http://vis.cs.brown.edu/results/bibtex/Cziemkiewicz-2012-AGO.bib bibtex])

Steven R. Gomez, Radu Jianu, Caroline Ziemkiewicz, Hua Guo, and David H. Laidlaw. Different Strokes for Different Folks: Visual Presentation Design Between Disciplines. In Proceedings of IEEE InfoVis, 2012. ([http://vis.cs.brown.edu/docs/pdf/bib/g/Gomez-2012-DSD.pdf.html pdf]) ([http://vis.cs.brown.edu/results/bibtex/Gomez-2012-DSD.bib bibtex]).

==Files and Data==

Videos and annotations of the observation sessions I ran for this project are under /map/gfx0/data/CarolineZ.

I used two annotation tools for these observations: [http://www.anvil-software.de/ Anvil] (files named *.anvil) and [http://www.lat-mpi.eu/tools/elan/ ELAN] (files named *.eaf). They're both free to download and cross-platform. Of the two, I quite strongly recommend ELAN.

In addition to the digital versions, there are hard copies of all the videos on tape with the lab camera.

==Contact==

My permanent email is [mailto:cziemki@gmail.com here]. Please feel free to contact me with any questions!

User:Caroline Ziemkiewicz

2012-05-29T17:20:06Z

Caroline Ziemkiewicz:

=Legacy Page=

Postdoc from Fall 2010 - Spring 2012. Worked on cognitive modeling and task analysis projects in scientific visualization. Now working at Aptima in Boston.

==Files and Data==

Videos and annotations of the observation sessions I ran for this project are under /map/gfx0/data/CarolineZ.

I used two annotation tools for these observations: [http://www.anvil-software.de/ Anvil] (files named *.anvil) and [http://www.lat-mpi.eu/tools/elan/ ELAN] (files named *.eaf). They're both free to download and cross-platform. Of the two, I quite strongly recommend ELAN.

In addition to the digital versions, there are hard copies of all the videos on tape with the lab camera.

==Contact==

My permanent email is [mailto:cziemki@gmail.com here]. Please feel free to contact me with any questions!

User:Caroline Ziemkiewicz

2012-05-29T17:15:11Z

Caroline Ziemkiewicz: /* Legacy Page */

=Legacy Page=

Postdoc from Fall 2010 - Spring 2012. Worked on cognitive modeling and task analysis projects in scientific visualization. Now working at Aptima in Boston.

==Files and Data==

Videos and annotations of the observation sessions I ran for this project are under /map/gfx0/data/CarolineZ. I used two annotation tools for these observations: [http://www.anvil-software.de/ Anvil], which is cross-platform (files named *.anvil), and [http://www.lat-mpi.eu/tools/elan/ Elan], which is for MacOS (files named *.??). Of the two, I quite strongly recommend Elan.

In addition to the digital files, there are hard copies of all the video files on tape with the lab camera.

==Contact==

My permanent email is [mailto:cziemki@gmail.com here]. Please feel free to contact me with any questions!

User:Caroline Ziemkiewicz

2012-05-29T17:12:13Z

Caroline Ziemkiewicz:

=Legacy Page=

Postdoc from Fall 2010 - Spring 2012. Worked on cognitive modeling and task analysis projects in scientific visualization. Now working at Aptima in Boston.

Videos and annotations of the observation sessions I ran for this project are under /gpfs/main/research/graphics/data/CarolineZ. I used two annotation tools for these observations: [http://www.anvil-software.de/ Anvil], which is cross-platform (files named *.anvil), and [http://www.lat-mpi.eu/tools/elan/ Elan], which is for MacOS (files named *.??). Of the two, I quite strongly recommend Elan.

In addition to the digital files, there are hard copies of all the video files on tape with the lab camera.

My permanent email is [mailto:cziemki@gmail.com here]. Please feel free to contact me with any questions!

Plans and Goals

2012-01-24T13:57:49Z

Caroline Ziemkiewicz: /* Caroline */

On this page the members of the [[VRL]] record and refine their goals for the current semester. This is a living document in which [[dhl]] will provide feedback. See the bottom of the page for links to past plans & goals documents.

== Current Schedule ==
Meetings are on Tuesdays. The authoritative list is in dhl's calendar. (updated here 9/24/10)

== Current Plans and Goals (Spring '12) ==

=== [[User:Brad Berg|Brad]] ===



----
=== [[User:Cagatay Demiralp|Çağatay]] ===



----
=== [[User:Caroline Ziemkiewicz|Caroline]] ===

* Get a job!
** Go on interviews, network, improve job talk, etc.
* Prepare a VisWeek submission extending task analysis work
* Help Steve's group on the whiteboard/explanations project; also aim for VisWeek
* Submit TVCG article based on VAST locus of control paper
* Service activities
** VisWeek program committee
** Reviews
** NSF panel?



----

=== David ===



----
=== Eni ===



----
=== [[User:Jadrian Miles|Jadrian]] ===

* [[:Image:Jadrian Miles PhD Proposal original 2009-06-22.pdf‎|Initial thesis proposal, submitted 2009-06-22]]
* [[:Image:Jadrian Miles PhD Proposal rev2 2009-10-15.pdf|Proposal second revision, prepared 2009-10-15]]

{{User:Jadrian Miles/OKRs}}


----
=== [[User:Nathan Malkin|Nathan]] ===

* UTRA application?
* Independent research
*# API for experimental platform to interact with Mechanical Turk
*# Using games from experimental economics (Prisoner's Dilemma, Trust Game, Ultimatum Game), attempt to replicate some of their findings
*# Test the effect of the incentive level on people's behavior
*# Test the effects various manifestations of online identity have on behavior
*#* Attempt to replicate findings with faces increasing other-regarding behavior
*#* Repeat experiment with avatars
*#* Also names and nicknames
*# Test effects of different interface elements
*#* Colors, borders, overall "prettiness", ...
*# Test effects of priming



----

=== Radu ===



----
=== Wenjin ===



----
=== [[User:Steven Gomez|Steve]] ===



----
=== Ryan ===



----
=== [[User:Hua Guo|Hua]] ===



----

== Past Plans and Goals ==
* [[/Summer-Fall 2011|Summer-Fall '11]]
* [[/Spring 2011|Spring '11]]
* [[/Summer-Fall 2010|Summer-Fall '10]]
* [[/Spring 2010|Spring '10]]
* [[/Fall 2009|Fall '09]]
* [[/Summer 2009|Summer '09]]
* [[/Spring 2009|Spring '09]]
* [[/Fall 2008|Fall '08]]
* [http://sites.google.com/a/vis.cs.brown.edu/collaboravis/Home/summer-08-group-goals Summer '08]

[[Category:VRL]]

CS295J/Project Schedule:Caroline

2011-11-22T20:59:41Z

Caroline Ziemkiewicz: /* Material */

== Caroline's Project Schedule ==

'''Nov 1''' -- Interview Eric Fields at Tufts and plan for observation session.

'''Nov 8''' -- Add complete related work to paper outline. Rewrite to reflect pilot-study scope. Set up contacts with 2-3 more EEG researchers.

'''Nov 15''' -- Detailed plan for observation sessions, including task description and methods for insight and cognitive load gathering. Schedule observation sessions.

'''Nov 22''' -- [UPDATED] Schedule session with Eric for the week after Thanksgiving. Have a complete procedure outline for the paper. (Note: project schedule changed due to delay in Eric's research.)

'''Nov 29''' -- [UPDATED] Fill out the paper's outline pieces with fully written text. Iterate on study procedure and finalize details with Eric's group.

'''Dec 6''' -- [UPDATED] Run observation session with Eric and perform primary data analysis. Outline updated results section.

'''Dec 13''' -- Final draft of two-page paper.

== Material ==

[http://vrl.cs.brown.edu/wiki/images/e/e6/Caroline_Outline_11-22.pdf Current Paper Outline - updated 11/22]

[http://www.eecs.tufts.edu/~wfairb01/CTemp_CNS.pdf Example Poster from Tufts Group] - shows types of visual representations used

[http://neurocog.psy.tufts.edu/wiki/doku.php?id=documentation Documentation of Tufts EEG Analysis Software]

[[/Fields_Interview|Eric Fields Interview]] (Rough notes)

== Boards ==

<center>
<gallery widths="180px" heights="250px" caption="Poster sketches and in-class critiques">
Image:caroline_project_sketch.png | Oct 13, 2011: Title and Results Sketch
Image:10-18-sketch.png | Oct 18, 2011: Paper Outline
Image:10-20-sketch.png | Oct 20, 2011: Related Work Outline
Image:Caroline 11-22.jpg | Nov 22, 2011: Updated Paper Outline
</gallery>
</center>

File:Caroline Outline 11-22.pdf

2011-11-22T20:58:35Z

Caroline Ziemkiewicz:

CS295J/Project Schedule:Caroline

2011-11-22T20:58:17Z

Caroline Ziemkiewicz: /* Boards */

== Caroline's Project Schedule ==

'''Nov 1''' -- Interview Eric Fields at Tufts and plan for observation session.

'''Nov 8''' -- Add complete related work to paper outline. Rewrite to reflect pilot-study scope. Set up contacts with 2-3 more EEG researchers.

'''Nov 15''' -- Detailed plan for observation sessions, including task description and methods for insight and cognitive load gathering. Schedule observation sessions.

'''Nov 22''' -- [UPDATED] Schedule session with Eric for the week after Thanksgiving. Have a complete procedure outline for the paper. (Note: project schedule changed due to delay in Eric's research.)

'''Nov 29''' -- [UPDATED] Fill out the paper's outline pieces with fully written text. Iterate on study procedure and finalize details with Eric's group.

'''Dec 6''' -- [UPDATED] Run observation session with Eric and perform primary data analysis. Outline updated results section.

'''Dec 13''' -- Final draft of two-page paper.

== Material ==

[http://vrl.cs.brown.edu/wiki/images/1/10/Caroline_Outline2.pdf Current Paper Outline - updated 11/8]

[http://www.eecs.tufts.edu/~wfairb01/CTemp_CNS.pdf Example Poster from Tufts Group] - shows types of visual representations used

[http://neurocog.psy.tufts.edu/wiki/doku.php?id=documentation Documentation of Tufts EEG Analysis Software]

[[/Fields_Interview|Eric Fields Interview]] (Rough notes)

== Boards ==

<center>
<gallery widths="180px" heights="250px" caption="Poster sketches and in-class critiques">
Image:caroline_project_sketch.png | Oct 13, 2011: Title and Results Sketch
Image:10-18-sketch.png | Oct 18, 2011: Paper Outline
Image:10-20-sketch.png | Oct 20, 2011: Related Work Outline
Image:Caroline 11-22.jpg | Nov 22, 2011: Updated Paper Outline
</gallery>
</center>

CS295J/Project Schedule:Caroline

2011-11-22T20:56:26Z

Caroline Ziemkiewicz: /* Boards */

== Caroline's Project Schedule ==

'''Nov 1''' -- Interview Eric Fields at Tufts and plan for observation session.

'''Nov 8''' -- Add complete related work to paper outline. Rewrite to reflect pilot-study scope. Set up contacts with 2-3 more EEG researchers.

'''Nov 15''' -- Detailed plan for observation sessions, including task description and methods for insight and cognitive load gathering. Schedule observation sessions.

'''Nov 22''' -- [UPDATED] Schedule session with Eric for the week after Thanksgiving. Have a complete procedure outline for the paper. (Note: project schedule changed due to delay in Eric's research.)

'''Nov 29''' -- [UPDATED] Fill out the paper's outline pieces with fully written text. Iterate on study procedure and finalize details with Eric's group.

'''Dec 6''' -- [UPDATED] Run observation session with Eric and perform primary data analysis. Outline updated results section.

'''Dec 13''' -- Final draft of two-page paper.

== Material ==

[http://vrl.cs.brown.edu/wiki/images/1/10/Caroline_Outline2.pdf Current Paper Outline - updated 11/8]

[http://www.eecs.tufts.edu/~wfairb01/CTemp_CNS.pdf Example Poster from Tufts Group] - shows types of visual representations used

[http://neurocog.psy.tufts.edu/wiki/doku.php?id=documentation Documentation of Tufts EEG Analysis Software]

[[/Fields_Interview|Eric Fields Interview]] (Rough notes)

== Boards ==

<center>
<gallery widths="250px" heights="250px" caption="Poster sketches and in-class critiques">
Image:caroline_project_sketch.png | Oct 13, 2011: Title and Results Sketch
Image:10-18-sketch.png | Oct 18, 2011: Paper Outline
Image:10-20-sketch.png | Oct 20, 2011: Related Work Outline
Image:Caroline 11-22.jpg | Nov 22, 2011: Updated Paper Outline
</gallery>
</center>

File:Caroline 11-22.jpg

2011-11-22T20:55:57Z

Caroline Ziemkiewicz:

CS295J

2011-11-22T20:55:39Z

Caroline Ziemkiewicz:

'''Cognition, Human-Computer Interaction, and Visual Analysis'''

In this graduate seminar we will learn about models of human cognition and perception and explore potential implications of the models on how computers and humans can interact effectively when performing scientific analyses. Participants will be responsible for reading assigned materials, taking turns guiding discussions of the readings, and preparing a final paper and presentation. It is recommended that participants have some background in at least one of the areas of study.

We will structure our work around revising a 5-year research proposal that
was submitted for funding. We will base the revisions on comments
from reviewers and on our own developing vision of where the research could go. Some revisions will involve learning more about related work and including that context in the writing. Some revisions will involve identifying risky aspects of the proposal and implementing demonstrations or preliminary results (code, math, date, or otherwise) that help quantify the risk. Some revisions will involve adding new elements to the proposed research.

Each student will write up and orally present their results.

*[[/People.11|People]]
*Notes from Class (weeks go fri-thu) -- link to details about 2009 class at bottom of page
**[[/Week 1.11|Week 1]] 6-8 sep 2011, [[/Theory of Visualization outline|Theory of Visualization outline]]
**[[/Week 2.11|Week 2]] 9-15 sep 2011 [[/Literature class 2.11|Literature to read for week 2]], [[media:Discussion_rankings_week_2.11.jpg|discussion scores from board]]
**[[/Week 3.11|Week 3]] 16-22 sep 2011 [[/Literature class 3.11|Literature to read for week 3]]
**[[/Week 4.11|Week 4]] 23-29 sep 2011, [[/Literature to read for week 4.11|Reading for week 4]] (2/13/09), [[/Project ideas from week 4|Project ideas from week 4]]
**[[/Week 5.11|Week 5]] 30 sep-6 oct 2011 [[/Literature to read for week 5.11|Reading for week 5]] (2/20/09), [[/Wordle for the list of literature titles from the first four weeks|Wordle for the list of literature titles from the first four weeks]]
**[[/Week 6.11|Week 6]] 7-13 oct 2011, [[/Literature to read for week 6.11|Reading for week 6]], [[/Application Critiques.11|Application Critiques]], [[/Experiment results from class 6.11|Experiment results from class 6]]
**[[/Week 7.11|Week 7]] 14-20 oct 2011, [[/Literature to read for week 7.11|Reading for week 7]], [[/Boards from class 7.11|Boards from class 7]] (preliminary results first sentence(s))
**[[/Week 8.11|Week 8]] 21-27 oct 2011, VisWeek in Providence! [[/Literature to read for class 8.11|Reading for class 8]], [[/Proposal reviews from class 8.11|Proposal reviews from class 8]], [[/Boards from class 8.11|Boards from class 8]] (big picture intro stuff)
**[[CS295J/Week 9.11|Week 9]] 28 oct-3 nov 2011, [[/Literature to read for class 9.11|Reading for class 9]]
**[[/Week 10.11|Week 10]] 4-10 nov 2011, [[/Literature to read for class 10.11|Reading for class 10]]
**[[/Week 11.11|Week 11]] 11-17 nov 2011, [[/Literature to read for class 11.11|Reading for class 11]]
**[[/Week 12.11|Week 12]] 18-27 nov 2011, Thanksgiving week!, [[/Literature to read for class 12.11|Reading for class 12]], [[/Boards from class 12.11|Boards from class 12]]
**[[/Week 13.11|Week 13]] 25 nov-1 dec 2011
**[[/Week 14.11|Week 14]] 2-8 dec 2011, [[/Literature to read for class 14.11|Reading for class 14]]

*[[/Assignments.11|Assignments]]

*Project Schedules
**[[/Project Schedule:Steve|Steve]] -- Analyzing and modeling imaging QA workflow
**[[/Project_Schedule:CKS-NM|Clara and Nathan]] -- Memory Modulated Visual Search Patterns as Explored on a Touch Screen Interfaces
**[[/Project Schedule:HG-DT|Hua & Diem]] -- Design & Evaluation of EEG Data Visualization Tool Based on Cognitive Task Analysis (Feel free to comment directly on the google doc)
**[https://docs.google.com/document/d/1jlV9TkCIzV4adXOe1OVpWiOXL9JPMO6T7cgnnNtFwpc/edit?hl=en_US Jenna + Michael] -- Evaluating Cognitive Load in Network Visualization
**[https://docs.google.com/a/brown.edu/document/d/1JxrFXvhubalpjcqkuGwiSZwAvXsIb-QdFetRfwydV7M/edit?hl=en_US Wenjun] -- Handedness perspective on mixed text/graphic data visualization for artists
**[[/Project Schedule:Caroline|Caroline]] -- A Method for Predicting Insight in a Brain Imaging Analysis Task

*'''[[/Final contributions.11|Final contributions]]'''
*'''[[/Research proposal before.11|Research proposal before]]'''
*'''[[/Research proposal after.11|Research proposal after]]'''
*[[/Relevant Journals and Conferences|Relevant Journals and Conferences]]
*[[/Literature|Literature]]
*[[/Review Response|Review Response]]
*[[/Rule Lists.11|Rule Lists]]
*[[/Model elements.11|Model elements]]
*[[/Project Concepts.11|Project Concepts]]

*[[/Archived 2009 Pages|Archived 2009 Pages]]

[[/How Tos|How Tos]] -- [[/Class Members' Pages.11|Class Members' Pages]] -- [http://groups.google.com/group/cs295j Mailing list archive]

[[Category:Courses]]

CS295J/Project Schedule:Caroline

2011-11-22T16:30:06Z

Caroline Ziemkiewicz: /* Caroline's Project Schedule */

== Caroline's Project Schedule ==

'''Nov 1''' -- Interview Eric Fields at Tufts and plan for observation session.

'''Nov 8''' -- Add complete related work to paper outline. Rewrite to reflect pilot-study scope. Set up contacts with 2-3 more EEG researchers.

'''Nov 15''' -- Detailed plan for observation sessions, including task description and methods for insight and cognitive load gathering. Schedule observation sessions.

'''Nov 22''' -- [UPDATED] Schedule session with Eric for the week after Thanksgiving. Have a complete procedure outline for the paper. (Note: project schedule changed due to delay in Eric's research.)

'''Nov 29''' -- [UPDATED] Fill out the paper's outline pieces with fully written text. Iterate on study procedure and finalize details with Eric's group.

'''Dec 6''' -- [UPDATED] Run observation session with Eric and perform primary data analysis. Outline updated results section.

'''Dec 13''' -- Final draft of two-page paper.

== Material ==

[http://vrl.cs.brown.edu/wiki/images/1/10/Caroline_Outline2.pdf Current Paper Outline - updated 11/8]

[http://www.eecs.tufts.edu/~wfairb01/CTemp_CNS.pdf Example Poster from Tufts Group] - shows types of visual representations used

[http://neurocog.psy.tufts.edu/wiki/doku.php?id=documentation Documentation of Tufts EEG Analysis Software]

[[/Fields_Interview|Eric Fields Interview]] (Rough notes)

== Boards ==

<center>
<gallery widths="250px" heights="250px" caption="Poster sketches and in-class critiques">
Image:caroline_project_sketch.png | Oct 13, 2011: Title and Results Sketch
Image:10-18-sketch.png | Oct 18, 2011: Paper Outline
Image:10-20-sketch.png | Oct 20, 2011: Related Work Outline
</gallery>
</center>

User:Steven Gomez/Project:Expression Representation Variability.11

2011-11-17T20:48:37Z

Caroline Ziemkiewicz: /* Codings */

== How do people with different background/training communicate ideas? ==

=== Related ===
[http://research.microsoft.com/en-us/um/people/nath/docs/whiteboardstudy_infovis2011.pdf "Visual Thinking in Action: Visualizations As Used On Whiteboards"], Walny et al., InfoVis 2011

=== Methods ===
First step, get slides from different folks, domains. Some basic background info ("what is the topic?", "how general is the audience? size?", "where presented?"). Ask GSBB, see how many we get.

=== Codings ===
We need to group people based on their fields.

Proposed: Bio-medical sciences, natural sciences, formal sciences (CS, math, engineering), humanities, visual art, social sciences

* After first pass: Things to add?
** Extending diagrams over time
** Equations
** Labels on [x] (where x is an image, diagram, equation, or other text)
** Tables
** Diagram overlaid on images
** Colors (background, highlighting)

==== Slideshows ====
* charts
* trees
* graphs
* flow-charts
* dense versus sparse
* separation versus grouping
* ellipses

* word lists
* words in spatial organizations
* simple diagrammatic constructs
* words in visual constructions
* mixed words and diagrams
* diagrams with labels
* pure diagrams

* focus and context

* words in sentences or paragraphs

==== Live Study ====
* charts
* trees
* graphs
* flow-charts
* dense versus sparse
* separation versus grouping
* ellipses

* word lists
* words in spatial organizations
* simple diagrammatic constructs
* words in visual constructions
* mixed words and diagrams
* diagrams with labels
* pure diagrams

* focus and context

* words in sentences or paragraphs

=== Concepts to explain, teach ===

==== Spatial Concepts ====
* Assembly instructions, or how simple mechanical devices work
** Doorknob
** Ballpoint pen (e.g., how ink rubs on on the papers and the ball is "re-inked" by gravity)
* Maps
** Drawing a map from Brown CIT down to Kennedy Plaza from memory -- what do subjects include?

==== Non-spatial (or, not observable) Concepts ====
* Chemical processes
** Acid + Base = Water + Salt
** Rust/Oxidation

* Astronomical processes
** Greenhouse gas build up
** Albedo -- light absorption/reflection
** How a cell phone calls another cell phone

* Computational processes
** Explaining concurrency, race conditions, deadlocks
** Explaining inductive proofs to non-math people
** Explaining the 'map' function (i.e., for all elements, or numbers, in a set apply some function to get a new set)
** Describe a traffic light "state machine" based on a set of written observations
*** See how they figure out the rules, then how they explain it to someone else

CS295J/Assignments.11

2011-11-16T17:57:22Z

Caroline Ziemkiewicz: /* Assignment 9 (out 15 nov 2011, due 22 nov 2011) */

== Assignment 9 (out 15 nov 2011, due dates through 22 nov 2011) ==

* For Thursday, read and rate two InfoVis papers from the spreadsheet in preparation for in-class discussion. Be ready to give a brief summary of any of the papers you've rated.
* For Tuesday, bring large-format (at least 2x2 pages per real page taped together) printout of one- to two-page paper outline. Be prepared to discuss and augment during class. The research contributions and results are the most important parts to have made progress on. Related work is second most important.
* Bring printout of David's [[Write a paper]] page with an evaluation of each point for your outline. Evaluate on a 1-10 scale, with 10 being the best.

== Assignment 7 (out 20 oct 2011, due dates through 28 oct 2011) ==

*By Tuesday, put both versions of your marked up posters on the wiki
*by Tuesday, create a project schedule and put it in a new page off the course main page. The schedule should include a two-page extended abstract as the final deliverable. That abstract should follow the guidelines in the “write a paper” wiki page and incorporate the feedback that we have been generating in our class critiques. The schedule should also include milestones for each Tuesday remaining in the semester. A milestone is something that we can evaluate concretely. “Do some research” is not a milestone. “Complete video capture of 3 users” is a milestone. "Complete reviewable draft of results section” is a milestone. If you have any question about how to structure milestones, please ask in the Google group.
*By Thursday, review all project schedules and send feedback to their author by e-mail
*on Tuesday at 8:30 AM be at the Rhode Island convention Center 5th floor for the VisWeek keynote presentation. If you do not hear otherwise from me, tell the people in the red vests that the password is "goredsox". I may be able to get a more official way to get in, I'm going to use this as the contingency plan.
*On Friday at 10:30 AM be at the Rhode Island convention Center 5th floor for the VisWeek capstone presentation.

== Assignment 6 (out 13 oct 2011 (ok, a little later...), two due dates through 20 oct 2011) ==

* For Tuesday, bring large-format printout of 1-page paper outline. Be prepared to discuss and augment during class much as we did with titles on Thursday. The research contributions and results are the most important parts to have made progress on. Related work is second most important. See David's [[Write a paper]] page for more thoughts.
* For Tuesday read "a cognitive model for the perception and understanding of graphs" [[http://dl.acm.org/citation.cfm?id=108844.108865]]
* For Thursday, complete related work outline. Should include all the citations you need to support the novelty and significance of research reported in your fictional paper. Be prepared to describe and defend the novelty and signficance in class.

== Assignment 5 (out 6 oct 2011, several due dates through 13 oct 2011) ==

* For Tuesday, bring large-format printout of 1-page paper outline. Be prepared to discuss and augment during class much as we did with titles on Thursday. The research contributions and results are the most important parts to have made progress on. Related work is second most important. See David's [/Write a paper] page for more thoughts.
* For Tuesday read "a cognitive model for the perception and understanding of graphs" [[http://dl.acm.org/citation.cfm?id=108844.108865]]
* For Thursday, complete related work outline. Should include all the citations you need to support the novelty and significance of research reported in your fictional paper. Be prepared to describe and defend the novelty and significance in class.
* might want to look at some of the boards and hand ins from last instantiation of class== Assignment 4 (out 29 sep 2011, several due dates through 6 oct 2011) ==

== Assignment 4 (several due dates through 5 oct 2011) ==

* by Tuesday, complete the draft response letter.
** This is the response that we already started.
** This is a group effort, so please do your fair share! That said, there shouldn't be more than two or three substantial comments per person. There will be more "addressed by response to comment 3.5".
** Label your responses with your name in square brackets.
** If you point out to a changed section in the proposal or pre-proposal where the change isn't clear, please put enough additional detail about what would be in that section so that others would be able to make the change. Put that additional detail in square brackets.
** All response verbiage to left margin.
** All responses clearly indicative of what they are responding to; perhaps most easily done by numbering parts of the review to indicate the start of what the next response addresses.
** All criticisms addressed. Be careful not to address one part of a criticism and miss a second part -- reviewers don't like that!
** All unnecessary parts of the original reviews removed. For example, a set of typos might be listed in a review; they can be addressed without repeating them in the letter by summarizing what was there. Often, a redundant set of comments can be removed, but this is riskier if it is not obviously redundant. It is also ok to paraphrase a part of the critique if you indicate that you are doing so. This is important when there are space issues.
** Any redundancies in the response removed by pointing to the first instance of the repeated answer.
* by Tuesday, read all of the papers we didn't finish in class Thursday. As a general rule, I expect everyone to read every paper. We had the "discussant" concept the first week, but I hadn't meant for it to continue. I'd really like everyone to be able to discuss every paper. Please come to class with your notes about every paper so you can refer back to them -- I don't expect everyone to *remember* everything, just to be able to recall it from notes.
* by Tuesday, add 1/2 (on average) of a paper to our [[CS295J/Literature to read for week 5.11|Literature to read for week 5]]. By that I mean half of you should add one paper; the other half will get a chance next time. First come first serve (but you can negotiate if someone added one and you'd really like to get yours in there sooner).
* by Thursday, read the five new papers, bring your notes on them to class, and bring additional project ideas

== Assignment 3 (out 23 sep 2011, several due dates through 29 sep 2011) ==

* pair up with someone in class and critique each others draft reviewer response
* revise your draft reviewer response to address the critique, which might include issues of clarity, appropriateness of a technique, significance, risk, etc.
* be prepared to have the group critique your revised response in class Tuesday
* by Monday noon add at least three papers to the literature page. The three should be relevant to a reviewer response. They might help establish a point that wasn't clear to the reviewers, provide evidence that some risky part is feasible, or be a basis for some additional preliminary work that could be added to the proposal. Summarize this relevance in the comments for each of the three papers.
* by Tuesday class select one of the three to own and put on the [[CS295J/Literature to read for week 4.11|Literature to read for week 4]]
* by Thursday class be prepared to spend 1-2 minutes reminding us about the paper you own and explaining the relationship to a revised proposal.
* by Thursday class read and be prepared to discuss all of the readings

== Assignment 2 (out 15 sep 2011, several due dates through 22 sep 2011) ==
* find readings that develop some of the areas we read about in the first group of papers
* by Monday noon add new readings to big literature page. If you've got a tentative summary evaluation, go ahead and add it. It's ok to edit folks' summary evaluations, but try to make the result more accurate or precise without losing information.
* by Tuesday class finish with tentative summary evaluations of your new readings and also "own" at least one as-relevant-as-possible reading as yours. Put into [[CS295J/Literature class 3.11|Literature to read for week 3]]
* by Thursday class -- author a summary description, less than 250 words, in the wiki of how the reading you own relates to our project. Be prepared to describe, in two (2!) minutes, how your reading relates to the project. Also be prepared for everyone in class to discuss your description. You may bring notes for yourself, but no slides. The wiki page for your reading will be displayed while you talk.
* by Thursday class -- read and be prepared to discuss all of the other readings (for an appropriate definition of "read")
* by Thursday class read proposal and pre-proposal reviews, identify a significant criticism, and draft a short response. Make it short enough that you can write it on the board during class. Please add to [[CS295J/Review Response|Response to Reviewers]]

== Assignment 1 (out 8 sep 2011, several due dates through 15 sep 2011) ==
* spend 10 hours adding to any part of the wiki you think is relevant
* read both [[CS295J/Research proposal before.11|Research proposal before]] proposals.
* by Monday noon add new readings. If you've got a tentative summary evaluation, go ahead and add it. It's ok to edit folks summary evaluations, but try to make the result more accurate or precise without losing information.
* by Tuesday class finish with tentative summary evaluations of your new readings and also identify at least one as-relevant-as-possible reading as yours. This can be one of the new readings you added or something someone else doesn't own. Put your name on that entry in the reading list as the "owner" so that there are no duplicates.
* by Wednesday 5pm -- select 2 additional relevant readings that are owned and that you will read by class Thursday and be prepared to discuss. Put your name as a "discussant" in the reading list; there should be a max of two discussants per reading.
* by Thursday class -- author a summary description, less than 250 words, in the wiki of how the reading you own relates to our project. Be prepared to describe, in two (2!) minutes, how your reading relates to the project. Also be prepared for everyone in class to discuss your description. You may bring notes for yourself, but no slides. The wiki page for your reading will be displayed while you talk.
* by Thursday class -- read and be prepared to discuss the other two readings you choose.
* Let me know if you have any kind of problems. You should be spending right around 8-10 hours each week -- if that's a problem, let's talk.
* The [[../How Tos|How Tos]] page has some tips. Edit or add as you find others.

CS295J/Project Schedule:Caroline

2011-11-08T19:01:22Z

Caroline Ziemkiewicz: /* Material */

== Caroline's Project Schedule ==

'''Nov 1''' -- Interview Eric Fields at Tufts and plan for observation session.

'''Nov 8''' -- Add complete related work to paper outline. Rewrite to reflect pilot-study scope. Set up contacts with 2-3 more EEG researchers.

'''Nov 15''' -- Detailed plan for observation sessions, including task description and methods for insight and cognitive load gathering. Schedule observation sessions.

'''Nov 22''' -- Gather data from observation sessions. Get data set up for analysis (process videos, transcribe notes, etc.)

'''Nov 29''' -- Same as last week for any remaining observation sessions. Analyze all data and outline actual results and discussion sections.

'''Dec 6''' -- Perform any necessary followup data analysis. Fully written draft of two-page paper. Get feedback on draft.

'''Dec 13''' -- Final draft of two-page paper.

== Material ==

[http://vrl.cs.brown.edu/wiki/images/1/10/Caroline_Outline2.pdf Current Paper Outline - updated 11/8]

[http://www.eecs.tufts.edu/~wfairb01/CTemp_CNS.pdf Example Poster from Tufts Group] - shows types of visual representations used

[http://neurocog.psy.tufts.edu/wiki/doku.php?id=documentation Documentation of Tufts EEG Analysis Software]

[[/Fields_Interview|Eric Fields Interview]] (Rough notes)

== Boards ==

<center>
<gallery widths="250px" heights="250px" caption="Poster sketches and in-class critiques">
Image:caroline_project_sketch.png | Oct 13, 2011: Title and Results Sketch
Image:10-18-sketch.png | Oct 18, 2011: Paper Outline
Image:10-20-sketch.png | Oct 20, 2011: Related Work Outline
</gallery>
</center>

CS295J/Project Schedule:Caroline

2011-11-08T17:58:00Z

Caroline Ziemkiewicz: /* Material */

== Caroline's Project Schedule ==

'''Nov 1''' -- Interview Eric Fields at Tufts and plan for observation session.

'''Nov 8''' -- Add complete related work to paper outline. Rewrite to reflect pilot-study scope. Set up contacts with 2-3 more EEG researchers.

'''Nov 15''' -- Detailed plan for observation sessions, including task description and methods for insight and cognitive load gathering. Schedule observation sessions.

'''Nov 22''' -- Gather data from observation sessions. Get data set up for analysis (process videos, transcribe notes, etc.)

'''Nov 29''' -- Same as last week for any remaining observation sessions. Analyze all data and outline actual results and discussion sections.

'''Dec 6''' -- Perform any necessary followup data analysis. Fully written draft of two-page paper. Get feedback on draft.

'''Dec 13''' -- Final draft of two-page paper.

== Material ==

[http://vrl.cs.brown.edu/wiki/images/b/b2/Caroline_Outline2.pdf Current Paper Outline - updated 11/8]

[http://www.eecs.tufts.edu/~wfairb01/CTemp_CNS.pdf Example Poster from Tufts Group] - shows types of visual representations used

[http://neurocog.psy.tufts.edu/wiki/doku.php?id=documentation Documentation of Tufts EEG Analysis Software]

[[/Fields_Interview|Eric Fields Interview]] (Rough notes)

== Boards ==

<center>
<gallery widths="250px" heights="250px" caption="Poster sketches and in-class critiques">
Image:caroline_project_sketch.png | Oct 13, 2011: Title and Results Sketch
Image:10-18-sketch.png | Oct 18, 2011: Paper Outline
Image:10-20-sketch.png | Oct 20, 2011: Related Work Outline
</gallery>
</center>

File:Caroline Outline2.pdf

2011-11-08T17:57:36Z

Caroline Ziemkiewicz:

File:Caroline Outline.pdf

2011-11-08T17:57:10Z

Caroline Ziemkiewicz: uploaded a new version of "Image:Caroline Outline.pdf"

File:Caroline Outline.pdf

2011-11-08T17:55:54Z

Caroline Ziemkiewicz: uploaded a new version of "Image:Caroline Outline.pdf"

CS295J/Project Schedule:Caroline

2011-11-03T20:35:05Z

Caroline Ziemkiewicz: /* Material */

== Caroline's Project Schedule ==

'''Nov 1''' -- Interview Eric Fields at Tufts and plan for observation session.

'''Nov 8''' -- Add complete related work to paper outline. Rewrite to reflect pilot-study scope. Set up contacts with 2-3 more EEG researchers.

'''Nov 15''' -- Detailed plan for observation sessions, including task description and methods for insight and cognitive load gathering. Schedule observation sessions.

'''Nov 22''' -- Gather data from observation sessions. Get data set up for analysis (process videos, transcribe notes, etc.)

'''Nov 29''' -- Same as last week for any remaining observation sessions. Analyze all data and outline actual results and discussion sections.

'''Dec 6''' -- Perform any necessary followup data analysis. Fully written draft of two-page paper. Get feedback on draft.

'''Dec 13''' -- Final draft of two-page paper.

== Material ==

[http://vrl.cs.brown.edu/wiki/images/b/b2/Caroline_Outline.pdf Current Paper Outline]

[http://www.eecs.tufts.edu/~wfairb01/CTemp_CNS.pdf Example Poster from Tufts Group] - shows types of visual representations used

[http://neurocog.psy.tufts.edu/wiki/doku.php?id=documentation Documentation of Tufts EEG Analysis Software]

[[/Fields_Interview|Eric Fields Interview]] (Rough notes)

== Boards ==

<center>
<gallery widths="250px" heights="250px" caption="Poster sketches and in-class critiques">
Image:caroline_project_sketch.png | Oct 13, 2011: Title and Results Sketch
Image:10-18-sketch.png | Oct 18, 2011: Paper Outline
Image:10-20-sketch.png | Oct 20, 2011: Related Work Outline
</gallery>
</center>

CS295J/Project Schedule:Caroline/Fields Interview

2011-11-03T20:34:50Z

Caroline Ziemkiewicz: New page: What's the data? What do you want out of it? Q's: language processing psycholinguistic questions how social/emotional factors affect language processing ERP is good for temporal sensiti...

What's the data? What do you want out of it?

Q's: language processing
psycholinguistic questions
how social/emotional factors affect language processing

ERP is good for temporal sensitivity - can go as far as millisecond level
interested in which *stages* of processing are being affected

MRI and MEG (magnetic encephalography) - similar to EEG
MEG differences: skull and scalp less opaque to magnetic imaging

combining data sources is difficult
complex math involved
mostly gathered and analyzed separately; Phil Holcomb at Tufts working on project to bring data together, but on a long time scale

they mostly gather EEG data at 5ms intervals; higher-res is possible, but not helpful

-- experiment room --

experiment: sentence comprehension task, varying a single word between positive, negative, and neutral across two conditions (sentence in third person or second person)
do people process the emotional stimulus differently when sentence is about them vs. other person?

output they see during experiment; waves from all the different electrodes (N? about 20) at 1.2s time scale
they'll cut up signal based on events later

waveform is very noisy, so you need a lot of trials - 30 events for each condition
ERP (event related potential) - way to analyze EEG

need to watch output during experiment to prevent bad data
- e.g. artifacts (like blinks)
can tell participants to stop blinking if it keeps happening!

output is a distribution of activation ("smear")
center of distribution is not necessarily the actual location in the brain... e.g. N400 response shows up on the opposite side of the skull from the brain area that generates it
inverse problem - infinite # of brain areas where a signal could have originated
want to localize within % of probability

combining this data with MRI or MEG can add spatial information (but again, hard)
distribution at least tells you if two events are happening in different areas or not

-- data processing and analysis --

lots of script files for data collection and pre-processing

events are coded according to a coding scheme for study events - just an excel file

BDF: bins of interest
select out time slices that represent events
categorize events based on the study conditions and other slices of interest
script generates the bins
(so each bin is a collection of small time-slices of waveform data around an experimental event)

scripts are just straight text, and they all seem to be processed in homebrew software made by a senior researcher in the lab
lots of different processing apps; users just drag and drop script files onto the appropriate executable.

bin generation outputs count of bins, events and .... what's that word supposed to be? wives? elves?
useful for error checking

RawView -> viewer for the raw waveform, also used for error checking
sample events
semi-automatic artifact detection (e.g. blinks)
researcher checks the automatic output for accuracy and changes thresholds as needed.
usually do need to change something
could do it all by hand, but time and consistency are problems
they prefer false negatives to false positives; better to throw out good data than keep bad (similar to CBDM researchers)
artifacts cause really strong signals, so they can throw off the analysis by a lot
checks errors by viewing the raw waveforms
eric seems to have an intuitive sense of what good and bad wave patterns look like

this process takes 10-15 minutes per participant (varies)

another script generates averages across each bin
check data again, and if bins seem small, return to rawview and filter on bin of interest
want to keep the number of events in each bin over 30+ or N gets too small for confidence
so if the number gets small you want to make sure no good data is getting thrown out and adjust the threshold again if you can

CalNormalize -> calibrate waveforms by a wave of known characteristics, then scale everything
bandpass and lowpass filter to remove noise
fourier taking out everything over 15hz

(again, these are all separate executables and he's just dragging and dropping script files on them)

that's the end of pre-processing, now you actually look at the data...

ERPView

this view shows the waveforms from each electrode, arranged in a roughly spatially-accurate formation
average waveform taken across participants
you can look at one bin or split it into multiple bins (e.g. conditions) per electrode
looking for components - common patterns in the waveform, associated with a type of brain activity
e.g. N400 -> associated with semantic processing
P600 -> late positivity, associated with deeper encoding

components have been discovered over the history of EEG research
not always 100% sure this component is a known component you're comparing it to... need to use distribution to check
certain components associated with particular electrode locations

also focus in and subtract two conditions to see difference between them
generate contour maps (rainbows kill people!!) over a head-shaped glyph - show activation distribution
can be misleading to novices, since activation location doesn't really correspond to brain areas
all this information can help confirm identity of a component

eric has trouble explaining what the contour maps are for
helps to connect spatial and temporal... but again, spatial is kind of misleading
separate effects across conditions

can look at individuals or averages
can switch channels

ERPManipulate -> more processing, missed something here

ERPMeasure -> converts all the data to numbers for stats analysis
components based on time and distribution
scale by a baseline

lots of raw text data
script file within SPSS used to process text data

need to divide up scalp spatially
don't want to treat every electrode as a factor in the ANOVA
but location at some level should be a factor
need a coarser spatial organization
but no standard way to divide up the head, you just make up a scheme and argue for it in the paper
regions analysis vs. column analysis

VBA script does SPSS processing so it's all invisible to the researcher
extracts an organized excel file
just lists significant effects, then followup tests to each effect
highlight shade to show significance level

time windows
needs to be adapted per analysis

dependent variable is either average amplitude over time range or peak amplitude
(avg amplitude is better, peak in an average is problematic)
also look at latency

do this whole process after each participant, then more analysis when all the data is in
go through the data lots of different ways
figure out time windows, scalp divisions

statistics is confirmatory; the bulk of the analysis is visual

CS295J/Project Schedule:Caroline

2011-11-03T20:34:10Z

Caroline Ziemkiewicz: /* Material */

== Caroline's Project Schedule ==

'''Nov 1''' -- Interview Eric Fields at Tufts and plan for observation session.

'''Nov 8''' -- Add complete related work to paper outline. Rewrite to reflect pilot-study scope. Set up contacts with 2-3 more EEG researchers.

'''Nov 15''' -- Detailed plan for observation sessions, including task description and methods for insight and cognitive load gathering. Schedule observation sessions.

'''Nov 22''' -- Gather data from observation sessions. Get data set up for analysis (process videos, transcribe notes, etc.)

'''Nov 29''' -- Same as last week for any remaining observation sessions. Analyze all data and outline actual results and discussion sections.

'''Dec 6''' -- Perform any necessary followup data analysis. Fully written draft of two-page paper. Get feedback on draft.

'''Dec 13''' -- Final draft of two-page paper.

== Material ==

[http://vrl.cs.brown.edu/wiki/images/b/b2/Caroline_Outline.pdf Current Paper Outline]

[http://www.eecs.tufts.edu/~wfairb01/CTemp_CNS.pdf Example Poster from Tufts Group] - shows types of visual representations used

[http://neurocog.psy.tufts.edu/wiki/doku.php?id=documentation Documentation of Tufts EEG Analysis Software]

[[/Fields_Interview|Eric Fields Interview]]

== Boards ==

<center>
<gallery widths="250px" heights="250px" caption="Poster sketches and in-class critiques">
Image:caroline_project_sketch.png | Oct 13, 2011: Title and Results Sketch
Image:10-18-sketch.png | Oct 18, 2011: Paper Outline
Image:10-20-sketch.png | Oct 20, 2011: Related Work Outline
</gallery>
</center>

CS295J/Project Schedule:Caroline

2011-11-03T20:22:05Z

Caroline Ziemkiewicz: /* Material */

== Caroline's Project Schedule ==

'''Nov 1''' -- Interview Eric Fields at Tufts and plan for observation session.

'''Nov 8''' -- Add complete related work to paper outline. Rewrite to reflect pilot-study scope. Set up contacts with 2-3 more EEG researchers.

'''Nov 15''' -- Detailed plan for observation sessions, including task description and methods for insight and cognitive load gathering. Schedule observation sessions.

'''Nov 22''' -- Gather data from observation sessions. Get data set up for analysis (process videos, transcribe notes, etc.)

'''Nov 29''' -- Same as last week for any remaining observation sessions. Analyze all data and outline actual results and discussion sections.

'''Dec 6''' -- Perform any necessary followup data analysis. Fully written draft of two-page paper. Get feedback on draft.

'''Dec 13''' -- Final draft of two-page paper.

== Material ==

[http://vrl.cs.brown.edu/wiki/images/b/b2/Caroline_Outline.pdf Current Paper Outline]

[http://www.eecs.tufts.edu/~wfairb01/CTemp_CNS.pdf Example Poster from Tufts Group] - shows types of visual representations used

[http://neurocog.psy.tufts.edu/wiki/doku.php?id=documentation Documentation of Tufts EEG Analysis Software]

== Boards ==

<center>
<gallery widths="250px" heights="250px" caption="Poster sketches and in-class critiques">
Image:caroline_project_sketch.png | Oct 13, 2011: Title and Results Sketch
Image:10-18-sketch.png | Oct 18, 2011: Paper Outline
Image:10-20-sketch.png | Oct 20, 2011: Related Work Outline
</gallery>
</center>

CS295J/Project Schedule:Caroline

2011-11-03T18:45:08Z

Caroline Ziemkiewicz: /* Material */

== Caroline's Project Schedule ==

'''Nov 1''' -- Interview Eric Fields at Tufts and plan for observation session.

'''Nov 8''' -- Add complete related work to paper outline. Rewrite to reflect pilot-study scope. Set up contacts with 2-3 more EEG researchers.

'''Nov 15''' -- Detailed plan for observation sessions, including task description and methods for insight and cognitive load gathering. Schedule observation sessions.

'''Nov 22''' -- Gather data from observation sessions. Get data set up for analysis (process videos, transcribe notes, etc.)

'''Nov 29''' -- Same as last week for any remaining observation sessions. Analyze all data and outline actual results and discussion sections.

'''Dec 6''' -- Perform any necessary followup data analysis. Fully written draft of two-page paper. Get feedback on draft.

'''Dec 13''' -- Final draft of two-page paper.

== Material ==

[http://vrl.cs.brown.edu/wiki/images/b/b2/Caroline_Outline.pdf Current Paper Outline]

[http://www.eecs.tufts.edu/~wfairb01/CTemp_CNS.pdf Example Poster from Tufts Group] - shows types of visual representations used

== Boards ==

<center>
<gallery widths="250px" heights="250px" caption="Poster sketches and in-class critiques">
Image:caroline_project_sketch.png | Oct 13, 2011: Title and Results Sketch
Image:10-18-sketch.png | Oct 18, 2011: Paper Outline
Image:10-20-sketch.png | Oct 20, 2011: Related Work Outline
</gallery>
</center>

CS295J/Project Schedule:Caroline

2011-11-03T16:25:03Z

Caroline Ziemkiewicz:

== Caroline's Project Schedule ==

'''Nov 1''' -- Interview Eric Fields at Tufts and plan for observation session.

'''Nov 8''' -- Add complete related work to paper outline. Rewrite to reflect pilot-study scope. Set up contacts with 2-3 more EEG researchers.

'''Nov 15''' -- Detailed plan for observation sessions, including task description and methods for insight and cognitive load gathering. Schedule observation sessions.

'''Nov 22''' -- Gather data from observation sessions. Get data set up for analysis (process videos, transcribe notes, etc.)

'''Nov 29''' -- Same as last week for any remaining observation sessions. Analyze all data and outline actual results and discussion sections.

'''Dec 6''' -- Perform any necessary followup data analysis. Fully written draft of two-page paper. Get feedback on draft.

'''Dec 13''' -- Final draft of two-page paper.

== Material ==

[http://vrl.cs.brown.edu/wiki/images/b/b2/Caroline_Outline.pdf Current Paper Outline]

== Boards ==

<center>
<gallery widths="250px" heights="250px" caption="Poster sketches and in-class critiques">
Image:caroline_project_sketch.png | Oct 13, 2011: Title and Results Sketch
Image:10-18-sketch.png | Oct 18, 2011: Paper Outline
Image:10-20-sketch.png | Oct 20, 2011: Related Work Outline
</gallery>
</center>

File:Caroline Outline.pdf

2011-11-03T16:21:18Z

Caroline Ziemkiewicz:

CS295J/Project Schedule:Caroline

2011-11-03T16:21:07Z

Caroline Ziemkiewicz:

CS295J/Project Schedule:Caroline

2011-11-03T16:18:41Z

Caroline Ziemkiewicz: /* Boards */

CS295J/Project Schedule:Caroline

2011-11-03T16:17:59Z

Caroline Ziemkiewicz: /* Boards */

File:10-18-sketch.png

2011-11-03T16:17:17Z

Caroline Ziemkiewicz:

CS295J/Project Schedule:Caroline

2011-11-02T19:02:20Z

Caroline Ziemkiewicz:

File:10-20-sketch.png

2011-11-02T15:41:55Z

Caroline Ziemkiewicz:

File:10-20-sketch.JPG

2011-11-02T15:41:09Z

Caroline Ziemkiewicz:

CS295J/Project Schedule:Caroline

2011-11-02T15:39:23Z

Caroline Ziemkiewicz: New page: == Caroline's Project Schedule == '''Nov 1''' -- '''Nov 8''' -- '''Nov 15''' -- '''Nov 22''' -- '''Nov 29''' -- '''Dec 6''' -- '''Dec 13''' -- == Boards == <center> <galler...

== Caroline's Project Schedule ==

'''Nov 1''' --

'''Nov 8''' --

'''Nov 15''' --

'''Nov 22''' --

'''Nov 29''' --

'''Dec 6''' --

'''Dec 13''' --

== Boards ==

<center>
<gallery widths="300px" heights="250px" caption="Poster sketches and in-class critiques ">
Image:caroline_project_sketch.png | Oct 13, 2011
</gallery>
</center>

CS295J

2011-11-02T15:37:48Z

Caroline Ziemkiewicz:

'''Cognition, Human-Computer Interaction, and Visual Analysis'''

In this graduate seminar we will learn about models of human cognition and perception and explore potential implications of the models on how computers and humans can interact effectively when performing scientific analyses. Participants will be responsible for reading assigned materials, taking turns guiding discussions of the readings, and preparing a final paper and presentation. It is recommended that participants have some background in at least one of the areas of study.

We will structure our work around revising a 5-year research proposal that
was submitted for funding. We will base the revisions on comments
from reviewers and on our own developing vision of where the research could go. Some revisions will involve learning more about related work and including that context in the writing. Some revisions will involve identifying risky aspects of the proposal and implementing demonstrations or preliminary results (code, math, date, or otherwise) that help quantify the risk. Some revisions will involve adding new elements to the proposed research.

Each student will write up and orally present their results.

*[[/People.11|People]]
*Notes from Class (weeks go fri-thu) -- link to details about 2009 class at bottom of page
**[[/Week 1.11|Week 1]] 6-8 sep 2011, [[/Theory of Visualization outline|Theory of Visualization outline]]
**[[/Week 2.11|Week 2]] 9-15 sep 2011 [[/Literature class 2.11|Literature to read for week 2]], [[media:Discussion_rankings_week_2.11.jpg|discussion scores from board]]
**[[/Week 3.11|Week 3]] 16-22 sep 2011 [[/Literature class 3.11|Literature to read for week 3]]
**[[/Week 4.11|Week 4]] 23-29 sep 2011, [[/Literature to read for week 4.11|Reading for week 4]] (2/13/09), [[/Project ideas from week 4|Project ideas from week 4]]
**[[/Week 5.11|Week 5]] 30 sep-6 oct 2011 [[/Literature to read for week 5.11|Reading for week 5]] (2/20/09), [[/Wordle for the list of literature titles from the first four weeks|Wordle for the list of literature titles from the first four weeks]]
**[[/Week 6.11|Week 6]] 7-13 oct 2011, [[/Literature to read for week 6.11|Reading for week 6]], [[/Application Critiques.11|Application Critiques]], [[/Experiment results from class 6.11|Experiment results from class 6]]
**[[/Week 7.11|Week 7]] 14-20 oct 2011, [[/Literature to read for week 7.11|Reading for week 7]], [[/Boards from class 7.11|Boards from class 7]] (preliminary results first sentence(s))
**[[/Week 8.11|Week 8]] 21-27 oct 2011, VisWeek in Providence! [[/Literature to read for class 8.11|Reading for class 8]], [[/Proposal reviews from class 8.11|Proposal reviews from class 8]], [[/Boards from class 8.11|Boards from class 8]] (big picture intro stuff)
**[[CS295J/Week 9.11|Week 9]] 28 oct-3 nov 2011, [[/Literature to read for class 9.11|Reading for class 9]]
**[[/Week 10.11|Week 10]] 4-10 nov 2011, [[/Literature to read for class 10.11|Reading for class 10]]
**[[/Week 11.11|Week 11]] 11-17 nov 2011, [[/Literature to read for class 11.11|Reading for class 11]]
**[[/Week 12.11|Week 12]] 18-27 nov 2011, Thanksgiving week!, [[/Literature to read for class 12.11|Reading for class 12]], [[/Boards from class 12.11|Boards from class 12]]
**[[/Week 13.11|Week 13]] 25 nov-1 dec 2011
**[[/Week 14.11|Week 14]] 2-8 dec 2011, [[/Literature to read for class 14.11|Reading for class 14]]

*[[/Assignments.11|Assignments]]

*Project Schedules
**[[/Project Schedule:Steve|Steve]] -- Analyzing and modeling imaging QA workflow
**[[/Project_Schedule:CKS-NM|Clara and Nathan]] -- Memory Modulated Visual Search Patterns as Explored on a Touch Screen Interfaces
**[https://docs.google.com/document/d/1JkBLdCf4kZAuE1W88tvCQCso5NLhZFrjmel56FynrMg/edit?hl=en_US Hua & Diem ] -- Design & Evaluation of EEG Data Visualization Tool Based on Cognitive Task Analysis (Feel free to comment directly on the google doc)
**[https://docs.google.com/document/d/1jlV9TkCIzV4adXOe1OVpWiOXL9JPMO6T7cgnnNtFwpc/edit?hl=en_US Jenna + Michael] -- Evaluating Cognitive Load in Network Visualization
**[https://docs.google.com/document/d/19OO5Qwnd2SKOvudP8TR52wTHXwkRjHlj8eePeb3f0S8/edit?hl=en_US Wenjun] -- Correlation between Handedness/ cerebral dominance and data visualization
**[[/Project Schedule:Caroline|Caroline]] -- Predicting Insight in a Brain Imaging Analysis Task

*'''[[/Final contributions.11|Final contributions]]'''
*'''[[/Research proposal before.11|Research proposal before]]'''
*'''[[/Research proposal after.11|Research proposal after]]'''
*[[/Relevant Journals and Conferences|Relevant Journals and Conferences]]
*[[/Literature|Literature]]
*[[/Review Response|Review Response]]
*[[/Rule Lists.11|Rule Lists]]
*[[/Model elements.11|Model elements]]
*[[/Project Concepts.11|Project Concepts]]

*[[/Archived 2009 Pages|Archived 2009 Pages]]

[[/How Tos|How Tos]] -- [[/Class Members' Pages.11|Class Members' Pages]] -- [http://groups.google.com/group/cs295j Mailing list archive]

[[Category:Courses]]

CS295J/Boards from class 7.11

2011-10-13T18:53:05Z

Caroline Ziemkiewicz: New page: === Identifying and Addressing Common Workflow Gaps in Brain Data Analysis (Caroline) === thumb Notes: Combine with some kind of cognitive load measur...

=== Identifying and Addressing Common Workflow Gaps in Brain Data Analysis (Caroline) ===
[[Image:Caroline project sketch.png|thumb]]
Notes: Combine with some kind of cognitive load measurement, insight evaluation, to discover patterns that help explain evaluation results?
<br style="clear: both" />

File:Caroline project sketch.png

2011-10-13T18:50:54Z

Caroline Ziemkiewicz:

CS295J/Review Response

2011-09-29T16:18:02Z

Caroline Ziemkiewicz:

== Full Proposal (SI2-SSI) Review and Response ==

Dear reviewers

Thank you so much for your insightful, constructive, and helpful comments. We particularly appreciated the positive evaluation of X, Y, and Z in the proposal.

N competitive evaluations

We have addressed the criticisms below in the context of the reviews received. Our responses are

italics, bold, whatever.

:''Panel Summary
:''
:''INTELLECTUAL MERIT (INCLUDING POTENTIAL TRANSFORMATIVE ASPECTS):
:''
:''POSITIVE ASPECTS OF THE PROPOSAL AND PROPOSED RESEARCH:
:''
:''The panel noted that the proposal was clear and comprehensive, and addressed the criteria for SI2 proposals as well as the general NSF criteria. The proposed software would enhance both visualization of data on brain function and the knowledge discovery process of researchers in this area.
:''
:''SHORTCOMINGS AND WEAKNESSES OF THE PROPOSAL AND PROPOSED RESEARCH:
:''
:''The panel's discussion focused on the sustainability issue beyond the end of the project support timeframe. While a section of the proposal talks about the Outreach, Education, and Sustainability Plan, the community outreach and sustainability aspects are treated somewhat cursorily. These two issues are closely related: without community support, the software is unlikely to be sustainable in the long run. On the other hand, the proposers appear to be well known in their field, which may enhance community uptake.
:''
:''The proposal was perhaps over-ambitious; aspects of the evaluation (for example, eye tracking) will create large amounts of data that will require correspondingly intensive data analysis. However, the panel felt that even if the project did not accomplish every detail of the proposal, it would still be highly worthwhile. Similarly, while some aspects of the project might be seen as risky, a certain amount of risk is acceptable in NSF proposals, or even expected. Moreover, any risk is mitigated by the qualifications of the PIs, as exemplified by their excellent track record.
:''
:''==========================
:''
:''BROADER IMPACTS:
:''
:''POSITIVE ASPECTS OF THE PROPOSAL AND PROPOSED RESEARCH:
:''
:''The panel believes that the framework of this project would be portable to such other fields as gene regulation and the analysis of other complex networks.
:''
:''SHORTCOMINGS AND WEAKNESSES OF THE PROPOSAL AND PROPOSED RESEARCH:
:''
:''
:''
:''==========================
:''
:''ADDITIONAL REVIEW CRITERIA:
:''
:''The proposed software would primarily be of use to brain researchers. Other fields would be impacted indirectly, in the sense that if this way of building software packages combining visualization with support for hypothesis testing and tracking of analyses succeeds, it might provide a pattern for those fields to follow in their own software development processes.
:''
:''The proposers have laid out a detailed five year plan. One panelist questioned whether sufficient attention had been paid to issues of sustainability, in particular there was no mention made of plans for software support beyond the end of the five year plan.
:''
:''==========================
:''
:''SYNTHESIS COMMENTS:
:''
:''The panel agreed that this was a highly competitive proposal. In particular, the proposal attacks a large but tractable problem, and the team's wide and deep expertise gives the project a high probability of success.
:''
:''==========================
:''
:''PANEL RECOMMENDATION (CHECK ONE):
:''[X] Competitive (C)
:''[ ] Not Competitive (NC)
:''
:''This panel summary was read by panelists who participated in the discussion of this proposal, and they concurred that the summary accurately reflects the panel discussion.
:''
:''
:''Panel Recommendation: Competitive
:''
:''
:''Review #1
:''
:''
:''Rating: Excellent
:''
:''

:''
:''REVIEW:
:''
:''What is the intellectual merit of the proposed activity?
:''
:''SSI proposal.
:''
:''A five year project to develop software for visualization and analysis of brain circuitry by working with brain researchers to analyze their cognitive processes that need to be supported. The software is to help link the visualization workflow to a "decisional" workflow, supporting "reasoning and analysis at a high level, rather than just displaying data."
:''
:''The development phase will include studies of user interaction at both low level (eye tracking, mouse click logs) and high level (decision making).
:''
:''What are the broader impacts of the proposed activity?
:''
:''The proposers see this as sort of a prototype of the way software could be developed to support other scientific endeavors; this effect appears to constitute most of the Broader Impact (I wouldn't count benefit to "the entire brain science research community", mentioned in the BI statement, as broader impact).
:''
:''Summary Statement
:''
:''While this work is mostly outside my domain of expertise, it looks like a well chosen problem and an interdisciplinary effort. In particular, the team appears to have the broad range of people they would need to pull this off, from experts from the target audience (who suggested the project, and is listed as a co-PI), to cognitive scientists and computer scientists. In addition to the Intellectual and Broader Impact criteria, the specific "additional criteria" listed in the Program Solicitation have been explicitly and (to the extent that I can tell) well addressed. The required supplemental documents address the required points as well. One quibble I have in the Management and Coordination Plan is the statement that "The Stanford researchers will visit Brown if face-to-face interactions become necessary." While electronic communication allows collaboration in ways that would not have been possible before, I think it's not a question of whether face-to-face will be necessary, but how often. Fortunately, this appears to have been built into the travel budget.
:''
:''
:''Review #2
:''
:''
:''
:''Proposal Number:
:''
:''
:''
:''1047832
:''
:''NSF Program:
:''
:''
:''
:''Software Institutes
:''
:''Principal Investigator:
:''
:''
:''
:''Laidlaw, David H
:''
:''Proposal Title:
:''
:''
:''
:''SI2-SSI: Collaborative Research: Cognition-aware Visual Analytics of Brain Circuits
:''
:''Rating:
:''
:''
:''
:''Excellent
:''
:''
:''
:''
:''
:''REVIEW:
:''
:''What is the intellectual merit of the proposed activity?
:''
:''The PI's of this project are proposing to develop, test, and deploy software tools for scientific study of brain circuits. The project will focus on building a cyber infrastructure software system that is intended to improve the speed at which those doing brain research are able to complete their data analysis and it will advance the understanding of human cognition. This project has potential to have impact in the way researchers in the field collect and analyze data by providing an a rich set of cyber infrastructure tools for use in studying and modeling brain circuits. The intellectual merit of the project is very high as it has the potential to greatly reduce the time required to collect and analyze data.
:''
:''What are the broader impacts of the proposed activity?
:''
:''This project will provide an enhanced set of software tools to researchers in several areas that are conducting research that is related to the human brain. Areas of research that the cyber infrastructure software can be used in included: gene regulation, protein signaling and even crime and terrorism analysis and all have the potential to benefit.
:''
:''Summary Statement
:''
:''This project is focused on an area of research that spans several disciplines that will be able to utilize the cyber infrastructure software that will be developed. The PI's have a proven record of accomplishment in prior research projects. The project has intellectual merit and will have a broad impact by providing an enhanced set of software tools that will facilitate the efforts of researchers doing work related to studying and modeling the brain.
:''
:''Review #3
:''
:''
:''
:''Proposal Number:
:''
:''
:''
:''1047832
:''
:''NSF Program:
:''
:''
:''
:''Software Institutes
:''
:''Principal Investigator:
:''
:''
:''
:''Laidlaw, David H
:''
:''Proposal Title:
:''
:''
:''
:''SI2-SSI: Collaborative Research: Cognition-aware Visual Analytics of Brain Circuits
:''
:''Rating:
:''
:''
:''
:''Very Good
:''
:''
:''
:''
:''
:''REVIEW:
:''
:''What is the intellectual merit of the proposed activity?
:''
:''The proposed activity will allow brain scientists to visualize brain functions more easily and in much more detail than in the past.Network models, coupled with sophisticated methods for dimensionality reduction, promise to offer unique insights into the workings of the human brain. Moreover, the proposed visualization tool will go through rigorous evaluation that will allow its constant improvement. The proposers form a very strong group of well-established researchers in brain science and data visualization, offering a unique collaboration.
:''
:''What are the broader impacts of the proposed activity?
:''
:''Researchers from other disciplines, e.g., who study gene regulation, protein signaling, or perform crime and terrorism analysis, etc., have the potential to be benefited by the proposed software.
:''
:''Summary Statement
:''
:''This unique collaboration between brain scientists and data visualization scientists promises to offer tremendous benefits to the scientific community. The software that will be developed, will allow researchers to understand the signal pathways in the human brain in more detail than ever before. The software will be analyzed through a rigorous process, using models of cognition.
:''Review #4
:''
:''
:''
:''Proposal Number:
:''
:''
:''
:''1047832
:''
:''NSF Program:
:''
:''
:''
:''Software Institutes
:''
:''Principal Investigator:
:''
:''
:''
:''Laidlaw, David H
:''
:''Proposal Title:
:''
:''
:''
:''SI2-SSI: Collaborative Research: Cognition-aware Visual Analytics of Brain Circuits
:''
:''Rating:
:''
:''
:''
:''Good
:''
:''
:''
:''
:''
:''REVIEW:
:''
:''What is the intellectual merit of the proposed activity?
:''
:''The goal of this proposal is to develop, test, and deploy interactive visualization tools for scientific study of brain circuits. The tools will help brain researchers view brain circuits at multiple scales and perform sophisticated analysis of research hypotheses. The team members have a decade of experience developing scientific visualization tools for scientific users and consist of experts in cognitive science, neuroscience, computer science, and visual design. They are well qualified to conduct the project.
:''
:''What are the broader impacts of the proposed activity?
:''
:''This is an interdisciplinary project and the target user community is brain scientists. The tools will be made available to the public and are expected to benefit the entire brain science research community as well as other disciplines studying linked types of data. The tools can also be used in classes to help students understand connectivity.
:''
:''As the proposal is for SI2-SSI, I would like to know more on what the team plans to do to ensure the sustainability of the software and develop open-source community support. It is also unclear what the team will do to integrate diversity into the proposed activity.
:''
'''As added in section G, the plan to create long-term sustainability now includes more detail. Specifically we will focus on creating a lasting open source community that provides frequent updates to the code centralized by the core research team following a Macro R&D development infrastructure.''' ([[User: Stephen Brawner | Stephen Brawner]])
:''
:''Summary Statement
:''
:''I can see the proposed work will be valuable for brain scientists to study connectivity and dynamics of neural circuits in intact brain as existing systems all have limitations and cannot satisfy the needs of brain scientists as discussed in the proposal. Other scientific domains may need similar tools.
:''
:''This proposal focuses on an interesting problem for which it also provides a novel solution. Therefore, I think this is a quality proposal and worthy of support.

== Pre-proposal (Expeditions) Reviews and Response ==
:''Proposal Information
:''
:''Proposal Number:
:''
:''
:''
:''1064261
:''
:''Proposal Title:
:''
:''
:''
:''Collaborative Research: Cognitive Optimization of Brain-Science Visual-Analysis Tools
:''
:''Received by NSF:
:''
:''
:''
:''09/10/10
:''
:''Principal Investigator:
:''
:''
:''
:''David Laidlaw
:''
:''Co-PI(s):
:''
:''
:''
:''David Badre
:''
:''
:''
:''Steven Sloman
:''
:''
:''
:''This Proposal has been Electronically Signed by the Authorized Organizational Representative (AOR).
:''
:''
:''
:''NSF Program Information
:''
:''NSF Division:
:''
:''
:''
:''Division of Computer and Communication Foundations
:''
:''NSF Program:
:''
:''
:''
:''Experimental Expeditions
:''
:''Program Officer:
:''
:''
:''
:''Mitra Basu
:''
:''PO Telephone:
:''
:''
:''
:''(703) 292-8649
:''
:''PO Email:
:''
:''
:''
:''mbasu@nsf.gov
:''
:''
:''
:''Panel Summary #1
:''
:''
:''
:''
:''Proposal Number: 1064261
:''
:''Panel Summary:
:''Panel Summary
:''
:''Panel Summary for Expeditions Preliminary Proposals
:''
:''Preliminary Proposal Summary (Vision/Goals of the Expedition)
:''This proposal is focused on blending the areas of cognitive science, neuroscience, and HCI to develop new tools that would help in understanding the interrelationships in complex interconnected data sets. The work would focus on brain science activities but would likely be applicable to many other areas that have complex interconnected data such as crime and terrorism analysis.
:''
:''The strength of this proposal is in the benefits it would bring to the intersections of cognitive science, neuroscience, and HCI.
:''
:''Intellectual Merit
:''The authors of this proposal are planning to build cognitive models of brain scientists' perception and reasoning in performing their research. They then intend to use these models to develop new and improved interaction and visualization techniques for tracing neural pathways, with the expectation that use of the cognitive models will reduce the trial and error required to produce effective tools. Additionally, the cognitive models may even result in the invention of new visualizations through a more systematic exploration of the design space. One novel aspect of this proposal is the inclusion of heuristic knowledge of artists and visual designers related to cognition and perception. However the proposal does not elaborate on how the PIs would use heuristic knowledge of artists and visual designers; this is only mentioned in the introduction but never developed in the proposal.
:''
:''
:''Overall the panel had difficulty coming to a common understanding of the proposal contents. The range of reviews mirrored the range of what people had read into the proposal and their enthusiasm for the proposal topic.
:''
:''That said, the proposal failed to convince the reviewers along a number of dimensions.
:''
:''First, the proposal fails to articulate a clear research plan and a clear set of outcomes. As an example, the proposal mentions a number of cognitive concepts that will be incorporated in the interface, such as causal reasoning and dual systems theory. Utilizing cognitive principles to inform this research is applauded, but we expected to see some indication of how they would be put into practice. The proposal is even less clear when it comes to goal maintenance.
:: '''We have outlined the design of a module aimed at facilitating 'goal maintenance' with analysts. It will integrate with the brain-diagram user interface to collect input, including logging and explicit communications (e.g., what the user declares himself/herself to be doing), then predict user goals and propose or prime facilitating sub-goals. We have added a section called "Task Analysis" that outlines how we will analyze and encode tasks and sub-tasks that prime or conflict, and will use these encodings in the predictive model used by the module.''' ([[User:Steven Gomez|Steven Gomez]])
:''

:''' "We have included a plan for implementing cognitive principles. Strategies (through eye-tracking and other means outlined above) will include monitoring users' preferences and cognitive strategies through reaction time data and verbal reports from users. Historically, such strategies have proved successful when gathering user information, input, and comprehension; we expect the same in our implementation. We also intend to use eye-tracking and reaction time procedures in order to best gauge cognitive load to the extent that it affects user experience, as we strive to understand user adaptation and navigation strategies within our applications."''' --Clara

:''A more ambitious undertaking is that of predicting user performance. This aspect of the work is motivated by previous studies showing that student performance on algebra problems can be predicted based on eye movements. This is a very interesting result, but it is not clear how it would apply to an entirely different domain that requires a different and more taxing set of cognitive skills. The proposal does not describe how user behavior will be measured (other than through eye trackers) or even how performance is going to be measured. Algebra problems are generally closed ended, with a well-defined solution, whereas exploratory data analysis of neuroinformatics data is an open-ended problem.

:''
:''
:''The proposal needs to be much more explicit about the techniques used to derive predictions including the track records of these techniques and the ways in which these techniques may need to be enhanced to be used in particular application domains.
:''
:''
:''Another concern is methodological. We suggest that the team identify benchmark tasks that would be representative of the cognitive skills that the interface attempts to capture, and would also vary in their degree of complexity.
:''
:''
:: '''Original: We have identified a set of benchmark tasks to evaluate the users’ cognitive skills when working with our interface. These tasks test how well the interface increases users’ performance in the following cognitive aspects: multi-tasking, task-switching, visual search, reasoning, decision-making and cognitive workload. More detailed description of the design of these tasks, including experiment procedure outline, involved equipments and techniques, and data analysis methods, can be found in section e.5 (Formal Testing).''' ([[User: Hua Guo|Hua]], Sep.22, 2011)
:: '''Revised: We will break down the brain circuitry exploration process into a set of simpler cognitive tasks through cognitive task analysis. Benchmark tests that involve solving simple problems related to brain circuitry analysis will then be designed based on the identified cognitive tasks. These benchmark tests can then be used to evaluate the users' cognitive performance when working with our system in comparison with existing systems. ([[User: Hua Guo|Hua]], Sep.27, 2011)

:: ''' As the brain exploration process may involve essential complex tasks that are not able to break down into simpler tasks, we choose to evaluate the tool qualitatively over a long-term period: observe and interview users regarding their experience with the tools in order to discover critical features that the existing tool is missing. The researcher will also provide technical support if necessary. ([[User:Diem Tran|Diem Tran]] 21:22, 28 September 2011 (EDT))'''
:''Broader Impacts
:''The proposed work would provide research opportunities for faculty, postdocs and students working in the project. A tool with the capabilities described in the proposal would mostly benefit the brain science research community. The tool will be used in two computer science courses at Brown.
:''
:''The panel thought that the proposal team could do more work in considering possibilities for broader societal impact and for designing more impactful outreach and education activities that would extend beyond the Brown community.
:''
:''
:''
:''Summary Rationale for the Recommendation
:''Despite enthusiasm for this topic and the potential for significant impact if successful, the panel could not support the pre-proposal at this time due to a lack of coherent vision and a realizable plan of implementation.
:''
:''This is a very ambitious proposal that seeks to develop a new generation of visualization tools for the analysis of neuroinformatics data. While this is the kind of big-picture, high-risk project that the Expeditions in Computing program is designed to support, the proposal itself fails to provide a plan for achieving its high-level, abstract objectives.
:''
:''
:''Put an X next to the appropriate category
:''
:''Invite
:''Invite-if-possible
:''Do-Not-Invite X
:''
:''
:''
:''
:''
:''This summary was read by/to the panel, and the panel concurred that the summary accurately reflects the panel discussion.
:''
:''
:''Panel Recommendation: Do Not Invite
:''
:''
:''
:''
:''
:''Review #1
:''
:''
:''
:''Proposal Number:
:''
:''
:''
:''1064261
:''
:''NSF Program:
:''
:''
:''
:''Experimental Expeditions
:''
:''Principal Investigator:
:''
:''
:''
:''Laidlaw, David H
:''
:''Proposal Title:
:''
:''
:''
:''Collaborative Research: Cognitive Optimization of Brain-Science Visual-Analysis Tools
:''
:''Rating:
:''
:''
:''
:''Very Good
:''
:''
:''
:''
:''
:''REVIEW:
:''
:''What is the intellectual merit of the proposed activity?
:''
:''This proposal is focused on blending the areas of cognitive science, neuroscience, and HCI to develop new tools that would help in understanding the interrelationships in complex interconnected data sets. One novel aspect of this proposal is the inclusion of heuristic knowledge of artists and visual designers related to cognition and perception. The work would focus on brain science activities but would likely be applicable to many other areas that have complex interconnect data such as crime and terrorism analysis. The strength of this proposal is in the benefits it would bring to the intersections of cognitive science, neuroscience, and HCI.
:''The leadership team of distinguished faculty and researchers is well qualified to conduct this research. The entire team is an appropriate blend of researchers representing all of the subordinate areas of research.
:''The quality of prior work from all participating members is uniformly outstanding and appropriate for this endeavor.
:''While there have been much work of a similar nature done in the past, this project would move our knowledge forward on a number of new fronts. In addition, the inclusion of heuristic knowledge of artists and visual designers related to cognition and perception is novel.
:''The proposal is well conceived and organized and clearly presented.
:''With the pupil tracking device requested in the budget, there appears to be sufficient access to all the required resources necessary for this undertaking.
:''There seems to be a wealth of available experimental facilities in the associated institutions. Some of the relevant equipment includes tiled display walls, stereo-enabled desktop displays, ultra-high-resolution Wheatstone stereoscope, haptic devices, and a virtual-reality cave to come online later this year.
:''The leadership plan appears to be appropriate for the small size of the project personnel. The team members have a good history of interaction on related academic activities.
:''Other than the nominal support for traditional computing need by the computer science department at Brown, there did not appear to be any specific institutional support for this proposal.
:''The budget is well thought out, clearly described, and justified.
:''The collaboration amongst the faculty of Brown, Stanford, and the Rhode Island School of Design appears to be very appropriate for this project. The project would bring together cognitive scientists, visualization experts, and other domain specialists to bridge the gap between theory and practice in this area of brain research. Clearly the synergy in this group would help to insure the success of this work.
:''
:''
:''What are the broader impacts of the proposed activity?
:''
:''The value of the proposed work appears to be very important in this specific area of research. While this is not my specific area of expertise, I have some concern whether this project meets the over arching goals of the Expeditions in Computing Initiative. In particular, it would seem to have the potential to stimulate interest in this area for graduate students in related disciplines, but may not be very successful in drawing attention to STEM studies amongst the K-12 age group.
:''A major focus of this proposal is the conceptualization, design, and development of a software framework for predicting user performance. It would gather information on specific models, user interfaces, and user goals and endeavor to produce probabilistic estimates of the state of users over time as predicted by the models.
:''I did not see anything in the proposal that indicated that it would be of particular interest to youth and underrepresented groups.
:''The single sentence on stimulating effective knowledge transfer did not seem convincing.
:''
:''
:''Summary Statement
:''
:''This is a good general research proposal in the area of brain research and the understanding of interconnected relationships of complex data sets. There are some novel research concepts, including the heuristic knowledge of artists and visual designers related to cognition and perception. The leadership team is well qualified to lead this endeavor and the outlook for good research results look promising. The proposed budget is in line with the proposed activities and personnel commitments. This proposal should fair well as a general unsolicited proposal for NSF. I would rank this proposal to be better than many of the other proposals of the Expeditions in Computing Initiative.
:''
:''
:''Review #2
:''
:''
:''
:''Proposal Number:
:''
:''
:''
:''1064261
:''
:''NSF Program:
:''
:''
:''
:''Experimental Expeditions
:''
:''Principal Investigator:
:''
:''
:''
:''Laidlaw, David H
:''
:''Proposal Title:
:''
:''
:''
:''Collaborative Research: Cognitive Optimization of Brain-Science Visual-Analysis Tools
:''
:''Rating:
:''
:''
:''
:''Good
:''
:''
:''
:''
:''
:''REVIEW:
:''
:''What is the intellectual merit of the proposed activity?
:''
:''Intellectual Merit: The main goal is to push the frontiers of data visualization, with secondary thrusts on improved learning in cognition about how we look at data. The strengths of the proposal are that the proposed tool is, as far as I know, ground-breaking in that it will actively change based on the user. Further, the researchers are well qualified, with expertise in the psychology as well as in the computer science. The major weakness was that I was not really clear how the software tool would eventually work. For instance, they talked a little about following pupils of the user - but I was not clear how they would capitalize off of that knowledge.
:''
:''Value added:
:''I think the proposed research could be complex and important enough to warrant this investment - however I could not find a clearly defined path of attack in the proposal. It fits well within the 3 program goals, with probably the greatest emphasis on the first. Intelligent data visualization tools that can react to the user will open many new doors in understanding science. It will impact and inspire future computer scientists, although I do no see a preference for underrepresented groups. Finally, it has the potential to stipulate new significant findings in science and in education.
:''
:''Leadership plan:
:''The leadership plan seemed well thought out. They have a diverse set of researchers, each with their unique skill set.
:''
:''
:''What are the broader impacts of the proposed activity?
:''
:''Broader Impact: The work will be distributed to all who want to use it and will be used in classes at Brown, affecting students at all levels (either as developers or clients). I found their vision here a little short-sighted.
:''
:''Summary Statement
:''
:''In their proposal entitled 'Collaborative Research: Cognitive Optimization of Brain-Science Visual-Analysis Tools,' the authors propose to use understandings from cognitive science, neuroscience and human - computer interaction to develop better tools for examining data. In particular, they will develop software to that will visualize neural connections in the brain. At the same time they will actively measure the client and use these data to predict what the client will want to see next. They present a compelling case that we need better visualization tools for understanding the brain.
:''
:''Review #3
:''
:''
:''
:''Proposal Number:
:''
:''
:''
:''1064261
:''
:''NSF Program:
:''
:''
:''
:''Experimental Expeditions
:''
:''Principal Investigator:
:''
:''
:''
:''Laidlaw, David H
:''
:''Proposal Title:
:''
:''
:''
:''Collaborative Research: Cognitive Optimization of Brain-Science Visual-Analysis Tools
:''
:''Rating:
:''
:''
:''
:''Excellent
:''
:''
:''
:''
:''
:''REVIEW:
:''
:''What is the intellectual merit of the proposed activity?
:''
:''The dominant approach to developing interactive systems is for the developers to interact with the envisioned users to gather the general requirements for the application and to construct software based on the developer's intuitions as to how the users will actually interact with the software. Depending on the sophistication of the organization, cycles of usability testing and re-design are used to refine the interface; alternately, they may simply release the software to the users and wait for the complaints or lack of sales.
:''An alternative to this expensive process is to construct explicit models of the perceptual and decision making processes of the users and then use these models to inform the design process. Work on cognitive models such as GOMS and ACT began about three decades ago and has progressed slowly but steadily throughout the time period and there have been a number of small demonstrations that such models can, in fact, eliminate most or all of the iteration previously required.
:''The authors of this are planning to build cognitive models of brain scientists' perception and reasoning in performing their research. They then intend to use these models to develop new and improved interaction and visualization techniques for tracing of neural pathways, with the expectation that use of the cognitive models will reduce the trial and error required to produce effective tools. Additionally, the cognitive models may even result in the invention of new visualizations through a more systematic exploration of the design space.
:''Although the focus of the recent work in cognitive models has been to develop engineering models which are capable of being used outside of the research setting, use of such models in the design of interactive systems has been slow to catch on. If nothing else, construction of the models requires a large amount of intellectual labor and, to date, impressive examples of the use of these models to justify that labor investment have been rare. This work has the potential for providing such a critical example and could be the impetus to finally move cognitive models into widespread use.
:''Additionally, work in as complex an area as brain science will ensure that the cognitive modeling tools can handle nearly any application.
:''Finally, if the models do result in improved tools, the research may result in new findings in the brain science field.
:''The primary risk in this proposal is that the cognitive models which can be created are too weak to support the design process. There is no guarantee that the computer scientists and psychologists doing this research will be able understand and model the cognitive processes of a brain scientist.
:'''We have included our experience of developing visualization tools for brain study at Brown, and also we have a detailed timeline, in each phase, brain scientists will be involved in the process of model evaluation, testing, and providing feedback.( --- [[User:Chen Xu|Chen Xu]])
:''Value-added of funding the activity as an Expedition
:''This work requires substantial commitment on the part of the computer scientists and psychologists to learn the brain science domain and on the part of the brain scientists for their interaction with the cognitive scientists. Such a commitment is unlikely to be obtained with smaller, more fragmented funding. Industry or venture capital are unlikely to fund this kind of research.
:''The main knowledge transfer methods will be the mentoring of graduate students and the addition of formal courses intended to teach about interdisciplinary collaboration. In addition, the software they develop will be made available for distribution. Except through the rather limited vehicle of scholarly publication, it is not clear how the cognitive models themselves are to be made available. The authors may want to consider using their own visualization capabilities to explain the models.
:''Leadership and Collaboration Plan
:''Only two institutions are involved in this work and the senior researchers are all located at one of the two institutions. This should minimize coordination problems.
:''Funding for the primary brain scientist in this research is at the 50% level and his supervisor has a nominal level of funding. This is a cause for concern, given the level of commitment required to support what is, essentially, someone else's area of research. A higher level of funding would be desirable, even if a substantial amount of the funded time is spent on pure brain science research.
:''
:''
:''What are the broader impacts of the proposed activity?
:''
:''The proposal includes training a number of graduate and post doctoral students; in fact, most of the funding requested is for student support.
:''As mentioned earlier, to the extent this work results in wider acceptance and usage of cognitive models, particularly in the development of scientific software, it will accelerate the construction of interactive systems which can be used efficiently.
:''
:''
:''Summary Statement
:''
:''This work should lead signficantly wider use of cognitive modeling in interactive systems design as well as provide researchers in brain science with superior tools.
:''
:''
:''
:''Review #4
:''
:''
:''
:''Proposal Number:
:''
:''
:''
:''1064261
:''
:''NSF Program:
:''
:''
:''
:''Experimental Expeditions
:''
:''Principal Investigator:
:''
:''
:''
:''Laidlaw, David H
:''
:''Proposal Title:
:''
:''
:''
:''Collaborative Research: Cognitive Optimization of Brain-Science Visual-Analysis Tools
:''
:''Rating:
:''
:''
:''
:''Fair
:''
:''
:''
:''
:''
:''REVIEW:
:''
:''What is the intellectual merit of the proposed activity?
:''
:''Proposal Summary
:''
:''This proposal seeks to develop new visualization techniques that will assist brain scientists with the interpretation of high-dimensional data. For this purpose, the PI will incorporate design principles and knowledge from cognitive science, neuroscience and human computer interaction. the visualization system will also capture data of scientists as they use the tool, and compare it with computational models from cognition, perception and art. The tool will also be able to predict user performance and user state over time. The tool will be released through an open-source license, and will be incorporated into two courses. The team has worked together for a number of years.
:''
:''
:''Criterion 1. What is the intellectual merit of the proposed activity?
:''
:''Strengths
:''
:''The kind of tool envisioned in this proposal would be invaluable, not only in neuroinformatics but also on other disciplines that deal with high-dimensional, multi-scale data, from social networks to geospatial information.
:''
:''Weaknesses
:''
:''Despite its laudable objective, this work is not ready for further scrutiny as a full proposal.
:''
:''First, the proposal fails to articulate a clear research plan and a clear set of outcomes. As an example, the proposal mentions a number of cognitive concepts that will be incorporated in the interface, such as causal reasoning and dual systems theory. How are these principles going to be used to design a better visualization, and how are they going to be tested? I very much like the idea of using cognitive principles, but would have expected to see some indication of how they would be put into practice. The proposal is even less clear when it comes to goal maintenance: "We will use these principles to determine which tasks to make easily accessible to users and which to put in the background." This is a general problem for interface design, not a solution to the problem.
:''
:'''Reviewer 4 indicated that the proposal was not explicit about how cognitive principles would be implemented in the project. In section ___ on page __, we have elucidated how the principles listed will concretely lead to better, more optimized visualizations, as well as the studies we plan to perform to show that the visualizations are indeed efficient. ([[User:Jenna Zeigen|Jenna Zeigen]])'''

:''A more ambitious undertaking is that of predicting user performance. This aspect of the work is motivated by previous studies showing that student performance on algebra problems can be predicted based on eye movements. This is a very interesting result, but it is not clear how it would apply to an entirely different domain that requires a different and more taxing set of cognitive skills. This is a wild extrapolation. The proposal does not describe how user behavior will be measured (other than through eye trackers) or even how is performance going to be measured. Algebra problems are generally closed ended, with a well-defined solution, whereas exploratory data analysis of neuroinformatics data is an open-ended problem.
:''
:'''We have made a detailed timeline, and in phase/year 3, we have included the way to measure user performance and evaluate the cognitive models. We evaluate the models prediction against actual user data, in addition to eye-tracking, computer interaction logging, video logging and skin conductance response will be adopted, and the results and help refine the models.( --- [[User:Chen Xu|Chen Xu]])
:''Another concern is methodological. Say that the proposal had articulated a clear plan and a reasonable set of deliverables for a new generation of visualization interfaces. Wouldn't it be better to test this interface on some benchmark problems, and see how it facilitates performance relative to a standard interface? These benchmark problems would be representative of the cognitive skills that the interface attempts to capture, and would also vary in their degree of complexity.
:''
:'''We mainly test the user interface performance by empirical ways. But they are based on the principles of perception and goal selection . We also have included some standards like performance time of an average user completing a unit task . We will include a comparison between the past and present interface in one unit task performance test. ( --- [[User:Wenjun Wang|Wenjun Wang]])
:''To what extent does the proposed activity suggest and explore creative, original, or potentially transformative concepts?
:''
:''The proposal is very ambitious in its overall objectives. A visualization tool having the characteristics suggested in the proposal would be invaluable to brain science as well as to other scientific disciplines dealing with high-dimensional complex data, such as genomics/proteomics, geospatial analysis, network analysis, etc. However, the proposal fails to turn a high-level concept into a realizable implementation.
:''
:'''Our framework allows more modules to be involved in as the system evolves in future. The interface facilitating viewing data and the model to make prediction can be both applied to other disciplines .( --- [[User:Wenjun Wang|Wenjun Wang]])
:''Is the work of sufficient import, scale, and/or complexity to justify this type of investment?
:''
:''The brain is one of the scientific frontiers for the 21st century. The proposal has the complexity and scale worthy of this type of investment, but the proposal fails to deliver a realistic plan (if any plan at all) or even specifications.
:''
:''Will the work contribute to realization of the EIC program goals and is it likely to demonstrate completion of these goals?
:''
:''Understanding the brain is one of our greatest scientific challenges. Unfortunately, without a clear research plan it is difficult to asses the likelihood that the proposal will be able to demonstrate completion of its overall goals.
:''
:''Value of the experimental systems or shared experimental facilities proposed
:''
:''The investigators will utilize some shared facilities in their research and will share software and data that they produce to allow further research by others. The proposed software testbed will be used across the collaborators to test models of cognition and perception in the context of HCI.
:''
:''Leadership and Collaboration Plan
:''
:''The investigators have worked together for a number of years, have taught classes together, and their students have attended classes from each other. No leadership or collaboration plan is discussed beyond this.
:''
:''
:''What are the broader impacts of the proposed activity?
:''
:''Criterion 2. What are the broader impacts of the proposed activity?
:''
:''Strengths
:''
:''The proposed work would provide research opportunities for faculty, postdocs and students working in the project. A tool with the capabilities described in the proposal would mostly benefit the brain science research community. The tool will be used in two computer science courses at Brown.
:''
:''Weaknesses
:''
:''Societal benefits of this tool would derive from its scientific merit to the extent that it would help understand the brain. Given the characteristics of this project, I wonder if other NSF funding opportunities would be more suitable, such as the FODAVA program or the interdisciplinary program in neuroscience at CISE. I also wonder whether this work should be funded instead by NIH (NIBIB, NIMH). The budget contains a request for $3,000 to cover costs of animal (mouse) care; why is this needed given that the proposal is for software development?

<b>While the scientific merit of this project is most directly applicable to understanding the brain, it should be noted that the goal to “convincingly demonstrate that the employed techniques facilitate better analysis” (described in section c.1) is unique to this proposal, and would have impacts that reached into any other discipline in which vast amounts of data needs to be interpreted. --Michael</b>

:''
:''
:''Summary Statement
:''
:''This is a very ambitious proposal that seeks to develop a new generation of visualization tools for the analysis of neuroinformatics data. The tools would allow brain scientists to explore high-dimensional data, and the tool would also predict user performance and state. The proposal is inspired by principles from cognitive science, neuroscience and HCI. While this is the kind of big-picture, high-risk project that the Expeditions in Computing program is designed to support, the proposal itself fails to provide a plan for achieving its high-level, abstract objectives. The proposal does not provide a leadership or collaboration plan.
:''
:''
:''Review #5
:''
:''
:''
:''Proposal Number:
:''
:''
:''
:''1064261
:''
:''NSF Program:
:''
:''
:''
:''Experimental Expeditions
:''
:''Principal Investigator:
:''
:''
:''
:''Laidlaw, David H
:''
:''Proposal Title:
:''
:''
:''
:''Collaborative Research: Cognitive Optimization of Brain-Science Visual-Analysis Tools
:''
:''Rating:
:''
:''
:''
:''Fair
:''
:''
:''
:''
:''
:''REVIEW:
:''
:''What is the intellectual merit of the proposed activity?
:''
:''PROPOSAL OBJECTIVES AND APPROACH
:''The proposal develops a variety of tools for interactive analysis and reasoning for brain scientists.
:''
:''INTELLECTUAL MERIT
:''The project addresses research in three areas: human-computer interaction, cognitive modeling and the connectivity in the brain. It lists11 items that are to be developed.
:''
:''The proposal is vague and has a lot of repeatability. It is not well written. It is not clear what research experiments are performed.
:''
:''The team is fine.
:''
:''
:''What are the broader impacts of the proposed activity?
:''
:''There is a great need for the tools by the computational neuroscience and cognitive science community. This project will develop some of these tools.
:''
:''The details of the educational plan are not given.
:''
:''
:''Summary Statement
:''
:''The proposal can be strengthened by focusing and making the challenges and ideas more clear.
:''
:''
:''Review #6
:''
:''
:''
:''Proposal Number:
:''
:''
:''
:''1064261
:''
:''NSF Program:
:''
:''
:''
:''Experimental Expeditions
:''
:''Principal Investigator:
:''
:''
:''
:''Laidlaw, David H
:''
:''Proposal Title:
:''
:''
:''
:''Collaborative Research: Cognitive Optimization of Brain-Science Visual-Analysis Tools
:''
:''Rating:
:''
:''
:''
:''Poor
:''
:''
:''
:''
:''
:''REVIEW:
:''
:''What is the intellectual merit of the proposed activity?
:''
:''The proposed activity of developing and improving neuroscience visual analysis tools is very important. Currently access to the genre of systems described in this proposal, especially in areas with complex sub-structure such as neuroscience, is lacking and the proposed activity could have a profound effect on the state-of-the-art in the field. The possible interplay between the user-interface experts and biomedical informatics developers is a possible strength. The available team and resources are very strong and capable with an excellent track record in the field.
:''
:''However, the proposed activities within the proposed are completely underspecified. Given the scale of the project and the challenging nature of the problems to be faced, the level of technical detail and planning presented in the proposal is insufficient and unconvincing. A great many claims are made in the proposal with no clear measurable end-points to determine how success within the project could be evaluated. Specifically, the authors claims to employ developments in concepts of cognition and perception to assist scientists reasoning. What aspects of reasoning? For which tasks? Within which discipline? If this is only related to tractography, what sort of scientific hypotheses to the applicants expect to address? Are they relating these representations of neural connectivity to studies in animals? Are they relating these analyses to other modalities of imaging data? How do they intend to reason over the complex semantics of these other experimental types? These are all glaring omissions from the proposal. The description of the cognitive science aspects of the project were marginally better specified, but I still found the details lacking. For example, the claim was made that 'our system will tune itself to individual work styles. How? What technical elements will the system exploit to accomplish this? In particular, the applicants must spend more time specifying the precise tasks that the system is designed to tackle before it is possible to improve or optimize performance at that task.

::Response: We have added a section called "Task Analysis," in which we outline a plan for specifying visual analysis tasks through user observation. We will begin with loosely structured observations to generate hypotheses about primary tasks and user strategies in this domain, and continue with cognitive task analysis methods to focus and test these hypotheses. We present preliminary results that show the viability of this plan. ([[User:Caroline Ziemkiewicz|Caroline Ziemkiewicz]] 10:48, 22 September 2011 (EDT))

:''
:''In particular, the statement "We expect the core element to evolve significantly through the five years of the project. It cannot be meaningfully defined without the data we will acquire from users, so details beyond this overview are not possible yet" is incredibly revealing and suggests that the applicants themselves do not have a clear idea of how they intend to solve these problems.
:''
:''The Figures presented in the proposal looked confusing and uninformative, adding nothing to the argument that these systems would actually help a scientist understand the underlying data.
:''
:''I very much liked the proposed idea of using the system directly in courses taught at Brown and other collaborating institutions. I think that this is a sizable innovation that would be very welcome in the field and might even form the basis of evaluation metrics for the success of the system (which could address one of my previous criticisms of the project).
:''
:''
:''What are the broader impacts of the proposed activity?
:''
:''The activity is well integrated with training and learning. There are a number of students in the group, all of whom would have the opportunity to work with professionals from a very different focus. The interdisciplinary nature of the work coupled with the need for analysis tools in biology would be an excellent synergy to cultivate. The presence of high-end graphics equipment (such as a virtual-reality cave and haptic displays, etc.) is a plus for the project but also is a hindrance to enable the developers to release their work to a broader audience. If the system is only available to the small number of people who have access to such facilities, then the impact of the work would be lessened.
:''
:''The proposed activity makes no specific claims to target or support underrepresented groups explicitly.
:''
:''The underspecification of the technical aspects of the project undermine the open-source distribution of the code. It is technically demanding to generate usable open source products for other people to use. Notably, browsing the co-PIs webpages, there were no easily accessible open-source software products noticeably available.

:#''' Since the review, the PI's lab has published a large selection of source code from previous projects, available from its website (vis.cs.brown.edu), with provisions for multiple platforms.
:#''' We are undertaking a project to make anonymized images of healthy and abnormal brains (and related data), from our projects and those of a number of collaborators worldwide, freely available to the scientific community.
:#''' Development will be done using a publicly visible source code repository (e.g., Github), so that other scientists and the public may be able to track progress, comment on changes, provide feedback, and even contribute pieces of code to the software.
::''' ([[User:Nathan Malkin|Nathan]])

:''
:''Summary Statement
:''
:''Although the high-level conceptualization of this project is exciting, the way that the project was described in the proposal was massively underspecified. Technical details were lacking and some fundamental aspects of the project's conceptualization in terms of the scientific domain under study were missing. There was no timetable, and no evaluation proposed to see how progress would be measured. The authors should be careful about making high-level claims concerning the possible impact of the proposed without a more carefully constructed argument to back up the claims.
:''