VrlWiki - User contributions [en]

Port existing project

2010-04-19T15:48:50Z

Trevor O'Brien: New page: Before porting your project, you may wish to take a look at documentation on the transition to the new $G. At this point, porting existing projects along with the...

Before porting your project, you may wish to take a look at documentation on the [[Transition to new $G|transition to the new $G]].

At this point, porting existing projects along with their source control histories is a tricky undertaking. Please see Brad for help with this process.

Check out projects

2010-04-19T15:43:51Z

Trevor O'Brien:

== Checking out a project ==
Before checking out a project, be sure that your development environment is [[$G in practice|set up for $G]].

$G uses CVS for source control and you can read more about it [http://www.nongnu.org/cvs/ on its official site] or [[w:Concurrent Versions System|on Wikipedia]].

You can work anywhere except under the $G directory.
At Brown ample disk space has been allocated here for students:

<pre>
/map/gfx0/users/$USER
</pre>

So, for example, if your user id is "goo" and you wanted to check out the "wrist" project, you should type:

<pre>
> mkdir /map/gfx0/users/goo
> cd /map/gfx0/users/goo
> cvs co project/wrist # Creates: /map/gfx0/users/goo/project/wrist/...
</pre>

Please do not waste space and retain only the files you need.
When you are done with your account please be sure to delete it.

== Building a project ==

Type "make all" to build a project. Continuing the wrist project example from above, you would type:

<pre>
> cd /map/gfx0/users/foo/project/wrist
> make all
</pre>

By convention simple projects that build a single program or library can be built just by typing "make".

Add a project

2010-04-19T15:39:20Z

Trevor O'Brien: New page: == Add a new project == == Port an existing project from legacy $G to the new $G == == Template apps for starting new projects ==

== Add a new project ==
== Port an existing project from legacy $G to the new $G ==
== Template apps for starting new projects ==

Put your code into CVS in $G

2010-04-19T15:37:18Z

Trevor O'Brien: Put your code into CVS in $G moved to Source Control

#REDIRECT [[Source Control]]

Source Control

2010-04-19T15:37:18Z

Trevor O'Brien: Put your code into CVS in $G moved to Source Control

== Using CVS ==

Active projects are being migrated to a new directory structure using a common set of make files. Have Brad help you set up any new project. Students actively working on legacy projects can also begin to migrate to the new directory structure.

CVS keeps track of changes to every file, allows concurrent editing of source code by multiple people, and provides an easy means for copying source trees to new locations. Before working with CVS you should read some introductory web pages about it until you are comfortable with the basic concepts. In order to maintain the integrity of our code base it is imperative that you understand the effects of any CVS command you use.

:[http://www.nongnu.org/cvs/#TOCintroduction Introduction to CVS]

=== Setting up CVS ===

On Linux add the following to your <tt>~/.environment</tt> file:
<pre width="50%">setenvvar CVSROOT /map/gfx0/cvsroot/</pre>

On Windows set CVSROOT to: <code>Y:\map\gfx0\cvsroot\</code>

The scripts in <code>$G/common/build/setup/</code> can be used to perform common CVS operations. They simplify check ins and use integrity checks to avoid common mistakes.

=== CVS commands you might find handy ===

<pre>
> cvs update -d <directory> # Bring a new directory into your sandbox.
> cvs log <file> # List log messages for a file.

> cvs add -m "add directory" <directory> # Create a new directory.

> cvs add -m "add file" <file> # Create a new file.
> cvs commit -m "commit file" <file>

> cvs delete <file> # To delete first remove the file from your sandbox.
> cvs commit -m "gone" <file>

> cvs export -D NOW <directory> # Snapshot without administative directories.
> tar cvf <project>.tar <directory> # Make a tar ball.
> gzip <project>.tar # Compress it into: <project>.tar.gz

> cvs checkout -D yesterday <directory> # Get previous versions of a directory.
> cvs checkout -D "<#> hour ago"
> cvs checkout -D "<month>/<day>/<year>"
</pre>

=== How Not To Use CVS ===

* Avoid checking in entire directory trees with junk files.
Clear out edit backups, binaries, and irrelevant file first or do selective check-ins.

* Check in source files only.
Do not check in files generated by tools.

* Overwriting other people’s changes is bad.
Revision control is not a substitute for coordinating with your peers.

* Don’t trust automated CVS features.
They are primitive and not bulletproof.

* Don't Lock (Reserve) Files.

* Think Twice about Creating Directories.
CVS does not have a command for renaming or deleting directories.

== Legacy Code in $G/src/ ==

This page describes how the bulk of existing code in <code>$G</code> is organized. Active projects are being migrated to a new directory structure using a common set of make files. Until the procedures are in place have Brad help you set up any new project. Students actively working on projects can also begin to migrate to the new directory structure.

The <code>$G</code> tree is not intended for users to write to directly, and future IT plans for our group include enforcing this restriction. Instead, we use the [[w:revision control system|revision control system]] [[w:Concurrent Versions System|CVS]] as an interface to the directory structure in <code>$G</code>. CVS keeps track of changes to every file, allows concurrent editing of source code by multiple people, and provides an easy means for copying source trees to new locations.

Ordinarily, the first steps of development for your own software project happens somewhere in your homedir. This doesn't allow other people to work with your code, though, so once your project is nontrivial you should add it to <code>$G</code> using the <code>gfxprojinit</code> script. This script creates a home for your project in <code>$G/src</code>, but not everything associated with a given project belongs there. To get everything into the right place, follow these steps:
# Go to the directory that has your source in it.
# Delete or move away anything you don't want to go into CVS. If you already have something compiled, do <code>make clean</code> to get rid of the object files --- you don't want to put them in CVS. In general, just leave the Makefile and the source files.
#* '''Note:''' even though non-source files are sometimes necessary for builds of complex projects, data and config files don't belong in <code>$G/src</code>! Carefully consider whether non-source files belong in the <code>$G/src</code>; they most likely do not and belong in <code>$G/lib</code> (or <code>$G/data</code> for example data on which your program runs) instead. Move these out of the directory temporarily; we'll check them into the appropriate locations later.
# If you have any binary files, like images, that you want to go in CVS, move them out of this directory temporarily. We will add them later because CVS requires non-text files to be added with a special flag. Once again, keep in mind that most binary files are program resources and belong in <code>$G/lib</code> rather than <code>$G/src</code>.
# Run the following command, replacing <projectname> with the name you want your project to have --- that will become a new directory under <code>$G/src</code>: <pre>gfxprojinit <projectname> .</pre>
#* Note the period at the end of that command. It might be hard to see, but it is important.
#* It's worth knowing, for those familiar with CVS, that <code>gfxprojinit</code> is essentially a fancy wrapper around <code>cvs import</code>. Please be sure to use <code>gfxprojinit</code> anyway, as it maintains important metadata and may do even more in the future.
# If you had some extra files you wanted to add to CVS, like build components, config files, program resources, or example data, you now want to check them into the relevant directories, which will be among <code>$G/src/<projectname></code>, <code>$G/lib/<projectname></code>, and <code>$G/doc/<projectname></code>. For each relevant directory, which we'll call <code>$X</code>, do the following:
## Create <code>$X</code>, import it into CVS, then delete it and check it back out.
## Copy each appropriate file into <code>$X</code>, and for each one run the command: <pre>cvs add -kb <filename></pre>
## After adding all the files, run: <pre>cvs commit -m "<commit message>"</pre>
# Your project should have been automatically added to the <code>$G/src</code> directory on the filesystem you are on now. To get it in the <code>$G/src</code> directory somewhere else, like at the [[Cave]], run this command from within <code>$G/src</code>: <pre>cvs update -d <projectname></pre> You may also need to use the same command in <code>$G/lib</code> and <code>$G/doc</code>.

[[Category:$G HOWTO]][[Category:Software Development]]

Check out projects

2010-04-19T15:35:30Z

Trevor O'Brien: New page: == Checking out a project == $G uses CVS for source control and you can read more about it [http://www.nongnu.org/cvs/ on its official site] or [[w:Concurrent Versions System|on Wikipedia...

== Checking out a project ==

$G uses CVS for source control and you can read more about it [http://www.nongnu.org/cvs/ on its official site] or [[w:Concurrent Versions System|on Wikipedia]].

You can work anywhere except under the $G directory.
At Brown ample disk space has been allocated here for students:

<pre>
/map/gfx0/users/$USER
</pre>

So, for example, if your user id is "goo" and you wanted to check out the "wrist" project, you should type:

<pre>
> mkdir /map/gfx0/users/goo
> cd /map/gfx0/users/goo
> cvs co project/wrist # Creates: /map/gfx0/users/goo/project/wrist/...
</pre>

Please do not waste space and retain only the files you need.
When you are done with your account please be sure to delete it.

== Building a project ==

Type "make all" to build a project. Continuing the wrist project example from above, you would type:

<pre>
> cd /map/gfx0/users/foo/project/wrist
> make all
</pre>

By convention simple projects that build a single program or library can be built just by typing "make".

$G

2010-04-19T15:33:14Z

Trevor O'Brien:

{{wikify}}

== $G in principle ==
GfxTools a.k.a $G is a software development framework for keeping projects organized and usable over time. Several of its key goals are:
# Standardization - using a common development framework makes it a lot easier for folks within our group to pitch in with development, use tools others have created, pick up someone else's project after he/she graduates, and make sure that your own project lives on after you graduate.
# Dependable demos and software libraries - $G provides an environment where software libraries and demos can be installed and used by the group.
# Easy cross-platform development - $G utilizes the same software organization and Makefiles across multiple development platforms.
# Source control for software projects - $G uses CVS to manage source projects.

== [[$G in practice]] ==

Here's how $G works: Whenever you start a shell, the environment variable "G" is set to point to the root of the $G directory tree. To change directories to go there, you would type "cd $G", hence the name "$G". The directory that $G points is actually different depending on the architecture of the machine that you are on, this means that you can trust that everything within $G/bin will run on your machine. Likewise, libraries in $G/lib are compiled for your machine/OS, etc..

There is a Unix-like directory structure within $G. Here's a description of the directories you will find inside each architecture specific $G tree:

* $G/bin: Installed executables and scripts.
* $G/data: A symlink to a data directory that is common amoung all the $G trees. Sci-vis data is organized and stored inside here.
* $G/doc: Project documentation in non-man format. Typically, html generated from javadoc or doxygen, README files for demos, etc..
* $G/help: Contains $G specific documentation, like this webpage.
* $G/lib: Installed libraries, shared libraries, and program resource files like config files, icons, sounds, etc.. inside project specific subdirectories.
* $G/man: Project documentation in unix man page format.
* $G/src: CVS controlled projects in subdirectories.

''While this page is under construction, you might find the [http://vis.cs.brown.edu/resources/doc/gfxtools-docs/index.html old online $G doc] useful.''

== [[Transition to new $G]] ==

----

'''In December 2008 [mailto:bb@cs.brown.edu Bradley Berg] began redesigning $G and the documentation on the transition can be found [[Dollarg 2009|here]].'''

----

[[Category:$G]][[Category:Software Development]][[Category:IT Tools]]

Write and run software tests

2010-04-19T15:25:31Z

Trevor O'Brien: Write and run software tests moved to Software tests

#REDIRECT [[Software tests]]

Software tests

2010-04-19T15:25:31Z

Trevor O'Brien: Write and run software tests moved to Software tests

== Running Tests ==

The test driver is installed on all CS department machines. A configuration file is used to launch tests and mail results to subscribers. You can subscribe to a test summary or receive notices when select tests fail. The default configuration file is: <tt>$G/common/build/site/test.platform.cfg</tt>

If you want to perform a private test run and suppress mail notifications run:

<pre>
> /systest/bin/vrl_test -noemail
</pre>

To run a set of select tests and specify alternative mail notifications (usually to yourself) you can run using a custom configuration file. Make a private copy of the default configuration file and modify it. The comments in the default configuration file describe the format. Then run using the new configuration:

<pre>
> /systest/bin/vrl_test -config <My work area>/<Custom configuration>.cfg
</pre>

Here is an example of a small custom configuration file.

<pre>
######################### test.platform.fast.cfg #############################
#
Developer: you@cs.brown.edu
Summary: none

nag none linux common/build/valid/test.project.py common/include/nag
libnr none linux common/build/valid/test.project.py common/gg/libnr
libgg none linux common/build/valid/test.project.py common/gg/libgg
libds none linux common/build/valid/test.project.py common/gg/libds
libggpp none linux common/build/valid/test.project.py common/gg/libggpp
libnum none linux common/build/valid/test.project.py common/gg/libnum
libcff2 none linux common/build/valid/test.project.py common/gg/libcff2
libmri none linux common/build/valid/test.project.py common/mri/libmri

#
####################### end test.platform.fast.cfg ##########################
</pre>

== Writing Tests ==

Create a make test target that runs a test that returns a zero status for success and a non-zero status for an error. The test should be able to run as a batch job (not interactive). Projects with multiple tests can use a target named <tt>test.<name></tt> for each individual test and a target <tt>alltest</tt> to run them all.

Test targets should limit themselves to commands supported on both linux and Windows or use Python scripts. Currently only the "rm -f" and "diff" commands are supported. As always, absolute path names are to be avoided.

Large raw input files such as MRI images should be copied to a directory named <tt>$GROOT/data/<project>/test/</tt>. Cooked data and test output should go to a test directory inside the project directory. Tests should clear output files from prior runs to avoid false positives. Regression tests that use diff to check output should comapare against a file named: <tt><root name>_base.<suffix></tt>

[[Category:HOWTO]][[Category:$G HOWTO]][[Category:Software Development]]

Coding conventions

2010-04-19T15:23:09Z

Trevor O'Brien:

$G in practice

2010-04-19T15:22:44Z

Trevor O'Brien: New page: == Development Environment == The graphics group has developed a system over many years that allows easy development of a large, shared base of software. We give the whole thing the nick...

== Development Environment ==

The graphics group has developed a system over many years that allows easy development of a large, shared base of software. We give the whole thing the nickname <tt>[[$G]]</tt> ("dollar gee"), named after the environment variable that represents the root of the source tree.

There source code our group has created is modified and backed up through a source control system called CVS (Concurrent Versioning System). You must have CVS set up properly in order to develop with our shared codebase.

Source control works by allowing an individual user to make structured copies of the master versions of files (this process is called "checking out" the files). The user then modifies his or her personal copies of the files, and then "checks in" the changes once they're certain that the code compiles and runs properly. The location where a user modifies a personal copy of shared code is called a "sandbox", because it is in some sense walled off from the master copy: destructive changes to the code in the sandbox do not affect the master copy, provided the user doesn't check them in.

=== Access to CVS ===

Environment variables are used to configure the development environment. General instructions for setting up environment variables and using them.

[[Setting Environment Variables]]

Linux: <code>CVSROOT is /map/gfx0/cvsroot/</code>

On Windows set CVSROOT to: <code>Y:\map\gfx0\cvsroot\</code>

=== Setting up <tt>$G</tt> ===

=== Creating a development sandbox ===

Your sandbox is the area where you work. It can be any directory, but the area <code>/map/gfx0/users/<Login Id></code> is set aside for students in the graphics group. Create a directory there to work in or if you'd like also create a subdirectory to work in. To create your own sandbox, do the following (the "<tt>></tt>" just indicates the beginning of each command):

<pre>
> cd /map/gfx0/users
> mkdir -p $USER
</pre>

You can get a private copy of all source code migrated to the [[dollarg 2009| new version of <tt>$G</tt>]] by executing the following command afterward:

<pre>
> cd <Sandbox>
> cvs co common project
</pre>

The '''common''' directory contains software shared by projects and the '''project''' directory contains student applications. There are two other directories of interest. One is the '''import''' directory which is where third party software is installed. The other is the '''install_linux''' or '''install_WIN32''' directory. This is where software that you build is installed when you perform a '''"make install"'''.

All of these directories can be arranged to suite your needs. If you are just getting started just create them all at the top level of your sandbox. For now just create the top level directories and populate them later.

<pre>
> mkdir import
> mkdir install_linux
</pre>

=== Setting Environment Variables ===

There are several environment variables to set up. You can set these up in your login scripts along with the setting for CVSROOT.

'''GARCH''': Designate the platform architecture.

GARCH = WIN32 # Use this setting on Windows.
GARCH = linux # Use this setting on Linux.

'''G''': This is used to access the G Build system and is set to the location of the common directory. By default it is also used to access the '''import''' directory. If your sandbox is at the top of your users' directory then set:

G = /map/gfx0/users/<login>

'''GROOT''': At Brown this variable is used to access these directories:

$GROOT/data # Contains large data sets.
$GROOT/users # Work areas for end users.

GROOT = /map/gfx0

'''GTO''': This is the location of the install directories. Usually you will want to keep them at the top of your sandbox then it will be the same as $G.

GTO = $G

Main Page

2010-04-19T15:18:57Z

Trevor O'Brien:

__NOTOC__ __NOEDITSECTION__

{| style="width:100%; background:#fcfcfc; margin-top:1.2em; border:1px solid #ccc; text-align:center; white-space:nowrap; color:#000;"
| 
<div style="font-size:162%; border:none; margin:0; padding:.1em; color:#000;">Welcome to the [[VrlWiki]],</div>
<div style="top:+0.2em; font-size:95%;">a [[wikipedia:Wiki|user-editable]] repository of information for the [http://www.brown.edu Brown University] [http://vis.cs.brown.edu Visualization Research Lab].</div>
<div id="articlecount" style="width:100%; text-align:center; font-size:85%;">There are [[Special:Statistics|{{NUMBEROFARTICLES}}]] articles so far in the wiki.</div>
Red links below indicate pages that should exist but haven't yet been created. If you know something, write something!
|}


{|style="margin:0px 0px 0px 0px; background:none;"
|class="MainPageBG" style="width:50%; border:1px solid #cef2e0; background:#f5fffa; vertical-align:top; color:#000;"|
{|width="100%" cellpadding="2" cellspacing="5" style="vertical-align:top; background:#f5fffa;"
! <h2 style="margin:0; background:#cef2e0; font-size:120%; font-weight:bold; border:1px solid #a3bfb1; text-align:left; color:#000; padding:0.2em 0.4em;">[[:Category:VRL|VRL]]</h2>
|-
|style="color:#000;"|
* [[VRL|About the VRL]]
* [[Plans and Goals]]
* [[Nascent Papers]]
* [[Job Jar]]
* [[Vis Nuggets 08]]
* [[SciVis Meetings|Research Group Meetings]]
* [[Advising Meetings]]
* [[SciVis Reading Group]]
* [[Courses]]
* [[Demo Menu]]
|-
! <h2 style="margin:0; background:#cef2e0; font-size:120%; font-weight:bold; border:1px solid #a3bfb1; text-align:left; color:#000; padding:0.2em 0.4em;">[[:Category:IT Tools|IT Tools]]</h2>
|-
|style="color:#000;"|
* [[VrlWiki]]
* [[Visweb]]
* [[$G]]
* [[Source Control]]
* [[Build System]]
* [[Computing Resources]]
|-
! <h2 style="margin:0; background:#cef2e0; font-size:120%; font-weight:bold; border:1px solid #a3bfb1; text-align:left; color:#000; padding:0.2em 0.4em;">[[:Category:Projects|Projects]]</h2>
|-
|style="color:#000;"|
* Interdisciplinary Research
** [[2D Scalar Fields]]
** [[Art and Visualization]]
** [[Bats]]
** [[Chewing]]
** [[Diffusion MRI]]
** [[Fluid Flow]]
** [[Protein Networks]]
** [[Wrists]]
* [[:Category:Software|Support Software]]
** [[G3D|G3D/VRG3D]]
** [[Gluebase]]
** [[Inspace]]
* [[:Category:Hardware|Support Hardware]]
** [[CAVE]]
** [[New CAVE Design]]
** [[Powerwall]]
** [[Fishtank]]
** [[Wheatstone]]
|}
|style="border:1px solid transparent"|
|class="MainPageBG" style="width:50%; border:1px solid #cedff2; background:#f5faff; vertical-align:top;"|
{| width="100%" cellpadding="2" cellspacing="5" style="vertical-align:top; background:#f5faff;"
! <h2 style="margin:0; background:#cedff2; font-size:120%; font-weight:bold; border:1px solid #a3b0bf; text-align:left; color:#000; padding:0.2em 0.4em;">[[:Category:HOWTO|HOWTO]]</h2>
|-
|style="color:#000;"|
* Research
** [[Find a project]]
** [[Write a paper]]
** [[Do a literature search]]
** [[Find collaborators]]
** [[Get certified to work with human subjects]]
** [[Get certified to run an MRI scanner]]
** [[Store data]]
** [[Statistics And Graphs|Analyze Data]]
** [[Write a grant proposal]]

* [[Software Development]]
** [[Coding conventions|Coding conventions and best practices]]
** [[$G in practice|Get started with $G]]
** [[Check out projects|Check out existing projects]]
** [[Contribute to projects|Contribute to existing projects]]
** [[Add a project]]
*** [[Add a new project]]
*** [[Port existing project|Port existing project to the new $G]]
** [[Software tests|Create and automate software tests]]
** [[Software demos|Create and install software demos]]
** [[Distribute software|Distribute software and data]]
** [[Install 3rd party code|Install 3rd-party programs or libraries]]

* Dissemination
** [[Pick a destination for your project]]
** [[Submit a paper]]
** [[Make a poster]]
** [[Give a talk]]
** [[Run a demo]]
** [[Distribute software tools]]

* Finishing Up
** [[Find a job]]
** [[Propose your PhD]]
** [[Defend you PhD]]
|}
|}

Software Development

2010-04-19T15:18:36Z

Trevor O'Brien:

* [[Coding conventions|Coding conventions and best practices]]
* [[$G in practice|Get started with $G]]
* [[Check out projects|Check out existing projects]]
* [[Contribute to projects|Contribute to existing projects]]
* [[Add a project]]
** [[Add a new project]]
** [[Port existing project|Port existing project to the new $G]]
* [[Software tests|Create and automate software tests]]
* [[Software demos|Create and install software demos]]
* [[Distribute software|Distribute software and data]]
* [[Install 3rd party code|Install 3rd-party programs or libraries]]

Coding conventions and best practices

2010-04-19T15:18:15Z

Trevor O'Brien: Coding conventions and best practices moved to Coding conventions: Long page names are lousy.

#REDIRECT [[Coding conventions]]

Coding conventions

2010-04-19T15:18:15Z

Trevor O'Brien: Coding conventions and best practices moved to Coding conventions: Long page names are lousy.

Just about everyone in the [[VRL]] ends up re-using code and programs that someone else wrote previously, and undoubtedly the code that they write will be used by someone else in the future. Therefore the golden rule applies well to software development in our lab: do unto others as you would have them do unto you. This page records best practices for getting started developing software in an environment where code lives on through many generations of researchers.

The primary things to keep in mind are that:
* Future users of your code need read and write permission; and
* Future users of your code can't read your mind, so it must be organized and well-documented.

== Quick Start Instructions ==

If you are comfortable in setting up your own programming environment you can follow these quick start instructions.

[[Quick Start for Linux in CIT]]

== File Permissions ==
On Linux, there are three basic permissions that may be modified for a file: '''r''', '''w''', and '''x''', which stand for '''r'''ead, '''w'''rite, and e'''x'''ecute. For a given file, the permissions may be different for three different classes of users, '''u''', '''g''', and '''o''', which stand for the '''u'''ser who owns the file, other members of the owner's '''g'''roup, and '''o'''ther users. Therefore there are a total of nine permission bits on each file. Each three-bit '''rwx''' code may be represented in binary as a number between 0 and 7:

{| border=1 | align=center | cellpadding=3px
|-
! # !! r !! w !! x !! Permission
|-
! 0
| 0 || 0 || 0 || No access
|-
! 1
| 0 || 0 || 1 || Execute only
|-
! 2
| 0 || 1 || 0 || Write only
|-
! 3
| 0 || 1 || 1 || Write and execute
|-
! 4
| 1 || 0 || 0 || Read only
|-
! 5
| 1 || 0 || 1 || Read and execute
|-
! 6
| 1 || 1 || 0 || Read and write
|-
! 7
| 1 || 1 || 1 || Read, write, and execute
|}

The entire set of permissions for a file or directory may therefore be compactly represented by a three-digit octal number. For example, <tt>754</tt> means that the owner may read, write, and execute the file, other members of the owner's group may read and execute it, and all others may only read.

=== Default permissions for new files ===

When a new file is created, it starts off having permissions <tt>666</tt> (read and write allowed for all users). For directories, the execute permission means that the user can access the directory, and so the initial permissions on a new directory are <tt>777</tt> (read, write, and access allowed for all users).

These initial permissions are modified by what's called the "umask" or "file mode creation mask". This is a nine-bit binary mask that is XOR'd with the initial permissions. By default, the umask is <tt>022</tt>, and therefore a new file will come out with permissions <tt>644</tt> = <tt>666 ^ 022</tt> (read and write for the owner, read only for all others).

Every member of our research group is a member of the <tt>graphics</tt> group on the CS department network. In order to facilitate code sharing and collaboration, we need to make sure that read and write permissions are set not just for the owner of a file, but for all members of the <tt>graphics</tt> group, too. This is easily accopmlished by setting the umask to <tt>002</tt> instead: <tt>666</tt> ^ <tt>002 = 664</tt> (read and write for the owner and members of the owner's group, read only for all others).

The <tt>umask</tt> command either views or sets the current value of the umask. The config file <tt>~/.environment</tt> is run every time you open a shell, and the default one created by the CS department for new users sets the umask to <tt>022</tt>. To set the umask to <tt>002</tt> at the beginning of every shell session, just edit your <tt>~/.environment</tt> file. Look for the line
<pre width="30%">umask 022</pre>
and change it to
<pre width="30%">umask 002</pre>

To make your change take effect immediately, execute the following command in all your open shells:
<pre width="30%">source ~/.environment</pre>
If that gives you an error, try the following instead:
<pre width="30%">. ~/.environment</pre>
You can check that it worked by running <tt>umask</tt>; the result should be <tt>002</tt>. From now on, any files you create from the command line will be writeable by the group.

=== Commands you should know for dealing with permissions ===
Type <code>man <command></code> at the command line to get information about any of the following:
* <tt>umask</tt> --- view or set the umask
* <tt>chmod</tt> --- change the permissions for a file or directory
* <tt>chown</tt> --- change the owner of a file or directory (only allowed if you are the owner)

== Development Environment ==

The graphics group has developed a system over many years that allows easy development of a large, shared base of software. We give the whole thing the nickname <tt>[[$G]]</tt> ("dollar gee"), named after the environment variable that represents the root of the source tree.

There source code our group has created is modified and backed up through a source control system called CVS (Concurrent Versioning System). You must have CVS set up properly in order to develop with our shared codebase.

Source control works by allowing an individual user to make structured copies of the master versions of files (this process is called "checking out" the files). The user then modifies his or her personal copies of the files, and then "checks in" the changes once they're certain that the code compiles and runs properly. The location where a user modifies a personal copy of shared code is called a "sandbox", because it is in some sense walled off from the master copy: destructive changes to the code in the sandbox do not affect the master copy, provided the user doesn't check them in.

=== Access to CVS ===

Environment variables are used to configure the development environment. General instructions for setting up environment variables and using them.

[[Setting Environment Variables]]

Linux: <code>CVSROOT is /map/gfx0/cvsroot/</code>

On Windows set CVSROOT to: <code>Y:\map\gfx0\cvsroot\</code>

=== Setting up <tt>$G</tt> ===

=== Creating a development sandbox ===

Your sandbox is the area where you work. It can be any directory, but the area <code>/map/gfx0/users/<Login Id></code> is set aside for students in the graphics group. Create a directory there to work in or if you'd like also create a subdirectory to work in. To create your own sandbox, do the following (the "<tt>></tt>" just indicates the beginning of each command):

<pre>
> cd /map/gfx0/users
> mkdir -p $USER
</pre>

You can get a private copy of all source code migrated to the [[dollarg 2009| new version of <tt>$G</tt>]] by executing the following command afterward:

<pre>
> cd <Sandbox>
> cvs co common project
</pre>

The '''common''' directory contains software shared by projects and the '''project''' directory contains student applications. There are two other directories of interest. One is the '''import''' directory which is where third party software is installed. The other is the '''install_linux''' or '''install_WIN32''' directory. This is where software that you build is installed when you perform a '''"make install"'''.

All of these directories can be arranged to suite your needs. If you are just getting started just create them all at the top level of your sandbox. For now just create the top level directories and populate them later.

<pre>
> mkdir import
> mkdir install_linux
</pre>

=== Setting Environment Variables ===

There are several environment variables to set up. You can set these up in your login scripts along with the setting for CVSROOT.

'''GARCH''': Designate the platform architecture.

GARCH = WIN32 # Use this setting on Windows.
GARCH = linux # Use this setting on Linux.

'''G''': This is used to access the G Build system and is set to the location of the common directory. By default it is also used to access the '''import''' directory. If your sandbox is at the top of your users' directory then set:

G = /map/gfx0/users/<login>

'''GROOT''': At Brown this variable is used to access these directories:

$GROOT/data # Contains large data sets.
$GROOT/users # Work areas for end users.

GROOT = /map/gfx0

'''GTO''': This is the location of the install directories. Usually you will want to keep them at the top of your sandbox then it will be the same as $G.

GTO = $G

== Development Best Practices ==

=== Minimal Coding Standards ===

* Tabs are only meaningful in your edit session.
Do not use tabs (except in Makefiles as needed).
Indent 4 spaces, but 3 is okay if you prefer.

* Name script files with suffixes.
Use Python for scripts (very portable).
Avoid reliance on Cygwin.

* Commenting code does not slow progress.
If you make it a habit it speeds things up.
At a minimum write a method’s purpose and parameter usage.

* Linux file system links are not portable. Never use them.

=== Coding in C++ ===

* Use .cpp and .h suffixes for c++ source file names. Not: .C and .H

* Avoid conditional compilation (#ifdef).

* Use brackets around code bodies over multiple lines.

<pre>
Evil: if (condition)
action;

Good: if (condition) action; if (condition) { action };

if (condition) if (condition)
{ action; {
} action;
}
</pre>

* Line up brackets so they are easier to match visually.

* Do not #include .cpp files.

[[Category:HOWTO]][[Category:$G HOWTO]][[Category:Software Development]]

Get started on software development

2010-04-19T15:17:40Z

Trevor O'Brien: Get started on software development moved to Coding conventions and best practices: Reorganization of the software development section of the vrl home page.

#REDIRECT [[Coding conventions and best practices]]

Coding conventions

2010-04-19T15:17:40Z

Trevor O'Brien: Get started on software development moved to Coding conventions and best practices: Reorganization of the software development section of the vrl home page.

Software Development

2010-04-19T15:16:35Z

Trevor O'Brien: New page: * Coding conventions and best practices * Get started with $G * Check out existing projects * [[Contribute to projects|Cont...

* [[Coding Conventions|Coding conventions and best practices]]
* [[$G in Practice|Get started with $G]]
* [[Check out projects|Check out existing projects]]
* [[Contribute to projects|Contribute to existing projects]]
* [[Add a project]]
** [[Add a new project]]
** [[Port existing project|Port existing project to the new $G]]
* [[Software tests|Create and automate software tests]]
* [[Software demos|Create and install software demos]]
* [[Distribute software|Distribute software and data]]
* [[Install 3rd party code|Install 3rd-party programs or libraries]]

Main Page

2010-04-19T15:15:35Z

Trevor O'Brien:

__NOTOC__ __NOEDITSECTION__

{| style="width:100%; background:#fcfcfc; margin-top:1.2em; border:1px solid #ccc; text-align:center; white-space:nowrap; color:#000;"
| 
<div style="font-size:162%; border:none; margin:0; padding:.1em; color:#000;">Welcome to the [[VrlWiki]],</div>
<div style="top:+0.2em; font-size:95%;">a [[wikipedia:Wiki|user-editable]] repository of information for the [http://www.brown.edu Brown University] [http://vis.cs.brown.edu Visualization Research Lab].</div>
<div id="articlecount" style="width:100%; text-align:center; font-size:85%;">There are [[Special:Statistics|{{NUMBEROFARTICLES}}]] articles so far in the wiki.</div>
Red links below indicate pages that should exist but haven't yet been created. If you know something, write something!
|}


{|style="margin:0px 0px 0px 0px; background:none;"
|class="MainPageBG" style="width:50%; border:1px solid #cef2e0; background:#f5fffa; vertical-align:top; color:#000;"|
{|width="100%" cellpadding="2" cellspacing="5" style="vertical-align:top; background:#f5fffa;"
! <h2 style="margin:0; background:#cef2e0; font-size:120%; font-weight:bold; border:1px solid #a3bfb1; text-align:left; color:#000; padding:0.2em 0.4em;">[[:Category:VRL|VRL]]</h2>
|-
|style="color:#000;"|
* [[VRL|About the VRL]]
* [[Plans and Goals]]
* [[Nascent Papers]]
* [[Job Jar]]
* [[Vis Nuggets 08]]
* [[SciVis Meetings|Research Group Meetings]]
* [[Advising Meetings]]
* [[SciVis Reading Group]]
* [[Courses]]
* [[Demo Menu]]
|-
! <h2 style="margin:0; background:#cef2e0; font-size:120%; font-weight:bold; border:1px solid #a3bfb1; text-align:left; color:#000; padding:0.2em 0.4em;">[[:Category:IT Tools|IT Tools]]</h2>
|-
|style="color:#000;"|
* [[VrlWiki]]
* [[Visweb]]
* [[$G]]
* [[Source Control]]
* [[Build System]]
* [[Computing Resources]]
|-
! <h2 style="margin:0; background:#cef2e0; font-size:120%; font-weight:bold; border:1px solid #a3bfb1; text-align:left; color:#000; padding:0.2em 0.4em;">[[:Category:Projects|Projects]]</h2>
|-
|style="color:#000;"|
* Interdisciplinary Research
** [[2D Scalar Fields]]
** [[Art and Visualization]]
** [[Bats]]
** [[Chewing]]
** [[Diffusion MRI]]
** [[Fluid Flow]]
** [[Protein Networks]]
** [[Wrists]]
* [[:Category:Software|Support Software]]
** [[G3D|G3D/VRG3D]]
** [[Gluebase]]
** [[Inspace]]
* [[:Category:Hardware|Support Hardware]]
** [[CAVE]]
** [[New CAVE Design]]
** [[Powerwall]]
** [[Fishtank]]
** [[Wheatstone]]
|}
|style="border:1px solid transparent"|
|class="MainPageBG" style="width:50%; border:1px solid #cedff2; background:#f5faff; vertical-align:top;"|
{| width="100%" cellpadding="2" cellspacing="5" style="vertical-align:top; background:#f5faff;"
! <h2 style="margin:0; background:#cedff2; font-size:120%; font-weight:bold; border:1px solid #a3b0bf; text-align:left; color:#000; padding:0.2em 0.4em;">[[:Category:HOWTO|HOWTO]]</h2>
|-
|style="color:#000;"|
* Research
** [[Find a project]]
** [[Write a paper]]
** [[Do a literature search]]
** [[Find collaborators]]
** [[Get certified to work with human subjects]]
** [[Get certified to run an MRI scanner]]
** [[Store data]]
** [[Statistics And Graphs|Analyze Data]]
** [[Write a grant proposal]]

* [[Software Development]]
** [[Coding Conventions|Coding conventions and best practices]]
** [[$G in Practice|Get started with $G]]
** [[Check out projects|Check out existing projects]]
** [[Contribute to projects|Contribute to existing projects]]
** [[Add a project]]
*** [[Add a new project]]
*** [[Port existing project|Port existing project to the new $G]]
** [[Software tests|Create and automate software tests]]
** [[Software demos|Create and install software demos]]
** [[Distribute software|Distribute software and data]]
** [[Install 3rd party code|Install 3rd-party programs or libraries]]

* Dissemination
** [[Pick a destination for your project]]
** [[Submit a paper]]
** [[Make a poster]]
** [[Give a talk]]
** [[Run a demo]]
** [[Distribute software tools]]

* Finishing Up
** [[Find a job]]
** [[Propose your PhD]]
** [[Defend you PhD]]
|}
|}

Plans and Goals

2010-01-12T18:36:10Z

Trevor O'Brien: /* Current Plans and Goals (Spring '10) */

On this page the members of the [[VRL]] record and refine their goals for the current semester. This is a living document in which [[dhl]] will provide feedback. See the bottom of the page for links to past plans & goals documents.

== Current Schedule ==
Meetings are on Tuesdays.

* 1:50 - jadrian
* 2:00 - wenjin
* 2:10 - eni
* 2:20 - jadrian
* 2:30 - bb
* 2:40 - dtr
* 2:50 - trevor
* 3:00 - cad
* 3:10 - steve
* 3:20 - radu

== Current Plans and Goals (Spring '10) ==

=== [[User:Brad Berg|Brad]] ===

Projects

* Support the ongoing projects including collaborators. Active projects are wrist, brain, adviser, and protein.

* Assist Radu to integrate a test and any recent code.

* Contribute to the design and development of the new cave. This includes integrating g3d, vrpn and hosting vrg3d. Once the software is running simplify the configuration and provide a quick start path for student projects.

* Set up a distribution of diffusion imaging data.

Infrastructure

* Continue to write Wiki entries as needed.

* Continue setting up external software for students to use. Current imported packages are: blitz, cg, g3d, qt, camino, dtk, vrpn, glut, and sdl.

* Finish extending make files for local installs and kits for source and binary distributions.

* Continue to make improvements to the make files so they are easier to use.

* Upgrade migrated Linux code to work with gcc 4.3.

* Continue to integrate any new tests written by students into nightly test runs.

* Finish integrating the new version of the Nag libraries.

Knowledge

* Learn more about the cave; particularly vrg3d software development.

* Meet (SciVis talk) with students to address any integration issues.

* Learn how to manage a project using Source Forge.

----
=== David ===
#get diffusion normals, one library, and one executable distributed
#wrist proposal submission (check with Trey?)
#support 5 phd students' + trevor's progress
#write a plan for HCI modeling work
#minimize professional service
----

=== [[User:Steven Gomez|Steve]] ===
# Staying on track for candidacy
## Complete programming comp (late January)
## Complete research comp proposal and presentation to committee (mid April)
## Do well in courses
# Research goals
## Explore current research in HCI related to creativity, and user interaction modeling
## Identify significant, well-scoped research project in this area for the research comp proposal
## Develop research skills, including paper reading/analysis and learning to communicate effectively
# Other academic goals
## Work on collaboration skills (finding and communicating with colleagues in or outside of CS)
## Work on time management and project planning
## Learn what working with DHL is like, his expectations, etc.
## Plan for the summer
----

=== [[User:Trevor O'Brien|Trevor]] ===

# I3D poster & demo
## Revised abstract: January 29
## Conference: February 19th (poster complete by February 17)
# Vis Papers - Iterate often
## Abstract: March 21
## Full paper: March 31
### Gene visualization paper
### Confocal microscopy paper
# Genome Research paper
# Grad TA for CS224
# Paper with Paul Kulesa
## Define goals contributions from Vis perspective
# Look for themes between work with Paul, Trey, and maybe Ben
## Coordinate spaces? (Tensor invariances?)
## Morphometrics? (Non-dimensionalization)
## Other feature detection?
# Organize code, papers, tools, other resources to be backed up, handed off

=== Wenjin ===
# Submit to MICCAI (March 11th) + good collaboration with Matt
# Prepare histology data - do homework about the right path to choose, acquire, and prepare the tissue
# Derive and test optimal double-PGSE imaging on Brown 3T scanner
# Propose by 10/15
# Prepare for MRM/TMI journal submission

== Current Plans and Goals (Fall '09) ==

=== Çağatay ===
# Incorporate feedback from the committee into the[[:Image:Çağatay's_thesis_ideas.pdf | proposal]]
# Start working realizing the finalized proposal.
# Submit the histogram embedding work to ISMRM.
# Submit the surface deformation work to SIGGRAPH.
----

=== [[User:Andy Forsberg|Andy]] ===
# mri/wrist
## tools
### registration, classification, distance-field calculation, bone proximity calc.
### graphical visualization/verification?
## modeling?
# $G (nightly tests, architecture, outside test-case = ADVISER)
# future funding candidates
## volume data visualization/analysis
## flow visualization w/ leopold/GK
## other?

----

=== [[User:Brad Berg|Brad]] ===

Projects

* Support the ongoing projects including collaborators. Active projects are wrist, brainapp, adviser, and protein.

* Assist Radu to integrate a test and any recent code.

* Contribute to the design of the new cave.

Infrastructure

* Continue to write Wiki entries as needed.

* Continue setting up external software for students to use. Current imported packages are: blitz, cg, g3d, qt, and xerces

* Extend make files for local installs and kits for source and binary distributions.

* Continue to make improvements to the make files so they are easier to use and to provide Windows support.

* Upgrade migrated Linux code to work with gcc 4.3.

* Continue to integrate any new tests written by students into nightly test runs.

* Integrate the new version of the Nag libraries.

Knowledge

* Learn more about the cave; particularly software development.

* Meet (SciVis talk) with students to address any integration issues.

----

=== Çağatay ===
# Incorporate feedback from the committee into the[[:Image:Çağatay's_thesis_ideas.pdf | proposal]]
# Start working realizing the finalized proposal.
# Submit the histogram embedding work to ISMRM.
# Submit the surface deformation work to SIGGRAPH.
----

=== David ===
#figure out software and data dissemination and get implemented with bb's help [some progress, more next semester]
#wrist proposal submission [not sure where this went with Trey]
#harvard stuff moving; read Janeway? [good meeting in Boston]
#support 6 phd students' progress [all proposals done!]
#write a plan for HCI modeling work (draft by 10/23, rev by 11/20) [not enough structure]
#minimize professional service [less successful than hoped]
#vis2011 sited, early committee defined after Vis by 10/16 [contract signed, delaying committee]
----

=== Eni ===

*submit tract integrity vs. cognitive tests abstract and histograms abstract to ismrm
*possibly write a paper on the wrist comparison study (old/new subjects)
*while working on different wrist problems, think of a problem I might want to focus on
*do well in the courses I'm taking

----

=== [[User:Jadrian Miles|Jadrian]] ===
[[:Image:Jadrian Miles PhD Proposal original 2009-06-22.pdf‎|Initial thesis proposal, submitted 2009-06-22]]

# Incorporate committee's comments into my proposal by 2009-10-15
#* Revision complete 2009-10-15
# Make a list of papers to spawn from the proposal by 2009-10-15
#* {{red|Incomplete}}
# Decide with dhl on the role of animal models in the proposal
#* Histological "wetwork" will play an important role in the research; significant planning and collaboration-building took place this semester
# Finish clustering code sufficient for an initial version of the backsolving system
#* {{red|Incomplete}}
# Find or build implementations of a more sophisticated tractography technique (Q-ball or Gordon's multi-tensor)
#* DTK & Trackvis installed and working with current data as of 2009-11-04
# Work with Ed Walsh, dhl, and wzhou to settle on some HARDI parameters
#* {{red|Scan parameters have not been decided}} but ongoing work with Steve's group is teaching us all more about parameter tradeoffs
# Take one HARDI scan by 2009-11-24
#* {{red|No new scan was acquired this fall}}, pending a decision about scan parameters
# Get DifSim (the MCell-based UCSD diffusion simulator) installed and working here
#* {{red|Not installed}} due to ongoing license issues, but I'm in communication with the developers and confident that it'll work out

----

=== Radu ===
# Deploy existing visualization tools by Oct 15 (in Labs and in CVS - Protein-version2, Clustering apps)
# Migrate BrainApp functionality to the new framework by Nov 1 (help from Cagatay and Eni)
# Test usefulness of the QT vis framework (on Cagatay); if useful deploy by Nov 1
# Advance Immgen
## Write report for next year
## Daphne-data vis prototype by Nov 1
## Genome-google maps operation by Oct1
## Broad display operational
## Paper on analyzing and disseminating large visualizations?
# Gather vis tools usage data
# Proposal
## Gather, read and summarize cogsci and linked visualization literature
# 2D interaction on stream-tubes study? Paper?
# Successful Vis talk
# Proposal is [http://cs.brown.edu/people/jr/proposal_jr.pdf here]

----

=== Trevor ===

# Research comps!
# Review and address EuroVis reviews for cell paper.
## Submit to PacVis?
# Research comps work --> i3d submission.
# Narrow long term focus.
# Foster collaborations:
## Paul Kulesa
## Trey Crisco
## Sharon Swartz
# Finish course requirements.

----

=== Wenjin ===

* Finalize [http://cs.brown.edu/~wzhou/proposal_wzhou.pdf Proposal]: incorporate feedback from committee in NIH format
* Survive NEUR 1650
* Analytical model tested on simulation data to accomplish 1-2 of the following (based on current limitations):
** estimate axon parameter for unknown fiber direction
** recover small axon radii < 4 micrometer
** estimation with clinically achievable acquisitions
** build and evaluate 4-compartment model with water exchange vs. 2-compartment
* Derive and test optimal imaging parameters on Brown 3T scanner
* submit to ISMRM (Nov. 10th) or MICCAI in March
* prepare for MRM/TMI journal submission

=== [[User:Steven Gomez|Steve]] ===

Course work
* Explore courses that may be relevant to research interests, or may help me think more outside-the-box
* Use project classes to explore interests in graphics, vis, hci

Research
* Draft R&R plan for spring semester by 11/15
* Start thinking about research comp ideas and proposal by 11/15
* Contribute to the VRL however I can
* Search through past ideas/projects worth investigating more by 10/1
** Online raytracing with voronoi diagrams?
** Interfaces that learn?

Personal
* Get adjusted to Brown, Providence, graduate school
** Work on time management, organization
* Pass programming comp in January

== Past Plans and Goals ==
* [[/Summer 2009|Summer '09]]
* [[/Spring 2009|Spring '09]]
* [[/Fall 2008|Fall '08]]
* [http://sites.google.com/a/vis.cs.brown.edu/collaboravis/Home/summer-08-group-goals Summer '08]

[[Category:VRL]]

Plans and Goals

2009-09-14T18:43:05Z

Trevor O'Brien: /* Trevor */

Plans and Goals

2009-05-04T18:15:23Z

Trevor O'Brien: /* Trevor */

On this page the members of the [[VRL]] record and refine their goals for the current semester. This is a living document in which [[dhl]] will provide feedback. See the bottom of the page for links to past plans & goals documents.

== Current Schedule ==
Meetings are on Monday.

* 2:00 - [[User:Andrew_Forsberg/Meeting_Notes|asf]]
* 2:10 - jadrian
* 2:20 - radu
* 2:30 - Jian
* 2:40 - dtr
* 2:50 - trevor
* 3:00 - wzhou
* 3:10 - Eni
* 3:20
* 3:30 - Brad
* 3:40 - cad
* 3:50 - Laura

== Current Plans and Goals ==

=== [[User:Andy Forsberg|Andy]] ===
# comparison of 3d vector vis methods
## run subjects, analysis, complete paper, submit to vis'09
# mri/wrist
## tools
### registration, classification, distance-field calculation, bone proximity calc.
### graphical visualization/verification?
## modeling?
# Potentially get involved in protein project, possibly pen UIs
# $G (nightly tests, architecture, outside test-case = ADVISER)
# future funding candidates
## volume data visualization/analysis
## flow visualization w/ leopold/GK
## other?

Important dates
# 2/27 finish flow eval paper (Vis09 abstracts due 3/21, papers due 3/31) http://vis.computer.org/VisWeek2009/vis/
# 2/27 - 3/1 - I3D in Boston http://graphics.cs.williams.edu/i3d09/index.html
# 4/15 - SBIM http://sbim09.cse.wustl.edu/pages/importantDates.php

----

=== [[User:Brad Berg|Brad]] ===

Projects

* Migrate all of the wrist project before Evan leaves.

* Support collaborators for the Protein project.

* Continue to migrate active projects.

* Begin migrating shared libraries. Some will need to be migrated in conjunction with projects.

Infrastructure

* Write Wiki entries for creating and building projects.

* Identify external software for students to use. Currently students download software based on their immediate needs. By providing support for a small number of external packages it will make it possible to distribute their software faster and with less effort. Students will have better support when they use common external software packages.

* Extend make files for tests, local installs and kits for source and binary distributions.

* Integrate new tests into nightly test runs. This might involve upgrading the nightly test mechanism. Tests will need to be written. In order for students to write tests clear and simple procedures need to be in place.

Knowledge

* It might be useful for me to sit in on CSCI 2950J.

* As project migrations complete make documentation available so that new student developers can easily get started.

* Provide support for novice CVS users. Students should be able to check code in and out of CVS with a minimum of effort. This will not only make it easier for students to participate, but will also protect the code base. It is very easy to unintentionally damage files in CVS and the damage might go undetected for some time. Recovery can be difficult.

----

=== Çağatay ===
# Present the phd thesis proposal.
# Submit reeb graph of dt invariants to miccai (done -- simcolor work submitted instead)
# Submit line-field coloring to vis (done -- two papers submitted, as a result of a fruitful collaboration w/ Radu)
# Submit surface deformation to a journal.
----

=== David ===
#cs295J (HCI+COG+VIS) runs smoothly: new research directions, class goals met
#improve group infrastructure: sw, data, dissemination, tools (with help from bb, asf)
#proto-papers incubator
#more brain project deliverables; paper skeletons
#trey stuff happy [distance pipeline ok, modeling needs attn next semester]
#joslin stuff happy
#support group members' goals and growth, particularly impact and products
#refine 5-year funding/sabbatical plan – write some proposals
#keep up on dept/Brown service (search, facil, brdd, academic advising, mrf)
#keep up on professional service

----

=== Eni ===

*publish a full metrics paper
*start working on the wrist project
*publish a paper on wrist work
*improve programming skills
*possibly get involved in another brain project, once metrics paper is sent out

----

=== [[User:Jadrian Miles|Jadrian]] ===
[[User:Jadrian Miles/PhD Plan]] provides more context for my goals and also serves as a working document.

# Develop a research plan that works as a spiral, not a waterfall
# Present a draft PhD proposal to dhl by 2/2/09
# Present a draft abstract to dhl by 2/2/09

----

=== Jian ===

----
Deliverables:
* Devon's paper by March 31st (vis deadline)
** full draft by March 12th
** iterations till the deadline
* Andy's paper by March 31st (vis deadline)
* A book chapter by March 31st (on domain-specific design)
* The display paper to CG&A by March 31st (CG&A)
* consider a biology paper (by Apr. 10th?)
* consider a TVCG paper (on scalable time-varying data visualization and interaction (?on the samsung display? is there any reason to use it?)
Job:
* Get a job
Project:
* Finish camber measurement
* Finish visualization method
* Finish the vis tool
Proposal:
* Short-term proposal that has potential to extend my stay
Community services:
* Review SIGGRAPH / TVCG papers

=== Laura ===

----

=== Radu ===
Phd proposal
* first rough skeleton draft by Feb 1
* lit search and revised draft by March 1
* more lit search and final proposal by April 1
* presentation by May 1
Vis submission
* paper sketch by Feb 1
* iteration 1 (iteration = coded concepts, evaluation scenario and feedback) by Feb 15
* iteration 2 by Mar 1
* first write-up and iteration 3 by Mar 15
* submission
Immgen
*new techniques, interactions and approaches to exploring multi-dimensionality
* expansion - idea and application brainstorming
* continue with Bang at Broad (more or less in the background)
* extract insights from genomic viewer (more evaluation)
* Immgen progress report (current + future) - by Feb 1
* Immgen progress report (current) - by May 15
TVCG submission
* tackle revision/resubmission if I get response
Other conference paper
* write 2-3 proposals for ideas (by April 15)
* select one and write paper sketch by May 1
Other
* transform the weekly todo in a habit
* improve multi-tasking/planning abilities
* more responsive and less ball droping
* cleaner submissions (of answers, code, writeups etc)

----

=== Sidra ===

----

=== Trevor ===

# Distortion correction paper.
## Finalize content and evaluation.
## Submit to TMI? Consider where this will have the most impact.
# <s>Work toward Vis submission on interaction histories, collaborative scientific visualization.</s>
## <s>Explore possible applications.</s>
## <s>Develop evaluation criterion. </s>
## <s>Consider software model as a contribution.</s>
# <s>Research comps proposal. (Interaction histories?)</s>
## <s>Draft complete by mid-February. (by February I mean April)</s>
## <s>Practice talk at SciVis meeting February 24th.</s>
# <s>Evaluate eye-tracking systems available at Brown.</s>
## <s>Develop webcam-based eye-tracking?</s>
# <s>Continue to explore and evaluate intern opportunities.</s>
# <s>Get through classes.</s>
## <s>CS224, CS295-J, Reading and Research.</s>
# Improve time management.
# <s>Improve writing.</s>
## <s>Explore opportunities for short submissions.</s>

----

=== Wenjin ===

Ph.D. Proposal Draft
* Abstract ready 2/23
* Specific contributions 3/9
* Preliminary Studies 3/16
* Research Design and Methods 4/6
* Background and Significance 4/20
* full draft 5/4

Improve Axon Modeling and Validation with Histology, submit to MICCAI
* Improve axon modeling
* Validation with Macaque Histology Data

ISMRM conference
* deliver a good presentation and poster
* connect with people and seek for collaboration
* gather inspiring project ideas

Aim for MRM journal submission
* Histology validation stuff
* Improvement on modeling

Prepare to get teaching experience in the fall

=== Wudan ===

== Past Plans and Goals ==
* [[/Fall 2008|Fall '08]]
* [http://sites.google.com/a/vis.cs.brown.edu/collaboravis/Home/summer-08-group-goals Summer '08]

[[Category:VRL]]

CS295J/Final contributions

2009-05-01T16:25:38Z

Trevor O'Brien: /* Semantic-level Interaction Chunking */

= Specific Contributions =

[I have removed almost everything that does not describe itself as a contribution. If your part is missing, a revision is in order and you can find what the former version on the page linked below (David)]

[[CS295J/Thursday 2pm version of final contribution|Thursday 2pm version of final contribution]]

== Workflow, Multi-tasking and Interruptions ==
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

'''Aims'''
* Develop a Theory of Multi-tasking Workflow and the Impact of Interruptions
* Build a Meta-work Assistance Tool
:# We will perform a series of ecologically-valid studies to compare user performance between a state of the art/popular task management system (control group) and our meta-work assistance tool (experimental group) [Didn't your study show that there is nothing close to a killer app in this department? - Steven] [By "state of the art" I mean a popular, well-recognized system (or systems), such as Outlook, that are used by large numbers of people.]
* Evaluate the Module-based Model of HCI by using it to predict the outcome of the above-mentioned evaluation; then compare with empirical results

'''Contributions'''
* Theory of Multi-tasking Workflow and the Impact of Interruptions
:# We will undertake detailed studies to help understand the following questions:
::# How does the size of a user's working set impact interruption resumption time?
::# How does the size of a user's working set, when used for rapid multi-tasking, impact performance metrics?
::# How does a user interface which supports multiple simultaneous working sets benefit interruption resumption time?
:* Internal dependencies: none
* Empirical Evaluation of Meta-work Assistance Tool in an Ecologically-valid Context
:* Internal dependencies: Completion of a testable meta-work tool
* Baseline Comparison Between Module-based Model of HCI and Core Multi-tasking Study
:# We will compare the results of the above-mentioned study in multi-tasking against results predicted by the module-based model of HCI in this proposal; this will give us an important baseline comparison, particularly given that multi-tasking and interruption involve higher brain functioning and are therefore likely difficult to predict
:* Internal dependencies: Dependent on core study completion, as well as most of the rest of the proposal being completed to the point of being testable

'''Demonstration (we might consider calling these deliverables)'''
* A predictive, quantiative model of controlled task performance and interruption resumption time for varying sized working sets
* An ecologically valid evaluation which shows that the completed meta-work assistance tool outperforms a state of the art system such as Outlook
* A comparison between the result predicted by the module-based model of HCI and the evaluation mentioned above; accompanied by a detailed analysis of factors contributing to the result

'''External Dependencies'''
* Dependent on most of the rest of the proposal being completed to the point of being usable to predict the outcome of the meta-work tool evaluation

'''Background and Related Work'''
* Salvucci et al's integrated theory of multi-tasking continuum integrates concurrent and sequential task performance
* Gloria Mark's research group has conducted a number of empirical studies into multi-tasking and interruption

'''Preliminary Work'''

* [[Study of Meta-work Tools and Strategies of 7 Information Workers]]
:* Results suggest there is not a single "ideal" meta-work tool today, motivating our proposal to build a new meta-work support tool based in part on a scientific theory of multi-tasking and interruptions.

'''Reviews'''
* Not in the expected format -- split into appropriate parts and label them as in the other sections (David)

== Recording User-Interaction Primitives ==

* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

'''Aims'''
* Develop system for logging low-level user-interactions within existing applications. By low-level interactions, we refer to those interactions for which a quantitative, predictive, GOMS-like model may be generated.[How do you propose to get around the problem Andrew brought up of accurately linking this low-level data to the interface/program's state at the moment it occurred? - Steven] [Low-level interactions will need to be logged along with a markup-language-style state description of the application itself. This would either require access to the source code, or a detailed interface description from the interface designer that would allow for complete simulation of the application.]
* At the same scale of granularity, integrate explicit interface-interactions with multimodal-sensing data. i.e. pupil-tracking, muscle-activity monitoring, auditory recognition, EEG data, and facial and posture-focused video.

'''Contributions'''
* Extensible, multi-modal, HCI framework for recording rich interaction-history data in existing applications.[Perhaps something should be said about sectioning this data off into modules - i.e. compiling all e-mail writing recordings to analyze separately from e-mail sorting recordings in order to facilitate analysis. - Steven] [I believe this is addressed in the "semantic-level chunking" section, given below.]

'''Demonstration'''
* Techniques for pupil-tracking, auditory-recognition, and muscle-activity monitoring will be evaluated with respect to accuracy, sampling rate, and computational cost in a series of quantitative studies. [Is there an explicit, quantifiable demonstration you can propose? Such some kind of conclusive comparison between automated capture techniques and manual techniques? Or maybe a demonstration of some kind of methodical, but still low-level, procedure for clustering related interaction primitives? [[User:E J Kalafarski|E J Kalafarski]] 15:02, 29 April 2009 (UTC)] [Things like pupil-tracking can be quantitatively evaluated through users tests. For instance, we could instruct users to shift the focus of their eyes from one on-screen object to another as quickly as possible some fixed number of times. We could then compare the results from our system to determine accuracy and speed, and by varying the size and distance of the target objects, we could obtain richer data for analysis. With respect to clustering low-level interactions, I believe that issue is addressed in the following section.][Should this take into account the need to match recordings to interface state? - Steven] [Possibly, if we go down the simulation route. If we're actually logging all of the events from within the source code itself, I believe the demonstration of syncing low-level interaction with application-state is self-evident.]

'''Dependencies'''
* Hardware for pupil-tracking and muscle activity monitoring. Commercial software packages for such software may be required.

'''Background and Related Work'''
* The utility of interaction histories with respect to assessing interface design has been demonstrated in <ref name = bob> [http://www.cs.brown.edu/people/trevor/Papers/Heer-2008-GraphicalHistories.pdf Graphical Histories for Visualization: Supporting Analysis, Communication, and Evaluation] (InfoVis 2008) </ref>. Can you briefly summarize the utility with one sentence? Or is that what the next bullet does? (Jon) [In the instance of this work, utility was demonstrated with respect to automatically generating presentations from data explorations, and providing quantitative interface-usage analysis that aided future decisions for interface designers. i.e., this operations was performed a lot, and this one was not; therefore we should focus on making the first more efficient, perhaps at the expense of the second. The results of the work are fairly basic, but they do demonstrate that logging histories is a useful practice for a number of reasons.]
* In addition, data management histories have been shown effective in the visualization community in <ref> [http://www.cs.brown.edu/people/trevor/Papers/Callahan-2006-MED.pdf Callahan-2006-MED] </ref> <ref> [http://www.cs.brown.edu/people/trevor/Papers/Callahan-2006-VVM.pdf Callahan-2006-VVM] </ref> <ref> [http://www.cs.brown.edu/people/trevor/Papers/Bavoil-2005-VEI.pdf Bavoil-2005-VEI]</ref>, providing visualizations by analogy <ref> [http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=4376187&isnumber=4376125 Querying and Creating Visualizations by Analogy] </ref> and offering automated suggestions <ref> [http://www.cs.utah.edu/~juliana/pub/tvcg-recommendation2008.pdf VisComplete: Automating Suggestions from Visualization Pipelines] </ref>, which we expect to generalize to user interaction history.

''''review''''
* This is recording everything possible during an interaction. It is necessary to do for our project. (Gideon)
* Two thumbs up. - Steven

== Semantic-level Interaction Chunking ==

* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

'''Aims'''
* Develop techniques for ''chunking'' low-level interaction primitives into ''semantic-level'' interactions, given an application's functionality and data-context. (And de-chunking? Invertible mapping needed?)
* Perform user-study evaluation to validate chunking methods.

'''Contributions'''
* Design and user-study evaluation of semantic-chunking techniques for interaction.

'''Demonstration'''
* These techniques will be validated quantitatively with respect to efficiency and user error rate on a set of timed-tasks over a series of case studies. In addition, qualitative feedback on user satisfaction will be incorporated in the evaluation of our methods. [As above, how will this be validated quantitatively? Comparing two methodologies for manual chunking? What would these differing methodologies be? [[User:E J Kalafarski|E J Kalafarski]] 15:06, 29 April 2009 (UTC)] [I don't intend for these techniques to be performed manually. Application-specific rules for chunking will be created manually, most likely by the interface designer, but those rules will be applied automatically to filter low-level primitives and application-states into semantic-level interactions. The success of such methods could be evaluated by having automatically-generated semantic-level histories compared to those manually created by a user. Of course, there's no guarantee that what a user manually creates is any good, so I think it would be more effective to have a quantitative study of user performance in which timed-tasks are completed with and without the ability to review and revisit semantic-level histories. The test without any history at all would serve as a baseline, and then performance with histories created using different chunking rules would speak to the effectiveness of different strategies for chunking.]

'''Dependencies'''
* Software framework for collecting user interactions.[I'd like to see this fleshed out in a theoretical manner. - Steven] [I believe the previous section speaks to this question, but perhaps not with enough detail. It seems appropriate to have this discussion in class.]
* Formal description of interface functionality.
* Description of data objects that can be manipulated through interaction.

'''review'''
* With what level of accuracy can this be performed. I like the idea, its worthy of a nice project in and of itself. (Gideon) [This is a great question. It really depends on what's meant by accuracy. If the only level of semantics we want is that defined by the interface-designer, then it's not really an interesting question. As we begin to talk about semantics with respect to what the user ''perceives'' or what the user's ''intent'' might be, it becomes more interesting, and the term accuracy becomes more ambiguous. I suppose accuracy could be tested by having users verbally speak their intent as they use an interface, and compare their descriptions with the output of the system, but I think that might be a messy study. Interesting to think about.]
Can you include something about the broader impact of this contribution? (Jon) [I think the broader impact here is offering interface designers tools to analyze interaction at a high enough level that they can begin to observe user's intent with an interface, and potentially infer overarching user-goals. If these tools prove effective, and could be implemented in real-time, there's a wealth of possibility in adaptive interfaces. In addition, the semantic-level interaction history construct has the potential to provide meaningful training data for machine-learning algorithms that could probabilistically model user-action and be used to offer suggested interactions.]

== Reconciling Usability Heuristics with Cognitive Theory ==
* Owner: [[User:E J Kalafarski|E J Kalafarski]] 14:56, 28 April 2009 (UTC)

'''Contribution''': A framework for the unification of established heuristic usability guidelines and accepted cognitive principles. We propose a matrix pairing cognitive principles and heuristic guidelines into specific interface improvement recommendations.

[David: I'm having trouble parsing this contribution. How do you weight a framework? Also, I'd like to have a little more sense of how this might fit into the bigger picture. The assignment didn't ask for that, but is there some way to provide some of that context?]
[I agree that the wording is is a little confusing. The framework is not weighted, but rather it weights the different guidelines/principles. It might also be worth explaining how this is a useful contribution, e.g. does it allow for more accurate interface evaluation? -Eric]

[Language cleaned up. While I still think it is important to give more "weight" to cognitive/heuristic principle pairs that agree more strongly than others and output more useful recommendations, I believe I've pitched it wrong and made it too prominent an attribute of this module. I've adjusted. [[User:E J Kalafarski|E J Kalafarski]] 15:34, 1 May 2009 (UTC)]

'''Demonstration''': Three groups of experts anecdotally apply cognitive principles, heuristic usability principles, and a combination of the two.
* A human "cognition expert," given a constrained, limited-functionality interface, develops an independent evaluative value for each interface element based on accepted cognitive principles.
* A human "usability expert" develops an independent evaluative value for each interface element based on accepted heuristic guidelines.
* A third expert applies several recommendations from a matrix of cogntive/heuristic principle pairs.
* User testing demonstrates the assumed efficacy and applicability of the principle pairs versus separate application of cognitive and heuristic guidelines.
* Matrix can be improved incrementally by applying "weight" to each cell of the matrix, increasing its influence on final output recommendations, based on the measurable success of this study.

[There is still the question of how these weights are determined. Is this the job of the third expert? Or is the third expert given these weights, and he just determines how to apply them? -Eric] [Added more explanatory description of methodology as a bullet under Demo. [[User:E J Kalafarski|E J Kalafarski]] 15:49, 1 May 2009 (UTC)]

'''Dependency''': A set of established cognitive principles, selected with an eye toward heuristic analogues.

'''Dependency''': A set of established heuristic design guidelines, selected with an eye toward cognitive analogues.

'''review'''

* This seems like the 2d matrix? Is this implemented as a module? (Gideon) [Yes. [[User:E J Kalafarski|E J Kalafarski]] 15:49, 1 May 2009 (UTC)]

* Gideon's comment brings up an important point - in what form will the weights be presented? - Steven [Values which contribute to the "influence" of each recommendation. [[User:E J Kalafarski|E J Kalafarski]] 15:49, 1 May 2009 (UTC)]
* Could be more explicit about whether the experts are simulated agents or real people (Jon)

== Evaluation for Recommendation and Incremental Improvement ==
* Owner: [[User:E J Kalafarski|E J Kalafarski]] 14:56, 28 April 2009 (UTC)

'''Contributions''': Using a set of interface evaluation modules for analysis, we demonstrate the proposed efficiency and accuracy of a method for aggregating these interface suggestions. In this preliminary demonstration, we limit the field to a strictly-defined interface and 2–3 CPM principles (e.g. Fitts' Law and Affordance). Unified recommendations are generated for application to the interface for incremental improvement in usability.

[David: I'm a little lost. Can you make this long sentence into a couple shorter simpler ones? I think I can imagine what you are getting at, but I'm not sure.]

'''Demonstration''': A limited but strictly-controlled study to demonstrate the efficacy and efficiency of automated aggregation of interface evaluation versus independent, analysis. Shows not total replacement, but a gain in speed of evaluation comparable to the loss of control and feature.
* Given a carefully-constrained interface, perhaps with as few as two buttons and a minimalist feature set, expert interface designer given the individual results of several basic evaluation modules makes recommendations and suggestions to the design of the interface.
* Aggregation meta-module conducts similar survey of module outputs, outputting recommendations and suggestions for improvement of given interface.
* Separate independent body of experts then implements the two sets of suggestions and performs user study on the resultant interfaces, analyzing usability change and comparing it to the time and resources committed by the evaluation expert and the aggregation module, respectively.

[David: nice demonstration!]

'''Dependency''': Module for the analysis of Fitts' Law as it applies to the individual elements of a given interface.

'''Dependency''': Module for the analysis of a Affordance as it applies to the individual elements of a given interface.

'''Dependency''': Framework for the input of a given interface to individual modules, and standardization of the output of said modules.

[David: I think this also has, as a dependency, some kind of framework for the modules. "narrow but comprehensive" sounds challenging. ]

'''review'''

* I see this as being the "black box" for our architecture? If so, good. Wouldn't the dependencies be any/all modules? (Gideon) [I limited the scope of this initial demonstration. [[User:E J Kalafarski|E J Kalafarski]] 15:46, 1 May 2009 (UTC)]
* [I think this is the same as my contribution, and that we should merge them together. You've covered some things that I didn't, such as a more detailed demonstration of how our framework is better than a human interface designer's evaluation. You also gave a demonstration of comparing recommendations, while I only covered evaluations. -Eric]
* I think this contribution and mine (the CPM-GOMS improvement) will need to be reconciled into a single unit. - Steven
[Is it feasible to combine all three of these contributions? [[User:E J Kalafarski|E J Kalafarski]] 15:47, 1 May 2009 (UTC)]
* Interesting to note that this could describe the overarching architecture but it depends upon components of the architecture to work. Would it depend on any other modules? (Jon) [Yes, several modules are listed as dependencies. [[User:E J Kalafarski|E J Kalafarski]] 15:46, 1 May 2009 (UTC)]

== Evaluation Metrics ==
Owner: [[User:Gideon Goldin | Gideon Goldin]]

:*Architecture Outputs
::* Time (time to complete task)
:::# In user studies, this module predicted time-to-completion more accurately than the standard CPM-GOMS model.
:::* Dependencies
:::# CPM-GOMS Module with proposed cognitive load extension
::* Cognitive Load
:::# User studies were conducted during which participants brain's were scanned with EEG to measure activation levels.
:::# Self-report measures were obtained describing the amount of concentration/effort required on the part of the user.
:::# The literature base provided an accurate assessment of cognitive load in the tasks involved.
:::# Facial recognition allows us to approximate spans of high-load (e.g., furled brow)
:::* Dependencies
::::# CPM-GOMS Module with cognitive load extension
::::# Facial Gesture Recognition Module
::::# Self-Report Measure
::::# Database of load per cognitive tasks
::* Frustration
:::# Frustration is measured by asking participants to respond on the self-report scale.
:::# Facial recognition allows us to approximate spans of frustration (e.g., frowning, sighing, furled brow)
:::# Galvanic Skin Response allows detection of emotional arousal
:::# Heart rate monitoring allows detection of emotional arousal
:::* Dependencies
::::# Facial Gesture Recognition Module
::::# Galvanic Skin Response input
::::# Heart rate response input
::::# Interface Efficiency Module

'''review'''

== Parallel Framework for Evaluation Modules ==
* Owner: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]]

''This section will describe in more detail the inputs, outputs and architecture that were presented in the introduction.''

'''Contribution:''' Create a framework that provides better interface evaluations than currently existing techniques, and a module weighting system that provides better evaluations than any of its modules taken in isolation.

'''Demonstration:''' Run a series of user studies and compare users' performance to expected performance, as given by the following interface evaluation methods:
# Traditional, manual interface evaluation
#* As a baseline.
# Using our system with a single module
#* "Are any of our individual modules better than currently existing methods of interface evaluation?".
# Using our system with multiple modules, but have aggregator give a fixed, equal weighting to each module
#* As a baseline for our aggregator: want to show that the value of adding the dynamic weighting.
# Using our system with multiple modules, and allow the aggregator to adjust weightings for each module, but have each module get a single weighting for all dimensions (time, fatigue, etc.)
#* For validating the use of a dynamic weighting system.
# Using our system with multiple modules, and allow the aggregator to adjust weightings for each module, and allow the module to give different weightings to every dimension of the module (time, fatigue, etc.)
#* For validating the use of weighting across multiple utility dimensions.

'''Dependencies:''' Requires a good set of modules to plug into the framework.

'''review'''

* What exactly are the differences between this and EJ's earlier contribution? I think that if they are the same, this one is a bit more clear, IMHO. (Gideon)
[I have a similar question to Gideon's. My hunch is that this defines a sort of "namespace" for the modules themselves, while the other contribution (the one you guys assigned to me) is more explicitly the aggregator of these outputs. Correct me if I'm wrong. But if that's the case, the two contributions might need to be validated similarly/together. [[User:E J Kalafarski|E J Kalafarski]] 15:17, 29 April 2009 (UTC)]
* I feel like this structure's clarity may come at expense of specificity, I'd like to know more than that it's parallel. - Steven
* Very thorough demonstration(s)! (Jon)
* I think that the contribution is much more clearly stated here -- I would argue for integrating them together. (Andrew)

== Sample Modules ==

=== Fitts's Law ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

'''Aims'''
*Provides an estimate of the required time to complete various tasks that have been decomposed into formalized sequences of interactions with interface elements, and will provide evaluations and recommendations for optimizing the time required to complete those tasks using the interface.
*Integrate this module into our novel cognitive framework for interface evaluation.

'''Contributions'''
* [Not sure here. Is this really novel?] [I like it. I think it's necessary. I think we demonstrated in class that this has not been formalized and standardized already. [[User:E J Kalafarski|E J Kalafarski]] 15:22, 29 April 2009 (UTC)]

'''Demonstrations'''
*We will demonstrate the feasibility of this module by generating a formal description of an interface for scientific visualization, a formal description of a task to perform with the interface, and we will correlate the estimated time to complete the task based on Fitt's law with the actual time required based on several user traces. [Check. [[User:E J Kalafarski|E J Kalafarski]] 15:22, 29 April 2009 (UTC)]

'''Dependencies'''
*Requires a formal description of the interface with graph nodes representing clickable interface elements and graph edges representing the physical (on-screen) distance between adjacent nodes. [Is this graph format you're suggesting part of the proposal, or is the "language" itself a dependency from literature? [[User:E J Kalafarski|E J Kalafarski]] 15:22, 29 April 2009 (UTC)]

'''Module Description'''

*Inputs [In theory could the non-semantic inputs be automatically read from the interface? - Steven]
**A formal description of the interface and its elements (e.g. buttons).
**A formal description of a particular task and the possible paths through a subset of interface elements that permit the user to accomplish that task.
**The physical distances between interface elements along those paths.
**The width of those elements along the most likely axes of motion.
**Device (e.g. mouse) characteristics including start/stop time and the inherent speed limitations of the device.

*Output
**The module will then use the Shannon formulation of Fitt's Law to compute the average time needed to complete the task along those paths.

=== Affordances ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

'''Aims'''
*To provide interface evaluations and recommendations based on a measure of the extent to which the user perceives the relevant affordances of the interface when performing specified tasks.

'''Contributions'''
*A quantitative measure of the extent to which an interface suggests to the user the actions that it is capable of performing.
*A quantitative, indirect measure of the extent to which an interface facilitates (or hinders) the use of fast perceptual mechanisms.
[Again, I'm a fan. I don't think this has been formalized already. [[User:E J Kalafarski|E J Kalafarski]] 15:24, 29 April 2009 (UTC)]

'''Demonstrations'''
*We will demonstrate the feasibility of this module through the following experiment:
**Specify a task for a user to perform with scientific visualization software.
**There should be several different ways to complete the task (paths through the space of possible interface actions).
**Some of these paths will be more direct than others.
**We will then measure the number of task-relevant affordances that were perceived and acted upon by analyzing the user trace, and the time required to complete the task.
**Use the formula: (affordances perceived) / [(relevant affordances present) * (time to complete task)].
**Correlate the resulting scores with verbal reports on naturalness and ease-of-use for the interface.

'''Dependencies'''
*Requires a method for capturing user traces that include formalized records of interface elements used for a particular, pre-specified task or set of tasks.
*Providing suggestions/recommendations will require interaction with other modules that analyze the perceptual salience of interface elements.
[Some kind of formalized list or arbitrary classification of affordances might be necessary to limit the scope of this contribution. [[User:E J Kalafarski|E J Kalafarski]] 15:24, 29 April 2009 (UTC)]

'''Description of the Module'''

*Module Inputs''
**Formalized descriptions of...
***Interface elements
***Their associated actions
***The functions of those actions
***A particular task
***User traces for that task [Could this module benefit from eye-tracking? - Steven]

*Processing
**Inputs (1-4) are then used to generate a "user-independent" space of possible functions that the interface is capable of performing with respect to a given task -- what the interface "affords" the user. From this set of possible interactions, our model will then determine the subset of optimal paths for performing a particular task. The user trace (5) is then used to determine what functions actually were performed in the course of a given task of interest and this information is then compared to the optimal path data to determine the extent to which affordances of the interface are present but not perceived.

*Output
**The output of this module is a simple ratio of (affordances perceived) / [(relevant affordances present) * (time to complete task)] which provides a quantitative measure of the extent to which the interface is "natural" to use for a particular task.

CS295J/Final contributions

2009-05-01T15:08:15Z

Trevor O'Brien: /* Recording User-Interaction Primitives */

= Specific Contributions =

[I have removed almost everything that does not describe itself as a contribution. If your part is missing, a revision is in order and you can find what the former version on the page linked below (David)]

[[CS295J/Thursday 2pm version of final contribution|Thursday 2pm version of final contribution]]

== Workflow, Multi-tasking and Interruptions ==
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

'''Aims'''
* Develop a Theory of Multi-tasking Workflow and the Impact of Interruptions
* Build a Meta-work Assistance Tool
:# We will perform a series of ecologically-valid studies to compare user performance between a state of the art/popular task management system (control group) and our meta-work assistance tool (experimental group) [Didn't your study show that there is nothing close to a killer app in this department? - Steven] [By "state of the art" I mean a popular, well-recognized system (or systems), such as Outlook, that are used by large numbers of people.]
* Evaluate the Module-based Model of HCI by using it to predict the outcome of the above-mentioned evaluation; then compare with empirical results

'''Contributions'''
* Theory of Multi-tasking Workflow and the Impact of Interruptions
:# We will undertake detailed studies to help understand the following questions:
::# How does the size of a user's working set impact interruption resumption time?
::# How does the size of a user's working set, when used for rapid multi-tasking, impact performance metrics?
::# How does a user interface which supports multiple simultaneous working sets benefit interruption resumption time?
:* Internal dependencies: none
* Empirical Evaluation of Meta-work Assistance Tool in an Ecologically-valid Context
:* Internal dependencies: Completion of a testable meta-work tool
* Baseline Comparison Between Module-based Model of HCI and Core Multi-tasking Study
:# We will compare the results of the above-mentioned study in multi-tasking against results predicted by the module-based model of HCI in this proposal; this will give us an important baseline comparison, particularly given that multi-tasking and interruption involve higher brain functioning and are therefore likely difficult to predict
:* Internal dependencies: Dependent on core study completion, as well as most of the rest of the proposal being completed to the point of being testable

'''Demonstration (we might consider calling these deliverables)'''
* A predictive, quantiative model of controlled task performance and interruption resumption time for varying sized working sets
* An ecologically valid evaluation which shows that the completed meta-work assistance tool outperforms a state of the art system such as Outlook
* A comparison between the result predicted by the module-based model of HCI and the evaluation mentioned above; accompanied by a detailed analysis of factors contributing to the result

'''External Dependencies'''
* Dependent on most of the rest of the proposal being completed to the point of being usable to predict the outcome of the meta-work tool evaluation

'''Background and Related Work'''
* Salvucci et al's integrated theory of multi-tasking continuum integrates concurrent and sequential task performance
* Gloria Mark's research group has conducted a number of empirical studies into multi-tasking and interruption

'''Preliminary Work'''

* [[Study of Meta-work Tools and Strategies of 7 Information Workers]]
:* Results suggest there is not a single "ideal" meta-work tool today, motivating our proposal to build a new meta-work support tool based in part on a scientific theory of multi-tasking and interruptions.

'''Reviews'''
* Not in the expected format -- split into appropriate parts and label them as in the other sections (David)

== Recording User-Interaction Primitives ==

* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

'''Aims'''
* Develop system for logging low-level user-interactions within existing applications. By low-level interactions, we refer to those interactions for which a quantitative, predictive, GOMS-like model may be generated.[How do you propose to get around the problem Andrew brought up of accurately linking this low-level data to the interface/program's state at the moment it occurred? - Steven] [Low-level interactions will need to be logged along with a markup-language-style state description of the application itself. This would either require access to the source code, or a detailed interface description from the interface designer that would allow for complete simulation of the application.]
* At the same scale of granularity, integrate explicit interface-interactions with multimodal-sensing data. i.e. pupil-tracking, muscle-activity monitoring, auditory recognition, EEG data, and facial and posture-focused video.

'''Contributions'''
* Extensible, multi-modal, HCI framework for recording rich interaction-history data in existing applications.[Perhaps something should be said about sectioning this data off into modules - i.e. compiling all e-mail writing recordings to analyze separately from e-mail sorting recordings in order to facilitate analysis. - Steven] [I believe this is addressed in the "semantic-level chunking" section, given below.]

'''Demonstration'''
* Techniques for pupil-tracking, auditory-recognition, and muscle-activity monitoring will be evaluated with respect to accuracy, sampling rate, and computational cost in a series of quantitative studies. [Is there an explicit, quantifiable demonstration you can propose? Such some kind of conclusive comparison between automated capture techniques and manual techniques? Or maybe a demonstration of some kind of methodical, but still low-level, procedure for clustering related interaction primitives? [[User:E J Kalafarski|E J Kalafarski]] 15:02, 29 April 2009 (UTC)] [Things like pupil-tracking can be quantitatively evaluated through users tests. For instance, we could instruct users to shift the focus of their eyes from one on-screen object to another as quickly as possible some fixed number of times. We could then compare the results from our system to determine accuracy and speed, and by varying the size and distance of the target objects, we could obtain richer data for analysis. With respect to clustering low-level interactions, I believe that issue is addressed in the following section.][Should this take into account the need to match recordings to interface state? - Steven] [Possibly, if we go down the simulation route. If we're actually logging all of the events from within the source code itself, I believe the demonstration of syncing low-level interaction with application-state is self-evident.]

'''Dependencies'''
* Hardware for pupil-tracking and muscle activity monitoring. Commercial software packages for such software may be required.

'''Background and Related Work'''
* The utility of interaction histories with respect to assessing interface design has been demonstrated in <ref name = bob> [http://www.cs.brown.edu/people/trevor/Papers/Heer-2008-GraphicalHistories.pdf Graphical Histories for Visualization: Supporting Analysis, Communication, and Evaluation] (InfoVis 2008) </ref>. Can you briefly summarize the utility with one sentence? Or is that what the next bullet does? (Jon) [In the instance of this work, utility was demonstrated with respect to automatically generating presentations from data explorations, and providing quantitative interface-usage analysis that aided future decisions for interface designers. i.e., this operations was performed a lot, and this one was not; therefore we should focus on making the first more efficient, perhaps at the expense of the second. The results of the work are fairly basic, but they do demonstrate that logging histories is a useful practice for a number of reasons.]
* In addition, data management histories have been shown effective in the visualization community in <ref> [http://www.cs.brown.edu/people/trevor/Papers/Callahan-2006-MED.pdf Callahan-2006-MED] </ref> <ref> [http://www.cs.brown.edu/people/trevor/Papers/Callahan-2006-VVM.pdf Callahan-2006-VVM] </ref> <ref> [http://www.cs.brown.edu/people/trevor/Papers/Bavoil-2005-VEI.pdf Bavoil-2005-VEI]</ref>, providing visualizations by analogy <ref> [http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=4376187&isnumber=4376125 Querying and Creating Visualizations by Analogy] </ref> and offering automated suggestions <ref> [http://www.cs.utah.edu/~juliana/pub/tvcg-recommendation2008.pdf VisComplete: Automating Suggestions from Visualization Pipelines] </ref>, which we expect to generalize to user interaction history.

''''review''''
* This is recording everything possible during an interaction. It is necessary to do for our project. (Gideon)
* Two thumbs up. - Steven

== Semantic-level Interaction Chunking ==

* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

'''Aims'''
* Develop techniques for ''chunking'' low-level interaction primitives into ''semantic-level'' interactions, given an application's functionality and data-context. (And de-chunking? Invertible mapping needed?)
* Perform user-study evaluation to validate chunking methods.

'''Contributions'''
* Design and user-study evaluation of semantic-chunking techniques for interaction.

'''Demonstration'''
* These techniques will be validated quantitatively with respect to efficiency and user error rate on a set of timed-tasks over a series of case studies. In addition, qualitative feedback on user satisfaction will be incorporated in the evaluation of our methods. [As above, how will this be validated quantitatively? Comparing two methodologies for manual chunking? What would these differing methodologies be? [[User:E J Kalafarski|E J Kalafarski]] 15:06, 29 April 2009 (UTC)]

'''Dependencies'''
* Software framework for collecting user interactions.[I'd like to see this fleshed out in a theoretical manner. - Steven]
* Formal description of interface functionality.
* Description of data objects that can be manipulated through interaction.

'''review'''
* With what level of accuracy can this be performed. I like the idea, its worthy of a nice project in and of itself. (Gideon)
Can you include something about the broader impact of this contribution? (Jon)

== Reconciling Usability Heuristics with Cognitive Theory ==
* Owner: [[User:E J Kalafarski|E J Kalafarski]] 14:56, 28 April 2009 (UTC)

'''Contribution''': A weighted framework for the unification of established heuristic usability guidelines and accepted cognitive principles.

[Andrew: Better word choice needed; technical terms that you are inventing should be defined -- for example, what is a "matricized analogue"? According to Wikipedia, matricization is defined as tensor unfolding - no doubt this is not what you intendend. Reviewers will not be gentle on points such as this, as they go to credibility.]

[David: I'm having trouble parsing this contribution. How do you weight a framework? Also, I'd like to have a little more sense of how this might fit into the bigger picture. The assignment didn't ask for that, but is there some way to provide some of that context?]
[I agree that the wording is is a little confusing. The framework is not weighted, but rather it weights the different guidelines/principles. It might also be worth explaining how this is a useful contribution, e.g. does it allow for more accurate interface evaluation? -Eric]

'''Demonstration''': Three groups of experts anecdotally apply cognitive principles, heuristic usability principles, and a combination of the two.
* A "cognition expert," given a constrained, limited-functionality interface, develops an independent evaluative value for each interface element based on accepted cognitive principles.
* A "usability expert" develops an independent evaluative value for each interface element based on accepted heuristic guidelines.
* A third expert applies several unified cognitive analogues from a matrix of weighted cognitive and [...HCI design principles? -Eric]
* User testing demonstrates the assumed efficacy and applicability of the matricized analogues versus independent application of analogued principles.

[There is still the question of how these weights are determined. Is this the job of the third expert? Or is the third expert given these weights, and he just determines how to apply them? -Eric]

'''Dependency''': A set of established cognitive principles, selected with an eye toward heuristic analogues.

'''Dependency''': A set of established heuristic design guidelines, selected with an eye toward cognitive analogues.

'''review'''

* This seems like the 2d matrix? Is this implemented as a module? (Gideon)
* Gideon's comment brings up an important point - in what form will the weights be presented? - Steven
* Could be more explicit about whether the experts are simulated agents or real people (Jon)

== Evaluation for Recommendation and Incremental Improvement ==
* Owner: [[User:E J Kalafarski|E J Kalafarski]] 14:56, 28 April 2009 (UTC)

'''Contributions''': Using a limited set of interface evaluation modules for analysis, we demonstrate, in a narrow and controlled manner, the proposed efficiency and accuracy of a method of aggregating individual interface suggestions based on accepted CPM principles (e.g. Fitts' Law and Affordance) and applying them to the incremental improvement of the interface.

[David: I'm a little lost. Can you make this long sentence into a couple shorter simpler ones? I think I can imagine what you are getting at, but I'm not sure.]

'''Demonstration''': A narrow but comprehensive study to demonstrate the efficacy and efficiency of automated aggregation of interface evaluation versus independent, analysis. Shows not total replacement, but a gain in speed of evaluation comparable to the loss of control and feature.
* Given a carefully-constrained interface, perhaps with as few as two buttons and a minimalist feature set, expert interface designer given the individual results of several basic evaluation modules makes recommendations and suggestions to the design of the interface.
* Aggregation meta-module conducts similar survey of module outputs, outputting recommendations and suggestions for improvement of given interface.
* Separate independent body of experts then implements the two sets of suggestions and performs user study on the resultant interfaces, analyzing usability change and comparing it to the time and resources committed by the evaluation expert and the aggregation module, respectively.

[David: nice demonstration!]

'''Dependency''': Module for the analysis of Fitts' Law as it applies to the individual elements of a given interface.

'''Dependency''': Module for the analysis of a Affordance as it applies to the individual elements of a given interface.

[David: I think this also has, as a dependency, some kind of framework for the modules. "narrow but comprehensive" sounds challenging. ]

'''review'''

* I see this as being the "black box" for our architecture? If so, good. Wouldn't the dependencies be any/all modules? (Gideon)
* [I think this is the same as my contribution, and that we should merge them together. You've covered some things that I didn't, such as a more detailed demonstration of how our framework is better than a human interface designer's evaluation. You also gave a demonstration of comparing recommendations, while I only covered evaluations. -Eric]
* I think this contribution and mine (the CPM-GOMS improvement) will need to be reconciled into a single unit. - Steven
* Interesting to note that this could describe the overarching architecture but it depends upon components of the architecture to work. Would it depend on any other modules? (Jon)

== Evaluation Metrics ==
Owner: [[User:Gideon Goldin | Gideon Goldin]]

:*Architecture Outputs
::* Time (time to complete task)
:::# In user studies, this module predicted time-to-completion more accurately than the standard CPM-GOMS model.
:::* Dependencies
:::# CPM-GOMS Module with proposed cognitive load extension
::* Cognitive Load
:::# User studies were conducted during which participants brain's were scanned with EEG to measure activation levels.
:::# Self-report measures were obtained describing the amount of concentration/effort required on the part of the user.
:::# The literature base provided an accurate assessment of cognitive load in the tasks involved.
:::# Facial recognition allows us to approximate spans of high-load (e.g., furled brow)
:::* Dependencies
::::# CPM-GOMS Module with cognitive load extension
::::# Facial Gesture Recognition Module
::::# Self-Report Measure
::::# Database of load per cognitive tasks
::* Frustration
:::# Frustration is measured by asking participants to respond on the self-report scale.
:::# Facial recognition allows us to approximate spans of frustration (e.g., frowning, sighing, furled brow)
:::# Galvanic Skin Response allows detection of emotional arousal
:::# Heart rate monitoring allows detection of emotional arousal
:::* Dependencies
::::# Facial Gesture Recognition Module
::::# Galvanic Skin Response input
::::# Heart rate response input
::::# Interface Efficiency Module

'''review'''

== Parallel Framework for Evaluation Modules ==
* Owner: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]]

''This section will describe in more detail the inputs, outputs and architecture that were presented in the introduction.''

'''Contribution:''' Create a framework that provides better interface evaluations than currently existing techniques, and a module weighting system that provides better evaluations than any of its modules taken in isolation.

'''Demonstration:''' Run a series of user studies and compare users' performance to expected performance, as given by the following interface evaluation methods:
# Traditional, manual interface evaluation
#* As a baseline.
# Using our system with a single module
#* "Are any of our individual modules better than currently existing methods of interface evaluation?".
# Using our system with multiple modules, but have aggregator give a fixed, equal weighting to each module
#* As a baseline for our aggregator: want to show that the value of adding the dynamic weighting.
# Using our system with multiple modules, and allow the aggregator to adjust weightings for each module, but have each module get a single weighting for all dimensions (time, fatigue, etc.)
#* For validating the use of a dynamic weighting system.
# Using our system with multiple modules, and allow the aggregator to adjust weightings for each module, and allow the module to give different weightings to every dimension of the module (time, fatigue, etc.)
#* For validating the use of weighting across multiple utility dimensions.

'''Dependencies:''' Requires a good set of modules to plug into the framework.

'''review'''

* What exactly are the differences between this and EJ's earlier contribution? I think that if they are the same, this one is a bit more clear, IMHO. (Gideon)
[I have a similar question to Gideon's. My hunch is that this defines a sort of "namespace" for the modules themselves, while the other contribution (the one you guys assigned to me) is more explicitly the aggregator of these outputs. Correct me if I'm wrong. But if that's the case, the two contributions might need to be validated similarly/together. [[User:E J Kalafarski|E J Kalafarski]] 15:17, 29 April 2009 (UTC)]
* I feel like this structure's clarity may come at expense of specificity, I'd like to know more than that it's parallel. - Steven
* Very thorough demonstration(s)! (Jon)
* I think that the contribution is much more clearly stated here -- I would argue for integrating them together. (Andrew)

== Sample Modules ==

=== Fitts's Law ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

'''Aims'''
*Provides an estimate of the required time to complete various tasks that have been decomposed into formalized sequences of interactions with interface elements, and will provide evaluations and recommendations for optimizing the time required to complete those tasks using the interface.
*Integrate this module into our novel cognitive framework for interface evaluation.

'''Contributions'''
* [Not sure here. Is this really novel?] [I like it. I think it's necessary. I think we demonstrated in class that this has not been formalized and standardized already. [[User:E J Kalafarski|E J Kalafarski]] 15:22, 29 April 2009 (UTC)]

'''Demonstrations'''
*We will demonstrate the feasibility of this module by generating a formal description of an interface for scientific visualization, a formal description of a task to perform with the interface, and we will correlate the estimated time to complete the task based on Fitt's law with the actual time required based on several user traces. [Check. [[User:E J Kalafarski|E J Kalafarski]] 15:22, 29 April 2009 (UTC)]

'''Dependencies'''
*Requires a formal description of the interface with graph nodes representing clickable interface elements and graph edges representing the physical (on-screen) distance between adjacent nodes. [Is this graph format you're suggesting part of the proposal, or is the "language" itself a dependency from literature? [[User:E J Kalafarski|E J Kalafarski]] 15:22, 29 April 2009 (UTC)]

'''Module Description'''

*Inputs [In theory could the non-semantic inputs be automatically read from the interface? - Steven]
**A formal description of the interface and its elements (e.g. buttons).
**A formal description of a particular task and the possible paths through a subset of interface elements that permit the user to accomplish that task.
**The physical distances between interface elements along those paths.
**The width of those elements along the most likely axes of motion.
**Device (e.g. mouse) characteristics including start/stop time and the inherent speed limitations of the device.

*Output
**The module will then use the Shannon formulation of Fitt's Law to compute the average time needed to complete the task along those paths.

=== Affordances ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

'''Aims'''
*To provide interface evaluations and recommendations based on a measure of the extent to which the user perceives the relevant affordances of the interface when performing specified tasks.

'''Contributions'''
*A quantitative measure of the extent to which an interface suggests to the user the actions that it is capable of performing.
*A quantitative, indirect measure of the extent to which an interface facilitates (or hinders) the use of fast perceptual mechanisms.
[Again, I'm a fan. I don't think this has been formalized already. [[User:E J Kalafarski|E J Kalafarski]] 15:24, 29 April 2009 (UTC)]

'''Demonstrations'''
*We will demonstrate the feasibility of this module through the following experiment:
**Specify a task for a user to perform with scientific visualization software.
**There should be several different ways to complete the task (paths through the space of possible interface actions).
**Some of these paths will be more direct than others.
**We will then measure the number of task-relevant affordances that were perceived and acted upon by analyzing the user trace, and the time required to complete the task.
**Use the formula: (affordances perceived) / [(relevant affordances present) * (time to complete task)].
**Correlate the resulting scores with verbal reports on naturalness and ease-of-use for the interface.

'''Dependencies'''
*Requires a method for capturing user traces that include formalized records of interface elements used for a particular, pre-specified task or set of tasks.
*Providing suggestions/recommendations will require interaction with other modules that analyze the perceptual salience of interface elements.
[Some kind of formalized list or arbitrary classification of affordances might be necessary to limit the scope of this contribution. [[User:E J Kalafarski|E J Kalafarski]] 15:24, 29 April 2009 (UTC)]

'''Description of the Module'''

*Module Inputs''
**Formalized descriptions of...
***Interface elements
***Their associated actions
***The functions of those actions
***A particular task
***User traces for that task [Could this module benefit from eye-tracking? - Steven]

*Processing
**Inputs (1-4) are then used to generate a "user-independent" space of possible functions that the interface is capable of performing with respect to a given task -- what the interface "affords" the user. From this set of possible interactions, our model will then determine the subset of optimal paths for performing a particular task. The user trace (5) is then used to determine what functions actually were performed in the course of a given task of interest and this information is then compared to the optimal path data to determine the extent to which affordances of the interface are present but not perceived.

*Output
**The output of this module is a simple ratio of (affordances perceived) / [(relevant affordances present) * (time to complete task)] which provides a quantitative measure of the extent to which the interface is "natural" to use for a particular task.

CS295J/Final contributions

2009-04-28T20:18:50Z

Trevor O'Brien: /* Semantic-level Interaction Chunking */

= Specific Aims and Contributions (to be separated later) =

[David: I have removed some unattributed stuff that wasn't in the format described in assignment 13. If that was in error, go back to an earlier version to recover the text, attribute it, and put in the correct format.]

== Recording User-Interaction Primitives ==

* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

'''Aims'''
* Develop system for logging low-level user-interactions within existing applications. By low-level interactions, we refer to those interactions for which a quantitative, predictive, GOMS-like model may be generated.
* At the same scale of granularity, integrate explicit interface-interactions with multimodal-sensing data. i.e. pupil-tracking, muscle-activity monitoring, auditory recognition, EEG data, and facial and posture-focused video.

'''Contributions'''
* Extensible, multi-modal, HCI framework for recording rich interaction-history data in existing applications.

'''Demonstration'''
* Techniques for pupil-tracking, auditory-recognition, and muscle-activity monitoring will be evaluated with respect to accuracy, sampling rate, and computational cost in a series of quantitative studies.

'''Dependencies'''
* Hardware for pupil-tracking and muscle activity monitoring. Commercial software packages for such software may be required.

'''Background and Related Work'''
* The utility of interaction histories with respect to assessing interface design has been demonstrated in <ref name = bob> [http://www.cs.brown.edu/people/trevor/Papers/Heer-2008-GraphicalHistories.pdf Graphical Histories for Visualization: Supporting Analysis, Communication, and Evaluation] (InfoVis 2008) </ref>.
* In addition, data management histories have been shown effective in the visualization community in <ref> [http://www.cs.brown.edu/people/trevor/Papers/Callahan-2006-MED.pdf Callahan-2006-MED] </ref> <ref> [http://www.cs.brown.edu/people/trevor/Papers/Callahan-2006-VVM.pdf Callahan-2006-VVM] </ref> <ref> [http://www.cs.brown.edu/people/trevor/Papers/Bavoil-2005-VEI.pdf Bavoil-2005-VEI]</ref>, providing visualizations by analogy <ref> [http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=4376187&isnumber=4376125 Querying and Creating Visualizations by Analogy] </ref> and offering automated suggestions <ref> [http://www.cs.utah.edu/~juliana/pub/tvcg-recommendation2008.pdf VisComplete: Automating Suggestions from Visualization Pipelines] </ref>, which we expect to generalize to user interaction history.

== Semantic-level Interaction Chunking ==

* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

'''Aims'''
* Develop techniques for ''chunking'' low-level interaction primitives into ''semantic-level'' interactions, given an application's functionality and data-context. (And de-chunking? Invertible mapping needed?)
* Perform user-study evaluation to validate chunking methods.

'''Contributions'''
* Design and user-study evaluation of semantic-chunking techniques for interaction.

'''Demonstration'''
* These techniques will be validated quantitatively with respect to efficiency and user error rate on a set of timed-tasks over a series of case studies. In addition, qualitative feedback on user satisfaction will be incorporated in the evaluation of our methods.

'''Dependencies'''
* Software framework for collecting user interactions.
* Formal description of interface functionality.
* Description of data objects that can be manipulated through interaction.

== Reconciling Usability Heuristics with Cognitive Theory ==
* Owner: [[User:E J Kalafarski|E J Kalafarski]] 14:56, 28 April 2009 (UTC)

'''Contribution''': A weighted framework for the unification of established heuristic usability guidelines and accepted cognitive principles.

[David: I'm having trouble parsing this contribution. How do you weight a framework? Also, I'd like to have a little more sense of how this might fit into the bigger picture. The assignment didn't ask for that, but is there some way to provide some of that context?]

'''Demonstration''': Three groups of experts anecdotally apply cognitive principles, heuristic usability principles, and a combination of the two.
* A "cognition expert," given a constrained, limited-functionality interface, develops an independent evaluative value for each interface element based on accepted cognitive principles.
* A "usability expert" develops an independent evaluative value for each interface element based on accepted heuristic guidelines.
* A third expert applies several unified cognitive analogues from a matrix of weighted cognitive and
* User testing demonstrates the assumed efficacy and applicability of the matricized analogues versus independent application of analogued principles.

'''Dependency''': A set of established cognitive principles, selected with an eye toward heuristic analogues.

'''Dependency''': A set of established heuristic design guidelines, selected with an eye toward cognitive analogues.

== Evaluation for Recommendation and Incremental Improvement ==
* Owner: [[User:E J Kalafarski|E J Kalafarski]] 14:56, 28 April 2009 (UTC)

'''Contributions''': Using a limited set of interface evaluation modules for analysis, we demonstrate, in a narrow and controlled manner, the proposed efficiency and accuracy of a method of aggregating individual interface suggestions based on accepted CPM principles (e.g. Fitts' Law and Affordance) and applying them to the incremental improvement of the interface.

[David: I'm a little lost. Can you make this long sentence into a couple shorter simpler ones? I think I can imagine what you are getting at, but I'm not sure.]

'''Demonstration''': A narrow but comprehensive study to demonstrate the efficacy and efficiency of automated aggregation of interface evaluation versus independent, analysis. Shows not total replacement, but a gain in speed of evaluation comparable to the loss of control and feature.
* Given a carefully-constrained interface, perhaps with as few as two buttons and a minimalist feature set, expert interface designer given the individual results of several basic evaluation modules makes recommendations and suggestions to the design of the interface.
* Aggregation meta-module conducts similar survey of module outputs, outputting recommendations and suggestions for improvement of given interface.
* Separate independent body of experts then implements the two sets of suggestions and performs user study on the resultant interfaces, analyzing usability change and comparing it to the time and resources committed by the evaluation expert and the aggregation module, respectively.

[David: nice demonstration!]

'''Dependency''': Module for the analysis of Fitts' Law as it applies to the individual elements of a given interface.

'''Dependency''': Module for the analysis of a Affordance as it applies to the individual elements of a given interface.

[David: I think this also has, as a dependency, some kind of framework for the modules. "narrow but comprehensive" sounds challenging. ]

== Evaluation Metrics ==
Owner: [[User:Gideon Goldin | Gideon Goldin]]

:*Architecture Outputs
::* Time (time to complete task)
:::# Performs as well or better than CPM-GOMS, demonstrated with user tasks
:::* Dependencies
:::# CPM-GOMS Module with cognitive load extension
::* Cognitive Load
:::# Predicts cognitive load during tasks, demonstrated with user tasks
:::* Dependencies
::::# CPM-GOMS Module with cognitive load extension
::::# Facial Gesture Recognition Module
::* Frustration
:::# Accurately predicts users' frustration levels, demonstrated with user tasks
:::* Dependencies
::::# Facial Gesture Recognition Module
::::# Galvanic Skin Response Module
::::# Interface Efficiency Module
::* Aesthetic Appeal
:::# Analyzes if the interface is aesthetically unpleasing, demonstrated with user tasks
:::* Dependencies
::::# Aesthetics Module
::* Simplicity
:::# Analyzes how simple the interface is, demonstrated with user tasks
:::* Dependencies
::::# Interface Efficiency Module

== Parallel Framework for Evaluation Modules ==
* Owner: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]]

''This section will describe in more detail the inputs, outputs and architecture that were presented in the introduction.''

==== Contributions ====

* '''Create a framework that provides better interface evaluations than currently existing techniques, and a module weighting system that provides better evaluations than any of its modules taken in isolation.'''

* Demonstrate by running user studies and comparing this performance to expected performance, as given by the following interface evaluation methods:
*# Traditional, manual interface evaluation
*#* As a baseline.
*# Using our system with a single module
*#* "Are any of our individual modules better than currently existing methods of interface evaluation?".
*# Using our system with multiple modules, but have aggregator give a fixed, equal weighting to each module
*#* As a baseline for our aggregator: want to show that the value of adding the dynamic weighting.
*# Using our system with multiple modules, and allow the aggregator to adjust weightings for each module, but have each module get a single weighting for all dimensions (time, fatigue, etc.)
*#* For validating the use of a dynamic weighting system.
*# Using our system with multiple modules, and allow the aggregator to adjust weightings for each module, and allow the module to give different weightings to every dimension of the module (time, fatigue, etc.)
*#* For validating the use of weighting across multiple utility dimensions.
*Dependencies: Having a good set of modules to plug into the framework.

[David: I have removed details of framework that seemed beyond contribs/demos/dependencies]

== Evaluation and Recommendation via Modules ==
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

A "meta-module" called the aggregator will be responsible for assembling and formatting the output of all other modules into a structure that is both extensible and immediately usable, by both an automated designer or a human designer.

===Requirements===
The aggregator's functionality, then, is defined by its '''inputs''', the outputs of the other modules, and the desired '''output''' of the system as a whole, per its position in the architecture. Its purpose is largely formatting and reconciliation of the products of the multitudinous (and extensible) modules. The output of the aggregator must meet several requirements: first, to generate a set of human-readable suggestions for the improvement of the given interface; second, to generate a machine-readable, but also analyzable, evaluation of the various characteristics of the interface and accompanying user traces.

From these specifications, it is logical to assume that a common language or format will be required for the output of individual modules. We propose an XML-based file format, allowing: (1) a section for the standardized identification of problem areas, applicable rules, and proposed improvements, generalized by the individual module and mapped to a single element, or group of elements, in the original interface specification; (2) a section for specification of generalizable "utility" functions, allowing a module to specify how much a measurable quantity of utility is positively or negatively affected by properties of the input interface; (3) new, user-definable sections for evaluations of the given interface not covered by the first two sections. The first two sections are capable of conveying the vast majority of module outputs predicted at this time, but the XML can extensibly allow modules to pass on whatever information may become prominent in the future.

===Specification===
<module id="Fitts-Law">
<interface-elements>
<element>
<desc>submit button</desc>
<problem>
<desc>size</desc>
<suggestion>width *= 2</suggestion>
<suggestion>height *= 2</suggestion>
<human-suggestion>Increase size relative to other elements</human-suggestion>
</problem>
</element>
</interface-elements>
<utility>
<dimension>
<desc>time</desc>
<value>0:15:35</value>
</dimension>
<dimension>
<desc>frustration</desc>
<value>pulling hair out</value>
</dimension>
<dimension>
<desc>efficiency</desc>
<value>13.2s/KPM task</value>
<value>0.56m/CPM task</value>
</dimension>
</utility>
<tasks>
<task>
<desc>complete form</desc>
</task>
<task>
<desc>lookup SSN</desc>
</task>
<task>
<desc>format phone number</desc>
</task>
</tasks>
</module>

===Logic===
This file provided by each module is then the input for the aggregator. The aggregator's most straightforward function is the compilation of the "problem areas," assembling them and noting problem areas and suggestions that are recommended by more than one module, and weighting them accordingly in its final report. These weightings can begin in an equal state, but the aggregator should be capable of learning iteratively which modules' results are most relevant to the user and update weightings accordingly. This may need to be accomplished with manual tuning, or a machine-learning algorithm capable of determining which modules most often agree with others.

Secondly, the aggregator compiles the utility functions provided by the module specs. This, again, is a summation of similarly-described values from the various modules.

When confronted with user-defined sections of the XML spec, the aggregator is primarily responsible for compiling them and sending them along to the output of the machine. Even if the aggregator does not recognize a section or property of the evaluative spec, if it sees the property reported by more than one module it should be capable of aggregating these intelligently. In future versions of the spec, it should be possible for a module to provide instructions for the aggregator on how to handle unrecognized sections of the XML.

From these compilations, then, the aggregator should be capable of outputting both aggregated human-readable suggestions on interface improvements for a human designer, as well as a comprehensive evaluation of the interface's effectiveness at the given task traces. Again, this is dependent on the specification of the system as a whole, but is likely to include measures and comparisons, graphings of task versus utility, and quantitative measures of an element's effectiveness.

''This section is necessarily defined by the output of the individual modules (which I already expect to be of varied and arbitrary structure) and the desired output of the machine as a whole. It will likely need to be revised heavily after other modules and the "Parellel Framework" section are defined.'' [[User:E J Kalafarski|E J Kalafarski]] 12:34, 24 April 2009 (UTC)

''This section describes the aggregator, which takes the output of multiple independent modules and aggregates the results to provide (1) an evaluation and (2) recommendations for the user interface. We should explain how the aggregator weights the output of different modules (this could be based on historical performance of each module, or perhaps based on E.J.'s cognitive/HCI guidelines).''

== Sample Modules ==

=== CPM-GOMS ===
* Owners: [[User:Steven Ellis | Steven Ellis]]

''This module will provide interface evaluations and suggestions based on a CPM-GOMS model of cognition for the given interface. It will provide a quantitative, predictive, cognition-based parameterization of usability. From empirically collected data, user trajectories through the model (critical paths) will be examined, highlighting bottlenecks within the interface, and offering suggested alterations to the interface to induce more optimal user trajectories.''

I’m hoping to have some input on this section, because it seems to be the crux of the “black box” into which we take the inputs of interface description, user traces, etc. and get our outputs (time, recommendations, etc.). I know at least a few people have pretty strong thoughts on the matter and we ought to discuss the final structure.

That said – my proposal for the module:
*In my opinion the concept of the Model Human Processor (at least as applied in CPM) is outdated – it’s too economic/overly parsimonious in its conception of human activity. I think we need to create a structure which accounts for more realistic conditions of HCI including multitasking, aspects of distributed cognition (and other relevant uses of tools – as far as I can tell CPM doesn’t take into account any sort of productivity aids), executive control processes of attention, etc. ACT-R appears to take steps towards this but we would probably need to look at their algorithms to know for sure.
*Critical paths will continue to play an important role – we should in fact emphasize that part of this tool’s purpose will be a description not only of ways in which the interface should be modified to best fit a critical path, but also ways in which the user’s ought to be instructed in their use of the path. This feedback mechanism could be bidirectional – if the model’s predictions of the user’s goals are incorrect, the critical path determined will also be incorrect and the interface inherently suboptimal. The user could be prompted with a tooltip explaining in brief why and how the interface has changed, along with options to revert, select other configurations (euphemized by goals), and to view a short video detailing how to properly use the interface.
*Call me crazy but, if we assume designers will be willing to code a model of their interfaces into our ACT-R-esque language, could we allow that model to be fairly transparent to the user, who could use a gui to input their goals to find an analogue in the program which would subsequently rearrange its interface to fit the user’s needs? Even if not useful to the users, such dynamic modeling could really help designers (IMO)
*I think the model should do its best to accept models written for ACT-R and whatever other cognitive models there are out there – gives us the best chance of early adoption
*I would particularly appreciate input on the number/complexity/type of inputs we’ll be using, as well as the same qualities for the output.

=== HCI Guidelines ===
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

Schneiderman's Eight Golden Rules and Jakob Nielsen's Ten Heuristics are perhaps the most famous and well-regarded heuristic design guidelines to emerge over the last twenty years. Although the explicit theoretical basis for such heuristics is controversial and not well-explored, the empirical success of these guidelines is established and accepted. This module will parse out up to three or four common (that is, intersecting) principles from these accepted guidelines and apply them to the input interface.

As an example, we identify an analogous principle that appears in Schneiderman ("Reduce short-term memory load")<ref>http://faculty.washington.edu/jtenenbg/courses/360/f04/sessions/schneidermanGoldenRules.html</ref> and Nielsen ("Recognition rather than recall/Minimize the user's memory load")<ref>http://www.useit.com/papers/heuristic/heuristic_list.html</ref>. The input interface is then evaluated for the consideration of the principle, based on an explicit formal description of the interface, such as XAML or XUL. The module attempts to determine how effectively the interface demonstrates the principle. When analyzing an interface for several principles that may be conflicting or opposing in a given context, the module makes use of a hard-coded but iterative (and evolving) weighting of these principles, based on (1) how often they appear in the training set of accepted sets of guidelines, (2) how analogues a heuristic principle is to a cognitive principle in a parallel training set, and (3) how effective the principle's associated suggestion is found to be using a feedback mechanism.

'''Inputs''' 
# A formal description of the interface and its elements (e.g. buttons).
# A formal description of a particular task and the possible paths through a subset of interface elements that permit the user to accomplish that task.

'''Output''' 
Standard XML-formatted file containing problem areas of the input interface, suggestions for each problem area based on principles that were found to have a strong application to a problem element and the problem itself, and a human-readable generated analysis of the element's affinity for the principle. Quantitative outputs will not be possible based on heuristic guidelines, and the "utility" section of this module's output is likely to be blank.

''This section could include an example or two of established design guidelines that could easily be implemented as modules.''

=== Fitts's Law ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

''This module provides an estimate of the required time to complete various tasks that have been decomposed into formalized sequences of interactions with interface elements, and will provide evaluations and recommendations for optimizing the time required to complete those tasks using the interface.''

'''Inputs'''

1. A formal description of the interface and its elements (e.g. buttons).

2. A formal description of a particular task and the possible paths through a subset of interface elements that permit the user to accomplish that task.

3. The physical distances between interface elements along those paths.

4. The width of those elements along the most likely axes of motion.

5. Device (e.g. mouse) characteristics including start/stop time and the inherent speed limitations of the device.

'''Output'''

The module will then use the Shannon formulation of Fitt's Law to compute the average time needed to complete the task along those paths.

=== Affordances ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

''This simple module will provide interface evaluations and recommendations based on a measure of the extent to which the user perceives the relevant affordances of the interface when performing a number of specified tasks.''

'''Inputs'''

Formalized descriptions of...

1. Interface elements

2. Their associated actions

3. The functions of those actions

4. A particular task

5. User traces for that task.

Inputs (1-4) are then used to generate a "user-independent" space of possible functions that the interface is capable of performing with respect to a given task -- what the interface "affords" the user. From this set of possible interactions, our model will then determine the subset of optimal paths for performing a particular task. The user trace (5) is then used to determine what functions actually were performed in the course of a given task of interest and this information is then compared to the optimal path data to determine the extent to which affordances of the interface are present but not perceived.

'''Output'''

The output of this module is a simple ratio of (affordances perceived) / [(relevant affordances present) * (time to complete task)] which provides a quantitative measure of the extent to which the interface is "natural" to use for a particular task.

=== Workflow, Multi-tasking and Interruptions ===
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

[David: I think that the work proposed here is interesting, but I'm not sure that it is organized by "contribution". Try labeling the contributions and the demonstrations explicitly. A "tool" is usually not a contribution -- an evaluation of the tool could be. If this distinction isn't clear, let's talk. For these particular contributions, phrasing and integrating them is a challenge because some of the contributions are demonstrations of other contributions...]

::* Scientific Study of Multi-tasking Workflow and the Impact of Interruptions
:::# We will undertake detailed studies to help understand the following questions:
::::# How does the size of a user's working set impact interruption resumption time?
::::# How does the size of a user's working set, when used for rapid multi-tasking, impact performance metrics?
::::# How does a user interface which supports multiple simultaneous working sets benefit interruption resumption time?
:::* No Dependencies
::* Meta-work Assistance Tool
:::# We will perform a series of ecologically-valid studies to compare user performance between a state of the art task management system (control group) and our meta-work assistance tool (experimental group)
:::* Dependent on core study completion, as some of the specific design decisions will be driven by the results of this study. However, it is worth pointing out that this separate contribution can be researched in parallel to the core study.
::* Baseline Comparison Between Module-based Model of HCI and Core Multi-tasking Study
:::# We will compare the results of the above-mentioned study in multi-tasking against results predicted by the module-based model of HCI in this proposal; this will give us an important baseline comparison, particularly given that multi-tasking and interruption involve higher brain functioning and are therefore likely difficult to predict
:::* Dependent on core study completion, as well as most of the rest of the proposal being completed to the point of being testable

Text for Assignment 12:

Add text here about how this can be used to evaluate automatic framework

There are, at least, two levels at which users work ([http://portal.acm.org/citation.cfm?id=985692.985707&coll=portal&dl=ACM&CFID=20781736&CFTOKEN=83176621 Gonzales, et al., 2004]). Users accomplish individual low-level tasks which are part of larger ''working spheres''; for example, an office worker might send several emails, create several Post-It (TM) note reminders, and then edit a word document, each of these smaller tasks being part of a single larger working sphere of "adding a new section to the website." Thus, it is important to understand this larger workflow context - which often involves extensive levels of multi-tasking, as well as switching between a variety of computing devices and traditional tools, such as notebooks. In this study it was found that the information workers surveyed typically switch individual tasks every 2 minutes and have many simultaneous working spheres which they switch between, on average every 12 minutes. This frenzied pace of switching tasks and switching working spheres suggests that users will not be using a single application or device for a long period of time, and that affordances to support this characteristic pattern of information work are important.

The purpose of this module is to integrate existing work on multi-tasking, interruption and higher-level workflow into a framework which can predict user recovery times from interruptions. Specifically, the goals of this framework will be to:

* Understand the role of the larger workflow context in user interfaces
* Understand the impact of interruptions on user workflow
* Understand how to design software which fits into the larger working spheres in which information work takes place

It is important to point out that because workflow and multi-tasking rely heavily on higher-level brain functioning, it is unrealistic within the scope of this grant to propose a system which can predict user performance given a description of a set of arbitrary software programs. Therefore, we believe this module will function much more in a qualitative role to provide context to the rest of the model. Specifically, our findings related to interruption and multi-tasking will advance the basic research question of "how do you users react to interruptions when using working sets of varying sizes?". This core HCI contribution will help to inform the rest of the outputs of the model in a qualitative manner.

'''Inputs'''

N/A

'''Outputs'''

N/A

=== Working Memory Load ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''This module measures how much information the user needs to retain in memory while interacting with the interface and makes suggestions for improvements.''

* Models
** Beddeley's Model of Working Memory<ref>http://en.wikipedia.org/wiki/Baddeley%27s_model_of_working_memory</ref>
*** Episodic Buffer
** George Miller's "The magic number 7 plus or minus 2"<ref>http://en.wikipedia.org/wiki/The_Magical_Number_Seven,_Plus_or_Minus_Two</ref>
** The magic number 4<ref>Cowan, N. (2001). The magical number 4 in short-term memory: A reconsideration of mental storage capacity. Behavioral and Brain Sciences, 24, 87-185.</ref>
* Chunking<ref>http://en.wikipedia.org/wiki/Chunking_(psychology)</ref>
* Priming<ref>http://en.wikipedia.org/wiki/Priming_(psychology)</ref>
* Subitizing and Counting<ref>http://en.wikipedia.org/wiki/Subitizing</ref>

'''Inputs'''
* Visual Stimuli
* Audio Stimuli
'''Outputs'''
* Remembered percepts
* Half-Life of percepts

=== Automaticity of Interaction ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''Measures how easily the interaction with the interface becomes automatic with experience and makes suggestions for improvements.''

* Learning<ref>http://en.wikipedia.org/wiki/Learning#Mathematical_models_of_learning</ref>
** Logan's instance theory of automatization<ref>http://74.125.95.132/search?q=cache:IZ-Zccsu3SEJ:psych.wisc.edu/ugstudies/psych733/logan_1988.pdf+logan+isntance+teory&cd=1&hl=en&ct=clnk&gl=us&client=firefox-a</ref>
* Fluency
** As meta-cognitive information <ref>http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6VH9-4SM7PFK-4&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=10cd279fa80958981fcc3c06684c09af</ref>
** As a cognitive ''heuristic''<ref>http://en.wikipedia.org/wiki/Fluency_heuristic</ref>

'''Inputs'''
* Interface
* User goals

'''Outputs'''
* Learning curve

=== Anti-Pattern Conflict Resolution ===
* Owner: [[User: Ian Spector | Ian Spector]]

Interface design patterns are defined as reusable elements which provide solutions to common problems. For instance, we expect that an arrow in the top left corner of a window which points to the left, when clicked upon, will take the user to a previous screen. Furthermore, we expect that buttons in the top left corner of a window will in some way relate to navigation.

An anti-pattern is a design pattern which breaks standard design convention, creating more problems than it solves. An example of an anti-pattern would be if in place of the 'back' button on your web browser, there would be a 'view history' button.

The purpose of this module would be to analyze an interface to see if any anti-patterns exist, identify where they are in the interface, and then suggest alternatives.

'''Inputs'''
*Formal interface description
*Tasks which can be performed within the interface
*A library of standard design patterns
*Outputs from the 'Affordances' module
*Uncommon / Custom additional pattern library (optional)

'''Outputs'''
*Identification of interface elements whose placement or function are contrary to the pattern library
*Recommendations for alternative functionality or placement of such elements.

== Integration into the Design Process ==
* Owner: [[User:Ian Spector | Ian Spector]]

''This section outlines the process of designing an interface and at what stages our proposal fits in and how.''

The interface design process is critical to the creation of a quality end product. The process of creating an interface can also be used as a model for analyzing a finished one.

There are a number of different philosophies on how to best design software and in turn, interface. Currently, agile development using an incremental process such as [http://en.wikipedia.org/wiki/SCRUM Scrum] has become a well known and generally practiced procedure.

The steps to create interfaces varies significantly from text to text, although the [http://http://cfg.cit.cornell.edu/design/process.html Common Front Group at Cornell] has succinctly been able to reduce this variety into six simple steps:

*Requirement Sketching
*Conceptual Design
*Logical Design
*Physical Design
*Construction
*Usability Testing

This can be broken down further into just information architecture design followed by physical design and testing.

As far as this proposal is concerned in the context of interface design, the goal of our proposal is to improve what is involved with later end of the middle two portions: logical and physical design. Prior to feeding an interface to the system we are proposing, designers should have already created a baseline model for review that should exhibit the majority, if not all, of the functionality listed in the interface requirements. Once this initial interface has been created, our system will aid in rapidly iterating through the physical design process. The ultimate end products are then subject to human usability testing.

CS295J/Final contributions

2009-04-28T20:16:53Z

Trevor O'Brien: /* Recording User-Interaction Primitives */

= Specific Aims and Contributions (to be separated later) =

[David: I have removed some unattributed stuff that wasn't in the format described in assignment 13. If that was in error, go back to an earlier version to recover the text, attribute it, and put in the correct format.]

== Recording User-Interaction Primitives ==

* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

'''Aims'''
* Develop system for logging low-level user-interactions within existing applications. By low-level interactions, we refer to those interactions for which a quantitative, predictive, GOMS-like model may be generated.
* At the same scale of granularity, integrate explicit interface-interactions with multimodal-sensing data. i.e. pupil-tracking, muscle-activity monitoring, auditory recognition, EEG data, and facial and posture-focused video.

'''Contributions'''
* Extensible, multi-modal, HCI framework for recording rich interaction-history data in existing applications.

'''Demonstration'''
* Techniques for pupil-tracking, auditory-recognition, and muscle-activity monitoring will be evaluated with respect to accuracy, sampling rate, and computational cost in a series of quantitative studies.

'''Dependencies'''
* Hardware for pupil-tracking and muscle activity monitoring. Commercial software packages for such software may be required.

'''Background and Related Work'''
* The utility of interaction histories with respect to assessing interface design has been demonstrated in <ref name = bob> [http://www.cs.brown.edu/people/trevor/Papers/Heer-2008-GraphicalHistories.pdf Graphical Histories for Visualization: Supporting Analysis, Communication, and Evaluation] (InfoVis 2008) </ref>.
* In addition, data management histories have been shown effective in the visualization community in <ref> [http://www.cs.brown.edu/people/trevor/Papers/Callahan-2006-MED.pdf Callahan-2006-MED] </ref> <ref> [http://www.cs.brown.edu/people/trevor/Papers/Callahan-2006-VVM.pdf Callahan-2006-VVM] </ref> <ref> [http://www.cs.brown.edu/people/trevor/Papers/Bavoil-2005-VEI.pdf Bavoil-2005-VEI]</ref>, providing visualizations by analogy <ref> [http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=4376187&isnumber=4376125 Querying and Creating Visualizations by Analogy] </ref> and offering automated suggestions <ref> [http://www.cs.utah.edu/~juliana/pub/tvcg-recommendation2008.pdf VisComplete: Automating Suggestions from Visualization Pipelines] </ref>, which we expect to generalize to user interaction history.

== Semantic-level Interaction Chunking ==
[David: This section doesn't list demonstrations or dependencies. It also has aims, significance, and background, which were not part of assignment 13. Can it be re-phrased to better match the assignment?]

* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

'''Aims'''
* Develop techniques for ''chunking'' low-level interaction primitives into ''semantic-level'' interactions, given an application's functionality and data-context. (And de-chunking? Invertible mapping needed?)
* Perform user-study evaluation to validate chunking methods.

'''Contributions'''
* Design and user-study evaluation of semantic-chunking techniques for interaction.

'''Demonstration'''
* These techniques will be validated quantitatively with respect to efficiency and user error rate on a set of timed-tasks over a series of case studies. In addition, qualitative feedback on user satisfaction will be incorporated in the evaluation of our methods.

'''Dependencies'''
* Software framework for collecting user interactions.
* Formal description of interface functionality.
* Description of data objects that can be manipulated through interaction.

== Reconciling Usability Heuristics with Cognitive Theory ==
* Owner: [[User:E J Kalafarski|E J Kalafarski]] 14:56, 28 April 2009 (UTC)

'''Contribution''': A weighted framework for the unification of established heuristic usability guidelines and accepted cognitive principles.

[David: I'm having trouble parsing this contribution. How do you weight a framework? Also, I'd like to have a little more sense of how this might fit into the bigger picture. The assignment didn't ask for that, but is there some way to provide some of that context?]

'''Demonstration''': Three groups of experts anecdotally apply cognitive principles, heuristic usability principles, and a combination of the two.
* A "cognition expert," given a constrained, limited-functionality interface, develops an independent evaluative value for each interface element based on accepted cognitive principles.
* A "usability expert" develops an independent evaluative value for each interface element based on accepted heuristic guidelines.
* A third expert applies several unified cognitive analogues from a matrix of weighted cognitive and
* User testing demonstrates the assumed efficacy and applicability of the matricized analogues versus independent application of analogued principles.

'''Dependency''': A set of established cognitive principles, selected with an eye toward heuristic analogues.

'''Dependency''': A set of established heuristic design guidelines, selected with an eye toward cognitive analogues.

== Evaluation for Recommendation and Incremental Improvement ==
* Owner: [[User:E J Kalafarski|E J Kalafarski]] 14:56, 28 April 2009 (UTC)

'''Contributions''': Using a limited set of interface evaluation modules for analysis, we demonstrate, in a narrow and controlled manner, the proposed efficiency and accuracy of a method of aggregating individual interface suggestions based on accepted CPM principles (e.g. Fitts' Law and Affordance) and applying them to the incremental improvement of the interface.

[David: I'm a little lost. Can you make this long sentence into a couple shorter simpler ones? I think I can imagine what you are getting at, but I'm not sure.]

'''Demonstration''': A narrow but comprehensive study to demonstrate the efficacy and efficiency of automated aggregation of interface evaluation versus independent, analysis. Shows not total replacement, but a gain in speed of evaluation comparable to the loss of control and feature.
* Given a carefully-constrained interface, perhaps with as few as two buttons and a minimalist feature set, expert interface designer given the individual results of several basic evaluation modules makes recommendations and suggestions to the design of the interface.
* Aggregation meta-module conducts similar survey of module outputs, outputting recommendations and suggestions for improvement of given interface.
* Separate independent body of experts then implements the two sets of suggestions and performs user study on the resultant interfaces, analyzing usability change and comparing it to the time and resources committed by the evaluation expert and the aggregation module, respectively.

[David: nice demonstration!]

'''Dependency''': Module for the analysis of Fitts' Law as it applies to the individual elements of a given interface.

'''Dependency''': Module for the analysis of a Affordance as it applies to the individual elements of a given interface.

[David: I think this also has, as a dependency, some kind of framework for the modules. "narrow but comprehensive" sounds challenging. ]

== Evaluation Metrics ==
Owner: [[User:Gideon Goldin | Gideon Goldin]]

:*Architecture Outputs
::* Time (time to complete task)
:::# Performs as well or better than CPM-GOMS, demonstrated with user tasks
:::* Dependencies
:::# CPM-GOMS Module with cognitive load extension
::* Cognitive Load
:::# Predicts cognitive load during tasks, demonstrated with user tasks
:::* Dependencies
::::# CPM-GOMS Module with cognitive load extension
::::# Facial Gesture Recognition Module
::* Frustration
:::# Accurately predicts users' frustration levels, demonstrated with user tasks
:::* Dependencies
::::# Facial Gesture Recognition Module
::::# Galvanic Skin Response Module
::::# Interface Efficiency Module
::* Aesthetic Appeal
:::# Analyzes if the interface is aesthetically unpleasing, demonstrated with user tasks
:::* Dependencies
::::# Aesthetics Module
::* Simplicity
:::# Analyzes how simple the interface is, demonstrated with user tasks
:::* Dependencies
::::# Interface Efficiency Module

== Parallel Framework for Evaluation Modules ==
* Owner: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]]

''This section will describe in more detail the inputs, outputs and architecture that were presented in the introduction.''

==== Contributions ====

* '''Create a framework that provides better interface evaluations than currently existing techniques, and a module weighting system that provides better evaluations than any of its modules taken in isolation.'''

* Demonstrate by running user studies and comparing this performance to expected performance, as given by the following interface evaluation methods:
*# Traditional, manual interface evaluation
*#* As a baseline.
*# Using our system with a single module
*#* "Are any of our individual modules better than currently existing methods of interface evaluation?".
*# Using our system with multiple modules, but have aggregator give a fixed, equal weighting to each module
*#* As a baseline for our aggregator: want to show that the value of adding the dynamic weighting.
*# Using our system with multiple modules, and allow the aggregator to adjust weightings for each module, but have each module get a single weighting for all dimensions (time, fatigue, etc.)
*#* For validating the use of a dynamic weighting system.
*# Using our system with multiple modules, and allow the aggregator to adjust weightings for each module, and allow the module to give different weightings to every dimension of the module (time, fatigue, etc.)
*#* For validating the use of weighting across multiple utility dimensions.
*Dependencies: Having a good set of modules to plug into the framework.

[David: I have removed details of framework that seemed beyond contribs/demos/dependencies]

== Evaluation and Recommendation via Modules ==
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

A "meta-module" called the aggregator will be responsible for assembling and formatting the output of all other modules into a structure that is both extensible and immediately usable, by both an automated designer or a human designer.

===Requirements===
The aggregator's functionality, then, is defined by its '''inputs''', the outputs of the other modules, and the desired '''output''' of the system as a whole, per its position in the architecture. Its purpose is largely formatting and reconciliation of the products of the multitudinous (and extensible) modules. The output of the aggregator must meet several requirements: first, to generate a set of human-readable suggestions for the improvement of the given interface; second, to generate a machine-readable, but also analyzable, evaluation of the various characteristics of the interface and accompanying user traces.

From these specifications, it is logical to assume that a common language or format will be required for the output of individual modules. We propose an XML-based file format, allowing: (1) a section for the standardized identification of problem areas, applicable rules, and proposed improvements, generalized by the individual module and mapped to a single element, or group of elements, in the original interface specification; (2) a section for specification of generalizable "utility" functions, allowing a module to specify how much a measurable quantity of utility is positively or negatively affected by properties of the input interface; (3) new, user-definable sections for evaluations of the given interface not covered by the first two sections. The first two sections are capable of conveying the vast majority of module outputs predicted at this time, but the XML can extensibly allow modules to pass on whatever information may become prominent in the future.

===Specification===
<module id="Fitts-Law">
<interface-elements>
<element>
<desc>submit button</desc>
<problem>
<desc>size</desc>
<suggestion>width *= 2</suggestion>
<suggestion>height *= 2</suggestion>
<human-suggestion>Increase size relative to other elements</human-suggestion>
</problem>
</element>
</interface-elements>
<utility>
<dimension>
<desc>time</desc>
<value>0:15:35</value>
</dimension>
<dimension>
<desc>frustration</desc>
<value>pulling hair out</value>
</dimension>
<dimension>
<desc>efficiency</desc>
<value>13.2s/KPM task</value>
<value>0.56m/CPM task</value>
</dimension>
</utility>
<tasks>
<task>
<desc>complete form</desc>
</task>
<task>
<desc>lookup SSN</desc>
</task>
<task>
<desc>format phone number</desc>
</task>
</tasks>
</module>

===Logic===
This file provided by each module is then the input for the aggregator. The aggregator's most straightforward function is the compilation of the "problem areas," assembling them and noting problem areas and suggestions that are recommended by more than one module, and weighting them accordingly in its final report. These weightings can begin in an equal state, but the aggregator should be capable of learning iteratively which modules' results are most relevant to the user and update weightings accordingly. This may need to be accomplished with manual tuning, or a machine-learning algorithm capable of determining which modules most often agree with others.

Secondly, the aggregator compiles the utility functions provided by the module specs. This, again, is a summation of similarly-described values from the various modules.

When confronted with user-defined sections of the XML spec, the aggregator is primarily responsible for compiling them and sending them along to the output of the machine. Even if the aggregator does not recognize a section or property of the evaluative spec, if it sees the property reported by more than one module it should be capable of aggregating these intelligently. In future versions of the spec, it should be possible for a module to provide instructions for the aggregator on how to handle unrecognized sections of the XML.

From these compilations, then, the aggregator should be capable of outputting both aggregated human-readable suggestions on interface improvements for a human designer, as well as a comprehensive evaluation of the interface's effectiveness at the given task traces. Again, this is dependent on the specification of the system as a whole, but is likely to include measures and comparisons, graphings of task versus utility, and quantitative measures of an element's effectiveness.

''This section is necessarily defined by the output of the individual modules (which I already expect to be of varied and arbitrary structure) and the desired output of the machine as a whole. It will likely need to be revised heavily after other modules and the "Parellel Framework" section are defined.'' [[User:E J Kalafarski|E J Kalafarski]] 12:34, 24 April 2009 (UTC)

''This section describes the aggregator, which takes the output of multiple independent modules and aggregates the results to provide (1) an evaluation and (2) recommendations for the user interface. We should explain how the aggregator weights the output of different modules (this could be based on historical performance of each module, or perhaps based on E.J.'s cognitive/HCI guidelines).''

== Sample Modules ==

=== CPM-GOMS ===
* Owners: [[User:Steven Ellis | Steven Ellis]]

''This module will provide interface evaluations and suggestions based on a CPM-GOMS model of cognition for the given interface. It will provide a quantitative, predictive, cognition-based parameterization of usability. From empirically collected data, user trajectories through the model (critical paths) will be examined, highlighting bottlenecks within the interface, and offering suggested alterations to the interface to induce more optimal user trajectories.''

I’m hoping to have some input on this section, because it seems to be the crux of the “black box” into which we take the inputs of interface description, user traces, etc. and get our outputs (time, recommendations, etc.). I know at least a few people have pretty strong thoughts on the matter and we ought to discuss the final structure.

That said – my proposal for the module:
*In my opinion the concept of the Model Human Processor (at least as applied in CPM) is outdated – it’s too economic/overly parsimonious in its conception of human activity. I think we need to create a structure which accounts for more realistic conditions of HCI including multitasking, aspects of distributed cognition (and other relevant uses of tools – as far as I can tell CPM doesn’t take into account any sort of productivity aids), executive control processes of attention, etc. ACT-R appears to take steps towards this but we would probably need to look at their algorithms to know for sure.
*Critical paths will continue to play an important role – we should in fact emphasize that part of this tool’s purpose will be a description not only of ways in which the interface should be modified to best fit a critical path, but also ways in which the user’s ought to be instructed in their use of the path. This feedback mechanism could be bidirectional – if the model’s predictions of the user’s goals are incorrect, the critical path determined will also be incorrect and the interface inherently suboptimal. The user could be prompted with a tooltip explaining in brief why and how the interface has changed, along with options to revert, select other configurations (euphemized by goals), and to view a short video detailing how to properly use the interface.
*Call me crazy but, if we assume designers will be willing to code a model of their interfaces into our ACT-R-esque language, could we allow that model to be fairly transparent to the user, who could use a gui to input their goals to find an analogue in the program which would subsequently rearrange its interface to fit the user’s needs? Even if not useful to the users, such dynamic modeling could really help designers (IMO)
*I think the model should do its best to accept models written for ACT-R and whatever other cognitive models there are out there – gives us the best chance of early adoption
*I would particularly appreciate input on the number/complexity/type of inputs we’ll be using, as well as the same qualities for the output.

=== HCI Guidelines ===
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

Schneiderman's Eight Golden Rules and Jakob Nielsen's Ten Heuristics are perhaps the most famous and well-regarded heuristic design guidelines to emerge over the last twenty years. Although the explicit theoretical basis for such heuristics is controversial and not well-explored, the empirical success of these guidelines is established and accepted. This module will parse out up to three or four common (that is, intersecting) principles from these accepted guidelines and apply them to the input interface.

As an example, we identify an analogous principle that appears in Schneiderman ("Reduce short-term memory load")<ref>http://faculty.washington.edu/jtenenbg/courses/360/f04/sessions/schneidermanGoldenRules.html</ref> and Nielsen ("Recognition rather than recall/Minimize the user's memory load")<ref>http://www.useit.com/papers/heuristic/heuristic_list.html</ref>. The input interface is then evaluated for the consideration of the principle, based on an explicit formal description of the interface, such as XAML or XUL. The module attempts to determine how effectively the interface demonstrates the principle. When analyzing an interface for several principles that may be conflicting or opposing in a given context, the module makes use of a hard-coded but iterative (and evolving) weighting of these principles, based on (1) how often they appear in the training set of accepted sets of guidelines, (2) how analogues a heuristic principle is to a cognitive principle in a parallel training set, and (3) how effective the principle's associated suggestion is found to be using a feedback mechanism.

'''Inputs''' 
# A formal description of the interface and its elements (e.g. buttons).
# A formal description of a particular task and the possible paths through a subset of interface elements that permit the user to accomplish that task.

'''Output''' 
Standard XML-formatted file containing problem areas of the input interface, suggestions for each problem area based on principles that were found to have a strong application to a problem element and the problem itself, and a human-readable generated analysis of the element's affinity for the principle. Quantitative outputs will not be possible based on heuristic guidelines, and the "utility" section of this module's output is likely to be blank.

''This section could include an example or two of established design guidelines that could easily be implemented as modules.''

=== Fitts's Law ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

''This module provides an estimate of the required time to complete various tasks that have been decomposed into formalized sequences of interactions with interface elements, and will provide evaluations and recommendations for optimizing the time required to complete those tasks using the interface.''

'''Inputs'''

1. A formal description of the interface and its elements (e.g. buttons).

2. A formal description of a particular task and the possible paths through a subset of interface elements that permit the user to accomplish that task.

3. The physical distances between interface elements along those paths.

4. The width of those elements along the most likely axes of motion.

5. Device (e.g. mouse) characteristics including start/stop time and the inherent speed limitations of the device.

'''Output'''

The module will then use the Shannon formulation of Fitt's Law to compute the average time needed to complete the task along those paths.

=== Affordances ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

''This simple module will provide interface evaluations and recommendations based on a measure of the extent to which the user perceives the relevant affordances of the interface when performing a number of specified tasks.''

'''Inputs'''

Formalized descriptions of...

1. Interface elements

2. Their associated actions

3. The functions of those actions

4. A particular task

5. User traces for that task.

Inputs (1-4) are then used to generate a "user-independent" space of possible functions that the interface is capable of performing with respect to a given task -- what the interface "affords" the user. From this set of possible interactions, our model will then determine the subset of optimal paths for performing a particular task. The user trace (5) is then used to determine what functions actually were performed in the course of a given task of interest and this information is then compared to the optimal path data to determine the extent to which affordances of the interface are present but not perceived.

'''Output'''

The output of this module is a simple ratio of (affordances perceived) / [(relevant affordances present) * (time to complete task)] which provides a quantitative measure of the extent to which the interface is "natural" to use for a particular task.

=== Workflow, Multi-tasking and Interruptions ===
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

[David: I think that the work proposed here is interesting, but I'm not sure that it is organized by "contribution". Try labeling the contributions and the demonstrations explicitly. A "tool" is usually not a contribution -- an evaluation of the tool could be. If this distinction isn't clear, let's talk. For these particular contributions, phrasing and integrating them is a challenge because some of the contributions are demonstrations of other contributions...]

::* Scientific Study of Multi-tasking Workflow and the Impact of Interruptions
:::# We will undertake detailed studies to help understand the following questions:
::::# How does the size of a user's working set impact interruption resumption time?
::::# How does the size of a user's working set, when used for rapid multi-tasking, impact performance metrics?
::::# How does a user interface which supports multiple simultaneous working sets benefit interruption resumption time?
:::* No Dependencies
::* Meta-work Assistance Tool
:::# We will perform a series of ecologically-valid studies to compare user performance between a state of the art task management system (control group) and our meta-work assistance tool (experimental group)
:::* Dependent on core study completion, as some of the specific design decisions will be driven by the results of this study. However, it is worth pointing out that this separate contribution can be researched in parallel to the core study.
::* Baseline Comparison Between Module-based Model of HCI and Core Multi-tasking Study
:::# We will compare the results of the above-mentioned study in multi-tasking against results predicted by the module-based model of HCI in this proposal; this will give us an important baseline comparison, particularly given that multi-tasking and interruption involve higher brain functioning and are therefore likely difficult to predict
:::* Dependent on core study completion, as well as most of the rest of the proposal being completed to the point of being testable

Text for Assignment 12:

Add text here about how this can be used to evaluate automatic framework

There are, at least, two levels at which users work ([http://portal.acm.org/citation.cfm?id=985692.985707&coll=portal&dl=ACM&CFID=20781736&CFTOKEN=83176621 Gonzales, et al., 2004]). Users accomplish individual low-level tasks which are part of larger ''working spheres''; for example, an office worker might send several emails, create several Post-It (TM) note reminders, and then edit a word document, each of these smaller tasks being part of a single larger working sphere of "adding a new section to the website." Thus, it is important to understand this larger workflow context - which often involves extensive levels of multi-tasking, as well as switching between a variety of computing devices and traditional tools, such as notebooks. In this study it was found that the information workers surveyed typically switch individual tasks every 2 minutes and have many simultaneous working spheres which they switch between, on average every 12 minutes. This frenzied pace of switching tasks and switching working spheres suggests that users will not be using a single application or device for a long period of time, and that affordances to support this characteristic pattern of information work are important.

The purpose of this module is to integrate existing work on multi-tasking, interruption and higher-level workflow into a framework which can predict user recovery times from interruptions. Specifically, the goals of this framework will be to:

* Understand the role of the larger workflow context in user interfaces
* Understand the impact of interruptions on user workflow
* Understand how to design software which fits into the larger working spheres in which information work takes place

It is important to point out that because workflow and multi-tasking rely heavily on higher-level brain functioning, it is unrealistic within the scope of this grant to propose a system which can predict user performance given a description of a set of arbitrary software programs. Therefore, we believe this module will function much more in a qualitative role to provide context to the rest of the model. Specifically, our findings related to interruption and multi-tasking will advance the basic research question of "how do you users react to interruptions when using working sets of varying sizes?". This core HCI contribution will help to inform the rest of the outputs of the model in a qualitative manner.

'''Inputs'''

N/A

'''Outputs'''

N/A

=== Working Memory Load ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''This module measures how much information the user needs to retain in memory while interacting with the interface and makes suggestions for improvements.''

* Models
** Beddeley's Model of Working Memory<ref>http://en.wikipedia.org/wiki/Baddeley%27s_model_of_working_memory</ref>
*** Episodic Buffer
** George Miller's "The magic number 7 plus or minus 2"<ref>http://en.wikipedia.org/wiki/The_Magical_Number_Seven,_Plus_or_Minus_Two</ref>
** The magic number 4<ref>Cowan, N. (2001). The magical number 4 in short-term memory: A reconsideration of mental storage capacity. Behavioral and Brain Sciences, 24, 87-185.</ref>
* Chunking<ref>http://en.wikipedia.org/wiki/Chunking_(psychology)</ref>
* Priming<ref>http://en.wikipedia.org/wiki/Priming_(psychology)</ref>
* Subitizing and Counting<ref>http://en.wikipedia.org/wiki/Subitizing</ref>

'''Inputs'''
* Visual Stimuli
* Audio Stimuli
'''Outputs'''
* Remembered percepts
* Half-Life of percepts

=== Automaticity of Interaction ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''Measures how easily the interaction with the interface becomes automatic with experience and makes suggestions for improvements.''

* Learning<ref>http://en.wikipedia.org/wiki/Learning#Mathematical_models_of_learning</ref>
** Logan's instance theory of automatization<ref>http://74.125.95.132/search?q=cache:IZ-Zccsu3SEJ:psych.wisc.edu/ugstudies/psych733/logan_1988.pdf+logan+isntance+teory&cd=1&hl=en&ct=clnk&gl=us&client=firefox-a</ref>
* Fluency
** As meta-cognitive information <ref>http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6VH9-4SM7PFK-4&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=10cd279fa80958981fcc3c06684c09af</ref>
** As a cognitive ''heuristic''<ref>http://en.wikipedia.org/wiki/Fluency_heuristic</ref>

'''Inputs'''
* Interface
* User goals

'''Outputs'''
* Learning curve

=== Anti-Pattern Conflict Resolution ===
* Owner: [[User: Ian Spector | Ian Spector]]

Interface design patterns are defined as reusable elements which provide solutions to common problems. For instance, we expect that an arrow in the top left corner of a window which points to the left, when clicked upon, will take the user to a previous screen. Furthermore, we expect that buttons in the top left corner of a window will in some way relate to navigation.

An anti-pattern is a design pattern which breaks standard design convention, creating more problems than it solves. An example of an anti-pattern would be if in place of the 'back' button on your web browser, there would be a 'view history' button.

The purpose of this module would be to analyze an interface to see if any anti-patterns exist, identify where they are in the interface, and then suggest alternatives.

'''Inputs'''
*Formal interface description
*Tasks which can be performed within the interface
*A library of standard design patterns
*Outputs from the 'Affordances' module
*Uncommon / Custom additional pattern library (optional)

'''Outputs'''
*Identification of interface elements whose placement or function are contrary to the pattern library
*Recommendations for alternative functionality or placement of such elements.

== Integration into the Design Process ==
* Owner: [[User:Ian Spector | Ian Spector]]

''This section outlines the process of designing an interface and at what stages our proposal fits in and how.''

The interface design process is critical to the creation of a quality end product. The process of creating an interface can also be used as a model for analyzing a finished one.

There are a number of different philosophies on how to best design software and in turn, interface. Currently, agile development using an incremental process such as [http://en.wikipedia.org/wiki/SCRUM Scrum] has become a well known and generally practiced procedure.

The steps to create interfaces varies significantly from text to text, although the [http://http://cfg.cit.cornell.edu/design/process.html Common Front Group at Cornell] has succinctly been able to reduce this variety into six simple steps:

*Requirement Sketching
*Conceptual Design
*Logical Design
*Physical Design
*Construction
*Usability Testing

This can be broken down further into just information architecture design followed by physical design and testing.

As far as this proposal is concerned in the context of interface design, the goal of our proposal is to improve what is involved with later end of the middle two portions: logical and physical design. Prior to feeding an interface to the system we are proposing, designers should have already created a baseline model for review that should exhibit the majority, if not all, of the functionality listed in the interface requirements. Once this initial interface has been created, our system will aid in rapidly iterating through the physical design process. The ultimate end products are then subject to human usability testing.

CS295J/Final contributions

2009-04-28T20:10:21Z

Trevor O'Brien: /* Semantic-level Interaction Chunking */

= Specific Aims and Contributions (to be separated later) =

[David: I have removed some unattributed stuff that wasn't in the format described in assignment 13. If that was in error, go back to an earlier version to recover the text, attribute it, and put in the correct format.]

== Recording User-Interaction Primitives ==

[David: This section doesn't list demonstrations or dependencies. It also has aims, significance, and background, which were not part of assignment 13. Can it be re-phrased to better match the assignment?]

* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

'''Aims'''
* Develop system for logging low-level user-interactions within existing applications. By low-level interactions, we refer to those interactions for which a quantitative, predictive, GOMS-like model may be generated.
* At the same scale of granularity, integrate explicit interface-interactions with multimodal-sensing data. i.e. pupil-tracking, muscle-activity monitoring, auditory recognition, EEG data, and facial and posture-focused video.

'''Contributions'''
* Extensible, multimodal, HCI framework for recording rich interaction-history data in existing applications.

'''Significance'''
* Provide rich history data to serve as the basis for novel quantitative interface evaluation.
* Aid expert HCI and visual designers in traditional design processes.
* Provide data for automated machine-learning strategies applied to interaction.

'''Background and Related Work'''
* The utility of interaction histories with respect to assessing interface design has been demonstrated in <ref name = bob> [http://www.cs.brown.edu/people/trevor/Papers/Heer-2008-GraphicalHistories.pdf Graphical Histories for Visualization: Supporting Analysis, Communication, and Evaluation] (InfoVis 2008) </ref>.
* In addition, data management histories have been shown effective in the visualization community in <ref> [http://www.cs.brown.edu/people/trevor/Papers/Callahan-2006-MED.pdf Callahan-2006-MED] </ref> <ref> [http://www.cs.brown.edu/people/trevor/Papers/Callahan-2006-VVM.pdf Callahan-2006-VVM] </ref> <ref> [http://www.cs.brown.edu/people/trevor/Papers/Bavoil-2005-VEI.pdf Bavoil-2005-VEI]</ref>, providing visualizations by analogy <ref> [http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=4376187&isnumber=4376125 Querying and Creating Visualizations by Analogy] </ref> and offering automated suggestions <ref> [http://www.cs.utah.edu/~juliana/pub/tvcg-recommendation2008.pdf VisComplete: Automating Suggestions from Visualization Pipelines] </ref>, which we expect to generalize to user interaction history.

== Semantic-level Interaction Chunking ==
[David: This section doesn't list demonstrations or dependencies. It also has aims, significance, and background, which were not part of assignment 13. Can it be re-phrased to better match the assignment?]

* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

'''Aims'''
* Develop techniques for ''chunking'' low-level interaction primitives into ''semantic-level'' interactions, given an application's functionality and data-context. (And de-chunking? Invertible mapping needed?)
* Perform user-study evaluation to validate chunking methods.

'''Contributions'''
* Design and user-study evaluation of semantic-chunking techniques for interaction.

'''Demonstration'''
* These techniques will be validated quantitatively with respect to efficiency and user error rate on a set of timed-tasks over a series of case studies. In addition, qualitative feedback on user satisfaction will be incorporated in the evaluation of our methods.

'''Dependencies'''
* Software framework for collecting user interactions.
* Formal description of interface functionality.
* Description of data objects that can be manipulated through interaction.

== Reconciling Usability Heuristics with Cognitive Theory ==
* Owner: [[User:E J Kalafarski|E J Kalafarski]] 14:56, 28 April 2009 (UTC)

'''Contribution''': A weighted framework for the unification of established heuristic usability guidelines and accepted cognitive principles.

[David: I'm having trouble parsing this contribution. How do you weight a framework? Also, I'd like to have a little more sense of how this might fit into the bigger picture. The assignment didn't ask for that, but is there some way to provide some of that context?]

'''Demonstration''': Three groups of experts anecdotally apply cognitive principles, heuristic usability principles, and a combination of the two.
* A "cognition expert," given a constrained, limited-functionality interface, develops an independent evaluative value for each interface element based on accepted cognitive principles.
* A "usability expert" develops an independent evaluative value for each interface element based on accepted heuristic guidelines.
* A third expert applies several unified cognitive analogues from a matrix of weighted cognitive and
* User testing demonstrates the assumed efficacy and applicability of the matricized analogues versus independent application of analogued principles.

'''Dependency''': A set of established cognitive principles, selected with an eye toward heuristic analogues.

'''Dependency''': A set of established heuristic design guidelines, selected with an eye toward cognitive analogues.

== Evaluation for Recommendation and Incremental Improvement ==
* Owner: [[User:E J Kalafarski|E J Kalafarski]] 14:56, 28 April 2009 (UTC)

'''Contributions''': Using a limited set of interface evaluation modules for analysis, we demonstrate, in a narrow and controlled manner, the proposed efficiency and accuracy of a method of aggregating individual interface suggestions based on accepted CPM principles (e.g. Fitts' Law and Affordance) and applying them to the incremental improvement of the interface.

[David: I'm a little lost. Can you make this long sentence into a couple shorter simpler ones? I think I can imagine what you are getting at, but I'm not sure.]

'''Demonstration''': A narrow but comprehensive study to demonstrate the efficacy and efficiency of automated aggregation of interface evaluation versus independent, analysis. Shows not total replacement, but a gain in speed of evaluation comparable to the loss of control and feature.
* Given a carefully-constrained interface, perhaps with as few as two buttons and a minimalist feature set, expert interface designer given the individual results of several basic evaluation modules makes recommendations and suggestions to the design of the interface.
* Aggregation meta-module conducts similar survey of module outputs, outputting recommendations and suggestions for improvement of given interface.
* Separate independent body of experts then implements the two sets of suggestions and performs user study on the resultant interfaces, analyzing usability change and comparing it to the time and resources committed by the evaluation expert and the aggregation module, respectively.

[David: nice demonstration!]

'''Dependency''': Module for the analysis of Fitts' Law as it applies to the individual elements of a given interface.

'''Dependency''': Module for the analysis of a Affordance as it applies to the individual elements of a given interface.

[David: I think this also has, as a dependency, some kind of framework for the modules. "narrow but comprehensive" sounds challenging. ]

== Evaluation Metrics ==
Owner: [[User:Gideon Goldin | Gideon Goldin]]

:*Architecture Outputs
::* Time (time to complete task)
:::# Performs as well or better than CPM-GOMS, demonstrated with user tasks
:::* Dependencies
:::# CPM-GOMS Module with cognitive load extension
::* Cognitive Load
:::# Predicts cognitive load during tasks, demonstrated with user tasks
:::* Dependencies
::::# CPM-GOMS Module with cognitive load extension
::::# Facial Gesture Recognition Module
::* Frustration
:::# Accurately predicts users' frustration levels, demonstrated with user tasks
:::* Dependencies
::::# Facial Gesture Recognition Module
::::# Galvanic Skin Response Module
::::# Interface Efficiency Module
::* Aesthetic Appeal
:::# Analyzes if the interface is aesthetically unpleasing, demonstrated with user tasks
:::* Dependencies
::::# Aesthetics Module
::* Simplicity
:::# Analyzes how simple the interface is, demonstrated with user tasks
:::* Dependencies
::::# Interface Efficiency Module

== Parallel Framework for Evaluation Modules ==
* Owner: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]]

''This section will describe in more detail the inputs, outputs and architecture that were presented in the introduction.''

==== Contributions ====

* '''Create a framework that provides better interface evaluations than currently existing techniques, and a module weighting system that provides better evaluations than any of its modules taken in isolation.'''

* Demonstrate by running user studies and comparing this performance to expected performance, as given by the following interface evaluation methods:
*# Traditional, manual interface evaluation
*#* As a baseline.
*# Using our system with a single module
*#* "Are any of our individual modules better than currently existing methods of interface evaluation?".
*# Using our system with multiple modules, but have aggregator give a fixed, equal weighting to each module
*#* As a baseline for our aggregator: want to show that the value of adding the dynamic weighting.
*# Using our system with multiple modules, and allow the aggregator to adjust weightings for each module, but have each module get a single weighting for all dimensions (time, fatigue, etc.)
*#* For validating the use of a dynamic weighting system.
*# Using our system with multiple modules, and allow the aggregator to adjust weightings for each module, and allow the module to give different weightings to every dimension of the module (time, fatigue, etc.)
*#* For validating the use of weighting across multiple utility dimensions.
*Dependencies: Having a good set of modules to plug into the framework.

[David: I have removed details of framework that seemed beyond contribs/demos/dependencies]

== Evaluation and Recommendation via Modules ==
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

A "meta-module" called the aggregator will be responsible for assembling and formatting the output of all other modules into a structure that is both extensible and immediately usable, by both an automated designer or a human designer.

===Requirements===
The aggregator's functionality, then, is defined by its '''inputs''', the outputs of the other modules, and the desired '''output''' of the system as a whole, per its position in the architecture. Its purpose is largely formatting and reconciliation of the products of the multitudinous (and extensible) modules. The output of the aggregator must meet several requirements: first, to generate a set of human-readable suggestions for the improvement of the given interface; second, to generate a machine-readable, but also analyzable, evaluation of the various characteristics of the interface and accompanying user traces.

From these specifications, it is logical to assume that a common language or format will be required for the output of individual modules. We propose an XML-based file format, allowing: (1) a section for the standardized identification of problem areas, applicable rules, and proposed improvements, generalized by the individual module and mapped to a single element, or group of elements, in the original interface specification; (2) a section for specification of generalizable "utility" functions, allowing a module to specify how much a measurable quantity of utility is positively or negatively affected by properties of the input interface; (3) new, user-definable sections for evaluations of the given interface not covered by the first two sections. The first two sections are capable of conveying the vast majority of module outputs predicted at this time, but the XML can extensibly allow modules to pass on whatever information may become prominent in the future.

===Specification===
<module id="Fitts-Law">
<interface-elements>
<element>
<desc>submit button</desc>
<problem>
<desc>size</desc>
<suggestion>width *= 2</suggestion>
<suggestion>height *= 2</suggestion>
<human-suggestion>Increase size relative to other elements</human-suggestion>
</problem>
</element>
</interface-elements>
<utility>
<dimension>
<desc>time</desc>
<value>0:15:35</value>
</dimension>
<dimension>
<desc>frustration</desc>
<value>pulling hair out</value>
</dimension>
<dimension>
<desc>efficiency</desc>
<value>13.2s/KPM task</value>
<value>0.56m/CPM task</value>
</dimension>
</utility>
<tasks>
<task>
<desc>complete form</desc>
</task>
<task>
<desc>lookup SSN</desc>
</task>
<task>
<desc>format phone number</desc>
</task>
</tasks>
</module>

===Logic===
This file provided by each module is then the input for the aggregator. The aggregator's most straightforward function is the compilation of the "problem areas," assembling them and noting problem areas and suggestions that are recommended by more than one module, and weighting them accordingly in its final report. These weightings can begin in an equal state, but the aggregator should be capable of learning iteratively which modules' results are most relevant to the user and update weightings accordingly. This may need to be accomplished with manual tuning, or a machine-learning algorithm capable of determining which modules most often agree with others.

Secondly, the aggregator compiles the utility functions provided by the module specs. This, again, is a summation of similarly-described values from the various modules.

When confronted with user-defined sections of the XML spec, the aggregator is primarily responsible for compiling them and sending them along to the output of the machine. Even if the aggregator does not recognize a section or property of the evaluative spec, if it sees the property reported by more than one module it should be capable of aggregating these intelligently. In future versions of the spec, it should be possible for a module to provide instructions for the aggregator on how to handle unrecognized sections of the XML.

From these compilations, then, the aggregator should be capable of outputting both aggregated human-readable suggestions on interface improvements for a human designer, as well as a comprehensive evaluation of the interface's effectiveness at the given task traces. Again, this is dependent on the specification of the system as a whole, but is likely to include measures and comparisons, graphings of task versus utility, and quantitative measures of an element's effectiveness.

''This section is necessarily defined by the output of the individual modules (which I already expect to be of varied and arbitrary structure) and the desired output of the machine as a whole. It will likely need to be revised heavily after other modules and the "Parellel Framework" section are defined.'' [[User:E J Kalafarski|E J Kalafarski]] 12:34, 24 April 2009 (UTC)

''This section describes the aggregator, which takes the output of multiple independent modules and aggregates the results to provide (1) an evaluation and (2) recommendations for the user interface. We should explain how the aggregator weights the output of different modules (this could be based on historical performance of each module, or perhaps based on E.J.'s cognitive/HCI guidelines).''

== Sample Modules ==

=== CPM-GOMS ===
* Owners: [[User:Steven Ellis | Steven Ellis]]

''This module will provide interface evaluations and suggestions based on a CPM-GOMS model of cognition for the given interface. It will provide a quantitative, predictive, cognition-based parameterization of usability. From empirically collected data, user trajectories through the model (critical paths) will be examined, highlighting bottlenecks within the interface, and offering suggested alterations to the interface to induce more optimal user trajectories.''

I’m hoping to have some input on this section, because it seems to be the crux of the “black box” into which we take the inputs of interface description, user traces, etc. and get our outputs (time, recommendations, etc.). I know at least a few people have pretty strong thoughts on the matter and we ought to discuss the final structure.

That said – my proposal for the module:
*In my opinion the concept of the Model Human Processor (at least as applied in CPM) is outdated – it’s too economic/overly parsimonious in its conception of human activity. I think we need to create a structure which accounts for more realistic conditions of HCI including multitasking, aspects of distributed cognition (and other relevant uses of tools – as far as I can tell CPM doesn’t take into account any sort of productivity aids), executive control processes of attention, etc. ACT-R appears to take steps towards this but we would probably need to look at their algorithms to know for sure.
*Critical paths will continue to play an important role – we should in fact emphasize that part of this tool’s purpose will be a description not only of ways in which the interface should be modified to best fit a critical path, but also ways in which the user’s ought to be instructed in their use of the path. This feedback mechanism could be bidirectional – if the model’s predictions of the user’s goals are incorrect, the critical path determined will also be incorrect and the interface inherently suboptimal. The user could be prompted with a tooltip explaining in brief why and how the interface has changed, along with options to revert, select other configurations (euphemized by goals), and to view a short video detailing how to properly use the interface.
*Call me crazy but, if we assume designers will be willing to code a model of their interfaces into our ACT-R-esque language, could we allow that model to be fairly transparent to the user, who could use a gui to input their goals to find an analogue in the program which would subsequently rearrange its interface to fit the user’s needs? Even if not useful to the users, such dynamic modeling could really help designers (IMO)
*I think the model should do its best to accept models written for ACT-R and whatever other cognitive models there are out there – gives us the best chance of early adoption
*I would particularly appreciate input on the number/complexity/type of inputs we’ll be using, as well as the same qualities for the output.

=== HCI Guidelines ===
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

Schneiderman's Eight Golden Rules and Jakob Nielsen's Ten Heuristics are perhaps the most famous and well-regarded heuristic design guidelines to emerge over the last twenty years. Although the explicit theoretical basis for such heuristics is controversial and not well-explored, the empirical success of these guidelines is established and accepted. This module will parse out up to three or four common (that is, intersecting) principles from these accepted guidelines and apply them to the input interface.

As an example, we identify an analogous principle that appears in Schneiderman ("Reduce short-term memory load")<ref>http://faculty.washington.edu/jtenenbg/courses/360/f04/sessions/schneidermanGoldenRules.html</ref> and Nielsen ("Recognition rather than recall/Minimize the user's memory load")<ref>http://www.useit.com/papers/heuristic/heuristic_list.html</ref>. The input interface is then evaluated for the consideration of the principle, based on an explicit formal description of the interface, such as XAML or XUL. The module attempts to determine how effectively the interface demonstrates the principle. When analyzing an interface for several principles that may be conflicting or opposing in a given context, the module makes use of a hard-coded but iterative (and evolving) weighting of these principles, based on (1) how often they appear in the training set of accepted sets of guidelines, (2) how analogues a heuristic principle is to a cognitive principle in a parallel training set, and (3) how effective the principle's associated suggestion is found to be using a feedback mechanism.

'''Inputs''' 
# A formal description of the interface and its elements (e.g. buttons).
# A formal description of a particular task and the possible paths through a subset of interface elements that permit the user to accomplish that task.

'''Output''' 
Standard XML-formatted file containing problem areas of the input interface, suggestions for each problem area based on principles that were found to have a strong application to a problem element and the problem itself, and a human-readable generated analysis of the element's affinity for the principle. Quantitative outputs will not be possible based on heuristic guidelines, and the "utility" section of this module's output is likely to be blank.

''This section could include an example or two of established design guidelines that could easily be implemented as modules.''

=== Fitts's Law ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

''This module provides an estimate of the required time to complete various tasks that have been decomposed into formalized sequences of interactions with interface elements, and will provide evaluations and recommendations for optimizing the time required to complete those tasks using the interface.''

'''Inputs'''

1. A formal description of the interface and its elements (e.g. buttons).

2. A formal description of a particular task and the possible paths through a subset of interface elements that permit the user to accomplish that task.

3. The physical distances between interface elements along those paths.

4. The width of those elements along the most likely axes of motion.

5. Device (e.g. mouse) characteristics including start/stop time and the inherent speed limitations of the device.

'''Output'''

The module will then use the Shannon formulation of Fitt's Law to compute the average time needed to complete the task along those paths.

=== Affordances ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

''This simple module will provide interface evaluations and recommendations based on a measure of the extent to which the user perceives the relevant affordances of the interface when performing a number of specified tasks.''

'''Inputs'''

Formalized descriptions of...

1. Interface elements

2. Their associated actions

3. The functions of those actions

4. A particular task

5. User traces for that task.

Inputs (1-4) are then used to generate a "user-independent" space of possible functions that the interface is capable of performing with respect to a given task -- what the interface "affords" the user. From this set of possible interactions, our model will then determine the subset of optimal paths for performing a particular task. The user trace (5) is then used to determine what functions actually were performed in the course of a given task of interest and this information is then compared to the optimal path data to determine the extent to which affordances of the interface are present but not perceived.

'''Output'''

The output of this module is a simple ratio of (affordances perceived) / [(relevant affordances present) * (time to complete task)] which provides a quantitative measure of the extent to which the interface is "natural" to use for a particular task.

=== Workflow, Multi-tasking and Interruptions ===
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

[David: I think that the work proposed here is interesting, but I'm not sure that it is organized by "contribution". Try labeling the contributions and the demonstrations explicitly. A "tool" is usually not a contribution -- an evaluation of the tool could be. If this distinction isn't clear, let's talk. For these particular contributions, phrasing and integrating them is a challenge because some of the contributions are demonstrations of other contributions...]

::* Scientific Study of Multi-tasking Workflow and the Impact of Interruptions
:::# We will undertake detailed studies to help understand the following questions:
::::# How does the size of a user's working set impact interruption resumption time?
::::# How does the size of a user's working set, when used for rapid multi-tasking, impact performance metrics?
::::# How does a user interface which supports multiple simultaneous working sets benefit interruption resumption time?
:::* No Dependencies
::* Meta-work Assistance Tool
:::# We will perform a series of ecologically-valid studies to compare user performance between a state of the art task management system (control group) and our meta-work assistance tool (experimental group)
:::* Dependent on core study completion, as some of the specific design decisions will be driven by the results of this study. However, it is worth pointing out that this separate contribution can be researched in parallel to the core study.
::* Baseline Comparison Between Module-based Model of HCI and Core Multi-tasking Study
:::# We will compare the results of the above-mentioned study in multi-tasking against results predicted by the module-based model of HCI in this proposal; this will give us an important baseline comparison, particularly given that multi-tasking and interruption involve higher brain functioning and are therefore likely difficult to predict
:::* Dependent on core study completion, as well as most of the rest of the proposal being completed to the point of being testable

Text for Assignment 12:

Add text here about how this can be used to evaluate automatic framework

There are, at least, two levels at which users work ([http://portal.acm.org/citation.cfm?id=985692.985707&coll=portal&dl=ACM&CFID=20781736&CFTOKEN=83176621 Gonzales, et al., 2004]). Users accomplish individual low-level tasks which are part of larger ''working spheres''; for example, an office worker might send several emails, create several Post-It (TM) note reminders, and then edit a word document, each of these smaller tasks being part of a single larger working sphere of "adding a new section to the website." Thus, it is important to understand this larger workflow context - which often involves extensive levels of multi-tasking, as well as switching between a variety of computing devices and traditional tools, such as notebooks. In this study it was found that the information workers surveyed typically switch individual tasks every 2 minutes and have many simultaneous working spheres which they switch between, on average every 12 minutes. This frenzied pace of switching tasks and switching working spheres suggests that users will not be using a single application or device for a long period of time, and that affordances to support this characteristic pattern of information work are important.

The purpose of this module is to integrate existing work on multi-tasking, interruption and higher-level workflow into a framework which can predict user recovery times from interruptions. Specifically, the goals of this framework will be to:

* Understand the role of the larger workflow context in user interfaces
* Understand the impact of interruptions on user workflow
* Understand how to design software which fits into the larger working spheres in which information work takes place

It is important to point out that because workflow and multi-tasking rely heavily on higher-level brain functioning, it is unrealistic within the scope of this grant to propose a system which can predict user performance given a description of a set of arbitrary software programs. Therefore, we believe this module will function much more in a qualitative role to provide context to the rest of the model. Specifically, our findings related to interruption and multi-tasking will advance the basic research question of "how do you users react to interruptions when using working sets of varying sizes?". This core HCI contribution will help to inform the rest of the outputs of the model in a qualitative manner.

'''Inputs'''

N/A

'''Outputs'''

N/A

=== Working Memory Load ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''This module measures how much information the user needs to retain in memory while interacting with the interface and makes suggestions for improvements.''

* Models
** Beddeley's Model of Working Memory<ref>http://en.wikipedia.org/wiki/Baddeley%27s_model_of_working_memory</ref>
*** Episodic Buffer
** George Miller's "The magic number 7 plus or minus 2"<ref>http://en.wikipedia.org/wiki/The_Magical_Number_Seven,_Plus_or_Minus_Two</ref>
** The magic number 4<ref>Cowan, N. (2001). The magical number 4 in short-term memory: A reconsideration of mental storage capacity. Behavioral and Brain Sciences, 24, 87-185.</ref>
* Chunking<ref>http://en.wikipedia.org/wiki/Chunking_(psychology)</ref>
* Priming<ref>http://en.wikipedia.org/wiki/Priming_(psychology)</ref>
* Subitizing and Counting<ref>http://en.wikipedia.org/wiki/Subitizing</ref>

'''Inputs'''
* Visual Stimuli
* Audio Stimuli
'''Outputs'''
* Remembered percepts
* Half-Life of percepts

=== Automaticity of Interaction ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''Measures how easily the interaction with the interface becomes automatic with experience and makes suggestions for improvements.''

* Learning<ref>http://en.wikipedia.org/wiki/Learning#Mathematical_models_of_learning</ref>
** Logan's instance theory of automatization<ref>http://74.125.95.132/search?q=cache:IZ-Zccsu3SEJ:psych.wisc.edu/ugstudies/psych733/logan_1988.pdf+logan+isntance+teory&cd=1&hl=en&ct=clnk&gl=us&client=firefox-a</ref>
* Fluency
** As meta-cognitive information <ref>http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6VH9-4SM7PFK-4&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=10cd279fa80958981fcc3c06684c09af</ref>
** As a cognitive ''heuristic''<ref>http://en.wikipedia.org/wiki/Fluency_heuristic</ref>

'''Inputs'''
* Interface
* User goals

'''Outputs'''
* Learning curve

=== Anti-Pattern Conflict Resolution ===
* Owner: [[User: Ian Spector | Ian Spector]]

Interface design patterns are defined as reusable elements which provide solutions to common problems. For instance, we expect that an arrow in the top left corner of a window which points to the left, when clicked upon, will take the user to a previous screen. Furthermore, we expect that buttons in the top left corner of a window will in some way relate to navigation.

An anti-pattern is a design pattern which breaks standard design convention, creating more problems than it solves. An example of an anti-pattern would be if in place of the 'back' button on your web browser, there would be a 'view history' button.

The purpose of this module would be to analyze an interface to see if any anti-patterns exist, identify where they are in the interface, and then suggest alternatives.

'''Inputs'''
*Formal interface description
*Tasks which can be performed within the interface
*A library of standard design patterns
*Outputs from the 'Affordances' module
*Uncommon / Custom additional pattern library (optional)

'''Outputs'''
*Identification of interface elements whose placement or function are contrary to the pattern library
*Recommendations for alternative functionality or placement of such elements.

== Integration into the Design Process ==
* Owner: [[User:Ian Spector | Ian Spector]]

''This section outlines the process of designing an interface and at what stages our proposal fits in and how.''

The interface design process is critical to the creation of a quality end product. The process of creating an interface can also be used as a model for analyzing a finished one.

There are a number of different philosophies on how to best design software and in turn, interface. Currently, agile development using an incremental process such as [http://en.wikipedia.org/wiki/SCRUM Scrum] has become a well known and generally practiced procedure.

The steps to create interfaces varies significantly from text to text, although the [http://http://cfg.cit.cornell.edu/design/process.html Common Front Group at Cornell] has succinctly been able to reduce this variety into six simple steps:

*Requirement Sketching
*Conceptual Design
*Logical Design
*Physical Design
*Construction
*Usability Testing

This can be broken down further into just information architecture design followed by physical design and testing.

As far as this proposal is concerned in the context of interface design, the goal of our proposal is to improve what is involved with later end of the middle two portions: logical and physical design. Prior to feeding an interface to the system we are proposing, designers should have already created a baseline model for review that should exhibit the majority, if not all, of the functionality listed in the interface requirements. Once this initial interface has been created, our system will aid in rapidly iterating through the physical design process. The ultimate end products are then subject to human usability testing.

CS295J/Final contributions

2009-04-28T20:09:48Z

Trevor O'Brien: /* Semantic-level Interaction Chunking */

= Specific Aims and Contributions (to be separated later) =

[David: I have removed some unattributed stuff that wasn't in the format described in assignment 13. If that was in error, go back to an earlier version to recover the text, attribute it, and put in the correct format.]

== Recording User-Interaction Primitives ==

[David: This section doesn't list demonstrations or dependencies. It also has aims, significance, and background, which were not part of assignment 13. Can it be re-phrased to better match the assignment?]

* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

'''Aims'''
* Develop system for logging low-level user-interactions within existing applications. By low-level interactions, we refer to those interactions for which a quantitative, predictive, GOMS-like model may be generated.
* At the same scale of granularity, integrate explicit interface-interactions with multimodal-sensing data. i.e. pupil-tracking, muscle-activity monitoring, auditory recognition, EEG data, and facial and posture-focused video.

'''Contributions'''
* Extensible, multimodal, HCI framework for recording rich interaction-history data in existing applications.

'''Significance'''
* Provide rich history data to serve as the basis for novel quantitative interface evaluation.
* Aid expert HCI and visual designers in traditional design processes.
* Provide data for automated machine-learning strategies applied to interaction.

'''Background and Related Work'''
* The utility of interaction histories with respect to assessing interface design has been demonstrated in <ref name = bob> [http://www.cs.brown.edu/people/trevor/Papers/Heer-2008-GraphicalHistories.pdf Graphical Histories for Visualization: Supporting Analysis, Communication, and Evaluation] (InfoVis 2008) </ref>.
* In addition, data management histories have been shown effective in the visualization community in <ref> [http://www.cs.brown.edu/people/trevor/Papers/Callahan-2006-MED.pdf Callahan-2006-MED] </ref> <ref> [http://www.cs.brown.edu/people/trevor/Papers/Callahan-2006-VVM.pdf Callahan-2006-VVM] </ref> <ref> [http://www.cs.brown.edu/people/trevor/Papers/Bavoil-2005-VEI.pdf Bavoil-2005-VEI]</ref>, providing visualizations by analogy <ref> [http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=4376187&isnumber=4376125 Querying and Creating Visualizations by Analogy] </ref> and offering automated suggestions <ref> [http://www.cs.utah.edu/~juliana/pub/tvcg-recommendation2008.pdf VisComplete: Automating Suggestions from Visualization Pipelines] </ref>, which we expect to generalize to user interaction history.

== Semantic-level Interaction Chunking ==
[David: This section doesn't list demonstrations or dependencies. It also has aims, significance, and background, which were not part of assignment 13. Can it be re-phrased to better match the assignment?]

* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

'''Aims'''
* Develop techniques for ''chunking'' low-level interaction primitives into ''semantic-level'' interactions, given an application's functionality and data-context. (And de-chunking? Invertible mapping needed?)
* Perform user-study evaluation to validate chunking methods.

'''Contributions'''
* Design and user-study evaluation of semantic chunking techniques.

'''Demonstration'''
* These techniques will be validated quantitatively with respect to efficiency and user error rate on a set of timed-tasks over a series of case studies. In addition, qualitative feedback on user satisfaction will be incorporated in the evaluation of our methods.

'''Dependencies'''
* Software framework for collecting user interactions.
* Formal description of interface functionality.
* Description of data objects that can be manipulated through interaction.

== Reconciling Usability Heuristics with Cognitive Theory ==
* Owner: [[User:E J Kalafarski|E J Kalafarski]] 14:56, 28 April 2009 (UTC)

'''Contribution''': A weighted framework for the unification of established heuristic usability guidelines and accepted cognitive principles.

[David: I'm having trouble parsing this contribution. How do you weight a framework? Also, I'd like to have a little more sense of how this might fit into the bigger picture. The assignment didn't ask for that, but is there some way to provide some of that context?]

'''Demonstration''': Three groups of experts anecdotally apply cognitive principles, heuristic usability principles, and a combination of the two.
* A "cognition expert," given a constrained, limited-functionality interface, develops an independent evaluative value for each interface element based on accepted cognitive principles.
* A "usability expert" develops an independent evaluative value for each interface element based on accepted heuristic guidelines.
* A third expert applies several unified cognitive analogues from a matrix of weighted cognitive and
* User testing demonstrates the assumed efficacy and applicability of the matricized analogues versus independent application of analogued principles.

'''Dependency''': A set of established cognitive principles, selected with an eye toward heuristic analogues.

'''Dependency''': A set of established heuristic design guidelines, selected with an eye toward cognitive analogues.

== Evaluation for Recommendation and Incremental Improvement ==
* Owner: [[User:E J Kalafarski|E J Kalafarski]] 14:56, 28 April 2009 (UTC)

'''Contributions''': Using a limited set of interface evaluation modules for analysis, we demonstrate, in a narrow and controlled manner, the proposed efficiency and accuracy of a method of aggregating individual interface suggestions based on accepted CPM principles (e.g. Fitts' Law and Affordance) and applying them to the incremental improvement of the interface.

[David: I'm a little lost. Can you make this long sentence into a couple shorter simpler ones? I think I can imagine what you are getting at, but I'm not sure.]

'''Demonstration''': A narrow but comprehensive study to demonstrate the efficacy and efficiency of automated aggregation of interface evaluation versus independent, analysis. Shows not total replacement, but a gain in speed of evaluation comparable to the loss of control and feature.
* Given a carefully-constrained interface, perhaps with as few as two buttons and a minimalist feature set, expert interface designer given the individual results of several basic evaluation modules makes recommendations and suggestions to the design of the interface.
* Aggregation meta-module conducts similar survey of module outputs, outputting recommendations and suggestions for improvement of given interface.
* Separate independent body of experts then implements the two sets of suggestions and performs user study on the resultant interfaces, analyzing usability change and comparing it to the time and resources committed by the evaluation expert and the aggregation module, respectively.

[David: nice demonstration!]

'''Dependency''': Module for the analysis of Fitts' Law as it applies to the individual elements of a given interface.

'''Dependency''': Module for the analysis of a Affordance as it applies to the individual elements of a given interface.

[David: I think this also has, as a dependency, some kind of framework for the modules. "narrow but comprehensive" sounds challenging. ]

== Evaluation Metrics ==
Owner: [[User:Gideon Goldin | Gideon Goldin]]

:*Architecture Outputs
::* Time (time to complete task)
:::# Performs as well or better than CPM-GOMS, demonstrated with user tasks
:::* Dependencies
:::# CPM-GOMS Module with cognitive load extension
::* Cognitive Load
:::# Predicts cognitive load during tasks, demonstrated with user tasks
:::* Dependencies
::::# CPM-GOMS Module with cognitive load extension
::::# Facial Gesture Recognition Module
::* Frustration
:::# Accurately predicts users' frustration levels, demonstrated with user tasks
:::* Dependencies
::::# Facial Gesture Recognition Module
::::# Galvanic Skin Response Module
::::# Interface Efficiency Module
::* Aesthetic Appeal
:::# Analyzes if the interface is aesthetically unpleasing, demonstrated with user tasks
:::* Dependencies
::::# Aesthetics Module
::* Simplicity
:::# Analyzes how simple the interface is, demonstrated with user tasks
:::* Dependencies
::::# Interface Efficiency Module

== Parallel Framework for Evaluation Modules ==
* Owner: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]]

''This section will describe in more detail the inputs, outputs and architecture that were presented in the introduction.''

==== Contributions ====

* '''Create a framework that provides better interface evaluations than currently existing techniques, and a module weighting system that provides better evaluations than any of its modules taken in isolation.'''

* Demonstrate by running user studies and comparing this performance to expected performance, as given by the following interface evaluation methods:
*# Traditional, manual interface evaluation
*#* As a baseline.
*# Using our system with a single module
*#* "Are any of our individual modules better than currently existing methods of interface evaluation?".
*# Using our system with multiple modules, but have aggregator give a fixed, equal weighting to each module
*#* As a baseline for our aggregator: want to show that the value of adding the dynamic weighting.
*# Using our system with multiple modules, and allow the aggregator to adjust weightings for each module, but have each module get a single weighting for all dimensions (time, fatigue, etc.)
*#* For validating the use of a dynamic weighting system.
*# Using our system with multiple modules, and allow the aggregator to adjust weightings for each module, and allow the module to give different weightings to every dimension of the module (time, fatigue, etc.)
*#* For validating the use of weighting across multiple utility dimensions.
*Dependencies: Having a good set of modules to plug into the framework.

[David: I have removed details of framework that seemed beyond contribs/demos/dependencies]

== Evaluation and Recommendation via Modules ==
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

A "meta-module" called the aggregator will be responsible for assembling and formatting the output of all other modules into a structure that is both extensible and immediately usable, by both an automated designer or a human designer.

===Requirements===
The aggregator's functionality, then, is defined by its '''inputs''', the outputs of the other modules, and the desired '''output''' of the system as a whole, per its position in the architecture. Its purpose is largely formatting and reconciliation of the products of the multitudinous (and extensible) modules. The output of the aggregator must meet several requirements: first, to generate a set of human-readable suggestions for the improvement of the given interface; second, to generate a machine-readable, but also analyzable, evaluation of the various characteristics of the interface and accompanying user traces.

From these specifications, it is logical to assume that a common language or format will be required for the output of individual modules. We propose an XML-based file format, allowing: (1) a section for the standardized identification of problem areas, applicable rules, and proposed improvements, generalized by the individual module and mapped to a single element, or group of elements, in the original interface specification; (2) a section for specification of generalizable "utility" functions, allowing a module to specify how much a measurable quantity of utility is positively or negatively affected by properties of the input interface; (3) new, user-definable sections for evaluations of the given interface not covered by the first two sections. The first two sections are capable of conveying the vast majority of module outputs predicted at this time, but the XML can extensibly allow modules to pass on whatever information may become prominent in the future.

===Specification===
<module id="Fitts-Law">
<interface-elements>
<element>
<desc>submit button</desc>
<problem>
<desc>size</desc>
<suggestion>width *= 2</suggestion>
<suggestion>height *= 2</suggestion>
<human-suggestion>Increase size relative to other elements</human-suggestion>
</problem>
</element>
</interface-elements>
<utility>
<dimension>
<desc>time</desc>
<value>0:15:35</value>
</dimension>
<dimension>
<desc>frustration</desc>
<value>pulling hair out</value>
</dimension>
<dimension>
<desc>efficiency</desc>
<value>13.2s/KPM task</value>
<value>0.56m/CPM task</value>
</dimension>
</utility>
<tasks>
<task>
<desc>complete form</desc>
</task>
<task>
<desc>lookup SSN</desc>
</task>
<task>
<desc>format phone number</desc>
</task>
</tasks>
</module>

===Logic===
This file provided by each module is then the input for the aggregator. The aggregator's most straightforward function is the compilation of the "problem areas," assembling them and noting problem areas and suggestions that are recommended by more than one module, and weighting them accordingly in its final report. These weightings can begin in an equal state, but the aggregator should be capable of learning iteratively which modules' results are most relevant to the user and update weightings accordingly. This may need to be accomplished with manual tuning, or a machine-learning algorithm capable of determining which modules most often agree with others.

Secondly, the aggregator compiles the utility functions provided by the module specs. This, again, is a summation of similarly-described values from the various modules.

When confronted with user-defined sections of the XML spec, the aggregator is primarily responsible for compiling them and sending them along to the output of the machine. Even if the aggregator does not recognize a section or property of the evaluative spec, if it sees the property reported by more than one module it should be capable of aggregating these intelligently. In future versions of the spec, it should be possible for a module to provide instructions for the aggregator on how to handle unrecognized sections of the XML.

From these compilations, then, the aggregator should be capable of outputting both aggregated human-readable suggestions on interface improvements for a human designer, as well as a comprehensive evaluation of the interface's effectiveness at the given task traces. Again, this is dependent on the specification of the system as a whole, but is likely to include measures and comparisons, graphings of task versus utility, and quantitative measures of an element's effectiveness.

''This section is necessarily defined by the output of the individual modules (which I already expect to be of varied and arbitrary structure) and the desired output of the machine as a whole. It will likely need to be revised heavily after other modules and the "Parellel Framework" section are defined.'' [[User:E J Kalafarski|E J Kalafarski]] 12:34, 24 April 2009 (UTC)

''This section describes the aggregator, which takes the output of multiple independent modules and aggregates the results to provide (1) an evaluation and (2) recommendations for the user interface. We should explain how the aggregator weights the output of different modules (this could be based on historical performance of each module, or perhaps based on E.J.'s cognitive/HCI guidelines).''

== Sample Modules ==

=== CPM-GOMS ===
* Owners: [[User:Steven Ellis | Steven Ellis]]

''This module will provide interface evaluations and suggestions based on a CPM-GOMS model of cognition for the given interface. It will provide a quantitative, predictive, cognition-based parameterization of usability. From empirically collected data, user trajectories through the model (critical paths) will be examined, highlighting bottlenecks within the interface, and offering suggested alterations to the interface to induce more optimal user trajectories.''

I’m hoping to have some input on this section, because it seems to be the crux of the “black box” into which we take the inputs of interface description, user traces, etc. and get our outputs (time, recommendations, etc.). I know at least a few people have pretty strong thoughts on the matter and we ought to discuss the final structure.

That said – my proposal for the module:
*In my opinion the concept of the Model Human Processor (at least as applied in CPM) is outdated – it’s too economic/overly parsimonious in its conception of human activity. I think we need to create a structure which accounts for more realistic conditions of HCI including multitasking, aspects of distributed cognition (and other relevant uses of tools – as far as I can tell CPM doesn’t take into account any sort of productivity aids), executive control processes of attention, etc. ACT-R appears to take steps towards this but we would probably need to look at their algorithms to know for sure.
*Critical paths will continue to play an important role – we should in fact emphasize that part of this tool’s purpose will be a description not only of ways in which the interface should be modified to best fit a critical path, but also ways in which the user’s ought to be instructed in their use of the path. This feedback mechanism could be bidirectional – if the model’s predictions of the user’s goals are incorrect, the critical path determined will also be incorrect and the interface inherently suboptimal. The user could be prompted with a tooltip explaining in brief why and how the interface has changed, along with options to revert, select other configurations (euphemized by goals), and to view a short video detailing how to properly use the interface.
*Call me crazy but, if we assume designers will be willing to code a model of their interfaces into our ACT-R-esque language, could we allow that model to be fairly transparent to the user, who could use a gui to input their goals to find an analogue in the program which would subsequently rearrange its interface to fit the user’s needs? Even if not useful to the users, such dynamic modeling could really help designers (IMO)
*I think the model should do its best to accept models written for ACT-R and whatever other cognitive models there are out there – gives us the best chance of early adoption
*I would particularly appreciate input on the number/complexity/type of inputs we’ll be using, as well as the same qualities for the output.

=== HCI Guidelines ===
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

Schneiderman's Eight Golden Rules and Jakob Nielsen's Ten Heuristics are perhaps the most famous and well-regarded heuristic design guidelines to emerge over the last twenty years. Although the explicit theoretical basis for such heuristics is controversial and not well-explored, the empirical success of these guidelines is established and accepted. This module will parse out up to three or four common (that is, intersecting) principles from these accepted guidelines and apply them to the input interface.

As an example, we identify an analogous principle that appears in Schneiderman ("Reduce short-term memory load")<ref>http://faculty.washington.edu/jtenenbg/courses/360/f04/sessions/schneidermanGoldenRules.html</ref> and Nielsen ("Recognition rather than recall/Minimize the user's memory load")<ref>http://www.useit.com/papers/heuristic/heuristic_list.html</ref>. The input interface is then evaluated for the consideration of the principle, based on an explicit formal description of the interface, such as XAML or XUL. The module attempts to determine how effectively the interface demonstrates the principle. When analyzing an interface for several principles that may be conflicting or opposing in a given context, the module makes use of a hard-coded but iterative (and evolving) weighting of these principles, based on (1) how often they appear in the training set of accepted sets of guidelines, (2) how analogues a heuristic principle is to a cognitive principle in a parallel training set, and (3) how effective the principle's associated suggestion is found to be using a feedback mechanism.

'''Inputs''' 
# A formal description of the interface and its elements (e.g. buttons).
# A formal description of a particular task and the possible paths through a subset of interface elements that permit the user to accomplish that task.

'''Output''' 
Standard XML-formatted file containing problem areas of the input interface, suggestions for each problem area based on principles that were found to have a strong application to a problem element and the problem itself, and a human-readable generated analysis of the element's affinity for the principle. Quantitative outputs will not be possible based on heuristic guidelines, and the "utility" section of this module's output is likely to be blank.

''This section could include an example or two of established design guidelines that could easily be implemented as modules.''

=== Fitts's Law ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

''This module provides an estimate of the required time to complete various tasks that have been decomposed into formalized sequences of interactions with interface elements, and will provide evaluations and recommendations for optimizing the time required to complete those tasks using the interface.''

'''Inputs'''

1. A formal description of the interface and its elements (e.g. buttons).

2. A formal description of a particular task and the possible paths through a subset of interface elements that permit the user to accomplish that task.

3. The physical distances between interface elements along those paths.

4. The width of those elements along the most likely axes of motion.

5. Device (e.g. mouse) characteristics including start/stop time and the inherent speed limitations of the device.

'''Output'''

The module will then use the Shannon formulation of Fitt's Law to compute the average time needed to complete the task along those paths.

=== Affordances ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

''This simple module will provide interface evaluations and recommendations based on a measure of the extent to which the user perceives the relevant affordances of the interface when performing a number of specified tasks.''

'''Inputs'''

Formalized descriptions of...

1. Interface elements

2. Their associated actions

3. The functions of those actions

4. A particular task

5. User traces for that task.

Inputs (1-4) are then used to generate a "user-independent" space of possible functions that the interface is capable of performing with respect to a given task -- what the interface "affords" the user. From this set of possible interactions, our model will then determine the subset of optimal paths for performing a particular task. The user trace (5) is then used to determine what functions actually were performed in the course of a given task of interest and this information is then compared to the optimal path data to determine the extent to which affordances of the interface are present but not perceived.

'''Output'''

The output of this module is a simple ratio of (affordances perceived) / [(relevant affordances present) * (time to complete task)] which provides a quantitative measure of the extent to which the interface is "natural" to use for a particular task.

=== Workflow, Multi-tasking and Interruptions ===
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

[David: I think that the work proposed here is interesting, but I'm not sure that it is organized by "contribution". Try labeling the contributions and the demonstrations explicitly. A "tool" is usually not a contribution -- an evaluation of the tool could be. If this distinction isn't clear, let's talk. For these particular contributions, phrasing and integrating them is a challenge because some of the contributions are demonstrations of other contributions...]

::* Scientific Study of Multi-tasking Workflow and the Impact of Interruptions
:::# We will undertake detailed studies to help understand the following questions:
::::# How does the size of a user's working set impact interruption resumption time?
::::# How does the size of a user's working set, when used for rapid multi-tasking, impact performance metrics?
::::# How does a user interface which supports multiple simultaneous working sets benefit interruption resumption time?
:::* No Dependencies
::* Meta-work Assistance Tool
:::# We will perform a series of ecologically-valid studies to compare user performance between a state of the art task management system (control group) and our meta-work assistance tool (experimental group)
:::* Dependent on core study completion, as some of the specific design decisions will be driven by the results of this study. However, it is worth pointing out that this separate contribution can be researched in parallel to the core study.
::* Baseline Comparison Between Module-based Model of HCI and Core Multi-tasking Study
:::# We will compare the results of the above-mentioned study in multi-tasking against results predicted by the module-based model of HCI in this proposal; this will give us an important baseline comparison, particularly given that multi-tasking and interruption involve higher brain functioning and are therefore likely difficult to predict
:::* Dependent on core study completion, as well as most of the rest of the proposal being completed to the point of being testable

Text for Assignment 12:

Add text here about how this can be used to evaluate automatic framework

There are, at least, two levels at which users work ([http://portal.acm.org/citation.cfm?id=985692.985707&coll=portal&dl=ACM&CFID=20781736&CFTOKEN=83176621 Gonzales, et al., 2004]). Users accomplish individual low-level tasks which are part of larger ''working spheres''; for example, an office worker might send several emails, create several Post-It (TM) note reminders, and then edit a word document, each of these smaller tasks being part of a single larger working sphere of "adding a new section to the website." Thus, it is important to understand this larger workflow context - which often involves extensive levels of multi-tasking, as well as switching between a variety of computing devices and traditional tools, such as notebooks. In this study it was found that the information workers surveyed typically switch individual tasks every 2 minutes and have many simultaneous working spheres which they switch between, on average every 12 minutes. This frenzied pace of switching tasks and switching working spheres suggests that users will not be using a single application or device for a long period of time, and that affordances to support this characteristic pattern of information work are important.

The purpose of this module is to integrate existing work on multi-tasking, interruption and higher-level workflow into a framework which can predict user recovery times from interruptions. Specifically, the goals of this framework will be to:

* Understand the role of the larger workflow context in user interfaces
* Understand the impact of interruptions on user workflow
* Understand how to design software which fits into the larger working spheres in which information work takes place

It is important to point out that because workflow and multi-tasking rely heavily on higher-level brain functioning, it is unrealistic within the scope of this grant to propose a system which can predict user performance given a description of a set of arbitrary software programs. Therefore, we believe this module will function much more in a qualitative role to provide context to the rest of the model. Specifically, our findings related to interruption and multi-tasking will advance the basic research question of "how do you users react to interruptions when using working sets of varying sizes?". This core HCI contribution will help to inform the rest of the outputs of the model in a qualitative manner.

'''Inputs'''

N/A

'''Outputs'''

N/A

=== Working Memory Load ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''This module measures how much information the user needs to retain in memory while interacting with the interface and makes suggestions for improvements.''

* Models
** Beddeley's Model of Working Memory<ref>http://en.wikipedia.org/wiki/Baddeley%27s_model_of_working_memory</ref>
*** Episodic Buffer
** George Miller's "The magic number 7 plus or minus 2"<ref>http://en.wikipedia.org/wiki/The_Magical_Number_Seven,_Plus_or_Minus_Two</ref>
** The magic number 4<ref>Cowan, N. (2001). The magical number 4 in short-term memory: A reconsideration of mental storage capacity. Behavioral and Brain Sciences, 24, 87-185.</ref>
* Chunking<ref>http://en.wikipedia.org/wiki/Chunking_(psychology)</ref>
* Priming<ref>http://en.wikipedia.org/wiki/Priming_(psychology)</ref>
* Subitizing and Counting<ref>http://en.wikipedia.org/wiki/Subitizing</ref>

'''Inputs'''
* Visual Stimuli
* Audio Stimuli
'''Outputs'''
* Remembered percepts
* Half-Life of percepts

=== Automaticity of Interaction ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''Measures how easily the interaction with the interface becomes automatic with experience and makes suggestions for improvements.''

* Learning<ref>http://en.wikipedia.org/wiki/Learning#Mathematical_models_of_learning</ref>
** Logan's instance theory of automatization<ref>http://74.125.95.132/search?q=cache:IZ-Zccsu3SEJ:psych.wisc.edu/ugstudies/psych733/logan_1988.pdf+logan+isntance+teory&cd=1&hl=en&ct=clnk&gl=us&client=firefox-a</ref>
* Fluency
** As meta-cognitive information <ref>http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6VH9-4SM7PFK-4&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=10cd279fa80958981fcc3c06684c09af</ref>
** As a cognitive ''heuristic''<ref>http://en.wikipedia.org/wiki/Fluency_heuristic</ref>

'''Inputs'''
* Interface
* User goals

'''Outputs'''
* Learning curve

=== Anti-Pattern Conflict Resolution ===
* Owner: [[User: Ian Spector | Ian Spector]]

Interface design patterns are defined as reusable elements which provide solutions to common problems. For instance, we expect that an arrow in the top left corner of a window which points to the left, when clicked upon, will take the user to a previous screen. Furthermore, we expect that buttons in the top left corner of a window will in some way relate to navigation.

An anti-pattern is a design pattern which breaks standard design convention, creating more problems than it solves. An example of an anti-pattern would be if in place of the 'back' button on your web browser, there would be a 'view history' button.

The purpose of this module would be to analyze an interface to see if any anti-patterns exist, identify where they are in the interface, and then suggest alternatives.

'''Inputs'''
*Formal interface description
*Tasks which can be performed within the interface
*A library of standard design patterns
*Outputs from the 'Affordances' module
*Uncommon / Custom additional pattern library (optional)

'''Outputs'''
*Identification of interface elements whose placement or function are contrary to the pattern library
*Recommendations for alternative functionality or placement of such elements.

== Integration into the Design Process ==
* Owner: [[User:Ian Spector | Ian Spector]]

''This section outlines the process of designing an interface and at what stages our proposal fits in and how.''

The interface design process is critical to the creation of a quality end product. The process of creating an interface can also be used as a model for analyzing a finished one.

There are a number of different philosophies on how to best design software and in turn, interface. Currently, agile development using an incremental process such as [http://en.wikipedia.org/wiki/SCRUM Scrum] has become a well known and generally practiced procedure.

The steps to create interfaces varies significantly from text to text, although the [http://http://cfg.cit.cornell.edu/design/process.html Common Front Group at Cornell] has succinctly been able to reduce this variety into six simple steps:

*Requirement Sketching
*Conceptual Design
*Logical Design
*Physical Design
*Construction
*Usability Testing

This can be broken down further into just information architecture design followed by physical design and testing.

As far as this proposal is concerned in the context of interface design, the goal of our proposal is to improve what is involved with later end of the middle two portions: logical and physical design. Prior to feeding an interface to the system we are proposing, designers should have already created a baseline model for review that should exhibit the majority, if not all, of the functionality listed in the interface requirements. Once this initial interface has been created, our system will aid in rapidly iterating through the physical design process. The ultimate end products are then subject to human usability testing.

CS295J/Research proposal (draft 2)

2009-04-24T16:59:36Z

Trevor O'Brien: /* Semantic-level Interaction Chunking */

= Introduction =
* Owners: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]], [[User: Trevor O'Brien | Trevor]]

'''Propose:''' The design, application and evaluation of a novel, cognition-based, computational framework for assessing interface design and providing automated suggestions to optimize usability.

'''Evaluation Methodology:''' Our techniques will be evaluated quantitatively through a series of user-study trials, as well as qualitatively by a team of expert interface designers.

'''Contributions and Significance:''' We expect this work to make the following contributions:
# design-space analysis and quantitative evaluation of cognition-based techniques for assessing user interfaces.
# design and quantitative evaluation of techniques for suggesting optimized interface-design changes.
# an extensible, multimodal software architecture for capturing user traces integrated with pupil-tracking data, auditory recognition, and muscle-activity monitoring.
# specification (language?) of how to define an interface evaluation module and how to integrate it into a larger system.
# (there may be more here, like testing different cognitive models, generating a markup language to represent interfaces, maybe even a unique metric space for interface usability)

--

We propose a framework for interface evaluation and recommendation that integrates behavioral models and design guidelines from both cognitive science and HCI. Our framework behaves like a committee of specialized experts, where each expert provides its own assessment of the interface, given its particular knowledge of HCI or cognitive science. For example, an expert may provide an evaluation based on the GOMS method, Fitts's law, Maeda's design principles, or cognitive models of learning and memory. An aggregator collects all of these assessments and weights the opinions of each expert, and outputs to the developer a merged evaluation score and a weighted set of recommendations.

Systematic methods of estimating human performance with computer interfaces are used only sparsely despite their obvious benefits, the reason being the overhead involved in implementing them. In order to test an interface, both manual coding systems like the GOMS variations and user simulations like those based on ACT-R/PM and EPIC require detailed pseudo-code descriptions of the user workflow with the application interface. Any change to the interface then requires extensive changes to the pseudo-code, a major problem because of the trial-and-error nature of interface design. Updating the models themselves is even more complicated. Even an expert in CPM-GOMS, for example, can't necessarily adapt it to take into account results from new cognitive research.

Our proposal makes automatic interface evaluation easier to use in several ways. First of all, we propose to divide the input to the system into three separate parts, functionality, user traces and interface. By separating the functionality from the interface, even radical interface changes will require updating only that part of the input. The user traces are also defined over the functionality so that they too translate across different interfaces. Second, the parallel modular architecture allows for a lower "entry cost" for using the tool. The system includes a broad array of evaluation modules some of which are very simple and some more complex. The simpler modules use only a subset of the input that a system like GOMS or ACT-R would require. This means that while more input will still lead to better output, interface designers can get minimal evaluations with only minimal information. For example, a visual search module may not require any functionality or user traces in order to determine whether all interface elements are distinct enough to be easy to find. Finally, a parallel modular architecture is much easier to augment with relevant cognitive and design evaluations.

= Background / Related Work =

Each person should add the background related to their specific aims.

* Steven Ellis - Cognitive models of HCI, including GOMS variations and ACT-R
* EJ - Design Guidelines
* Jon - Perception and Action
* Andrew - Multiple task environments
* Gideon - Cognition and dual systems
* Ian - Interface design process
* Trevor - User trace collection methods (especially any eye-tracking, EEG, ... you want to suggest using)

== Cognitive Models ==
I plan to port over most of the background on cognitive models of HCI from the old proposal

Additions will comprise of:
*CPM-GOMS as a bridge from GOMS architecture to the promising procedural optimization of the Model Human Processor
**Context of CPM development, discuss its relation to original GOMS and KLM
***Establish the tasks which were relevant for optimization when CPM was developed and note that its obsolescence may have been unavoidable
**Focus on CPM as the first step in transitioning from descriptive data, provided by mounting efforts in the cognitive sciences realm to discover the nature of task processing and accomplishment, to prescriptive algorithms which can predict an interface’s efficiency and suggest improvements
**CPM’s purpose as an abstraction of cognitive processing – a symbolic representation not designed for accuracy but precision
**CPM’s successful trials, e.g. Ernestine
***Implications of this project include CPM’s ability to accurately estimate processing at a psychomotor level
***Project does suggest limitations, however, when one attempts to examine more complex tasks which involve deeper and more numerous cognitive processes
*ACT-R as an example of a progressive cognitive modeling tool
**A tool clearly built by and for cognitive scientists, and as a result presents a much more accurate view of human processing – helpful for our research
**Built-in automation, which now seems to be a standard feature of cognitive modeling tools
**Still an abstraction of cognitive processing, but makes adaptation to cutting-edge cognitive research findings an integral aspect of its modular structure
**Expand on its focus on multi-tasking, taking what was a huge advance between GOMS and its CPM variation and bringing the simulation several steps closer to approximating the nature of cognition in regards to HCI
**Far more accessible both for researchers and the lay user/designer in its portability to LISP, pre-construction of modules representing cognitive capacities and underlying algorithms modeling paths of cognitive processing

==Design guidelines==
A multitude of rule sets exist for the design of not only interface, but architecture, city planning, and software development. They can range in scale from one primary rule to as many Christopher Alexander's 253 rules for urban environments,<ref>[http://hci.rwth-aachen.de/materials/publications/borchers2000a.pdf Borchers, Jan O. "A Pattern Approach to Interaction Design." 2000.]</ref> which he introduced with the concept design patterns in the 1970s. Study has likewise been conducted on the use of these rules:<ref>http://stl.cs.queensu.ca/~graham/cisc836/lectures/readings/tetzlaff-guidelines.pdf</ref> guidelines are often only partially understood, indistinct to the developer, and "fraught" with potential usability problems given a real-world situation.

===Application to AUE===
And yet, the vast majority of guideline sets, including the most popular rulesets, have been arrived at heuristically. The most successful, such as Raskin's and Schneiderman's, have been forged from years of observation instead of empirical study and experimentation. The problem is similar to the problem of circular logic faced by automated usability evaluations: an automated system is limited in the suggestions it can offer to a set of preprogrammed guidelines which have often not been subjected to rigorous experimentation.<ref>[http://www.eecs.berkeley.edu/Pubs/TechRpts/2000/CSD-00-1105.pdf Ivory, M and Hearst, M. "The State of the Art in Automated Usability Evaluation of User Interfaces." ACM Computing Surveys (CSUR), 2001.]</ref> In the vast majority of existing studies, emphasis has traditionally been placed on either the development of guidelines or the application of existing guidelines to automated evaluation. A mutually-reinforcing development of both simultaneously has not been attempted.

Overlap between rulesets is inevitable and unavoidable. For our purposes of evaluating existing rulesets efficiently, without extracting and analyzing each rule individually, it may be desirable to identify the the overarching ''principles'' or ''philosophy'' (max. 2 or 3) for a given ruleset and determining their quantitative relevance to problems of cognition.

===Popular and seminal examples===
Schneiderman's [http://faculty.washington.edu/jtenenbg/courses/360/f04/sessions/schneidermanGoldenRules.html Eight Golden Rules] date to 1987 and are arguably the most-cited. They are heuristic, but can be somewhat classified by cognitive objective: at least two rules apply primarily to ''repeated use'', versus ''discoverability''. Up to five of Schneiderman's rules emphasize ''predictability'' in the outcomes of operations and ''increased feedback and control'' in the agency of the user. His final rule, paradoxically, removes control from the user by suggesting a reduced short-term memory load, which we can arguably classify as ''simplicity''.

Raskin's [http://www.mprove.de/script/02/raskin/designrules.html Design Rules] are classified into five principles by the author, augmented by definitions and supporting rules. While one principle is primarily aesthetic (a design problem arguably out of the bounds of this proposal) and one is a basic endorsement of testing, the remaining three begin to reflect philosophies similar to Schneiderman's: reliability or ''predictability'', ''simplicity'' or ''efficiency'' (which we can construe as two sides of the same coin), and finally he introduces a concept of ''uninterruptibility''.

Maeda's [http://lawsofsimplicity.com/?cat=5&order=ASC Laws of Simplicity] are fewer, and ostensibly emphasize ''simplicity'' exclusively, although elements of ''use'' as related by Schneiderman's rules and ''efficiency'' as defined by Raskin may be facets of this simplicity. Google's corporate mission statement presents [http://www.google.com/corporate/ux.html Ten Principles], only half of which can be considered true interface guidelines. ''Efficiency'' and ''simplicity'' are cited explicitly, aesthetics are once again noted as crucial, and working within a user's trust is another application of ''predictability''.

===Elements and goals of a guideline set===
Myriad rulesets exist, but variation becomes scarce—it indeed seems possible to parse these common rulesets into overarching principles that can be converted to or associated with quantifiable cognitive properties. For example, it is likely ''simplicity'' has an analogue in the short-term memory retention or visual retention of the user, vis a vis the rule of [http://books.google.com/books?hl=en&lr=&id=j5q0VvOGExYC&oi=fnd&pg=PA357&dq=seven+plus+or+minus+two&ots=prI3PKJBar&sig=vOZnqpnkXKGYWxK6_XlA4I_CRyI Seven, Plus or Minus Two]. ''Predictability'' likewise may have an analogue in Activity Theory, in regards to a user's perceptual expectations for a given action; ''uninterruptibility'' has implications in cognitive task-switching;<ref>[http://portal.acm.org/citation.cfm?id=985692.985715&coll=Portal&dl=ACM&CFID=21136843&CFTOKEN=23841774 Czerwinski, Horvitz, and White. "A Diary Study of Task Switching and Interruptions." Proceedings of the SIGCHI conference on Human factors in computing systems, 2004.]</ref> and so forth.

Within the scope of this proposal, we aim to reduce and refine these philosophies found in seminal rulesets and identify their logical cognitive analogues. By assigning a quantifiable taxonomy to these principles, we will be able to rank and weight them with regard to their real-world applicability, developing a set of "meta-guidelines" and rules for applying them to a given interface in an automated manner. Combined with cognitive models and multi-modal HCI analysis, we seek to develop, in parallel with these guidelines, the interface evaluation system responsible for their application.

== Perception and Action (in progress) ==

*Information Processing Approach
**Advantages
***Formalism eases translation of theory into scripting language
**Disadvantages
***Assumes symbolic representation

*Ecological (Gibsonian) Approach
**Advantages
***Emphasis on bodily and environmental constraints
**Disadvantages
***Lack of formalism hinders translation of theory into scripting language

= Specific Aims and Contributions (to be separated later) =

'''Specific Aims'''
* Incorporate interaction history mechanisms into a set of existing applications.
* Perform user-study evaluation of history-collection techniques.
* Distill a set of cognitive principles/models, and evaluate empirically?
* Build/buy sensing system to include pupil-tracking, muscle-activity monitoring, auditory recognition.
* Design techniques for manual/semi-automated/automated construction of <insert favorite cognitive model here> from interaction histories and sensing data.
* Design system for posterior analysis of interaction history w.r.t. <insert favorite cognitive model here>, evaluating critical path <or equivalent> trajectories.
* Design cognition-based techniques for detecting bottlenecks in critical paths, and offering optimized alternatives.
* Perform quantitative user-study evaluations, collect qualitative feedback from expert interface designers.

'''Contributions'''

* Design-space analysis and quantitative evaluation of cognition-based techniques for assessing user interfaces.
* Design and quantitative evaluation of techniques for suggesting optimized interface-design changes.
* An extensible, multimodal software architecture for capturing user traces integrated with pupil-tracking data, auditory recognition, and muscle-activity monitoring.
* (there may be more here, like testing different cognitive models, generating a markup language to represent interfaces, maybe even a unique metric space for interface usability)

--

See the [http://vrl.cs.brown.edu/wiki/images/b/b1/Flowchart2.pdf flowchart] for a visual overview of our aims.

In order to use this framework, a designer will have to provide:
* Functional specification - what are the possible interactions between the user and the application. This can be thought of as method signatures, with a name (e.g., setVolume), direction (to user or from user) and a list of value types (boolean, number, text, video, ...) for each interaction.
* GUI specification - a mapping of interactions to interface elements (e.g., setVolume is mapped to the grey knob in the bottom left corner with clockwise turning increasing the input number).
* Functional user traces - sequences of representative ways in which the application is used. Instead of writing them, the designer could have users use the application with a trial interface and then use our methods to generalize the user traces beyond the specific interface (The second method is depicted in the diagram). As a form of pre-processing, the system also generates an interaction transition matrix which lists the probability of each type of interaction given the previous interaction.
* Utility function - this is a weighting of various performance metrics (time, cognitive load, fatigue, etc.), where the weighting expresses the importance of a particular dimension to the user. For example, a user at NASA probably cares more about interface accuracy than speed. By passing this information to our committee of experts, we can create interfaces that are tuned to maximize the utility of a particular user type.

Each of the modules can use all of this information or a subset of it. Our approach stresses flexibility and the ability to give more meaningful feedback the more information is provided. After processing the information sent by the system of experts, the aggregator will output:
* An evaluation of the interface. Evaluations are expressed both in terms of the utility function components (i.e. time, fatigue, cognitive load, etc.), and in terms of the overall utility for this interface (as defined by the utility function). These evaluations are given in the form of an efficiency curve, where the utility received on each dimension can change as the user becomes more accustomed to the interface.
* Suggested improvements for the GUI are also output. These suggestions are meant to optimize the utility function that was input to the system. If a user values accuracy over time, interface suggestions will be made accordingly.

== Recording User-Interaction Primitives ==
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

'''Aims'''
* Develop system for logging low-level user-interactions within existing applications. By low-level interactions, we refer to those interactions for which a quantitative, predictive, GOMS-like model may be generated.
* At the same scale of granularity, integrate explicit interface-interactions with multimodal-sensing data. i.e. pupil-tracking, muscle-activity monitoring, auditory recognition, EEG data, and facial and posture-focused video.

'''Contributions'''
* Extensible, multimodal, HCI framework for recording rich interaction-history data in existing applications.

'''Significance'''
* Provide rich history data to serve as the basis for novel quantitative interface evaluation.
* Aid expert HCI and visual designers in traditional design processes.
* Provide data for automated machine-learning strategies applied to interaction.

'''Background and Related Work'''
* The utility of interaction histories with respect to assessing interface design has been demonstrated in <ref name = bob> [http://www.cs.brown.edu/people/trevor/Papers/Heer-2008-GraphicalHistories.pdf Graphical Histories for Visualization: Supporting Analysis, Communication, and Evaluation] (InfoVis 2008) </ref>.
* In addition, data management histories have been shown effective in the visualization community in <ref> [http://www.cs.brown.edu/people/trevor/Papers/Callahan-2006-MED.pdf Callahan-2006-MED] </ref> <ref> [http://www.cs.brown.edu/people/trevor/Papers/Callahan-2006-VVM.pdf Callahan-2006-VVM] </ref> <ref> [http://www.cs.brown.edu/people/trevor/Papers/Bavoil-2005-VEI.pdf Bavoil-2005-VEI]</ref>, providing visualizations by analogy <ref> [http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=4376187&isnumber=4376125 Querying and Creating Visualizations by Analogy] </ref> and offering automated suggestions <ref> [http://www.cs.utah.edu/~juliana/pub/tvcg-recommendation2008.pdf VisComplete: Automating Suggestions from Visualization Pipelines] </ref>, which we expect to generalize to user interaction history.

== Semantic-level Interaction Chunking ==
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

'''Aims'''
* Develop techniques for ''chunking'' low-level interaction primitives into ''semantic-level'' interactions, given an application's functionality and data-context. (And de-chunking? Invertible mapping needed?)
* Perform user-study evaluation to validate chunking methods.

'''Contributions'''
* Design and user-study evaluation of semantic chunking techniques.

'''Significance'''
* Low-level <--> semantic-level mapping allows for cognitive-modeling to be applied at a functionality level, where low-level interaction techniques can be swapped out. This will allow our interface assessment system to make feasible suggestions for more optimal interface design.

'''Background and Related Work'''
* Chunking interactions has been studied in the HCI community as in <ref name = bob/> [http://www.cs.brown.edu/people/trevor/Papers/Heer-2008-GraphicalHistories.pdf Graphical Histories for Visualization: Supporting Analysis, Communication, and Evaluation] (InfoVis 2008).

== Parallel Framework for Evaluation Modules ==
* Owner: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]]

''This section will describe in more detail the inputs, outputs and architecture that were presented in the introduction.''

==== Module Inputs (Incomplete) ====
* A set of utility dimensions {d1, d2, ...} are defined in the framework. These could be {d1="time", d2="fatigue", ...}
* A set of interaction functions. These specify all of the information that the application wants to give users or get from them. It is not tied to a specific interface. For example, the fact that an applications shows videos would be included here. Whether it displays them embedded, in a pop-up window or fullscreen would not.
* A mapping of interaction functions to interface elements (e.g., buttons, windows, dials,...). Lots of optional information describing visual properties, associated text, physical interactions (e.g., turning the dial clockwise increases the input value) and timing.
* Functional user traces - sequences of interaction functions that represent typical user interactions with the application. Could include a goal hierarchy, in which case the function sequence is at the bottom of the hierarchy.

==== Module Outputs ====
* Every module ouputs at least one of the following:
** An evaluation of the interface
*** This can be on any or all of the utility dimensions, e.g. evaluation={d1=score1, d2=score2, ...}
*** This can alternately be an overall evaluation, ignoring dimensions, e.g. evaluation={score}
**** In this case, the aggregator will treat this as the module giving the same score to all dimensions. Which dimension this evaluator actually predicts well on can be learned by the aggregator over time.
** Recommendation(s) for improving the interface
***This can be a textual description of what changes the designer should make
***This can alternately be a transformation that can automatically be applied to the interface language (without designer intervention)
***In addition to the textual or transformational description of the recommendation, a "change in evaluation" is output to describe how specifically the value will improve the interface
****Recommendation = {description="make this change", Δevaluation={d1=score1, d2=score2, ...}
****Like before, this Δevaluation can cover any number of dimensions, or it can be generic.
***Either a single recommendation or a set of recommendations can be output

==== Aggregator Inputs ====
The aggregator receives as input the outputs of all the modules.

==== Aggregator Outputs ====

Outputs for the aggregator are the same as the outputs for each module. The difference is that the aggregator will consider all the module outputs, and arrive at a merged output based on the past performance of each of the modules.

== Evaluation and Recommendation via Modules ==
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

A "meta-module" called the aggregator will be responsible for assembling and formatting the output of all other modules into a structure that is both extensible and immediately usable, by both an automated designer or a human designer.

===Requirements===
The aggregator's functionality, then, is defined by its '''inputs''', the outputs of the other modules, and the desired '''output''' of the system as a whole, per its position in the architecture. Its purpose is largely formatting and reconciliation of the products of the multitudinous (and extensible) modules. The output of the aggregator must meet several requirements: first, to generate a set of human-readable suggestions for the improvement of the given interface; second, to generate a machine-readable, but also analyzable, evaluation of the various characteristics of the interface and accompanying user traces.

From these specifications, it is logical to assume that a common language or format will be required for the output of individual modules. We propose an XML-based file format, allowing: (1) a section for the standardized identification of problem areas, applicable rules, and proposed improvements, generalized by the individual module and mapped to a single element, or group of elements, in the original interface specification; (2) a section for specification of generalizable "utility" functions, allowing a module to specify how much a measurable quantity of utility is positively or negatively affected by properties of the input interface; (3) new, user-definable sections for evaluations of the given interface not covered by the first two sections. The first two sections are capable of conveying the vast majority of module outputs predicted at this time, but the XML can extensibly allow modules to pass on whatever information may become prominent in the future.

===Specification===
<module id="Fitts-Law">
<interface-elements>
<element>
<desc>submit button</desc>
<problem>
<desc>size</desc>
<suggestion>width *= 2</suggestion>
<suggetsion>height *= 2</suggestions>
<human-suggestion>Increase size relative to other elements</human-suggestion>
</problem>
</element>
</interface-elements>
<utility>
<dimension>
<desc>time</desc>
<value>0:15:35</value>
</dimension>
<dimension>
<desc>frustration</desc>
<value>pulling hair out</value>
</dimension>
<dimension>
<desc>efficiency</desc>
<value>13.2s/KPM task</value>
<value>0.56m/CPM task</value>
</dimension>
</utility>
<tasks>
<task>
<desc>complete form</desc>
</task>
<task>
<desc>lookup SSN</desc>
</task>
<task>
<desc>format phone number</desc>
</task>
</tasks>
</module>

===Logic===
This file provided by each module is then the input for the aggregator. The aggregator's most straightforward function is the compilation of the "problem areas," assembling them and noting problem areas and suggestions that are recommended by more than one module, and weighting them accordingly in its final report. These weightings can begin in an equal state, but the aggregator should be capable of learning iteratively which modules' results are most relevant to the user and update weightings accordingly. This may need to be accomplished with manual tuning, or a machine-learning algorithm capable of determining which modules most often agree with others.

Secondly, the aggregator compiles the utility functions provided by the module specs. This, again, is a summation of similarly-described values from the various modules.

When confronted with user-defined sections of the XML spec, the aggregator is primarily responsible for compiling them and sending them along to the output of the machine. Even if the aggregator does not recognize a section or property of the evaluative spec, if it sees the property reported by more than one module it should be capable of aggregating these intelligently. In future versions of the spec, it should be possible for a module to provide instructions for the aggregator on how to handle unrecognized sections of the XML.

From these compilations, then, the aggregator should be capable of outputting both aggregated human-readable suggestions on interface improvements for a human designer, as well as a comprehensive evaluation of the interface's effectiveness at the given task traces. Again, this is dependent on the specification of the system as a whole, but is likely to include measures and comparisons, graphings of task versus utility, and quantitative measures of an element's effectiveness.

''This section is necessarily defined by the output of the individual modules (which I already expect to be of varied and arbitrary structure) and the desired output of the machine as a whole. It will likely need to be revised heavily after other modules and the "Parellel Framework" section are defined.'' [[User:E J Kalafarski|E J Kalafarski]] 12:34, 24 April 2009 (UTC)

''This section describes the aggregator, which takes the output of multiple independent modules and aggregates the results to provide (1) an evaluation and (2) recommendations for the user interface. We should explain how the aggregator weights the output of different modules (this could be based on historical performance of each module, or perhaps based on E.J.'s cognitive/HCI guidelines).''

== Sample Modules ==

=== CPM-GOMS ===
* Owners: [[User:Steven Ellis | Steven Ellis]]

''This module will provide interface evaluations and suggestions based on a CPM-GOMS model of cognition for the given interface. It will provide a quantitative, predictive, cognition-based parameterization of usability. From empirically collected data, user trajectories through the model (critical paths) will be examined, highlighting bottlenecks within the interface, and offering suggested alterations to the interface to induce more optimal user trajectories.''

I’m hoping to have some input on this section, because it seems to be the crux of the “black box” into which we take the inputs of interface description, user traces, etc. and get our outputs (time, recommendations, etc.). I know at least a few people have pretty strong thoughts on the matter and we ought to discuss the final structure.

That said – my proposal for the module:
*In my opinion the concept of the Model Human Processor (at least as applied in CPM) is outdated – it’s too economic/overly parsimonious in its conception of human activity. I think we need to create a structure which accounts for more realistic conditions of HCI including multitasking, aspects of distributed cognition (and other relevant uses of tools – as far as I can tell CPM doesn’t take into account any sort of productivity aids), executive control processes of attention, etc. ACT-R appears to take steps towards this but we would probably need to look at their algorithms to know for sure.
*Critical paths will continue to play an important role – we should in fact emphasize that part of this tool’s purpose will be a description not only of ways in which the interface should be modified to best fit a critical path, but also ways in which the user’s ought to be instructed in their use of the path. This feedback mechanism could be bidirectional – if the model’s predictions of the user’s goals are incorrect, the critical path determined will also be incorrect and the interface inherently suboptimal. The user could be prompted with a tooltip explaining in brief why and how the interface has changed, along with options to revert, select other configurations (euphemized by goals), and to view a short video detailing how to properly use the interface.
*Call me crazy but, if we assume designers will be willing to code a model of their interfaces into our ACT-R-esque language, could we allow that model to be fairly transparent to the user, who could use a gui to input their goals to find an analogue in the program which would subsequently rearrange its interface to fit the user’s needs? Even if not useful to the users, such dynamic modeling could really help designers (IMO)
*I think the model should do its best to accept models written for ACT-R and whatever other cognitive models there are out there – gives us the best chance of early adoption
*I would particularly appreciate input on the number/complexity/type of inputs we’ll be using, as well as the same qualities for the output.

=== HCI Guidelines ===
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

Schneiderman's Eight Golden Rules and Jakob Nielsen's Ten Heuristics are perhaps the most famous and well-regarded heuristic design guidelines to emerge over the last twenty years. Although the explicit theoretical basis for such heuristics is controversial and not well-explored, the empirical success of these guidelines is established and accepted. This module will parse out up to three or four common (that is, intersecting) principles from these accepted guidelines and apply them to the input interface.

As an example, we identify an analogous principle that appears in Schneiderman ("Reduce short-term memory load")<ref>http://faculty.washington.edu/jtenenbg/courses/360/f04/sessions/schneidermanGoldenRules.html</ref> and Nielsen ("Recognition rather than recall/Minimize the user's memory load")<ref>http://www.useit.com/papers/heuristic/heuristic_list.html</ref>. The input interface is then evaluated for the consideration of the principle, based on an explicit formal description of the interface, such as XAML or XUL. The module attempts to determine how effectively the interface demonstrates the principle. When analyzing an interface for several principles that may be conflicting or opposing in a given context, the module makes use of a hard-coded but iterative (and evolving) weighting of these principles, based on (1) how often they appear in the training set of accepted sets of guidelines, (2) how analogues a heuristic principle is to a cognitive principle in a parallel training set, and (3) how effective the principle's associated suggestion is found to be using a feedback mechanism.

'''Inputs''' 
# A formal description of the interface and its elements (e.g. buttons).
# A formal description of a particular task and the possible paths through a subset of interface elements that permit the user to accomplish that task.

'''Output''' 
Standard XML-formatted file containing problem areas of the input interface, suggestions for each problem area based on principles that were found to have a strong application to a problem element and the problem itself, and a human-readable generated analysis of the element's affinity for the principle. Quantitative outputs will not be possible based on heuristic guidelines, and the "utility" section of this module's output is likely to be blank.

''This section could include an example or two of established design guidelines that could easily be implemented as modules.''

=== Fitts's Law ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

''This module provides an estimate of the required time to complete various tasks that have been decomposed into formalized sequences of interactions with interface elements, and will provide evaluations and recommendations for optimizing the time required to complete those tasks using the interface.''

'''Inputs'''

1. A formal description of the interface and its elements (e.g. buttons).

2. A formal description of a particular task and the possible paths through a subset of interface elements that permit the user to accomplish that task.

3. The physical distances between interface elements along those paths.

4. The width of those elements along the most likely axes of motion.

5. Device (e.g. mouse) characteristics including start/stop time and the inherent speed limitations of the device.

'''Output'''

The module will then use the Shannon formulation of Fitt's Law to compute the average time needed to complete the task along those paths.

=== Affordances ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

''This simple module will provide interface evaluations and recommendations based on a measure of the extent to which the user perceives the relevant affordances of the interface when performing a number of specified tasks.''

'''Inputs'''

Formalized descriptions of...

1. Interface elements

2. Their associated actions

3. The functions of those actions

4. A particular task

5. User traces for that task.

Inputs (1-4) are then used to generate a "user-independent" space of possible functions that the interface is capable of performing with respect to a given task -- what the interface "affords" the user. From this set of possible interactions, our model will then determine the subset of optimal paths for performing a particular task. The user trace (5) is then used to determine what functions actually were performed in the course of a given task of interest and this information is then compared to the optimal path data to determine the extent to which affordances of the interface are present but not perceived.

'''Output'''

The output of this module is a simple ratio of (affordances perceived) / [(relevant affordances present) * (time to complete task)] which provides a quantitative measure of the extent to which the interface is "natural" to use for a particular task.

=== Workflow, Multi-tasking and Interruptions ===
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

There are, at least, two levels at which users work ([http://portal.acm.org/citation.cfm?id=985692.985707&coll=portal&dl=ACM&CFID=20781736&CFTOKEN=83176621 Gonzales, et al., 2004]). Users accomplish individual low-level tasks which are part of larger ''working spheres''; for example, an office worker might send several emails, create several Post-It (TM) note reminders, and then edit a word document, each of these smaller tasks being part of a single larger working sphere of "adding a new section to the website." Thus, it is important to understand this larger workflow context - which often involves extensive levels of multi-tasking, as well as switching between a variety of computing devices and traditional tools, such as notebooks. In this study it was found that the information workers surveyed typically switch individual tasks every 2 minutes and have many simultaneous working spheres which they switch between, on average every 12 minutes. This frenzied pace of switching tasks and switching working spheres suggests that users will not be using a single application or device for a long period of time, and that affordances to support this characteristic pattern of information work are important.

The purpose of this module is to integrate existing work on multi-tasking, interruption and higher-level workflow into a framework which can predict user recovery times from interruptions. Specifically, the goals of this framework will be to:

* Understand the role of the larger workflow context in user interfaces
* Understand the impact of interruptions on user workflow
* Understand how to design software which fits into the larger working spheres in which information work takes place

It is important to point out that because workflow and multi-tasking rely heavily on higher-level brain functioning, it is unrealistic within the scope of this grant to propose a system which can predict user performance given a description of a set of arbitrary software programs. Therefore, we believe this module will function much more in a qualitative role to provide context to the rest of the model. Specifically, our findings related to interruption and multi-tasking will advance the basic research question of "how do you users react to interruptions when using working sets of varying sizes?". This core HCI contribution will help to inform the rest of the outputs of the model in a qualitative manner.

'''Inputs'''

N/A

'''Outputs'''

N/A

=== Working Memory Load ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''This module measures how much information the user needs to retain in memory while interacting with the interface and makes suggestions for improvements.''

* Models
** Beddeley's Model of Working Memory<ref>http://en.wikipedia.org/wiki/Baddeley%27s_model_of_working_memory</ref>
*** Episodic Buffer
** George Miller's "The magic number 7 plus or minus 2"<ref>http://en.wikipedia.org/wiki/The_Magical_Number_Seven,_Plus_or_Minus_Two</ref>
** The magic number 4<ref>Cowan, N. (2001). The magical number 4 in short-term memory: A reconsideration of mental storage capacity. Behavioral and Brain Sciences, 24, 87-185.</ref>
* Chunking<ref>http://en.wikipedia.org/wiki/Chunking_(psychology)</ref>
* Priming<ref>http://en.wikipedia.org/wiki/Priming_(psychology)</ref>
* Subitizing and Counting<ref>http://en.wikipedia.org/wiki/Subitizing</ref>

'''Inputs'''
* Visual Stimuli
* Audio Stimuli
'''Outputs'''
* Remembered percepts
* Half-Life of percepts

=== Automaticity of Interaction ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''Measures how easily the interaction with the interface becomes automatic with experience and makes suggestions for improvements.''

* Learning<ref>http://en.wikipedia.org/wiki/Learning#Mathematical_models_of_learning</ref>
** Logan's instance theory of automatization<ref>http://74.125.95.132/search?q=cache:IZ-Zccsu3SEJ:psych.wisc.edu/ugstudies/psych733/logan_1988.pdf+logan+isntance+teory&cd=1&hl=en&ct=clnk&gl=us&client=firefox-a</ref>
* Fluency
** As meta-cognitive information <ref>http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6VH9-4SM7PFK-4&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=10cd279fa80958981fcc3c06684c09af</ref>
** As a cognitive ''heuristic''<ref>http://en.wikipedia.org/wiki/Fluency_heuristic</ref>

'''Inputs'''
* Interface
* User goals

'''Outputs'''
* Learning curve

== Integration into the Design Process ==
* Owner: [[User:Ian Spector | Ian Spector]]

''This section outlines the process of designing an HCI interface and at what stages our proposal fits in and how.''

= Preliminary Results =

===Workflow, Multi-tasking, and Interruption===

====I. '''Goals'''====

The goals of the preliminary work are to gain qualitative insight into how information workers practice metawork, and to determine whether people might be better-supported with software which facillitates metawork and interruptions. Thus, the preliminary work should investigate, and demonstrate, the need and impact of the core goals of the project.

====II. '''Methodology'''====

Seven information workers, ages 20-38 (5 male, 2 female), were interviewed to determine which methods they use to "stay organized". An initial list of metawork strategies was established from two pilot interviews, and then a final list was compiled. Participants then responded to a series of 17 questions designed to gain insight into their metawork strategies and process. In addition, verbal interviews were conducted to get additional open-ended feedback.

====III. '''Final Results'''====
A histogram of methods people use to "stay organized" in terms of tracking things they need to do (TODOs), appointments and meetings, etc. is shown in the figure below.

[[Image:AcbGraph.jpg]]

In addition to these methods, participants also used a number of other methods, including:

* iCal
* Notes written in xterms
* "Inbox zero" method of email organization
* iGoogle Notepad (for tasks)
* Tag emails as "TODO", "Important", etc.
* Things (Organizer Software)
* Physical items placed to "remind me of things"
* Sometimes arranging windows on desk
* Keeping browser tabs open
* Bookmarking web pages
* Keep programs/files open scrolled to certain locations sometimes with things selected

In addition, three participants said that when interrupted they "rarely" or "very rarely" were able to resume the task they were working on prior to the interruption. Three of the participants said that they would not actively recommend their metawork strategies for other people, and two said that staying organized was "difficult".

Four participants were neutral to the idea of new tools to help them stay organized and three said that they would like to have such a tool/tools.

====IV. '''Discussion'''====
These results qunatiatively support our hypothesis that there is no clearly dominant set of metawork strategies employed by information workers. This highly fragemented landscape is surprising, even though most information workers work in a similar environment - at a desk, on the phone, in meetings - and with the same types of tools - computers, pens, paper, etc. We believe that this suggests that there are complex tradeoffs between these methods and that no single method is sufficient. We therefore believe that users will be better supported with a new set of software-based metawork tools.

= [Criticisms] =
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

''Any criticisms or questions we have regarding the proposal can go here.''

=References=
<references/>

CS295J/Research proposal (draft 2)

2009-04-24T16:59:02Z

Trevor O'Brien: /* Recording User-Interaction Primitives */

= Introduction =
* Owners: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]], [[User: Trevor O'Brien | Trevor]]

'''Propose:''' The design, application and evaluation of a novel, cognition-based, computational framework for assessing interface design and providing automated suggestions to optimize usability.

'''Evaluation Methodology:''' Our techniques will be evaluated quantitatively through a series of user-study trials, as well as qualitatively by a team of expert interface designers.

'''Contributions and Significance:''' We expect this work to make the following contributions:
# design-space analysis and quantitative evaluation of cognition-based techniques for assessing user interfaces.
# design and quantitative evaluation of techniques for suggesting optimized interface-design changes.
# an extensible, multimodal software architecture for capturing user traces integrated with pupil-tracking data, auditory recognition, and muscle-activity monitoring.
# specification (language?) of how to define an interface evaluation module and how to integrate it into a larger system.
# (there may be more here, like testing different cognitive models, generating a markup language to represent interfaces, maybe even a unique metric space for interface usability)

--

We propose a framework for interface evaluation and recommendation that integrates behavioral models and design guidelines from both cognitive science and HCI. Our framework behaves like a committee of specialized experts, where each expert provides its own assessment of the interface, given its particular knowledge of HCI or cognitive science. For example, an expert may provide an evaluation based on the GOMS method, Fitts's law, Maeda's design principles, or cognitive models of learning and memory. An aggregator collects all of these assessments and weights the opinions of each expert, and outputs to the developer a merged evaluation score and a weighted set of recommendations.

Systematic methods of estimating human performance with computer interfaces are used only sparsely despite their obvious benefits, the reason being the overhead involved in implementing them. In order to test an interface, both manual coding systems like the GOMS variations and user simulations like those based on ACT-R/PM and EPIC require detailed pseudo-code descriptions of the user workflow with the application interface. Any change to the interface then requires extensive changes to the pseudo-code, a major problem because of the trial-and-error nature of interface design. Updating the models themselves is even more complicated. Even an expert in CPM-GOMS, for example, can't necessarily adapt it to take into account results from new cognitive research.

Our proposal makes automatic interface evaluation easier to use in several ways. First of all, we propose to divide the input to the system into three separate parts, functionality, user traces and interface. By separating the functionality from the interface, even radical interface changes will require updating only that part of the input. The user traces are also defined over the functionality so that they too translate across different interfaces. Second, the parallel modular architecture allows for a lower "entry cost" for using the tool. The system includes a broad array of evaluation modules some of which are very simple and some more complex. The simpler modules use only a subset of the input that a system like GOMS or ACT-R would require. This means that while more input will still lead to better output, interface designers can get minimal evaluations with only minimal information. For example, a visual search module may not require any functionality or user traces in order to determine whether all interface elements are distinct enough to be easy to find. Finally, a parallel modular architecture is much easier to augment with relevant cognitive and design evaluations.

= Background / Related Work =

Each person should add the background related to their specific aims.

* Steven Ellis - Cognitive models of HCI, including GOMS variations and ACT-R
* EJ - Design Guidelines
* Jon - Perception and Action
* Andrew - Multiple task environments
* Gideon - Cognition and dual systems
* Ian - Interface design process
* Trevor - User trace collection methods (especially any eye-tracking, EEG, ... you want to suggest using)

== Cognitive Models ==
I plan to port over most of the background on cognitive models of HCI from the old proposal

Additions will comprise of:
*CPM-GOMS as a bridge from GOMS architecture to the promising procedural optimization of the Model Human Processor
**Context of CPM development, discuss its relation to original GOMS and KLM
***Establish the tasks which were relevant for optimization when CPM was developed and note that its obsolescence may have been unavoidable
**Focus on CPM as the first step in transitioning from descriptive data, provided by mounting efforts in the cognitive sciences realm to discover the nature of task processing and accomplishment, to prescriptive algorithms which can predict an interface’s efficiency and suggest improvements
**CPM’s purpose as an abstraction of cognitive processing – a symbolic representation not designed for accuracy but precision
**CPM’s successful trials, e.g. Ernestine
***Implications of this project include CPM’s ability to accurately estimate processing at a psychomotor level
***Project does suggest limitations, however, when one attempts to examine more complex tasks which involve deeper and more numerous cognitive processes
*ACT-R as an example of a progressive cognitive modeling tool
**A tool clearly built by and for cognitive scientists, and as a result presents a much more accurate view of human processing – helpful for our research
**Built-in automation, which now seems to be a standard feature of cognitive modeling tools
**Still an abstraction of cognitive processing, but makes adaptation to cutting-edge cognitive research findings an integral aspect of its modular structure
**Expand on its focus on multi-tasking, taking what was a huge advance between GOMS and its CPM variation and bringing the simulation several steps closer to approximating the nature of cognition in regards to HCI
**Far more accessible both for researchers and the lay user/designer in its portability to LISP, pre-construction of modules representing cognitive capacities and underlying algorithms modeling paths of cognitive processing

==Design guidelines==
A multitude of rule sets exist for the design of not only interface, but architecture, city planning, and software development. They can range in scale from one primary rule to as many Christopher Alexander's 253 rules for urban environments,<ref>[http://hci.rwth-aachen.de/materials/publications/borchers2000a.pdf Borchers, Jan O. "A Pattern Approach to Interaction Design." 2000.]</ref> which he introduced with the concept design patterns in the 1970s. Study has likewise been conducted on the use of these rules:<ref>http://stl.cs.queensu.ca/~graham/cisc836/lectures/readings/tetzlaff-guidelines.pdf</ref> guidelines are often only partially understood, indistinct to the developer, and "fraught" with potential usability problems given a real-world situation.

===Application to AUE===
And yet, the vast majority of guideline sets, including the most popular rulesets, have been arrived at heuristically. The most successful, such as Raskin's and Schneiderman's, have been forged from years of observation instead of empirical study and experimentation. The problem is similar to the problem of circular logic faced by automated usability evaluations: an automated system is limited in the suggestions it can offer to a set of preprogrammed guidelines which have often not been subjected to rigorous experimentation.<ref>[http://www.eecs.berkeley.edu/Pubs/TechRpts/2000/CSD-00-1105.pdf Ivory, M and Hearst, M. "The State of the Art in Automated Usability Evaluation of User Interfaces." ACM Computing Surveys (CSUR), 2001.]</ref> In the vast majority of existing studies, emphasis has traditionally been placed on either the development of guidelines or the application of existing guidelines to automated evaluation. A mutually-reinforcing development of both simultaneously has not been attempted.

Overlap between rulesets is inevitable and unavoidable. For our purposes of evaluating existing rulesets efficiently, without extracting and analyzing each rule individually, it may be desirable to identify the the overarching ''principles'' or ''philosophy'' (max. 2 or 3) for a given ruleset and determining their quantitative relevance to problems of cognition.

===Popular and seminal examples===
Schneiderman's [http://faculty.washington.edu/jtenenbg/courses/360/f04/sessions/schneidermanGoldenRules.html Eight Golden Rules] date to 1987 and are arguably the most-cited. They are heuristic, but can be somewhat classified by cognitive objective: at least two rules apply primarily to ''repeated use'', versus ''discoverability''. Up to five of Schneiderman's rules emphasize ''predictability'' in the outcomes of operations and ''increased feedback and control'' in the agency of the user. His final rule, paradoxically, removes control from the user by suggesting a reduced short-term memory load, which we can arguably classify as ''simplicity''.

Raskin's [http://www.mprove.de/script/02/raskin/designrules.html Design Rules] are classified into five principles by the author, augmented by definitions and supporting rules. While one principle is primarily aesthetic (a design problem arguably out of the bounds of this proposal) and one is a basic endorsement of testing, the remaining three begin to reflect philosophies similar to Schneiderman's: reliability or ''predictability'', ''simplicity'' or ''efficiency'' (which we can construe as two sides of the same coin), and finally he introduces a concept of ''uninterruptibility''.

Maeda's [http://lawsofsimplicity.com/?cat=5&order=ASC Laws of Simplicity] are fewer, and ostensibly emphasize ''simplicity'' exclusively, although elements of ''use'' as related by Schneiderman's rules and ''efficiency'' as defined by Raskin may be facets of this simplicity. Google's corporate mission statement presents [http://www.google.com/corporate/ux.html Ten Principles], only half of which can be considered true interface guidelines. ''Efficiency'' and ''simplicity'' are cited explicitly, aesthetics are once again noted as crucial, and working within a user's trust is another application of ''predictability''.

===Elements and goals of a guideline set===
Myriad rulesets exist, but variation becomes scarce—it indeed seems possible to parse these common rulesets into overarching principles that can be converted to or associated with quantifiable cognitive properties. For example, it is likely ''simplicity'' has an analogue in the short-term memory retention or visual retention of the user, vis a vis the rule of [http://books.google.com/books?hl=en&lr=&id=j5q0VvOGExYC&oi=fnd&pg=PA357&dq=seven+plus+or+minus+two&ots=prI3PKJBar&sig=vOZnqpnkXKGYWxK6_XlA4I_CRyI Seven, Plus or Minus Two]. ''Predictability'' likewise may have an analogue in Activity Theory, in regards to a user's perceptual expectations for a given action; ''uninterruptibility'' has implications in cognitive task-switching;<ref>[http://portal.acm.org/citation.cfm?id=985692.985715&coll=Portal&dl=ACM&CFID=21136843&CFTOKEN=23841774 Czerwinski, Horvitz, and White. "A Diary Study of Task Switching and Interruptions." Proceedings of the SIGCHI conference on Human factors in computing systems, 2004.]</ref> and so forth.

Within the scope of this proposal, we aim to reduce and refine these philosophies found in seminal rulesets and identify their logical cognitive analogues. By assigning a quantifiable taxonomy to these principles, we will be able to rank and weight them with regard to their real-world applicability, developing a set of "meta-guidelines" and rules for applying them to a given interface in an automated manner. Combined with cognitive models and multi-modal HCI analysis, we seek to develop, in parallel with these guidelines, the interface evaluation system responsible for their application.

== Perception and Action (in progress) ==

*Information Processing Approach
**Advantages
***Formalism eases translation of theory into scripting language
**Disadvantages
***Assumes symbolic representation

*Ecological (Gibsonian) Approach
**Advantages
***Emphasis on bodily and environmental constraints
**Disadvantages
***Lack of formalism hinders translation of theory into scripting language

= Specific Aims and Contributions (to be separated later) =

'''Specific Aims'''
* Incorporate interaction history mechanisms into a set of existing applications.
* Perform user-study evaluation of history-collection techniques.
* Distill a set of cognitive principles/models, and evaluate empirically?
* Build/buy sensing system to include pupil-tracking, muscle-activity monitoring, auditory recognition.
* Design techniques for manual/semi-automated/automated construction of <insert favorite cognitive model here> from interaction histories and sensing data.
* Design system for posterior analysis of interaction history w.r.t. <insert favorite cognitive model here>, evaluating critical path <or equivalent> trajectories.
* Design cognition-based techniques for detecting bottlenecks in critical paths, and offering optimized alternatives.
* Perform quantitative user-study evaluations, collect qualitative feedback from expert interface designers.

'''Contributions'''

* Design-space analysis and quantitative evaluation of cognition-based techniques for assessing user interfaces.
* Design and quantitative evaluation of techniques for suggesting optimized interface-design changes.
* An extensible, multimodal software architecture for capturing user traces integrated with pupil-tracking data, auditory recognition, and muscle-activity monitoring.
* (there may be more here, like testing different cognitive models, generating a markup language to represent interfaces, maybe even a unique metric space for interface usability)

--

See the [http://vrl.cs.brown.edu/wiki/images/b/b1/Flowchart2.pdf flowchart] for a visual overview of our aims.

In order to use this framework, a designer will have to provide:
* Functional specification - what are the possible interactions between the user and the application. This can be thought of as method signatures, with a name (e.g., setVolume), direction (to user or from user) and a list of value types (boolean, number, text, video, ...) for each interaction.
* GUI specification - a mapping of interactions to interface elements (e.g., setVolume is mapped to the grey knob in the bottom left corner with clockwise turning increasing the input number).
* Functional user traces - sequences of representative ways in which the application is used. Instead of writing them, the designer could have users use the application with a trial interface and then use our methods to generalize the user traces beyond the specific interface (The second method is depicted in the diagram). As a form of pre-processing, the system also generates an interaction transition matrix which lists the probability of each type of interaction given the previous interaction.
* Utility function - this is a weighting of various performance metrics (time, cognitive load, fatigue, etc.), where the weighting expresses the importance of a particular dimension to the user. For example, a user at NASA probably cares more about interface accuracy than speed. By passing this information to our committee of experts, we can create interfaces that are tuned to maximize the utility of a particular user type.

Each of the modules can use all of this information or a subset of it. Our approach stresses flexibility and the ability to give more meaningful feedback the more information is provided. After processing the information sent by the system of experts, the aggregator will output:
* An evaluation of the interface. Evaluations are expressed both in terms of the utility function components (i.e. time, fatigue, cognitive load, etc.), and in terms of the overall utility for this interface (as defined by the utility function). These evaluations are given in the form of an efficiency curve, where the utility received on each dimension can change as the user becomes more accustomed to the interface.
* Suggested improvements for the GUI are also output. These suggestions are meant to optimize the utility function that was input to the system. If a user values accuracy over time, interface suggestions will be made accordingly.

== Recording User-Interaction Primitives ==
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

'''Aims'''
* Develop system for logging low-level user-interactions within existing applications. By low-level interactions, we refer to those interactions for which a quantitative, predictive, GOMS-like model may be generated.
* At the same scale of granularity, integrate explicit interface-interactions with multimodal-sensing data. i.e. pupil-tracking, muscle-activity monitoring, auditory recognition, EEG data, and facial and posture-focused video.

'''Contributions'''
* Extensible, multimodal, HCI framework for recording rich interaction-history data in existing applications.

'''Significance'''
* Provide rich history data to serve as the basis for novel quantitative interface evaluation.
* Aid expert HCI and visual designers in traditional design processes.
* Provide data for automated machine-learning strategies applied to interaction.

'''Background and Related Work'''
* The utility of interaction histories with respect to assessing interface design has been demonstrated in <ref name = bob> [http://www.cs.brown.edu/people/trevor/Papers/Heer-2008-GraphicalHistories.pdf Graphical Histories for Visualization: Supporting Analysis, Communication, and Evaluation] (InfoVis 2008) </ref>.
* In addition, data management histories have been shown effective in the visualization community in <ref> [http://www.cs.brown.edu/people/trevor/Papers/Callahan-2006-MED.pdf Callahan-2006-MED] </ref> <ref> [http://www.cs.brown.edu/people/trevor/Papers/Callahan-2006-VVM.pdf Callahan-2006-VVM] </ref> <ref> [http://www.cs.brown.edu/people/trevor/Papers/Bavoil-2005-VEI.pdf Bavoil-2005-VEI]</ref>, providing visualizations by analogy <ref> [http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=4376187&isnumber=4376125 Querying and Creating Visualizations by Analogy] </ref> and offering automated suggestions <ref> [http://www.cs.utah.edu/~juliana/pub/tvcg-recommendation2008.pdf VisComplete: Automating Suggestions from Visualization Pipelines] </ref>, which we expect to generalize to user interaction history.

== Semantic-level Interaction Chunking ==
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

'''Aims'''
* Develop techniques for ''chunking'' low-level interaction primitives into ''semantic-level'' interactions, given an application's functionality and data-context. (And de-chunking? Invertible mapping needed?)
* Perform user-study evaluation to validate chunking methods.

'''Contributions'''
* Design and user-study evaluation of semantic chunking techniques.

'''Significance'''
* Low-level <--> semantic-level mapping allows for cognitive-modeling to be applied at a functionality level, where low-level interaction techniques can be swapped out. This will allow our interface assessment system to make feasible suggestions for more optimal interface design.

'''Background and Related Work'''
* Chunking interactions has been studied in the HCI community as in <ref> [http://www.cs.brown.edu/people/trevor/Papers/Heer-2008-GraphicalHistories.pdf Graphical Histories for Visualization: Supporting Analysis, Communication, and Evaluation] (InfoVis 2008) </ref>.

== Parallel Framework for Evaluation Modules ==
* Owner: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]]

''This section will describe in more detail the inputs, outputs and architecture that were presented in the introduction.''

==== Module Inputs (Incomplete) ====
* A set of utility dimensions {d1, d2, ...} are defined in the framework. These could be {d1="time", d2="fatigue", ...}
* A set of interaction functions. These specify all of the information that the application wants to give users or get from them. It is not tied to a specific interface. For example, the fact that an applications shows videos would be included here. Whether it displays them embedded, in a pop-up window or fullscreen would not.
* A mapping of interaction functions to interface elements (e.g., buttons, windows, dials,...). Lots of optional information describing visual properties, associated text, physical interactions (e.g., turning the dial clockwise increases the input value) and timing.
* Functional user traces - sequences of interaction functions that represent typical user interactions with the application. Could include a goal hierarchy, in which case the function sequence is at the bottom of the hierarchy.

==== Module Outputs ====
* Every module ouputs at least one of the following:
** An evaluation of the interface
*** This can be on any or all of the utility dimensions, e.g. evaluation={d1=score1, d2=score2, ...}
*** This can alternately be an overall evaluation, ignoring dimensions, e.g. evaluation={score}
**** In this case, the aggregator will treat this as the module giving the same score to all dimensions. Which dimension this evaluator actually predicts well on can be learned by the aggregator over time.
** Recommendation(s) for improving the interface
***This can be a textual description of what changes the designer should make
***This can alternately be a transformation that can automatically be applied to the interface language (without designer intervention)
***In addition to the textual or transformational description of the recommendation, a "change in evaluation" is output to describe how specifically the value will improve the interface
****Recommendation = {description="make this change", Δevaluation={d1=score1, d2=score2, ...}
****Like before, this Δevaluation can cover any number of dimensions, or it can be generic.
***Either a single recommendation or a set of recommendations can be output

==== Aggregator Inputs ====
The aggregator receives as input the outputs of all the modules.

==== Aggregator Outputs ====

Outputs for the aggregator are the same as the outputs for each module. The difference is that the aggregator will consider all the module outputs, and arrive at a merged output based on the past performance of each of the modules.

== Evaluation and Recommendation via Modules ==
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

A "meta-module" called the aggregator will be responsible for assembling and formatting the output of all other modules into a structure that is both extensible and immediately usable, by both an automated designer or a human designer.

===Requirements===
The aggregator's functionality, then, is defined by its '''inputs''', the outputs of the other modules, and the desired '''output''' of the system as a whole, per its position in the architecture. Its purpose is largely formatting and reconciliation of the products of the multitudinous (and extensible) modules. The output of the aggregator must meet several requirements: first, to generate a set of human-readable suggestions for the improvement of the given interface; second, to generate a machine-readable, but also analyzable, evaluation of the various characteristics of the interface and accompanying user traces.

From these specifications, it is logical to assume that a common language or format will be required for the output of individual modules. We propose an XML-based file format, allowing: (1) a section for the standardized identification of problem areas, applicable rules, and proposed improvements, generalized by the individual module and mapped to a single element, or group of elements, in the original interface specification; (2) a section for specification of generalizable "utility" functions, allowing a module to specify how much a measurable quantity of utility is positively or negatively affected by properties of the input interface; (3) new, user-definable sections for evaluations of the given interface not covered by the first two sections. The first two sections are capable of conveying the vast majority of module outputs predicted at this time, but the XML can extensibly allow modules to pass on whatever information may become prominent in the future.

===Specification===
<module id="Fitts-Law">
<interface-elements>
<element>
<desc>submit button</desc>
<problem>
<desc>size</desc>
<suggestion>width *= 2</suggestion>
<suggetsion>height *= 2</suggestions>
<human-suggestion>Increase size relative to other elements</human-suggestion>
</problem>
</element>
</interface-elements>
<utility>
<dimension>
<desc>time</desc>
<value>0:15:35</value>
</dimension>
<dimension>
<desc>frustration</desc>
<value>pulling hair out</value>
</dimension>
<dimension>
<desc>efficiency</desc>
<value>13.2s/KPM task</value>
<value>0.56m/CPM task</value>
</dimension>
</utility>
<tasks>
<task>
<desc>complete form</desc>
</task>
<task>
<desc>lookup SSN</desc>
</task>
<task>
<desc>format phone number</desc>
</task>
</tasks>
</module>

===Logic===
This file provided by each module is then the input for the aggregator. The aggregator's most straightforward function is the compilation of the "problem areas," assembling them and noting problem areas and suggestions that are recommended by more than one module, and weighting them accordingly in its final report. These weightings can begin in an equal state, but the aggregator should be capable of learning iteratively which modules' results are most relevant to the user and update weightings accordingly. This may need to be accomplished with manual tuning, or a machine-learning algorithm capable of determining which modules most often agree with others.

Secondly, the aggregator compiles the utility functions provided by the module specs. This, again, is a summation of similarly-described values from the various modules.

When confronted with user-defined sections of the XML spec, the aggregator is primarily responsible for compiling them and sending them along to the output of the machine. Even if the aggregator does not recognize a section or property of the evaluative spec, if it sees the property reported by more than one module it should be capable of aggregating these intelligently. In future versions of the spec, it should be possible for a module to provide instructions for the aggregator on how to handle unrecognized sections of the XML.

From these compilations, then, the aggregator should be capable of outputting both aggregated human-readable suggestions on interface improvements for a human designer, as well as a comprehensive evaluation of the interface's effectiveness at the given task traces. Again, this is dependent on the specification of the system as a whole, but is likely to include measures and comparisons, graphings of task versus utility, and quantitative measures of an element's effectiveness.

''This section is necessarily defined by the output of the individual modules (which I already expect to be of varied and arbitrary structure) and the desired output of the machine as a whole. It will likely need to be revised heavily after other modules and the "Parellel Framework" section are defined.'' [[User:E J Kalafarski|E J Kalafarski]] 12:34, 24 April 2009 (UTC)

''This section describes the aggregator, which takes the output of multiple independent modules and aggregates the results to provide (1) an evaluation and (2) recommendations for the user interface. We should explain how the aggregator weights the output of different modules (this could be based on historical performance of each module, or perhaps based on E.J.'s cognitive/HCI guidelines).''

== Sample Modules ==

=== CPM-GOMS ===
* Owners: [[User:Steven Ellis | Steven Ellis]]

''This module will provide interface evaluations and suggestions based on a CPM-GOMS model of cognition for the given interface. It will provide a quantitative, predictive, cognition-based parameterization of usability. From empirically collected data, user trajectories through the model (critical paths) will be examined, highlighting bottlenecks within the interface, and offering suggested alterations to the interface to induce more optimal user trajectories.''

I’m hoping to have some input on this section, because it seems to be the crux of the “black box” into which we take the inputs of interface description, user traces, etc. and get our outputs (time, recommendations, etc.). I know at least a few people have pretty strong thoughts on the matter and we ought to discuss the final structure.

That said – my proposal for the module:
*In my opinion the concept of the Model Human Processor (at least as applied in CPM) is outdated – it’s too economic/overly parsimonious in its conception of human activity. I think we need to create a structure which accounts for more realistic conditions of HCI including multitasking, aspects of distributed cognition (and other relevant uses of tools – as far as I can tell CPM doesn’t take into account any sort of productivity aids), executive control processes of attention, etc. ACT-R appears to take steps towards this but we would probably need to look at their algorithms to know for sure.
*Critical paths will continue to play an important role – we should in fact emphasize that part of this tool’s purpose will be a description not only of ways in which the interface should be modified to best fit a critical path, but also ways in which the user’s ought to be instructed in their use of the path. This feedback mechanism could be bidirectional – if the model’s predictions of the user’s goals are incorrect, the critical path determined will also be incorrect and the interface inherently suboptimal. The user could be prompted with a tooltip explaining in brief why and how the interface has changed, along with options to revert, select other configurations (euphemized by goals), and to view a short video detailing how to properly use the interface.
*Call me crazy but, if we assume designers will be willing to code a model of their interfaces into our ACT-R-esque language, could we allow that model to be fairly transparent to the user, who could use a gui to input their goals to find an analogue in the program which would subsequently rearrange its interface to fit the user’s needs? Even if not useful to the users, such dynamic modeling could really help designers (IMO)
*I think the model should do its best to accept models written for ACT-R and whatever other cognitive models there are out there – gives us the best chance of early adoption
*I would particularly appreciate input on the number/complexity/type of inputs we’ll be using, as well as the same qualities for the output.

=== HCI Guidelines ===
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

Schneiderman's Eight Golden Rules and Jakob Nielsen's Ten Heuristics are perhaps the most famous and well-regarded heuristic design guidelines to emerge over the last twenty years. Although the explicit theoretical basis for such heuristics is controversial and not well-explored, the empirical success of these guidelines is established and accepted. This module will parse out up to three or four common (that is, intersecting) principles from these accepted guidelines and apply them to the input interface.

As an example, we identify an analogous principle that appears in Schneiderman ("Reduce short-term memory load")<ref>http://faculty.washington.edu/jtenenbg/courses/360/f04/sessions/schneidermanGoldenRules.html</ref> and Nielsen ("Recognition rather than recall/Minimize the user's memory load")<ref>http://www.useit.com/papers/heuristic/heuristic_list.html</ref>. The input interface is then evaluated for the consideration of the principle, based on an explicit formal description of the interface, such as XAML or XUL. The module attempts to determine how effectively the interface demonstrates the principle. When analyzing an interface for several principles that may be conflicting or opposing in a given context, the module makes use of a hard-coded but iterative (and evolving) weighting of these principles, based on (1) how often they appear in the training set of accepted sets of guidelines, (2) how analogues a heuristic principle is to a cognitive principle in a parallel training set, and (3) how effective the principle's associated suggestion is found to be using a feedback mechanism.

'''Inputs''' 
# A formal description of the interface and its elements (e.g. buttons).
# A formal description of a particular task and the possible paths through a subset of interface elements that permit the user to accomplish that task.

'''Output''' 
Standard XML-formatted file containing problem areas of the input interface, suggestions for each problem area based on principles that were found to have a strong application to a problem element and the problem itself, and a human-readable generated analysis of the element's affinity for the principle. Quantitative outputs will not be possible based on heuristic guidelines, and the "utility" section of this module's output is likely to be blank.

''This section could include an example or two of established design guidelines that could easily be implemented as modules.''

=== Fitts's Law ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

''This module provides an estimate of the required time to complete various tasks that have been decomposed into formalized sequences of interactions with interface elements, and will provide evaluations and recommendations for optimizing the time required to complete those tasks using the interface.''

'''Inputs'''

1. A formal description of the interface and its elements (e.g. buttons).

2. A formal description of a particular task and the possible paths through a subset of interface elements that permit the user to accomplish that task.

3. The physical distances between interface elements along those paths.

4. The width of those elements along the most likely axes of motion.

5. Device (e.g. mouse) characteristics including start/stop time and the inherent speed limitations of the device.

'''Output'''

The module will then use the Shannon formulation of Fitt's Law to compute the average time needed to complete the task along those paths.

=== Affordances ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

''This simple module will provide interface evaluations and recommendations based on a measure of the extent to which the user perceives the relevant affordances of the interface when performing a number of specified tasks.''

'''Inputs'''

Formalized descriptions of...

1. Interface elements

2. Their associated actions

3. The functions of those actions

4. A particular task

5. User traces for that task.

Inputs (1-4) are then used to generate a "user-independent" space of possible functions that the interface is capable of performing with respect to a given task -- what the interface "affords" the user. From this set of possible interactions, our model will then determine the subset of optimal paths for performing a particular task. The user trace (5) is then used to determine what functions actually were performed in the course of a given task of interest and this information is then compared to the optimal path data to determine the extent to which affordances of the interface are present but not perceived.

'''Output'''

The output of this module is a simple ratio of (affordances perceived) / [(relevant affordances present) * (time to complete task)] which provides a quantitative measure of the extent to which the interface is "natural" to use for a particular task.

=== Workflow, Multi-tasking and Interruptions ===
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

There are, at least, two levels at which users work ([http://portal.acm.org/citation.cfm?id=985692.985707&coll=portal&dl=ACM&CFID=20781736&CFTOKEN=83176621 Gonzales, et al., 2004]). Users accomplish individual low-level tasks which are part of larger ''working spheres''; for example, an office worker might send several emails, create several Post-It (TM) note reminders, and then edit a word document, each of these smaller tasks being part of a single larger working sphere of "adding a new section to the website." Thus, it is important to understand this larger workflow context - which often involves extensive levels of multi-tasking, as well as switching between a variety of computing devices and traditional tools, such as notebooks. In this study it was found that the information workers surveyed typically switch individual tasks every 2 minutes and have many simultaneous working spheres which they switch between, on average every 12 minutes. This frenzied pace of switching tasks and switching working spheres suggests that users will not be using a single application or device for a long period of time, and that affordances to support this characteristic pattern of information work are important.

The purpose of this module is to integrate existing work on multi-tasking, interruption and higher-level workflow into a framework which can predict user recovery times from interruptions. Specifically, the goals of this framework will be to:

* Understand the role of the larger workflow context in user interfaces
* Understand the impact of interruptions on user workflow
* Understand how to design software which fits into the larger working spheres in which information work takes place

It is important to point out that because workflow and multi-tasking rely heavily on higher-level brain functioning, it is unrealistic within the scope of this grant to propose a system which can predict user performance given a description of a set of arbitrary software programs. Therefore, we believe this module will function much more in a qualitative role to provide context to the rest of the model. Specifically, our findings related to interruption and multi-tasking will advance the basic research question of "how do you users react to interruptions when using working sets of varying sizes?". This core HCI contribution will help to inform the rest of the outputs of the model in a qualitative manner.

'''Inputs'''

N/A

'''Outputs'''

N/A

=== Working Memory Load ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''This module measures how much information the user needs to retain in memory while interacting with the interface and makes suggestions for improvements.''

* Models
** Beddeley's Model of Working Memory<ref>http://en.wikipedia.org/wiki/Baddeley%27s_model_of_working_memory</ref>
*** Episodic Buffer
** George Miller's "The magic number 7 plus or minus 2"<ref>http://en.wikipedia.org/wiki/The_Magical_Number_Seven,_Plus_or_Minus_Two</ref>
** The magic number 4<ref>Cowan, N. (2001). The magical number 4 in short-term memory: A reconsideration of mental storage capacity. Behavioral and Brain Sciences, 24, 87-185.</ref>
* Chunking<ref>http://en.wikipedia.org/wiki/Chunking_(psychology)</ref>
* Priming<ref>http://en.wikipedia.org/wiki/Priming_(psychology)</ref>
* Subitizing and Counting<ref>http://en.wikipedia.org/wiki/Subitizing</ref>

'''Inputs'''
* Visual Stimuli
* Audio Stimuli
'''Outputs'''
* Remembered percepts
* Half-Life of percepts

=== Automaticity of Interaction ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''Measures how easily the interaction with the interface becomes automatic with experience and makes suggestions for improvements.''

* Learning<ref>http://en.wikipedia.org/wiki/Learning#Mathematical_models_of_learning</ref>
** Logan's instance theory of automatization<ref>http://74.125.95.132/search?q=cache:IZ-Zccsu3SEJ:psych.wisc.edu/ugstudies/psych733/logan_1988.pdf+logan+isntance+teory&cd=1&hl=en&ct=clnk&gl=us&client=firefox-a</ref>
* Fluency
** As meta-cognitive information <ref>http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6VH9-4SM7PFK-4&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=10cd279fa80958981fcc3c06684c09af</ref>
** As a cognitive ''heuristic''<ref>http://en.wikipedia.org/wiki/Fluency_heuristic</ref>

'''Inputs'''
* Interface
* User goals

'''Outputs'''
* Learning curve

== Integration into the Design Process ==
* Owner: [[User:Ian Spector | Ian Spector]]

''This section outlines the process of designing an HCI interface and at what stages our proposal fits in and how.''

= Preliminary Results =

===Workflow, Multi-tasking, and Interruption===

====I. '''Goals'''====

The goals of the preliminary work are to gain qualitative insight into how information workers practice metawork, and to determine whether people might be better-supported with software which facillitates metawork and interruptions. Thus, the preliminary work should investigate, and demonstrate, the need and impact of the core goals of the project.

====II. '''Methodology'''====

Seven information workers, ages 20-38 (5 male, 2 female), were interviewed to determine which methods they use to "stay organized". An initial list of metawork strategies was established from two pilot interviews, and then a final list was compiled. Participants then responded to a series of 17 questions designed to gain insight into their metawork strategies and process. In addition, verbal interviews were conducted to get additional open-ended feedback.

====III. '''Final Results'''====
A histogram of methods people use to "stay organized" in terms of tracking things they need to do (TODOs), appointments and meetings, etc. is shown in the figure below.

[[Image:AcbGraph.jpg]]

In addition to these methods, participants also used a number of other methods, including:

* iCal
* Notes written in xterms
* "Inbox zero" method of email organization
* iGoogle Notepad (for tasks)
* Tag emails as "TODO", "Important", etc.
* Things (Organizer Software)
* Physical items placed to "remind me of things"
* Sometimes arranging windows on desk
* Keeping browser tabs open
* Bookmarking web pages
* Keep programs/files open scrolled to certain locations sometimes with things selected

In addition, three participants said that when interrupted they "rarely" or "very rarely" were able to resume the task they were working on prior to the interruption. Three of the participants said that they would not actively recommend their metawork strategies for other people, and two said that staying organized was "difficult".

Four participants were neutral to the idea of new tools to help them stay organized and three said that they would like to have such a tool/tools.

====IV. '''Discussion'''====
These results qunatiatively support our hypothesis that there is no clearly dominant set of metawork strategies employed by information workers. This highly fragemented landscape is surprising, even though most information workers work in a similar environment - at a desk, on the phone, in meetings - and with the same types of tools - computers, pens, paper, etc. We believe that this suggests that there are complex tradeoffs between these methods and that no single method is sufficient. We therefore believe that users will be better supported with a new set of software-based metawork tools.

= [Criticisms] =
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

''Any criticisms or questions we have regarding the proposal can go here.''

=References=
<references/>

CS295J/Research proposal (draft 2)

2009-04-24T16:58:18Z

Trevor O'Brien: /* Semantic-level Interaction Chunking */

= Introduction =
* Owners: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]], [[User: Trevor O'Brien | Trevor]]

'''Propose:''' The design, application and evaluation of a novel, cognition-based, computational framework for assessing interface design and providing automated suggestions to optimize usability.

'''Evaluation Methodology:''' Our techniques will be evaluated quantitatively through a series of user-study trials, as well as qualitatively by a team of expert interface designers.

'''Contributions and Significance:''' We expect this work to make the following contributions:
# design-space analysis and quantitative evaluation of cognition-based techniques for assessing user interfaces.
# design and quantitative evaluation of techniques for suggesting optimized interface-design changes.
# an extensible, multimodal software architecture for capturing user traces integrated with pupil-tracking data, auditory recognition, and muscle-activity monitoring.
# specification (language?) of how to define an interface evaluation module and how to integrate it into a larger system.
# (there may be more here, like testing different cognitive models, generating a markup language to represent interfaces, maybe even a unique metric space for interface usability)

--

We propose a framework for interface evaluation and recommendation that integrates behavioral models and design guidelines from both cognitive science and HCI. Our framework behaves like a committee of specialized experts, where each expert provides its own assessment of the interface, given its particular knowledge of HCI or cognitive science. For example, an expert may provide an evaluation based on the GOMS method, Fitts's law, Maeda's design principles, or cognitive models of learning and memory. An aggregator collects all of these assessments and weights the opinions of each expert, and outputs to the developer a merged evaluation score and a weighted set of recommendations.

Systematic methods of estimating human performance with computer interfaces are used only sparsely despite their obvious benefits, the reason being the overhead involved in implementing them. In order to test an interface, both manual coding systems like the GOMS variations and user simulations like those based on ACT-R/PM and EPIC require detailed pseudo-code descriptions of the user workflow with the application interface. Any change to the interface then requires extensive changes to the pseudo-code, a major problem because of the trial-and-error nature of interface design. Updating the models themselves is even more complicated. Even an expert in CPM-GOMS, for example, can't necessarily adapt it to take into account results from new cognitive research.

Our proposal makes automatic interface evaluation easier to use in several ways. First of all, we propose to divide the input to the system into three separate parts, functionality, user traces and interface. By separating the functionality from the interface, even radical interface changes will require updating only that part of the input. The user traces are also defined over the functionality so that they too translate across different interfaces. Second, the parallel modular architecture allows for a lower "entry cost" for using the tool. The system includes a broad array of evaluation modules some of which are very simple and some more complex. The simpler modules use only a subset of the input that a system like GOMS or ACT-R would require. This means that while more input will still lead to better output, interface designers can get minimal evaluations with only minimal information. For example, a visual search module may not require any functionality or user traces in order to determine whether all interface elements are distinct enough to be easy to find. Finally, a parallel modular architecture is much easier to augment with relevant cognitive and design evaluations.

= Background / Related Work =

Each person should add the background related to their specific aims.

* Steven Ellis - Cognitive models of HCI, including GOMS variations and ACT-R
* EJ - Design Guidelines
* Jon - Perception and Action
* Andrew - Multiple task environments
* Gideon - Cognition and dual systems
* Ian - Interface design process
* Trevor - User trace collection methods (especially any eye-tracking, EEG, ... you want to suggest using)

== Cognitive Models ==
I plan to port over most of the background on cognitive models of HCI from the old proposal

Additions will comprise of:
*CPM-GOMS as a bridge from GOMS architecture to the promising procedural optimization of the Model Human Processor
**Context of CPM development, discuss its relation to original GOMS and KLM
***Establish the tasks which were relevant for optimization when CPM was developed and note that its obsolescence may have been unavoidable
**Focus on CPM as the first step in transitioning from descriptive data, provided by mounting efforts in the cognitive sciences realm to discover the nature of task processing and accomplishment, to prescriptive algorithms which can predict an interface’s efficiency and suggest improvements
**CPM’s purpose as an abstraction of cognitive processing – a symbolic representation not designed for accuracy but precision
**CPM’s successful trials, e.g. Ernestine
***Implications of this project include CPM’s ability to accurately estimate processing at a psychomotor level
***Project does suggest limitations, however, when one attempts to examine more complex tasks which involve deeper and more numerous cognitive processes
*ACT-R as an example of a progressive cognitive modeling tool
**A tool clearly built by and for cognitive scientists, and as a result presents a much more accurate view of human processing – helpful for our research
**Built-in automation, which now seems to be a standard feature of cognitive modeling tools
**Still an abstraction of cognitive processing, but makes adaptation to cutting-edge cognitive research findings an integral aspect of its modular structure
**Expand on its focus on multi-tasking, taking what was a huge advance between GOMS and its CPM variation and bringing the simulation several steps closer to approximating the nature of cognition in regards to HCI
**Far more accessible both for researchers and the lay user/designer in its portability to LISP, pre-construction of modules representing cognitive capacities and underlying algorithms modeling paths of cognitive processing

==Design guidelines==
A multitude of rule sets exist for the design of not only interface, but architecture, city planning, and software development. They can range in scale from one primary rule to as many Christopher Alexander's 253 rules for urban environments,<ref>[http://hci.rwth-aachen.de/materials/publications/borchers2000a.pdf Borchers, Jan O. "A Pattern Approach to Interaction Design." 2000.]</ref> which he introduced with the concept design patterns in the 1970s. Study has likewise been conducted on the use of these rules:<ref>http://stl.cs.queensu.ca/~graham/cisc836/lectures/readings/tetzlaff-guidelines.pdf</ref> guidelines are often only partially understood, indistinct to the developer, and "fraught" with potential usability problems given a real-world situation.

===Application to AUE===
And yet, the vast majority of guideline sets, including the most popular rulesets, have been arrived at heuristically. The most successful, such as Raskin's and Schneiderman's, have been forged from years of observation instead of empirical study and experimentation. The problem is similar to the problem of circular logic faced by automated usability evaluations: an automated system is limited in the suggestions it can offer to a set of preprogrammed guidelines which have often not been subjected to rigorous experimentation.<ref>[http://www.eecs.berkeley.edu/Pubs/TechRpts/2000/CSD-00-1105.pdf Ivory, M and Hearst, M. "The State of the Art in Automated Usability Evaluation of User Interfaces." ACM Computing Surveys (CSUR), 2001.]</ref> In the vast majority of existing studies, emphasis has traditionally been placed on either the development of guidelines or the application of existing guidelines to automated evaluation. A mutually-reinforcing development of both simultaneously has not been attempted.

Overlap between rulesets is inevitable and unavoidable. For our purposes of evaluating existing rulesets efficiently, without extracting and analyzing each rule individually, it may be desirable to identify the the overarching ''principles'' or ''philosophy'' (max. 2 or 3) for a given ruleset and determining their quantitative relevance to problems of cognition.

===Popular and seminal examples===
Schneiderman's [http://faculty.washington.edu/jtenenbg/courses/360/f04/sessions/schneidermanGoldenRules.html Eight Golden Rules] date to 1987 and are arguably the most-cited. They are heuristic, but can be somewhat classified by cognitive objective: at least two rules apply primarily to ''repeated use'', versus ''discoverability''. Up to five of Schneiderman's rules emphasize ''predictability'' in the outcomes of operations and ''increased feedback and control'' in the agency of the user. His final rule, paradoxically, removes control from the user by suggesting a reduced short-term memory load, which we can arguably classify as ''simplicity''.

Raskin's [http://www.mprove.de/script/02/raskin/designrules.html Design Rules] are classified into five principles by the author, augmented by definitions and supporting rules. While one principle is primarily aesthetic (a design problem arguably out of the bounds of this proposal) and one is a basic endorsement of testing, the remaining three begin to reflect philosophies similar to Schneiderman's: reliability or ''predictability'', ''simplicity'' or ''efficiency'' (which we can construe as two sides of the same coin), and finally he introduces a concept of ''uninterruptibility''.

Maeda's [http://lawsofsimplicity.com/?cat=5&order=ASC Laws of Simplicity] are fewer, and ostensibly emphasize ''simplicity'' exclusively, although elements of ''use'' as related by Schneiderman's rules and ''efficiency'' as defined by Raskin may be facets of this simplicity. Google's corporate mission statement presents [http://www.google.com/corporate/ux.html Ten Principles], only half of which can be considered true interface guidelines. ''Efficiency'' and ''simplicity'' are cited explicitly, aesthetics are once again noted as crucial, and working within a user's trust is another application of ''predictability''.

===Elements and goals of a guideline set===
Myriad rulesets exist, but variation becomes scarce—it indeed seems possible to parse these common rulesets into overarching principles that can be converted to or associated with quantifiable cognitive properties. For example, it is likely ''simplicity'' has an analogue in the short-term memory retention or visual retention of the user, vis a vis the rule of [http://books.google.com/books?hl=en&lr=&id=j5q0VvOGExYC&oi=fnd&pg=PA357&dq=seven+plus+or+minus+two&ots=prI3PKJBar&sig=vOZnqpnkXKGYWxK6_XlA4I_CRyI Seven, Plus or Minus Two]. ''Predictability'' likewise may have an analogue in Activity Theory, in regards to a user's perceptual expectations for a given action; ''uninterruptibility'' has implications in cognitive task-switching;<ref>[http://portal.acm.org/citation.cfm?id=985692.985715&coll=Portal&dl=ACM&CFID=21136843&CFTOKEN=23841774 Czerwinski, Horvitz, and White. "A Diary Study of Task Switching and Interruptions." Proceedings of the SIGCHI conference on Human factors in computing systems, 2004.]</ref> and so forth.

Within the scope of this proposal, we aim to reduce and refine these philosophies found in seminal rulesets and identify their logical cognitive analogues. By assigning a quantifiable taxonomy to these principles, we will be able to rank and weight them with regard to their real-world applicability, developing a set of "meta-guidelines" and rules for applying them to a given interface in an automated manner. Combined with cognitive models and multi-modal HCI analysis, we seek to develop, in parallel with these guidelines, the interface evaluation system responsible for their application.

== Perception and Action (in progress) ==

*Information Processing Approach
**Advantages
***Formalism eases translation of theory into scripting language
**Disadvantages
***Assumes symbolic representation

*Ecological (Gibsonian) Approach
**Advantages
***Emphasis on bodily and environmental constraints
**Disadvantages
***Lack of formalism hinders translation of theory into scripting language

= Specific Aims and Contributions (to be separated later) =

'''Specific Aims'''
* Incorporate interaction history mechanisms into a set of existing applications.
* Perform user-study evaluation of history-collection techniques.
* Distill a set of cognitive principles/models, and evaluate empirically?
* Build/buy sensing system to include pupil-tracking, muscle-activity monitoring, auditory recognition.
* Design techniques for manual/semi-automated/automated construction of <insert favorite cognitive model here> from interaction histories and sensing data.
* Design system for posterior analysis of interaction history w.r.t. <insert favorite cognitive model here>, evaluating critical path <or equivalent> trajectories.
* Design cognition-based techniques for detecting bottlenecks in critical paths, and offering optimized alternatives.
* Perform quantitative user-study evaluations, collect qualitative feedback from expert interface designers.

'''Contributions'''

* Design-space analysis and quantitative evaluation of cognition-based techniques for assessing user interfaces.
* Design and quantitative evaluation of techniques for suggesting optimized interface-design changes.
* An extensible, multimodal software architecture for capturing user traces integrated with pupil-tracking data, auditory recognition, and muscle-activity monitoring.
* (there may be more here, like testing different cognitive models, generating a markup language to represent interfaces, maybe even a unique metric space for interface usability)

--

See the [http://vrl.cs.brown.edu/wiki/images/b/b1/Flowchart2.pdf flowchart] for a visual overview of our aims.

In order to use this framework, a designer will have to provide:
* Functional specification - what are the possible interactions between the user and the application. This can be thought of as method signatures, with a name (e.g., setVolume), direction (to user or from user) and a list of value types (boolean, number, text, video, ...) for each interaction.
* GUI specification - a mapping of interactions to interface elements (e.g., setVolume is mapped to the grey knob in the bottom left corner with clockwise turning increasing the input number).
* Functional user traces - sequences of representative ways in which the application is used. Instead of writing them, the designer could have users use the application with a trial interface and then use our methods to generalize the user traces beyond the specific interface (The second method is depicted in the diagram). As a form of pre-processing, the system also generates an interaction transition matrix which lists the probability of each type of interaction given the previous interaction.
* Utility function - this is a weighting of various performance metrics (time, cognitive load, fatigue, etc.), where the weighting expresses the importance of a particular dimension to the user. For example, a user at NASA probably cares more about interface accuracy than speed. By passing this information to our committee of experts, we can create interfaces that are tuned to maximize the utility of a particular user type.

Each of the modules can use all of this information or a subset of it. Our approach stresses flexibility and the ability to give more meaningful feedback the more information is provided. After processing the information sent by the system of experts, the aggregator will output:
* An evaluation of the interface. Evaluations are expressed both in terms of the utility function components (i.e. time, fatigue, cognitive load, etc.), and in terms of the overall utility for this interface (as defined by the utility function). These evaluations are given in the form of an efficiency curve, where the utility received on each dimension can change as the user becomes more accustomed to the interface.
* Suggested improvements for the GUI are also output. These suggestions are meant to optimize the utility function that was input to the system. If a user values accuracy over time, interface suggestions will be made accordingly.

== Recording User-Interaction Primitives ==
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

'''Aims'''
* Develop system for logging low-level user-interactions within existing applications. By low-level interactions, we refer to those interactions for which a quantitative, predictive, GOMS-like model may be generated.
* At the same scale of granularity, integrate explicit interface-interactions with multimodal-sensing data. i.e. pupil-tracking, muscle-activity monitoring, auditory recognition, EEG data, and facial and posture-focused video.

'''Contributions'''
* Extensible, multimodal, HCI framework for recording rich interaction-history data in existing applications.

'''Significance'''
* Provide rich history data to serve as the basis for novel quantitative interface evaluation.
* Aid expert HCI and visual designers in traditional design processes.
* Provide data for automated machine-learning strategies applied to interaction.

'''Background and Related Work'''
* The utility of interaction histories with respect to assessing interface design has been demonstrated in <ref> [http://www.cs.brown.edu/people/trevor/Papers/Heer-2008-GraphicalHistories.pdf Graphical Histories for Visualization: Supporting Analysis, Communication, and Evaluation] (InfoVis 2008) </ref>.
* In addition, data management histories have been shown effective in the visualization community in <ref> [http://www.cs.brown.edu/people/trevor/Papers/Callahan-2006-MED.pdf Callahan-2006-MED] </ref> <ref> [http://www.cs.brown.edu/people/trevor/Papers/Callahan-2006-VVM.pdf Callahan-2006-VVM] </ref> <ref> [http://www.cs.brown.edu/people/trevor/Papers/Bavoil-2005-VEI.pdf Bavoil-2005-VEI]</ref>, providing visualizations by analogy <ref> [http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=4376187&isnumber=4376125 Querying and Creating Visualizations by Analogy] </ref> and offering automated suggestions <ref> [http://www.cs.utah.edu/~juliana/pub/tvcg-recommendation2008.pdf VisComplete: Automating Suggestions from Visualization Pipelines] </ref>, which we expect to generalize to user interaction history.

== Semantic-level Interaction Chunking ==
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

'''Aims'''
* Develop techniques for ''chunking'' low-level interaction primitives into ''semantic-level'' interactions, given an application's functionality and data-context. (And de-chunking? Invertible mapping needed?)
* Perform user-study evaluation to validate chunking methods.

'''Contributions'''
* Design and user-study evaluation of semantic chunking techniques.

'''Significance'''
* Low-level <--> semantic-level mapping allows for cognitive-modeling to be applied at a functionality level, where low-level interaction techniques can be swapped out. This will allow our interface assessment system to make feasible suggestions for more optimal interface design.

'''Background and Related Work'''
* Chunking interactions has been studied in the HCI community as in <ref> [http://www.cs.brown.edu/people/trevor/Papers/Heer-2008-GraphicalHistories.pdf Graphical Histories for Visualization: Supporting Analysis, Communication, and Evaluation] (InfoVis 2008) </ref>.

== Parallel Framework for Evaluation Modules ==
* Owner: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]]

''This section will describe in more detail the inputs, outputs and architecture that were presented in the introduction.''

==== Module Inputs (Incomplete) ====
* A set of utility dimensions {d1, d2, ...} are defined in the framework. These could be {d1="time", d2="fatigue", ...}
* A set of interaction functions. These specify all of the information that the application wants to give users or get from them. It is not tied to a specific interface. For example, the fact that an applications shows videos would be included here. Whether it displays them embedded, in a pop-up window or fullscreen would not.
* A mapping of interaction functions to interface elements (e.g., buttons, windows, dials,...). Lots of optional information describing visual properties, associated text, physical interactions (e.g., turning the dial clockwise increases the input value) and timing.
* Functional user traces - sequences of interaction functions that represent typical user interactions with the application. Could include a goal hierarchy, in which case the function sequence is at the bottom of the hierarchy.

==== Module Outputs ====
* Every module ouputs at least one of the following:
** An evaluation of the interface
*** This can be on any or all of the utility dimensions, e.g. evaluation={d1=score1, d2=score2, ...}
*** This can alternately be an overall evaluation, ignoring dimensions, e.g. evaluation={score}
**** In this case, the aggregator will treat this as the module giving the same score to all dimensions. Which dimension this evaluator actually predicts well on can be learned by the aggregator over time.
** Recommendation(s) for improving the interface
***This can be a textual description of what changes the designer should make
***This can alternately be a transformation that can automatically be applied to the interface language (without designer intervention)
***In addition to the textual or transformational description of the recommendation, a "change in evaluation" is output to describe how specifically the value will improve the interface
****Recommendation = {description="make this change", Δevaluation={d1=score1, d2=score2, ...}
****Like before, this Δevaluation can cover any number of dimensions, or it can be generic.
***Either a single recommendation or a set of recommendations can be output

==== Aggregator Inputs ====
The aggregator receives as input the outputs of all the modules.

==== Aggregator Outputs ====

Outputs for the aggregator are the same as the outputs for each module. The difference is that the aggregator will consider all the module outputs, and arrive at a merged output based on the past performance of each of the modules.

== Evaluation and Recommendation via Modules ==
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

A "meta-module" called the aggregator will be responsible for assembling and formatting the output of all other modules into a structure that is both extensible and immediately usable, by both an automated designer or a human designer.

===Requirements===
The aggregator's functionality, then, is defined by its '''inputs''', the outputs of the other modules, and the desired '''output''' of the system as a whole, per its position in the architecture. Its purpose is largely formatting and reconciliation of the products of the multitudinous (and extensible) modules. The output of the aggregator must meet several requirements: first, to generate a set of human-readable suggestions for the improvement of the given interface; second, to generate a machine-readable, but also analyzable, evaluation of the various characteristics of the interface and accompanying user traces.

From these specifications, it is logical to assume that a common language or format will be required for the output of individual modules. We propose an XML-based file format, allowing: (1) a section for the standardized identification of problem areas, applicable rules, and proposed improvements, generalized by the individual module and mapped to a single element, or group of elements, in the original interface specification; (2) a section for specification of generalizable "utility" functions, allowing a module to specify how much a measurable quantity of utility is positively or negatively affected by properties of the input interface; (3) new, user-definable sections for evaluations of the given interface not covered by the first two sections. The first two sections are capable of conveying the vast majority of module outputs predicted at this time, but the XML can extensibly allow modules to pass on whatever information may become prominent in the future.

===Specification===
<module id="Fitts-Law">
<interface-elements>
<element>
<desc>submit button</desc>
<problem>
<desc>size</desc>
<suggestion>width *= 2</suggestion>
<suggetsion>height *= 2</suggestions>
<human-suggestion>Increase size relative to other elements</human-suggestion>
</problem>
</element>
</interface-elements>
<utility>
<dimension>
<desc>time</desc>
<value>0:15:35</value>
</dimension>
<dimension>
<desc>frustration</desc>
<value>pulling hair out</value>
</dimension>
<dimension>
<desc>efficiency</desc>
<value>13.2s/KPM task</value>
<value>0.56m/CPM task</value>
</dimension>
</utility>
<tasks>
<task>
<desc>complete form</desc>
</task>
<task>
<desc>lookup SSN</desc>
</task>
<task>
<desc>format phone number</desc>
</task>
</tasks>
</module>

===Logic===
This file provided by each module is then the input for the aggregator. The aggregator's most straightforward function is the compilation of the "problem areas," assembling them and noting problem areas and suggestions that are recommended by more than one module, and weighting them accordingly in its final report. These weightings can begin in an equal state, but the aggregator should be capable of learning iteratively which modules' results are most relevant to the user and update weightings accordingly. This may need to be accomplished with manual tuning, or a machine-learning algorithm capable of determining which modules most often agree with others.

Secondly, the aggregator compiles the utility functions provided by the module specs. This, again, is a summation of similarly-described values from the various modules.

When confronted with user-defined sections of the XML spec, the aggregator is primarily responsible for compiling them and sending them along to the output of the machine. Even if the aggregator does not recognize a section or property of the evaluative spec, if it sees the property reported by more than one module it should be capable of aggregating these intelligently. In future versions of the spec, it should be possible for a module to provide instructions for the aggregator on how to handle unrecognized sections of the XML.

From these compilations, then, the aggregator should be capable of outputting both aggregated human-readable suggestions on interface improvements for a human designer, as well as a comprehensive evaluation of the interface's effectiveness at the given task traces. Again, this is dependent on the specification of the system as a whole, but is likely to include measures and comparisons, graphings of task versus utility, and quantitative measures of an element's effectiveness.

''This section is necessarily defined by the output of the individual modules (which I already expect to be of varied and arbitrary structure) and the desired output of the machine as a whole. It will likely need to be revised heavily after other modules and the "Parellel Framework" section are defined.'' [[User:E J Kalafarski|E J Kalafarski]] 12:34, 24 April 2009 (UTC)

''This section describes the aggregator, which takes the output of multiple independent modules and aggregates the results to provide (1) an evaluation and (2) recommendations for the user interface. We should explain how the aggregator weights the output of different modules (this could be based on historical performance of each module, or perhaps based on E.J.'s cognitive/HCI guidelines).''

== Sample Modules ==

=== CPM-GOMS ===
* Owners: [[User:Steven Ellis | Steven Ellis]]

''This module will provide interface evaluations and suggestions based on a CPM-GOMS model of cognition for the given interface. It will provide a quantitative, predictive, cognition-based parameterization of usability. From empirically collected data, user trajectories through the model (critical paths) will be examined, highlighting bottlenecks within the interface, and offering suggested alterations to the interface to induce more optimal user trajectories.''

I’m hoping to have some input on this section, because it seems to be the crux of the “black box” into which we take the inputs of interface description, user traces, etc. and get our outputs (time, recommendations, etc.). I know at least a few people have pretty strong thoughts on the matter and we ought to discuss the final structure.

That said – my proposal for the module:
*In my opinion the concept of the Model Human Processor (at least as applied in CPM) is outdated – it’s too economic/overly parsimonious in its conception of human activity. I think we need to create a structure which accounts for more realistic conditions of HCI including multitasking, aspects of distributed cognition (and other relevant uses of tools – as far as I can tell CPM doesn’t take into account any sort of productivity aids), executive control processes of attention, etc. ACT-R appears to take steps towards this but we would probably need to look at their algorithms to know for sure.
*Critical paths will continue to play an important role – we should in fact emphasize that part of this tool’s purpose will be a description not only of ways in which the interface should be modified to best fit a critical path, but also ways in which the user’s ought to be instructed in their use of the path. This feedback mechanism could be bidirectional – if the model’s predictions of the user’s goals are incorrect, the critical path determined will also be incorrect and the interface inherently suboptimal. The user could be prompted with a tooltip explaining in brief why and how the interface has changed, along with options to revert, select other configurations (euphemized by goals), and to view a short video detailing how to properly use the interface.
*Call me crazy but, if we assume designers will be willing to code a model of their interfaces into our ACT-R-esque language, could we allow that model to be fairly transparent to the user, who could use a gui to input their goals to find an analogue in the program which would subsequently rearrange its interface to fit the user’s needs? Even if not useful to the users, such dynamic modeling could really help designers (IMO)
*I think the model should do its best to accept models written for ACT-R and whatever other cognitive models there are out there – gives us the best chance of early adoption
*I would particularly appreciate input on the number/complexity/type of inputs we’ll be using, as well as the same qualities for the output.

=== HCI Guidelines ===
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

Schneiderman's Eight Golden Rules and Jakob Nielsen's Ten Heuristics are perhaps the most famous and well-regarded heuristic design guidelines to emerge over the last twenty years. Although the explicit theoretical basis for such heuristics is controversial and not well-explored, the empirical success of these guidelines is established and accepted. This module will parse out up to three or four common (that is, intersecting) principles from these accepted guidelines and apply them to the input interface.

As an example, we identify an analogous principle that appears in Schneiderman ("Reduce short-term memory load")<ref>http://faculty.washington.edu/jtenenbg/courses/360/f04/sessions/schneidermanGoldenRules.html</ref> and Nielsen ("Recognition rather than recall/Minimize the user's memory load")<ref>http://www.useit.com/papers/heuristic/heuristic_list.html</ref>. The input interface is then evaluated for the consideration of the principle, based on an explicit formal description of the interface, such as XAML or XUL. The module attempts to determine how effectively the interface demonstrates the principle. When analyzing an interface for several principles that may be conflicting or opposing in a given context, the module makes use of a hard-coded but iterative (and evolving) weighting of these principles, based on (1) how often they appear in the training set of accepted sets of guidelines, (2) how analogues a heuristic principle is to a cognitive principle in a parallel training set, and (3) how effective the principle's associated suggestion is found to be using a feedback mechanism.

'''Inputs''' 
# A formal description of the interface and its elements (e.g. buttons).
# A formal description of a particular task and the possible paths through a subset of interface elements that permit the user to accomplish that task.

'''Output''' 
Standard XML-formatted file containing problem areas of the input interface, suggestions for each problem area based on principles that were found to have a strong application to a problem element and the problem itself, and a human-readable generated analysis of the element's affinity for the principle. Quantitative outputs will not be possible based on heuristic guidelines, and the "utility" section of this module's output is likely to be blank.

''This section could include an example or two of established design guidelines that could easily be implemented as modules.''

=== Fitts's Law ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

''This module provides an estimate of the required time to complete various tasks that have been decomposed into formalized sequences of interactions with interface elements, and will provide evaluations and recommendations for optimizing the time required to complete those tasks using the interface.''

'''Inputs'''

1. A formal description of the interface and its elements (e.g. buttons).

2. A formal description of a particular task and the possible paths through a subset of interface elements that permit the user to accomplish that task.

3. The physical distances between interface elements along those paths.

4. The width of those elements along the most likely axes of motion.

5. Device (e.g. mouse) characteristics including start/stop time and the inherent speed limitations of the device.

'''Output'''

The module will then use the Shannon formulation of Fitt's Law to compute the average time needed to complete the task along those paths.

=== Affordances ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

''This simple module will provide interface evaluations and recommendations based on a measure of the extent to which the user perceives the relevant affordances of the interface when performing a number of specified tasks.''

'''Inputs'''

Formalized descriptions of...

1. Interface elements

2. Their associated actions

3. The functions of those actions

4. A particular task

5. User traces for that task.

Inputs (1-4) are then used to generate a "user-independent" space of possible functions that the interface is capable of performing with respect to a given task -- what the interface "affords" the user. From this set of possible interactions, our model will then determine the subset of optimal paths for performing a particular task. The user trace (5) is then used to determine what functions actually were performed in the course of a given task of interest and this information is then compared to the optimal path data to determine the extent to which affordances of the interface are present but not perceived.

'''Output'''

The output of this module is a simple ratio of (affordances perceived) / [(relevant affordances present) * (time to complete task)] which provides a quantitative measure of the extent to which the interface is "natural" to use for a particular task.

=== Workflow, Multi-tasking and Interruptions ===
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

There are, at least, two levels at which users work ([http://portal.acm.org/citation.cfm?id=985692.985707&coll=portal&dl=ACM&CFID=20781736&CFTOKEN=83176621 Gonzales, et al., 2004]). Users accomplish individual low-level tasks which are part of larger ''working spheres''; for example, an office worker might send several emails, create several Post-It (TM) note reminders, and then edit a word document, each of these smaller tasks being part of a single larger working sphere of "adding a new section to the website." Thus, it is important to understand this larger workflow context - which often involves extensive levels of multi-tasking, as well as switching between a variety of computing devices and traditional tools, such as notebooks. In this study it was found that the information workers surveyed typically switch individual tasks every 2 minutes and have many simultaneous working spheres which they switch between, on average every 12 minutes. This frenzied pace of switching tasks and switching working spheres suggests that users will not be using a single application or device for a long period of time, and that affordances to support this characteristic pattern of information work are important.

The purpose of this module is to integrate existing work on multi-tasking, interruption and higher-level workflow into a framework which can predict user recovery times from interruptions. Specifically, the goals of this framework will be to:

* Understand the role of the larger workflow context in user interfaces
* Understand the impact of interruptions on user workflow
* Understand how to design software which fits into the larger working spheres in which information work takes place

It is important to point out that because workflow and multi-tasking rely heavily on higher-level brain functioning, it is unrealistic within the scope of this grant to propose a system which can predict user performance given a description of a set of arbitrary software programs. Therefore, we believe this module will function much more in a qualitative role to provide context to the rest of the model. Specifically, our findings related to interruption and multi-tasking will advance the basic research question of "how do you users react to interruptions when using working sets of varying sizes?". This core HCI contribution will help to inform the rest of the outputs of the model in a qualitative manner.

'''Inputs'''

N/A

'''Outputs'''

N/A

=== Working Memory Load ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''This module measures how much information the user needs to retain in memory while interacting with the interface and makes suggestions for improvements.''

* Models
** Beddeley's Model of Working Memory<ref>http://en.wikipedia.org/wiki/Baddeley%27s_model_of_working_memory</ref>
*** Episodic Buffer
** George Miller's "The magic number 7 plus or minus 2"<ref>http://en.wikipedia.org/wiki/The_Magical_Number_Seven,_Plus_or_Minus_Two</ref>
** The magic number 4<ref>Cowan, N. (2001). The magical number 4 in short-term memory: A reconsideration of mental storage capacity. Behavioral and Brain Sciences, 24, 87-185.</ref>
* Chunking<ref>http://en.wikipedia.org/wiki/Chunking_(psychology)</ref>
* Priming<ref>http://en.wikipedia.org/wiki/Priming_(psychology)</ref>
* Subitizing and Counting<ref>http://en.wikipedia.org/wiki/Subitizing</ref>

'''Inputs'''
* Visual Stimuli
* Audio Stimuli
'''Outputs'''
* Remembered percepts
* Half-Life of percepts

=== Automaticity of Interaction ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''Measures how easily the interaction with the interface becomes automatic with experience and makes suggestions for improvements.''

* Learning<ref>http://en.wikipedia.org/wiki/Learning#Mathematical_models_of_learning</ref>
** Logan's instance theory of automatization<ref>http://74.125.95.132/search?q=cache:IZ-Zccsu3SEJ:psych.wisc.edu/ugstudies/psych733/logan_1988.pdf+logan+isntance+teory&cd=1&hl=en&ct=clnk&gl=us&client=firefox-a</ref>
* Fluency
** As meta-cognitive information <ref>http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6VH9-4SM7PFK-4&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=10cd279fa80958981fcc3c06684c09af</ref>
** As a cognitive ''heuristic''<ref>http://en.wikipedia.org/wiki/Fluency_heuristic</ref>

'''Inputs'''
* Interface
* User goals

'''Outputs'''
* Learning curve

== Integration into the Design Process ==
* Owner: [[User:Ian Spector | Ian Spector]]

''This section outlines the process of designing an HCI interface and at what stages our proposal fits in and how.''

= Preliminary Results =

===Workflow, Multi-tasking, and Interruption===

====I. '''Goals'''====

The goals of the preliminary work are to gain qualitative insight into how information workers practice metawork, and to determine whether people might be better-supported with software which facillitates metawork and interruptions. Thus, the preliminary work should investigate, and demonstrate, the need and impact of the core goals of the project.

====II. '''Methodology'''====

Seven information workers, ages 20-38 (5 male, 2 female), were interviewed to determine which methods they use to "stay organized". An initial list of metawork strategies was established from two pilot interviews, and then a final list was compiled. Participants then responded to a series of 17 questions designed to gain insight into their metawork strategies and process. In addition, verbal interviews were conducted to get additional open-ended feedback.

====III. '''Final Results'''====
A histogram of methods people use to "stay organized" in terms of tracking things they need to do (TODOs), appointments and meetings, etc. is shown in the figure below.

[[Image:AcbGraph.jpg]]

In addition to these methods, participants also used a number of other methods, including:

* iCal
* Notes written in xterms
* "Inbox zero" method of email organization
* iGoogle Notepad (for tasks)
* Tag emails as "TODO", "Important", etc.
* Things (Organizer Software)
* Physical items placed to "remind me of things"
* Sometimes arranging windows on desk
* Keeping browser tabs open
* Bookmarking web pages
* Keep programs/files open scrolled to certain locations sometimes with things selected

In addition, three participants said that when interrupted they "rarely" or "very rarely" were able to resume the task they were working on prior to the interruption. Three of the participants said that they would not actively recommend their metawork strategies for other people, and two said that staying organized was "difficult".

Four participants were neutral to the idea of new tools to help them stay organized and three said that they would like to have such a tool/tools.

====IV. '''Discussion'''====
These results qunatiatively support our hypothesis that there is no clearly dominant set of metawork strategies employed by information workers. This highly fragemented landscape is surprising, even though most information workers work in a similar environment - at a desk, on the phone, in meetings - and with the same types of tools - computers, pens, paper, etc. We believe that this suggests that there are complex tradeoffs between these methods and that no single method is sufficient. We therefore believe that users will be better supported with a new set of software-based metawork tools.

= [Criticisms] =
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

''Any criticisms or questions we have regarding the proposal can go here.''

=References=
<references/>

CS295J/Research proposal (draft 2)

2009-04-24T16:57:46Z

Trevor O'Brien: /* Semantic-level Interaction Chunking */

= Introduction =
* Owners: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]], [[User: Trevor O'Brien | Trevor]]

'''Propose:''' The design, application and evaluation of a novel, cognition-based, computational framework for assessing interface design and providing automated suggestions to optimize usability.

'''Evaluation Methodology:''' Our techniques will be evaluated quantitatively through a series of user-study trials, as well as qualitatively by a team of expert interface designers.

'''Contributions and Significance:''' We expect this work to make the following contributions:
# design-space analysis and quantitative evaluation of cognition-based techniques for assessing user interfaces.
# design and quantitative evaluation of techniques for suggesting optimized interface-design changes.
# an extensible, multimodal software architecture for capturing user traces integrated with pupil-tracking data, auditory recognition, and muscle-activity monitoring.
# specification (language?) of how to define an interface evaluation module and how to integrate it into a larger system.
# (there may be more here, like testing different cognitive models, generating a markup language to represent interfaces, maybe even a unique metric space for interface usability)

--

We propose a framework for interface evaluation and recommendation that integrates behavioral models and design guidelines from both cognitive science and HCI. Our framework behaves like a committee of specialized experts, where each expert provides its own assessment of the interface, given its particular knowledge of HCI or cognitive science. For example, an expert may provide an evaluation based on the GOMS method, Fitts's law, Maeda's design principles, or cognitive models of learning and memory. An aggregator collects all of these assessments and weights the opinions of each expert, and outputs to the developer a merged evaluation score and a weighted set of recommendations.

Systematic methods of estimating human performance with computer interfaces are used only sparsely despite their obvious benefits, the reason being the overhead involved in implementing them. In order to test an interface, both manual coding systems like the GOMS variations and user simulations like those based on ACT-R/PM and EPIC require detailed pseudo-code descriptions of the user workflow with the application interface. Any change to the interface then requires extensive changes to the pseudo-code, a major problem because of the trial-and-error nature of interface design. Updating the models themselves is even more complicated. Even an expert in CPM-GOMS, for example, can't necessarily adapt it to take into account results from new cognitive research.

Our proposal makes automatic interface evaluation easier to use in several ways. First of all, we propose to divide the input to the system into three separate parts, functionality, user traces and interface. By separating the functionality from the interface, even radical interface changes will require updating only that part of the input. The user traces are also defined over the functionality so that they too translate across different interfaces. Second, the parallel modular architecture allows for a lower "entry cost" for using the tool. The system includes a broad array of evaluation modules some of which are very simple and some more complex. The simpler modules use only a subset of the input that a system like GOMS or ACT-R would require. This means that while more input will still lead to better output, interface designers can get minimal evaluations with only minimal information. For example, a visual search module may not require any functionality or user traces in order to determine whether all interface elements are distinct enough to be easy to find. Finally, a parallel modular architecture is much easier to augment with relevant cognitive and design evaluations.

= Background / Related Work =

Each person should add the background related to their specific aims.

* Steven Ellis - Cognitive models of HCI, including GOMS variations and ACT-R
* EJ - Design Guidelines
* Jon - Perception and Action
* Andrew - Multiple task environments
* Gideon - Cognition and dual systems
* Ian - Interface design process
* Trevor - User trace collection methods (especially any eye-tracking, EEG, ... you want to suggest using)

== Cognitive Models ==
I plan to port over most of the background on cognitive models of HCI from the old proposal

Additions will comprise of:
*CPM-GOMS as a bridge from GOMS architecture to the promising procedural optimization of the Model Human Processor
**Context of CPM development, discuss its relation to original GOMS and KLM
***Establish the tasks which were relevant for optimization when CPM was developed and note that its obsolescence may have been unavoidable
**Focus on CPM as the first step in transitioning from descriptive data, provided by mounting efforts in the cognitive sciences realm to discover the nature of task processing and accomplishment, to prescriptive algorithms which can predict an interface’s efficiency and suggest improvements
**CPM’s purpose as an abstraction of cognitive processing – a symbolic representation not designed for accuracy but precision
**CPM’s successful trials, e.g. Ernestine
***Implications of this project include CPM’s ability to accurately estimate processing at a psychomotor level
***Project does suggest limitations, however, when one attempts to examine more complex tasks which involve deeper and more numerous cognitive processes
*ACT-R as an example of a progressive cognitive modeling tool
**A tool clearly built by and for cognitive scientists, and as a result presents a much more accurate view of human processing – helpful for our research
**Built-in automation, which now seems to be a standard feature of cognitive modeling tools
**Still an abstraction of cognitive processing, but makes adaptation to cutting-edge cognitive research findings an integral aspect of its modular structure
**Expand on its focus on multi-tasking, taking what was a huge advance between GOMS and its CPM variation and bringing the simulation several steps closer to approximating the nature of cognition in regards to HCI
**Far more accessible both for researchers and the lay user/designer in its portability to LISP, pre-construction of modules representing cognitive capacities and underlying algorithms modeling paths of cognitive processing

==Design guidelines==
A multitude of rule sets exist for the design of not only interface, but architecture, city planning, and software development. They can range in scale from one primary rule to as many Christopher Alexander's 253 rules for urban environments,<ref>[http://hci.rwth-aachen.de/materials/publications/borchers2000a.pdf Borchers, Jan O. "A Pattern Approach to Interaction Design." 2000.]</ref> which he introduced with the concept design patterns in the 1970s. Study has likewise been conducted on the use of these rules:<ref>http://stl.cs.queensu.ca/~graham/cisc836/lectures/readings/tetzlaff-guidelines.pdf</ref> guidelines are often only partially understood, indistinct to the developer, and "fraught" with potential usability problems given a real-world situation.

===Application to AUE===
And yet, the vast majority of guideline sets, including the most popular rulesets, have been arrived at heuristically. The most successful, such as Raskin's and Schneiderman's, have been forged from years of observation instead of empirical study and experimentation. The problem is similar to the problem of circular logic faced by automated usability evaluations: an automated system is limited in the suggestions it can offer to a set of preprogrammed guidelines which have often not been subjected to rigorous experimentation.<ref>[http://www.eecs.berkeley.edu/Pubs/TechRpts/2000/CSD-00-1105.pdf Ivory, M and Hearst, M. "The State of the Art in Automated Usability Evaluation of User Interfaces." ACM Computing Surveys (CSUR), 2001.]</ref> In the vast majority of existing studies, emphasis has traditionally been placed on either the development of guidelines or the application of existing guidelines to automated evaluation. A mutually-reinforcing development of both simultaneously has not been attempted.

Overlap between rulesets is inevitable and unavoidable. For our purposes of evaluating existing rulesets efficiently, without extracting and analyzing each rule individually, it may be desirable to identify the the overarching ''principles'' or ''philosophy'' (max. 2 or 3) for a given ruleset and determining their quantitative relevance to problems of cognition.

===Popular and seminal examples===
Schneiderman's [http://faculty.washington.edu/jtenenbg/courses/360/f04/sessions/schneidermanGoldenRules.html Eight Golden Rules] date to 1987 and are arguably the most-cited. They are heuristic, but can be somewhat classified by cognitive objective: at least two rules apply primarily to ''repeated use'', versus ''discoverability''. Up to five of Schneiderman's rules emphasize ''predictability'' in the outcomes of operations and ''increased feedback and control'' in the agency of the user. His final rule, paradoxically, removes control from the user by suggesting a reduced short-term memory load, which we can arguably classify as ''simplicity''.

Raskin's [http://www.mprove.de/script/02/raskin/designrules.html Design Rules] are classified into five principles by the author, augmented by definitions and supporting rules. While one principle is primarily aesthetic (a design problem arguably out of the bounds of this proposal) and one is a basic endorsement of testing, the remaining three begin to reflect philosophies similar to Schneiderman's: reliability or ''predictability'', ''simplicity'' or ''efficiency'' (which we can construe as two sides of the same coin), and finally he introduces a concept of ''uninterruptibility''.

Maeda's [http://lawsofsimplicity.com/?cat=5&order=ASC Laws of Simplicity] are fewer, and ostensibly emphasize ''simplicity'' exclusively, although elements of ''use'' as related by Schneiderman's rules and ''efficiency'' as defined by Raskin may be facets of this simplicity. Google's corporate mission statement presents [http://www.google.com/corporate/ux.html Ten Principles], only half of which can be considered true interface guidelines. ''Efficiency'' and ''simplicity'' are cited explicitly, aesthetics are once again noted as crucial, and working within a user's trust is another application of ''predictability''.

===Elements and goals of a guideline set===
Myriad rulesets exist, but variation becomes scarce—it indeed seems possible to parse these common rulesets into overarching principles that can be converted to or associated with quantifiable cognitive properties. For example, it is likely ''simplicity'' has an analogue in the short-term memory retention or visual retention of the user, vis a vis the rule of [http://books.google.com/books?hl=en&lr=&id=j5q0VvOGExYC&oi=fnd&pg=PA357&dq=seven+plus+or+minus+two&ots=prI3PKJBar&sig=vOZnqpnkXKGYWxK6_XlA4I_CRyI Seven, Plus or Minus Two]. ''Predictability'' likewise may have an analogue in Activity Theory, in regards to a user's perceptual expectations for a given action; ''uninterruptibility'' has implications in cognitive task-switching;<ref>[http://portal.acm.org/citation.cfm?id=985692.985715&coll=Portal&dl=ACM&CFID=21136843&CFTOKEN=23841774 Czerwinski, Horvitz, and White. "A Diary Study of Task Switching and Interruptions." Proceedings of the SIGCHI conference on Human factors in computing systems, 2004.]</ref> and so forth.

Within the scope of this proposal, we aim to reduce and refine these philosophies found in seminal rulesets and identify their logical cognitive analogues. By assigning a quantifiable taxonomy to these principles, we will be able to rank and weight them with regard to their real-world applicability, developing a set of "meta-guidelines" and rules for applying them to a given interface in an automated manner. Combined with cognitive models and multi-modal HCI analysis, we seek to develop, in parallel with these guidelines, the interface evaluation system responsible for their application.

== Perception and Action (in progress) ==

*Information Processing Approach
**Advantages
***Formalism eases translation of theory into scripting language
**Disadvantages
***Assumes symbolic representation

*Ecological (Gibsonian) Approach
**Advantages
***Emphasis on bodily and environmental constraints
**Disadvantages
***Lack of formalism hinders translation of theory into scripting language

= Specific Aims and Contributions (to be separated later) =

'''Specific Aims'''
* Incorporate interaction history mechanisms into a set of existing applications.
* Perform user-study evaluation of history-collection techniques.
* Distill a set of cognitive principles/models, and evaluate empirically?
* Build/buy sensing system to include pupil-tracking, muscle-activity monitoring, auditory recognition.
* Design techniques for manual/semi-automated/automated construction of <insert favorite cognitive model here> from interaction histories and sensing data.
* Design system for posterior analysis of interaction history w.r.t. <insert favorite cognitive model here>, evaluating critical path <or equivalent> trajectories.
* Design cognition-based techniques for detecting bottlenecks in critical paths, and offering optimized alternatives.
* Perform quantitative user-study evaluations, collect qualitative feedback from expert interface designers.

'''Contributions'''

* Design-space analysis and quantitative evaluation of cognition-based techniques for assessing user interfaces.
* Design and quantitative evaluation of techniques for suggesting optimized interface-design changes.
* An extensible, multimodal software architecture for capturing user traces integrated with pupil-tracking data, auditory recognition, and muscle-activity monitoring.
* (there may be more here, like testing different cognitive models, generating a markup language to represent interfaces, maybe even a unique metric space for interface usability)

--

See the [http://vrl.cs.brown.edu/wiki/images/b/b1/Flowchart2.pdf flowchart] for a visual overview of our aims.

In order to use this framework, a designer will have to provide:
* Functional specification - what are the possible interactions between the user and the application. This can be thought of as method signatures, with a name (e.g., setVolume), direction (to user or from user) and a list of value types (boolean, number, text, video, ...) for each interaction.
* GUI specification - a mapping of interactions to interface elements (e.g., setVolume is mapped to the grey knob in the bottom left corner with clockwise turning increasing the input number).
* Functional user traces - sequences of representative ways in which the application is used. Instead of writing them, the designer could have users use the application with a trial interface and then use our methods to generalize the user traces beyond the specific interface (The second method is depicted in the diagram). As a form of pre-processing, the system also generates an interaction transition matrix which lists the probability of each type of interaction given the previous interaction.
* Utility function - this is a weighting of various performance metrics (time, cognitive load, fatigue, etc.), where the weighting expresses the importance of a particular dimension to the user. For example, a user at NASA probably cares more about interface accuracy than speed. By passing this information to our committee of experts, we can create interfaces that are tuned to maximize the utility of a particular user type.

Each of the modules can use all of this information or a subset of it. Our approach stresses flexibility and the ability to give more meaningful feedback the more information is provided. After processing the information sent by the system of experts, the aggregator will output:
* An evaluation of the interface. Evaluations are expressed both in terms of the utility function components (i.e. time, fatigue, cognitive load, etc.), and in terms of the overall utility for this interface (as defined by the utility function). These evaluations are given in the form of an efficiency curve, where the utility received on each dimension can change as the user becomes more accustomed to the interface.
* Suggested improvements for the GUI are also output. These suggestions are meant to optimize the utility function that was input to the system. If a user values accuracy over time, interface suggestions will be made accordingly.

== Recording User-Interaction Primitives ==
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

'''Aims'''
* Develop system for logging low-level user-interactions within existing applications. By low-level interactions, we refer to those interactions for which a quantitative, predictive, GOMS-like model may be generated.
* At the same scale of granularity, integrate explicit interface-interactions with multimodal-sensing data. i.e. pupil-tracking, muscle-activity monitoring, auditory recognition, EEG data, and facial and posture-focused video.

'''Contributions'''
* Extensible, multimodal, HCI framework for recording rich interaction-history data in existing applications.

'''Significance'''
* Provide rich history data to serve as the basis for novel quantitative interface evaluation.
* Aid expert HCI and visual designers in traditional design processes.
* Provide data for automated machine-learning strategies applied to interaction.

'''Background and Related Work'''
* The utility of interaction histories with respect to assessing interface design has been demonstrated in <ref> [http://www.cs.brown.edu/people/trevor/Papers/Heer-2008-GraphicalHistories.pdf Graphical Histories for Visualization: Supporting Analysis, Communication, and Evaluation] (InfoVis 2008) </ref>.
* In addition, data management histories have been shown effective in the visualization community in <ref> [http://www.cs.brown.edu/people/trevor/Papers/Callahan-2006-MED.pdf Callahan-2006-MED] </ref> <ref> [http://www.cs.brown.edu/people/trevor/Papers/Callahan-2006-VVM.pdf Callahan-2006-VVM] </ref> <ref> [http://www.cs.brown.edu/people/trevor/Papers/Bavoil-2005-VEI.pdf Bavoil-2005-VEI]</ref>, providing visualizations by analogy <ref> [http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=4376187&isnumber=4376125 Querying and Creating Visualizations by Analogy] </ref> and offering automated suggestions <ref> [http://www.cs.utah.edu/~juliana/pub/tvcg-recommendation2008.pdf VisComplete: Automating Suggestions from Visualization Pipelines] </ref>, which we expect to generalize to user interaction history.

== Semantic-level Interaction Chunking ==
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

'''Aims'''
* Develop techniques for ''chunking'' low-level interaction primitives into ''semantic-level'' interactions, given an application's functionality and data-context. (And de-chunking? Invertible mapping needed?)
* Perform user-study evaluation to validate chunking methods.

'''Contributions'''
* Design and user-study evaluation of semantic chunking techniques.

'''Significance'''
* Low-level <--> semantic-level mapping allows for cognitive-modeling to be applied at a functionality level, where low-level interaction techniques can be swapped out. This will allow our interface assessment system to make feasible suggestions for more optimal interface design.

'''Background and Related Work'''
* Chunking interactions has been studied in the HCI community as in <ref> [http://www.cs.brown.edu/people/trevor/Papers/Heer-2008-GraphicalHistories.pdf Graphical Histories for Visualization: Supporting Analysis, Communication, and Evaluation] (InfoVis 2008) <\ref>.

== Parallel Framework for Evaluation Modules ==
* Owner: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]]

''This section will describe in more detail the inputs, outputs and architecture that were presented in the introduction.''

==== Module Inputs (Incomplete) ====
* A set of utility dimensions {d1, d2, ...} are defined in the framework. These could be {d1="time", d2="fatigue", ...}
* A set of interaction functions. These specify all of the information that the application wants to give users or get from them. It is not tied to a specific interface. For example, the fact that an applications shows videos would be included here. Whether it displays them embedded, in a pop-up window or fullscreen would not.
* A mapping of interaction functions to interface elements (e.g., buttons, windows, dials,...). Lots of optional information describing visual properties, associated text, physical interactions (e.g., turning the dial clockwise increases the input value) and timing.
* Functional user traces - sequences of interaction functions that represent typical user interactions with the application. Could include a goal hierarchy, in which case the function sequence is at the bottom of the hierarchy.

==== Module Outputs ====
* Every module ouputs at least one of the following:
** An evaluation of the interface
*** This can be on any or all of the utility dimensions, e.g. evaluation={d1=score1, d2=score2, ...}
*** This can alternately be an overall evaluation, ignoring dimensions, e.g. evaluation={score}
**** In this case, the aggregator will treat this as the module giving the same score to all dimensions. Which dimension this evaluator actually predicts well on can be learned by the aggregator over time.
** Recommendation(s) for improving the interface
***This can be a textual description of what changes the designer should make
***This can alternately be a transformation that can automatically be applied to the interface language (without designer intervention)
***In addition to the textual or transformational description of the recommendation, a "change in evaluation" is output to describe how specifically the value will improve the interface
****Recommendation = {description="make this change", Δevaluation={d1=score1, d2=score2, ...}
****Like before, this Δevaluation can cover any number of dimensions, or it can be generic.
***Either a single recommendation or a set of recommendations can be output

==== Aggregator Inputs ====
The aggregator receives as input the outputs of all the modules.

==== Aggregator Outputs ====

Outputs for the aggregator are the same as the outputs for each module. The difference is that the aggregator will consider all the module outputs, and arrive at a merged output based on the past performance of each of the modules.

== Evaluation and Recommendation via Modules ==
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

A "meta-module" called the aggregator will be responsible for assembling and formatting the output of all other modules into a structure that is both extensible and immediately usable, by both an automated designer or a human designer.

===Requirements===
The aggregator's functionality, then, is defined by its '''inputs''', the outputs of the other modules, and the desired '''output''' of the system as a whole, per its position in the architecture. Its purpose is largely formatting and reconciliation of the products of the multitudinous (and extensible) modules. The output of the aggregator must meet several requirements: first, to generate a set of human-readable suggestions for the improvement of the given interface; second, to generate a machine-readable, but also analyzable, evaluation of the various characteristics of the interface and accompanying user traces.

From these specifications, it is logical to assume that a common language or format will be required for the output of individual modules. We propose an XML-based file format, allowing: (1) a section for the standardized identification of problem areas, applicable rules, and proposed improvements, generalized by the individual module and mapped to a single element, or group of elements, in the original interface specification; (2) a section for specification of generalizable "utility" functions, allowing a module to specify how much a measurable quantity of utility is positively or negatively affected by properties of the input interface; (3) new, user-definable sections for evaluations of the given interface not covered by the first two sections. The first two sections are capable of conveying the vast majority of module outputs predicted at this time, but the XML can extensibly allow modules to pass on whatever information may become prominent in the future.

===Specification===
<module id="Fitts-Law">
<interface-elements>
<element>
<desc>submit button</desc>
<problem>
<desc>size</desc>
<suggestion>width *= 2</suggestion>
<suggetsion>height *= 2</suggestions>
<human-suggestion>Increase size relative to other elements</human-suggestion>
</problem>
</element>
</interface-elements>
<utility>
<dimension>
<desc>time</desc>
<value>0:15:35</value>
</dimension>
<dimension>
<desc>frustration</desc>
<value>pulling hair out</value>
</dimension>
<dimension>
<desc>efficiency</desc>
<value>13.2s/KPM task</value>
<value>0.56m/CPM task</value>
</dimension>
</utility>
<tasks>
<task>
<desc>complete form</desc>
</task>
<task>
<desc>lookup SSN</desc>
</task>
<task>
<desc>format phone number</desc>
</task>
</tasks>
</module>

===Logic===
This file provided by each module is then the input for the aggregator. The aggregator's most straightforward function is the compilation of the "problem areas," assembling them and noting problem areas and suggestions that are recommended by more than one module, and weighting them accordingly in its final report. These weightings can begin in an equal state, but the aggregator should be capable of learning iteratively which modules' results are most relevant to the user and update weightings accordingly. This may need to be accomplished with manual tuning, or a machine-learning algorithm capable of determining which modules most often agree with others.

Secondly, the aggregator compiles the utility functions provided by the module specs. This, again, is a summation of similarly-described values from the various modules.

When confronted with user-defined sections of the XML spec, the aggregator is primarily responsible for compiling them and sending them along to the output of the machine. Even if the aggregator does not recognize a section or property of the evaluative spec, if it sees the property reported by more than one module it should be capable of aggregating these intelligently. In future versions of the spec, it should be possible for a module to provide instructions for the aggregator on how to handle unrecognized sections of the XML.

From these compilations, then, the aggregator should be capable of outputting both aggregated human-readable suggestions on interface improvements for a human designer, as well as a comprehensive evaluation of the interface's effectiveness at the given task traces. Again, this is dependent on the specification of the system as a whole, but is likely to include measures and comparisons, graphings of task versus utility, and quantitative measures of an element's effectiveness.

''This section is necessarily defined by the output of the individual modules (which I already expect to be of varied and arbitrary structure) and the desired output of the machine as a whole. It will likely need to be revised heavily after other modules and the "Parellel Framework" section are defined.'' [[User:E J Kalafarski|E J Kalafarski]] 12:34, 24 April 2009 (UTC)

''This section describes the aggregator, which takes the output of multiple independent modules and aggregates the results to provide (1) an evaluation and (2) recommendations for the user interface. We should explain how the aggregator weights the output of different modules (this could be based on historical performance of each module, or perhaps based on E.J.'s cognitive/HCI guidelines).''

== Sample Modules ==

=== CPM-GOMS ===
* Owners: [[User:Steven Ellis | Steven Ellis]]

''This module will provide interface evaluations and suggestions based on a CPM-GOMS model of cognition for the given interface. It will provide a quantitative, predictive, cognition-based parameterization of usability. From empirically collected data, user trajectories through the model (critical paths) will be examined, highlighting bottlenecks within the interface, and offering suggested alterations to the interface to induce more optimal user trajectories.''

I’m hoping to have some input on this section, because it seems to be the crux of the “black box” into which we take the inputs of interface description, user traces, etc. and get our outputs (time, recommendations, etc.). I know at least a few people have pretty strong thoughts on the matter and we ought to discuss the final structure.

That said – my proposal for the module:
*In my opinion the concept of the Model Human Processor (at least as applied in CPM) is outdated – it’s too economic/overly parsimonious in its conception of human activity. I think we need to create a structure which accounts for more realistic conditions of HCI including multitasking, aspects of distributed cognition (and other relevant uses of tools – as far as I can tell CPM doesn’t take into account any sort of productivity aids), executive control processes of attention, etc. ACT-R appears to take steps towards this but we would probably need to look at their algorithms to know for sure.
*Critical paths will continue to play an important role – we should in fact emphasize that part of this tool’s purpose will be a description not only of ways in which the interface should be modified to best fit a critical path, but also ways in which the user’s ought to be instructed in their use of the path. This feedback mechanism could be bidirectional – if the model’s predictions of the user’s goals are incorrect, the critical path determined will also be incorrect and the interface inherently suboptimal. The user could be prompted with a tooltip explaining in brief why and how the interface has changed, along with options to revert, select other configurations (euphemized by goals), and to view a short video detailing how to properly use the interface.
*Call me crazy but, if we assume designers will be willing to code a model of their interfaces into our ACT-R-esque language, could we allow that model to be fairly transparent to the user, who could use a gui to input their goals to find an analogue in the program which would subsequently rearrange its interface to fit the user’s needs? Even if not useful to the users, such dynamic modeling could really help designers (IMO)
*I think the model should do its best to accept models written for ACT-R and whatever other cognitive models there are out there – gives us the best chance of early adoption
*I would particularly appreciate input on the number/complexity/type of inputs we’ll be using, as well as the same qualities for the output.

=== HCI Guidelines ===
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

Schneiderman's Eight Golden Rules and Jakob Nielsen's Ten Heuristics are perhaps the most famous and well-regarded heuristic design guidelines to emerge over the last twenty years. Although the explicit theoretical basis for such heuristics is controversial and not well-explored, the empirical success of these guidelines is established and accepted. This module will parse out up to three or four common (that is, intersecting) principles from these accepted guidelines and apply them to the input interface.

As an example, we identify an analogous principle that appears in Schneiderman ("Reduce short-term memory load")<ref>http://faculty.washington.edu/jtenenbg/courses/360/f04/sessions/schneidermanGoldenRules.html</ref> and Nielsen ("Recognition rather than recall/Minimize the user's memory load")<ref>http://www.useit.com/papers/heuristic/heuristic_list.html</ref>. The input interface is then evaluated for the consideration of the principle, based on an explicit formal description of the interface, such as XAML or XUL. The module attempts to determine how effectively the interface demonstrates the principle. When analyzing an interface for several principles that may be conflicting or opposing in a given context, the module makes use of a hard-coded but iterative (and evolving) weighting of these principles, based on (1) how often they appear in the training set of accepted sets of guidelines, (2) how analogues a heuristic principle is to a cognitive principle in a parallel training set, and (3) how effective the principle's associated suggestion is found to be using a feedback mechanism.

'''Inputs''' 
# A formal description of the interface and its elements (e.g. buttons).
# A formal description of a particular task and the possible paths through a subset of interface elements that permit the user to accomplish that task.

'''Output''' 
Standard XML-formatted file containing problem areas of the input interface, suggestions for each problem area based on principles that were found to have a strong application to a problem element and the problem itself, and a human-readable generated analysis of the element's affinity for the principle. Quantitative outputs will not be possible based on heuristic guidelines, and the "utility" section of this module's output is likely to be blank.

''This section could include an example or two of established design guidelines that could easily be implemented as modules.''

=== Fitts's Law ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

''This module provides an estimate of the required time to complete various tasks that have been decomposed into formalized sequences of interactions with interface elements, and will provide evaluations and recommendations for optimizing the time required to complete those tasks using the interface.''

'''Inputs'''

1. A formal description of the interface and its elements (e.g. buttons).

2. A formal description of a particular task and the possible paths through a subset of interface elements that permit the user to accomplish that task.

3. The physical distances between interface elements along those paths.

4. The width of those elements along the most likely axes of motion.

5. Device (e.g. mouse) characteristics including start/stop time and the inherent speed limitations of the device.

'''Output'''

The module will then use the Shannon formulation of Fitt's Law to compute the average time needed to complete the task along those paths.

=== Affordances ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

''This simple module will provide interface evaluations and recommendations based on a measure of the extent to which the user perceives the relevant affordances of the interface when performing a number of specified tasks.''

'''Inputs'''

Formalized descriptions of...

1. Interface elements

2. Their associated actions

3. The functions of those actions

4. A particular task

5. User traces for that task.

Inputs (1-4) are then used to generate a "user-independent" space of possible functions that the interface is capable of performing with respect to a given task -- what the interface "affords" the user. From this set of possible interactions, our model will then determine the subset of optimal paths for performing a particular task. The user trace (5) is then used to determine what functions actually were performed in the course of a given task of interest and this information is then compared to the optimal path data to determine the extent to which affordances of the interface are present but not perceived.

'''Output'''

The output of this module is a simple ratio of (affordances perceived) / [(relevant affordances present) * (time to complete task)] which provides a quantitative measure of the extent to which the interface is "natural" to use for a particular task.

=== Workflow, Multi-tasking and Interruptions ===
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

There are, at least, two levels at which users work ([http://portal.acm.org/citation.cfm?id=985692.985707&coll=portal&dl=ACM&CFID=20781736&CFTOKEN=83176621 Gonzales, et al., 2004]). Users accomplish individual low-level tasks which are part of larger ''working spheres''; for example, an office worker might send several emails, create several Post-It (TM) note reminders, and then edit a word document, each of these smaller tasks being part of a single larger working sphere of "adding a new section to the website." Thus, it is important to understand this larger workflow context - which often involves extensive levels of multi-tasking, as well as switching between a variety of computing devices and traditional tools, such as notebooks. In this study it was found that the information workers surveyed typically switch individual tasks every 2 minutes and have many simultaneous working spheres which they switch between, on average every 12 minutes. This frenzied pace of switching tasks and switching working spheres suggests that users will not be using a single application or device for a long period of time, and that affordances to support this characteristic pattern of information work are important.

The purpose of this module is to integrate existing work on multi-tasking, interruption and higher-level workflow into a framework which can predict user recovery times from interruptions. Specifically, the goals of this framework will be to:

* Understand the role of the larger workflow context in user interfaces
* Understand the impact of interruptions on user workflow
* Understand how to design software which fits into the larger working spheres in which information work takes place

It is important to point out that because workflow and multi-tasking rely heavily on higher-level brain functioning, it is unrealistic within the scope of this grant to propose a system which can predict user performance given a description of a set of arbitrary software programs. Therefore, we believe this module will function much more in a qualitative role to provide context to the rest of the model. Specifically, our findings related to interruption and multi-tasking will advance the basic research question of "how do you users react to interruptions when using working sets of varying sizes?". This core HCI contribution will help to inform the rest of the outputs of the model in a qualitative manner.

'''Inputs'''

N/A

'''Outputs'''

N/A

=== Working Memory Load ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''This module measures how much information the user needs to retain in memory while interacting with the interface and makes suggestions for improvements.''

* Models
** Beddeley's Model of Working Memory<ref>http://en.wikipedia.org/wiki/Baddeley%27s_model_of_working_memory</ref>
*** Episodic Buffer
** George Miller's "The magic number 7 plus or minus 2"<ref>http://en.wikipedia.org/wiki/The_Magical_Number_Seven,_Plus_or_Minus_Two</ref>
** The magic number 4<ref>Cowan, N. (2001). The magical number 4 in short-term memory: A reconsideration of mental storage capacity. Behavioral and Brain Sciences, 24, 87-185.</ref>
* Chunking<ref>http://en.wikipedia.org/wiki/Chunking_(psychology)</ref>
* Priming<ref>http://en.wikipedia.org/wiki/Priming_(psychology)</ref>
* Subitizing and Counting<ref>http://en.wikipedia.org/wiki/Subitizing</ref>

'''Inputs'''
* Visual Stimuli
* Audio Stimuli
'''Outputs'''
* Remembered percepts
* Half-Life of percepts

=== Automaticity of Interaction ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''Measures how easily the interaction with the interface becomes automatic with experience and makes suggestions for improvements.''

* Learning<ref>http://en.wikipedia.org/wiki/Learning#Mathematical_models_of_learning</ref>
** Logan's instance theory of automatization<ref>http://74.125.95.132/search?q=cache:IZ-Zccsu3SEJ:psych.wisc.edu/ugstudies/psych733/logan_1988.pdf+logan+isntance+teory&cd=1&hl=en&ct=clnk&gl=us&client=firefox-a</ref>
* Fluency
** As meta-cognitive information <ref>http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6VH9-4SM7PFK-4&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=10cd279fa80958981fcc3c06684c09af</ref>
** As a cognitive ''heuristic''<ref>http://en.wikipedia.org/wiki/Fluency_heuristic</ref>

'''Inputs'''
* Interface
* User goals

'''Outputs'''
* Learning curve

== Integration into the Design Process ==
* Owner: [[User:Ian Spector | Ian Spector]]

''This section outlines the process of designing an HCI interface and at what stages our proposal fits in and how.''

= Preliminary Results =

===Workflow, Multi-tasking, and Interruption===

====I. '''Goals'''====

The goals of the preliminary work are to gain qualitative insight into how information workers practice metawork, and to determine whether people might be better-supported with software which facillitates metawork and interruptions. Thus, the preliminary work should investigate, and demonstrate, the need and impact of the core goals of the project.

====II. '''Methodology'''====

Seven information workers, ages 20-38 (5 male, 2 female), were interviewed to determine which methods they use to "stay organized". An initial list of metawork strategies was established from two pilot interviews, and then a final list was compiled. Participants then responded to a series of 17 questions designed to gain insight into their metawork strategies and process. In addition, verbal interviews were conducted to get additional open-ended feedback.

====III. '''Final Results'''====
A histogram of methods people use to "stay organized" in terms of tracking things they need to do (TODOs), appointments and meetings, etc. is shown in the figure below.

[[Image:AcbGraph.jpg]]

In addition to these methods, participants also used a number of other methods, including:

* iCal
* Notes written in xterms
* "Inbox zero" method of email organization
* iGoogle Notepad (for tasks)
* Tag emails as "TODO", "Important", etc.
* Things (Organizer Software)
* Physical items placed to "remind me of things"
* Sometimes arranging windows on desk
* Keeping browser tabs open
* Bookmarking web pages
* Keep programs/files open scrolled to certain locations sometimes with things selected

In addition, three participants said that when interrupted they "rarely" or "very rarely" were able to resume the task they were working on prior to the interruption. Three of the participants said that they would not actively recommend their metawork strategies for other people, and two said that staying organized was "difficult".

Four participants were neutral to the idea of new tools to help them stay organized and three said that they would like to have such a tool/tools.

====IV. '''Discussion'''====
These results qunatiatively support our hypothesis that there is no clearly dominant set of metawork strategies employed by information workers. This highly fragemented landscape is surprising, even though most information workers work in a similar environment - at a desk, on the phone, in meetings - and with the same types of tools - computers, pens, paper, etc. We believe that this suggests that there are complex tradeoffs between these methods and that no single method is sufficient. We therefore believe that users will be better supported with a new set of software-based metawork tools.

= [Criticisms] =
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

''Any criticisms or questions we have regarding the proposal can go here.''

=References=
<references/>

CS295J/Research proposal (draft 2)

2009-04-24T16:52:28Z

Trevor O'Brien: /* Recording User-Interaction Primitives */

= Introduction =
* Owners: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]], [[User: Trevor O'Brien | Trevor]]

'''Propose:''' The design, application and evaluation of a novel, cognition-based, computational framework for assessing interface design and providing automated suggestions to optimize usability.

'''Evaluation Methodology:''' Our techniques will be evaluated quantitatively through a series of user-study trials, as well as qualitatively by a team of expert interface designers.

'''Contributions and Significance:''' We expect this work to make the following contributions:
# design-space analysis and quantitative evaluation of cognition-based techniques for assessing user interfaces.
# design and quantitative evaluation of techniques for suggesting optimized interface-design changes.
# an extensible, multimodal software architecture for capturing user traces integrated with pupil-tracking data, auditory recognition, and muscle-activity monitoring.
# specification (language?) of how to define an interface evaluation module and how to integrate it into a larger system.
# (there may be more here, like testing different cognitive models, generating a markup language to represent interfaces, maybe even a unique metric space for interface usability)

--

We propose a framework for interface evaluation and recommendation that integrates behavioral models and design guidelines from both cognitive science and HCI. Our framework behaves like a committee of specialized experts, where each expert provides its own assessment of the interface, given its particular knowledge of HCI or cognitive science. For example, an expert may provide an evaluation based on the GOMS method, Fitts's law, Maeda's design principles, or cognitive models of learning and memory. An aggregator collects all of these assessments and weights the opinions of each expert, and outputs to the developer a merged evaluation score and a weighted set of recommendations.

Systematic methods of estimating human performance with computer interfaces are used only sparsely despite their obvious benefits, the reason being the overhead involved in implementing them. In order to test an interface, both manual coding systems like the GOMS variations and user simulations like those based on ACT-R/PM and EPIC require detailed pseudo-code descriptions of the user workflow with the application interface. Any change to the interface then requires extensive changes to the pseudo-code, a major problem because of the trial-and-error nature of interface design. Updating the models themselves is even more complicated. Even an expert in CPM-GOMS, for example, can't necessarily adapt it to take into account results from new cognitive research.

Our proposal makes automatic interface evaluation easier to use in several ways. First of all, we propose to divide the input to the system into three separate parts, functionality, user traces and interface. By separating the functionality from the interface, even radical interface changes will require updating only that part of the input. The user traces are also defined over the functionality so that they too translate across different interfaces. Second, the parallel modular architecture allows for a lower "entry cost" for using the tool. The system includes a broad array of evaluation modules some of which are very simple and some more complex. The simpler modules use only a subset of the input that a system like GOMS or ACT-R would require. This means that while more input will still lead to better output, interface designers can get minimal evaluations with only minimal information. For example, a visual search module may not require any functionality or user traces in order to determine whether all interface elements are distinct enough to be easy to find. Finally, a parallel modular architecture is much easier to augment with relevant cognitive and design evaluations.

= Background / Related Work =

Each person should add the background related to their specific aims.

* Steven Ellis - Cognitive models of HCI, including GOMS variations and ACT-R
* EJ - Design Guidelines
* Jon - Perception and Action
* Andrew - Multiple task environments
* Gideon - Cognition and dual systems
* Ian - Interface design process
* Trevor - User trace collection methods (especially any eye-tracking, EEG, ... you want to suggest using)

== Cognitive Models ==
I plan to port over most of the background on cognitive models of HCI from the old proposal

Additions will comprise of:
*CPM-GOMS as a bridge from GOMS architecture to the promising procedural optimization of the Model Human Processor
**Context of CPM development, discuss its relation to original GOMS and KLM
***Establish the tasks which were relevant for optimization when CPM was developed and note that its obsolescence may have been unavoidable
**Focus on CPM as the first step in transitioning from descriptive data, provided by mounting efforts in the cognitive sciences realm to discover the nature of task processing and accomplishment, to prescriptive algorithms which can predict an interface’s efficiency and suggest improvements
**CPM’s purpose as an abstraction of cognitive processing – a symbolic representation not designed for accuracy but precision
**CPM’s successful trials, e.g. Ernestine
***Implications of this project include CPM’s ability to accurately estimate processing at a psychomotor level
***Project does suggest limitations, however, when one attempts to examine more complex tasks which involve deeper and more numerous cognitive processes
*ACT-R as an example of a progressive cognitive modeling tool
**A tool clearly built by and for cognitive scientists, and as a result presents a much more accurate view of human processing – helpful for our research
**Built-in automation, which now seems to be a standard feature of cognitive modeling tools
**Still an abstraction of cognitive processing, but makes adaptation to cutting-edge cognitive research findings an integral aspect of its modular structure
**Expand on its focus on multi-tasking, taking what was a huge advance between GOMS and its CPM variation and bringing the simulation several steps closer to approximating the nature of cognition in regards to HCI
**Far more accessible both for researchers and the lay user/designer in its portability to LISP, pre-construction of modules representing cognitive capacities and underlying algorithms modeling paths of cognitive processing

==Design guidelines==
A multitude of rule sets exist for the design of not only interface, but architecture, city planning, and software development. They can range in scale from one primary rule to as many Christopher Alexander's 253 rules for urban environments,<ref>[http://hci.rwth-aachen.de/materials/publications/borchers2000a.pdf Borchers, Jan O. "A Pattern Approach to Interaction Design." 2000.]</ref> which he introduced with the concept design patterns in the 1970s. Study has likewise been conducted on the use of these rules:<ref>http://stl.cs.queensu.ca/~graham/cisc836/lectures/readings/tetzlaff-guidelines.pdf</ref> guidelines are often only partially understood, indistinct to the developer, and "fraught" with potential usability problems given a real-world situation.

===Application to AUE===
And yet, the vast majority of guideline sets, including the most popular rulesets, have been arrived at heuristically. The most successful, such as Raskin's and Schneiderman's, have been forged from years of observation instead of empirical study and experimentation. The problem is similar to the problem of circular logic faced by automated usability evaluations: an automated system is limited in the suggestions it can offer to a set of preprogrammed guidelines which have often not been subjected to rigorous experimentation.<ref>[http://www.eecs.berkeley.edu/Pubs/TechRpts/2000/CSD-00-1105.pdf Ivory, M and Hearst, M. "The State of the Art in Automated Usability Evaluation of User Interfaces." ACM Computing Surveys (CSUR), 2001.]</ref> In the vast majority of existing studies, emphasis has traditionally been placed on either the development of guidelines or the application of existing guidelines to automated evaluation. A mutually-reinforcing development of both simultaneously has not been attempted.

Overlap between rulesets is inevitable and unavoidable. For our purposes of evaluating existing rulesets efficiently, without extracting and analyzing each rule individually, it may be desirable to identify the the overarching ''principles'' or ''philosophy'' (max. 2 or 3) for a given ruleset and determining their quantitative relevance to problems of cognition.

===Popular and seminal examples===
Schneiderman's [http://faculty.washington.edu/jtenenbg/courses/360/f04/sessions/schneidermanGoldenRules.html Eight Golden Rules] date to 1987 and are arguably the most-cited. They are heuristic, but can be somewhat classified by cognitive objective: at least two rules apply primarily to ''repeated use'', versus ''discoverability''. Up to five of Schneiderman's rules emphasize ''predictability'' in the outcomes of operations and ''increased feedback and control'' in the agency of the user. His final rule, paradoxically, removes control from the user by suggesting a reduced short-term memory load, which we can arguably classify as ''simplicity''.

Raskin's [http://www.mprove.de/script/02/raskin/designrules.html Design Rules] are classified into five principles by the author, augmented by definitions and supporting rules. While one principle is primarily aesthetic (a design problem arguably out of the bounds of this proposal) and one is a basic endorsement of testing, the remaining three begin to reflect philosophies similar to Schneiderman's: reliability or ''predictability'', ''simplicity'' or ''efficiency'' (which we can construe as two sides of the same coin), and finally he introduces a concept of ''uninterruptibility''.

Maeda's [http://lawsofsimplicity.com/?cat=5&order=ASC Laws of Simplicity] are fewer, and ostensibly emphasize ''simplicity'' exclusively, although elements of ''use'' as related by Schneiderman's rules and ''efficiency'' as defined by Raskin may be facets of this simplicity. Google's corporate mission statement presents [http://www.google.com/corporate/ux.html Ten Principles], only half of which can be considered true interface guidelines. ''Efficiency'' and ''simplicity'' are cited explicitly, aesthetics are once again noted as crucial, and working within a user's trust is another application of ''predictability''.

===Elements and goals of a guideline set===
Myriad rulesets exist, but variation becomes scarce—it indeed seems possible to parse these common rulesets into overarching principles that can be converted to or associated with quantifiable cognitive properties. For example, it is likely ''simplicity'' has an analogue in the short-term memory retention or visual retention of the user, vis a vis the rule of [http://books.google.com/books?hl=en&lr=&id=j5q0VvOGExYC&oi=fnd&pg=PA357&dq=seven+plus+or+minus+two&ots=prI3PKJBar&sig=vOZnqpnkXKGYWxK6_XlA4I_CRyI Seven, Plus or Minus Two]. ''Predictability'' likewise may have an analogue in Activity Theory, in regards to a user's perceptual expectations for a given action; ''uninterruptibility'' has implications in cognitive task-switching;<ref>[http://portal.acm.org/citation.cfm?id=985692.985715&coll=Portal&dl=ACM&CFID=21136843&CFTOKEN=23841774 Czerwinski, Horvitz, and White. "A Diary Study of Task Switching and Interruptions." Proceedings of the SIGCHI conference on Human factors in computing systems, 2004.]</ref> and so forth.

Within the scope of this proposal, we aim to reduce and refine these philosophies found in seminal rulesets and identify their logical cognitive analogues. By assigning a quantifiable taxonomy to these principles, we will be able to rank and weight them with regard to their real-world applicability, developing a set of "meta-guidelines" and rules for applying them to a given interface in an automated manner. Combined with cognitive models and multi-modal HCI analysis, we seek to develop, in parallel with these guidelines, the interface evaluation system responsible for their application.

== Perception and Action (in progress) ==

*Information Processing Approach
**Advantages
***Formalism eases translation of theory into scripting language
**Disadvantages
***Assumes symbolic representation

*Ecological (Gibsonian) Approach
**Advantages
***Emphasis on bodily and environmental constraints
**Disadvantages
***Lack of formalism hinders translation of theory into scripting language

= Specific Aims and Contributions (to be separated later) =

'''Specific Aims'''
* Incorporate interaction history mechanisms into a set of existing applications.
* Perform user-study evaluation of history-collection techniques.
* Distill a set of cognitive principles/models, and evaluate empirically?
* Build/buy sensing system to include pupil-tracking, muscle-activity monitoring, auditory recognition.
* Design techniques for manual/semi-automated/automated construction of <insert favorite cognitive model here> from interaction histories and sensing data.
* Design system for posterior analysis of interaction history w.r.t. <insert favorite cognitive model here>, evaluating critical path <or equivalent> trajectories.
* Design cognition-based techniques for detecting bottlenecks in critical paths, and offering optimized alternatives.
* Perform quantitative user-study evaluations, collect qualitative feedback from expert interface designers.

'''Contributions'''

* Design-space analysis and quantitative evaluation of cognition-based techniques for assessing user interfaces.
* Design and quantitative evaluation of techniques for suggesting optimized interface-design changes.
* An extensible, multimodal software architecture for capturing user traces integrated with pupil-tracking data, auditory recognition, and muscle-activity monitoring.
* (there may be more here, like testing different cognitive models, generating a markup language to represent interfaces, maybe even a unique metric space for interface usability)

--

See the [http://vrl.cs.brown.edu/wiki/images/b/b1/Flowchart2.pdf flowchart] for a visual overview of our aims.

In order to use this framework, a designer will have to provide:
* Functional specification - what are the possible interactions between the user and the application. This can be thought of as method signatures, with a name (e.g., setVolume), direction (to user or from user) and a list of value types (boolean, number, text, video, ...) for each interaction.
* GUI specification - a mapping of interactions to interface elements (e.g., setVolume is mapped to the grey knob in the bottom left corner with clockwise turning increasing the input number).
* Functional user traces - sequences of representative ways in which the application is used. Instead of writing them, the designer could have users use the application with a trial interface and then use our methods to generalize the user traces beyond the specific interface (The second method is depicted in the diagram). As a form of pre-processing, the system also generates an interaction transition matrix which lists the probability of each type of interaction given the previous interaction.
* Utility function - this is a weighting of various performance metrics (time, cognitive load, fatigue, etc.), where the weighting expresses the importance of a particular dimension to the user. For example, a user at NASA probably cares more about interface accuracy than speed. By passing this information to our committee of experts, we can create interfaces that are tuned to maximize the utility of a particular user type.

Each of the modules can use all of this information or a subset of it. Our approach stresses flexibility and the ability to give more meaningful feedback the more information is provided. After processing the information sent by the system of experts, the aggregator will output:
* An evaluation of the interface. Evaluations are expressed both in terms of the utility function components (i.e. time, fatigue, cognitive load, etc.), and in terms of the overall utility for this interface (as defined by the utility function). These evaluations are given in the form of an efficiency curve, where the utility received on each dimension can change as the user becomes more accustomed to the interface.
* Suggested improvements for the GUI are also output. These suggestions are meant to optimize the utility function that was input to the system. If a user values accuracy over time, interface suggestions will be made accordingly.

== Recording User-Interaction Primitives ==
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

'''Aims'''
* Develop system for logging low-level user-interactions within existing applications. By low-level interactions, we refer to those interactions for which a quantitative, predictive, GOMS-like model may be generated.
* At the same scale of granularity, integrate explicit interface-interactions with multimodal-sensing data. i.e. pupil-tracking, muscle-activity monitoring, auditory recognition, EEG data, and facial and posture-focused video.

'''Contributions'''
* Extensible, multimodal, HCI framework for recording rich interaction-history data in existing applications.

'''Significance'''
* Provide rich history data to serve as the basis for novel quantitative interface evaluation.
* Aid expert HCI and visual designers in traditional design processes.
* Provide data for automated machine-learning strategies applied to interaction.

'''Background and Related Work'''
* The utility of interaction histories with respect to assessing interface design has been demonstrated in <ref> [http://www.cs.brown.edu/people/trevor/Papers/Heer-2008-GraphicalHistories.pdf Graphical Histories for Visualization: Supporting Analysis, Communication, and Evaluation] (InfoVis 2008) </ref>.
* In addition, data management histories have been shown effective in the visualization community in <ref> [http://www.cs.brown.edu/people/trevor/Papers/Callahan-2006-MED.pdf Callahan-2006-MED] </ref> <ref> [http://www.cs.brown.edu/people/trevor/Papers/Callahan-2006-VVM.pdf Callahan-2006-VVM] </ref> <ref> [http://www.cs.brown.edu/people/trevor/Papers/Bavoil-2005-VEI.pdf Bavoil-2005-VEI]</ref>, providing visualizations by analogy <ref> [http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=4376187&isnumber=4376125 Querying and Creating Visualizations by Analogy] </ref> and offering automated suggestions <ref> [http://www.cs.utah.edu/~juliana/pub/tvcg-recommendation2008.pdf VisComplete: Automating Suggestions from Visualization Pipelines] </ref>, which we expect to generalize to user interaction history.

== Semantic-level Interaction Chunking ==
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

'''Aims'''
* Develop techniques for ''chunking'' low-level interaction primitives into ''semantic-level'' interactions, given an application's functionality and data-context. (And de-chunking? Invertible mapping needed?)
* Perform user-study evaluation to validate chunking methods.

'''Contributions'''
* Design and user-study evaluation of semantic chunking techniques.

'''Significance'''
* Low-level <--> semantic-level mapping allows for cognitive-modeling to be applied at a functionality level, where low-level interaction techniques can be swapped out. This will allow our interface assessment system to make feasible suggestions for more optimal interface design.

== Parallel Framework for Evaluation Modules ==
* Owner: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]]

''This section will describe in more detail the inputs, outputs and architecture that were presented in the introduction.''

==== Module Inputs (Incomplete) ====
* A set of utility dimensions {d1, d2, ...} are defined in the framework. These could be {d1="time", d2="fatigue", ...}
* A set of interaction functions. These specify all of the information that the application wants to give users or get from them. It is not tied to a specific interface. For example, the fact that an applications shows videos would be included here. Whether it displays them embedded, in a pop-up window or fullscreen would not.
* A mapping of interaction functions to interface elements (e.g., buttons, windows, dials,...). Lots of optional information describing visual properties, associated text, physical interactions (e.g., turning the dial clockwise increases the input value) and timing.
* Functional user traces - sequences of interaction functions that represent typical user interactions with the application. Could include a goal hierarchy, in which case the function sequence is at the bottom of the hierarchy.

==== Module Outputs ====
* Every module ouputs at least one of the following:
** An evaluation of the interface
*** This can be on any or all of the utility dimensions, e.g. evaluation={d1=score1, d2=score2, ...}
*** This can alternately be an overall evaluation, ignoring dimensions, e.g. evaluation={score}
**** In this case, the aggregator will treat this as the module giving the same score to all dimensions. Which dimension this evaluator actually predicts well on can be learned by the aggregator over time.
** Recommendation(s) for improving the interface
***This can be a textual description of what changes the designer should make
***This can alternately be a transformation that can automatically be applied to the interface language (without designer intervention)
***In addition to the textual or transformational description of the recommendation, a "change in evaluation" is output to describe how specifically the value will improve the interface
****Recommendation = {description="make this change", Δevaluation={d1=score1, d2=score2, ...}
****Like before, this Δevaluation can cover any number of dimensions, or it can be generic.
***Either a single recommendation or a set of recommendations can be output

==== Aggregator Inputs ====
The aggregator receives as input the outputs of all the modules.

==== Aggregator Outputs ====

Outputs for the aggregator are the same as the outputs for each module. The difference is that the aggregator will consider all the module outputs, and arrive at a merged output based on the past performance of each of the modules.

== Evaluation and Recommendation via Modules ==
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

A "meta-module" called the aggregator will be responsible for assembling and formatting the output of all other modules into a structure that is both extensible and immediately usable, by both an automated designer or a human designer.

===Requirements===
The aggregator's functionality, then, is defined by its '''inputs''', the outputs of the other modules, and the desired '''output''' of the system as a whole, per its position in the architecture. Its purpose is largely formatting and reconciliation of the products of the multitudinous (and extensible) modules. The output of the aggregator must meet several requirements: first, to generate a set of human-readable suggestions for the improvement of the given interface; second, to generate a machine-readable, but also analyzable, evaluation of the various characteristics of the interface and accompanying user traces.

From these specifications, it is logical to assume that a common language or format will be required for the output of individual modules. We propose an XML-based file format, allowing: (1) a section for the standardized identification of problem areas, applicable rules, and proposed improvements, generalized by the individual module and mapped to a single element, or group of elements, in the original interface specification; (2) a section for specification of generalizable "utility" functions, allowing a module to specify how much a measurable quantity of utility is positively or negatively affected by properties of the input interface; (3) new, user-definable sections for evaluations of the given interface not covered by the first two sections. The first two sections are capable of conveying the vast majority of module outputs predicted at this time, but the XML can extensibly allow modules to pass on whatever information may become prominent in the future.

===Specification===
<module id="Fitts-Law">
<interface-elements>
<element>
<desc>submit button</desc>
<problem>
<desc>size</desc>
<suggestion>width *= 2</suggestion>
<suggetsion>height *= 2</suggestions>
<human-suggestion>Increase size relative to other elements</human-suggestion>
</problem>
</element>
</interface-elements>
<utility>
<dimension>
<desc>time</desc>
<value>0:15:35</value>
</dimension>
<dimension>
<desc>frustration</desc>
<value>pulling hair out</value>
</dimension>
<dimension>
<desc>efficiency</desc>
<value>13.2s/KPM task</value>
<value>0.56m/CPM task</value>
</dimension>
</utility>
<tasks>
<task>
<desc>complete form</desc>
</task>
<task>
<desc>lookup SSN</desc>
</task>
<task>
<desc>format phone number</desc>
</task>
</tasks>
</module>

===Logic===
This file provided by each module is then the input for the aggregator. The aggregator's most straightforward function is the compilation of the "problem areas," assembling them and noting problem areas and suggestions that are recommended by more than one module, and weighting them accordingly in its final report. These weightings can begin in an equal state, but the aggregator should be capable of learning iteratively which modules' results are most relevant to the user and update weightings accordingly. This may need to be accomplished with manual tuning, or a machine-learning algorithm capable of determining which modules most often agree with others.

Secondly, the aggregator compiles the utility functions provided by the module specs. This, again, is a summation of similarly-described values from the various modules.

When confronted with user-defined sections of the XML spec, the aggregator is primarily responsible for compiling them and sending them along to the output of the machine. Even if the aggregator does not recognize a section or property of the evaluative spec, if it sees the property reported by more than one module it should be capable of aggregating these intelligently. In future versions of the spec, it should be possible for a module to provide instructions for the aggregator on how to handle unrecognized sections of the XML.

From these compilations, then, the aggregator should be capable of outputting both aggregated human-readable suggestions on interface improvements for a human designer, as well as a comprehensive evaluation of the interface's effectiveness at the given task traces. Again, this is dependent on the specification of the system as a whole, but is likely to include measures and comparisons, graphings of task versus utility, and quantitative measures of an element's effectiveness.

''This section is necessarily defined by the output of the individual modules (which I already expect to be of varied and arbitrary structure) and the desired output of the machine as a whole. It will likely need to be revised heavily after other modules and the "Parellel Framework" section are defined.'' [[User:E J Kalafarski|E J Kalafarski]] 12:34, 24 April 2009 (UTC)

''This section describes the aggregator, which takes the output of multiple independent modules and aggregates the results to provide (1) an evaluation and (2) recommendations for the user interface. We should explain how the aggregator weights the output of different modules (this could be based on historical performance of each module, or perhaps based on E.J.'s cognitive/HCI guidelines).''

== Sample Modules ==

=== CPM-GOMS ===
* Owners: [[User:Steven Ellis | Steven Ellis]]

''This module will provide interface evaluations and suggestions based on a CPM-GOMS model of cognition for the given interface. It will provide a quantitative, predictive, cognition-based parameterization of usability. From empirically collected data, user trajectories through the model (critical paths) will be examined, highlighting bottlenecks within the interface, and offering suggested alterations to the interface to induce more optimal user trajectories.''

I’m hoping to have some input on this section, because it seems to be the crux of the “black box” into which we take the inputs of interface description, user traces, etc. and get our outputs (time, recommendations, etc.). I know at least a few people have pretty strong thoughts on the matter and we ought to discuss the final structure.

That said – my proposal for the module:
*In my opinion the concept of the Model Human Processor (at least as applied in CPM) is outdated – it’s too economic/overly parsimonious in its conception of human activity. I think we need to create a structure which accounts for more realistic conditions of HCI including multitasking, aspects of distributed cognition (and other relevant uses of tools – as far as I can tell CPM doesn’t take into account any sort of productivity aids), executive control processes of attention, etc. ACT-R appears to take steps towards this but we would probably need to look at their algorithms to know for sure.
*Critical paths will continue to play an important role – we should in fact emphasize that part of this tool’s purpose will be a description not only of ways in which the interface should be modified to best fit a critical path, but also ways in which the user’s ought to be instructed in their use of the path. This feedback mechanism could be bidirectional – if the model’s predictions of the user’s goals are incorrect, the critical path determined will also be incorrect and the interface inherently suboptimal. The user could be prompted with a tooltip explaining in brief why and how the interface has changed, along with options to revert, select other configurations (euphemized by goals), and to view a short video detailing how to properly use the interface.
*Call me crazy but, if we assume designers will be willing to code a model of their interfaces into our ACT-R-esque language, could we allow that model to be fairly transparent to the user, who could use a gui to input their goals to find an analogue in the program which would subsequently rearrange its interface to fit the user’s needs? Even if not useful to the users, such dynamic modeling could really help designers (IMO)
*I think the model should do its best to accept models written for ACT-R and whatever other cognitive models there are out there – gives us the best chance of early adoption
*I would particularly appreciate input on the number/complexity/type of inputs we’ll be using, as well as the same qualities for the output.

=== HCI Guidelines ===
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

Schneiderman's Eight Golden Rules and Jakob Nielsen's Ten Heuristics are perhaps the most famous and well-regarded heuristic design guidelines to emerge over the last twenty years. Although the explicit theoretical basis for such heuristics is controversial and not well-explored, the empirical success of these guidelines is established and accepted. This module will parse out up to three or four common (that is, intersecting) principles from these accepted guidelines and apply them to the input interface.

As an example, we identify an analogous principle that appears in Schneiderman ("Reduce short-term memory load")<ref>http://faculty.washington.edu/jtenenbg/courses/360/f04/sessions/schneidermanGoldenRules.html</ref> and Nielsen ("Recognition rather than recall/Minimize the user's memory load")<ref>http://www.useit.com/papers/heuristic/heuristic_list.html</ref>. The input interface is then evaluated for the consideration of the principle, based on an explicit formal description of the interface, such as XAML or XUL. The module attempts to determine how effectively the interface demonstrates the principle. When analyzing an interface for several principles that may be conflicting or opposing in a given context, the module makes use of a hard-coded but iterative (and evolving) weighting of these principles, based on (1) how often they appear in the training set of accepted sets of guidelines, (2) how analogues a heuristic principle is to a cognitive principle in a parallel training set, and (3) how effective the principle's associated suggestion is found to be using a feedback mechanism.

'''Inputs''' 
# A formal description of the interface and its elements (e.g. buttons).
# A formal description of a particular task and the possible paths through a subset of interface elements that permit the user to accomplish that task.

'''Output''' 
Standard XML-formatted file containing problem areas of the input interface, suggestions for each problem area based on principles that were found to have a strong application to a problem element and the problem itself, and a human-readable generated analysis of the element's affinity for the principle. Quantitative outputs will not be possible based on heuristic guidelines, and the "utility" section of this module's output is likely to be blank.

''This section could include an example or two of established design guidelines that could easily be implemented as modules.''

=== Fitts's Law ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

''This module provides an estimate of the required time to complete various tasks that have been decomposed into formalized sequences of interactions with interface elements, and will provide evaluations and recommendations for optimizing the time required to complete those tasks using the interface.''

'''Inputs'''

1. A formal description of the interface and its elements (e.g. buttons).

2. A formal description of a particular task and the possible paths through a subset of interface elements that permit the user to accomplish that task.

3. The physical distances between interface elements along those paths.

4. The width of those elements along the most likely axes of motion.

5. Device (e.g. mouse) characteristics including start/stop time and the inherent speed limitations of the device.

'''Output'''

The module will then use the Shannon formulation of Fitt's Law to compute the average time needed to complete the task along those paths.

=== Affordances ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

''This simple module will provide interface evaluations and recommendations based on a measure of the extent to which the user perceives the relevant affordances of the interface when performing a number of specified tasks.''

'''Inputs'''

Formalized descriptions of...

1. Interface elements

2. Their associated actions

3. The functions of those actions

4. A particular task

5. User traces for that task.

Inputs (1-4) are then used to generate a "user-independent" space of possible functions that the interface is capable of performing with respect to a given task -- what the interface "affords" the user. From this set of possible interactions, our model will then determine the subset of optimal paths for performing a particular task. The user trace (5) is then used to determine what functions actually were performed in the course of a given task of interest and this information is then compared to the optimal path data to determine the extent to which affordances of the interface are present but not perceived.

'''Output'''

The output of this module is a simple ratio of (affordances perceived) / [(relevant affordances present) * (time to complete task)] which provides a quantitative measure of the extent to which the interface is "natural" to use for a particular task.

=== Workflow, Multi-tasking and Interruptions ===
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

There are, at least, two levels at which users work ([http://portal.acm.org/citation.cfm?id=985692.985707&coll=portal&dl=ACM&CFID=20781736&CFTOKEN=83176621 Gonzales, et al., 2004]). Users accomplish individual low-level tasks which are part of larger ''working spheres''; for example, an office worker might send several emails, create several Post-It (TM) note reminders, and then edit a word document, each of these smaller tasks being part of a single larger working sphere of "adding a new section to the website." Thus, it is important to understand this larger workflow context - which often involves extensive levels of multi-tasking, as well as switching between a variety of computing devices and traditional tools, such as notebooks. In this study it was found that the information workers surveyed typically switch individual tasks every 2 minutes and have many simultaneous working spheres which they switch between, on average every 12 minutes. This frenzied pace of switching tasks and switching working spheres suggests that users will not be using a single application or device for a long period of time, and that affordances to support this characteristic pattern of information work are important.

The purpose of this module is to integrate existing work on multi-tasking, interruption and higher-level workflow into a framework which can predict user recovery times from interruptions. Specifically, the goals of this framework will be to:

* Understand the role of the larger workflow context in user interfaces
* Understand the impact of interruptions on user workflow
* Understand how to design software which fits into the larger working spheres in which information work takes place

It is important to point out that because workflow and multi-tasking rely heavily on higher-level brain functioning, it is unrealistic within the scope of this grant to propose a system which can predict user performance given a description of a set of arbitrary software programs. Therefore, we believe this module will function much more in a qualitative role to provide context to the rest of the model. Specifically, our findings related to interruption and multi-tasking will advance the basic research question of "how do you users react to interruptions when using working sets of varying sizes?". This core HCI contribution will help to inform the rest of the outputs of the model in a qualitative manner.

'''Inputs'''

N/A

'''Outputs'''

N/A

=== Working Memory Load ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''This module measures how much information the user needs to retain in memory while interacting with the interface and makes suggestions for improvements.''

* Models
** Beddeley's Model of Working Memory<ref>http://en.wikipedia.org/wiki/Baddeley%27s_model_of_working_memory</ref>
*** Episodic Buffer
** George Miller's "The magic number 7 plus or minus 2"<ref>http://en.wikipedia.org/wiki/The_Magical_Number_Seven,_Plus_or_Minus_Two</ref>
** The magic number 4<ref>Cowan, N. (2001). The magical number 4 in short-term memory: A reconsideration of mental storage capacity. Behavioral and Brain Sciences, 24, 87-185.</ref>
* Chunking<ref>http://en.wikipedia.org/wiki/Chunking_(psychology)</ref>
* Priming<ref>http://en.wikipedia.org/wiki/Priming_(psychology)</ref>
* Subitizing and Counting<ref>http://en.wikipedia.org/wiki/Subitizing</ref>

'''Inputs'''
* Visual Stimuli
* Audio Stimuli
'''Outputs'''
* Remembered percepts
* Half-Life of percepts

=== Automaticity of Interaction ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''Measures how easily the interaction with the interface becomes automatic with experience and makes suggestions for improvements.''

* Learning<ref>http://en.wikipedia.org/wiki/Learning#Mathematical_models_of_learning</ref>
** Logan's instance theory of automatization<ref>http://74.125.95.132/search?q=cache:IZ-Zccsu3SEJ:psych.wisc.edu/ugstudies/psych733/logan_1988.pdf+logan+isntance+teory&cd=1&hl=en&ct=clnk&gl=us&client=firefox-a</ref>
* Fluency
** As meta-cognitive information <ref>http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6VH9-4SM7PFK-4&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=10cd279fa80958981fcc3c06684c09af</ref>
** As a cognitive ''heuristic''<ref>http://en.wikipedia.org/wiki/Fluency_heuristic</ref>

'''Inputs'''
* Interface
* User goals

'''Outputs'''
* Learning curve

== Integration into the Design Process ==
* Owner: [[User:Ian Spector | Ian Spector]]

''This section outlines the process of designing an HCI interface and at what stages our proposal fits in and how.''

= Preliminary Results =

===Workflow, Multi-tasking, and Interruption===

====I. '''Goals'''====

The goals of the preliminary work are to gain qualitative insight into how information workers practice metawork, and to determine whether people might be better-supported with software which facillitates metawork and interruptions. Thus, the preliminary work should investigate, and demonstrate, the need and impact of the core goals of the project.

====II. '''Methodology'''====

Seven information workers, ages 20-38 (5 male, 2 female), were interviewed to determine which methods they use to "stay organized". An initial list of metawork strategies was established from two pilot interviews, and then a final list was compiled. Participants then responded to a series of 17 questions designed to gain insight into their metawork strategies and process. In addition, verbal interviews were conducted to get additional open-ended feedback.

====III. '''Final Results'''====
A histogram of methods people use to "stay organized" in terms of tracking things they need to do (TODOs), appointments and meetings, etc. is shown in the figure below.

[[Image:AcbGraph.jpg]]

In addition to these methods, participants also used a number of other methods, including:

* iCal
* Notes written in xterms
* "Inbox zero" method of email organization
* iGoogle Notepad (for tasks)
* Tag emails as "TODO", "Important", etc.
* Things (Organizer Software)
* Physical items placed to "remind me of things"
* Sometimes arranging windows on desk
* Keeping browser tabs open
* Bookmarking web pages
* Keep programs/files open scrolled to certain locations sometimes with things selected

In addition, three participants said that when interrupted they "rarely" or "very rarely" were able to resume the task they were working on prior to the interruption. Three of the participants said that they would not actively recommend their metawork strategies for other people, and two said that staying organized was "difficult".

Four participants were neutral to the idea of new tools to help them stay organized and three said that they would like to have such a tool/tools.

====IV. '''Discussion'''====
These results qunatiatively support our hypothesis that there is no clearly dominant set of metawork strategies employed by information workers. This highly fragemented landscape is surprising, even though most information workers work in a similar environment - at a desk, on the phone, in meetings - and with the same types of tools - computers, pens, paper, etc. We believe that this suggests that there are complex tradeoffs between these methods and that no single method is sufficient. We therefore believe that users will be better supported with a new set of software-based metawork tools.

= [Criticisms] =
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

''Any criticisms or questions we have regarding the proposal can go here.''

=References=
<references/>

CS295J/Research proposal (draft 2)

2009-04-24T16:39:48Z

Trevor O'Brien: /* Semantic-level Interaction Chunking */

= Introduction =
* Owners: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]], [[User: Trevor O'Brien | Trevor]]

'''Propose:''' The design, application and evaluation of a novel, cognition-based, computational framework for assessing interface design and providing automated suggestions to optimize usability.

'''Evaluation Methodology:''' Our techniques will be evaluated quantitatively through a series of user-study trials, as well as qualitatively by a team of expert interface designers.

'''Contributions and Significance:''' We expect this work to make the following contributions:
# design-space analysis and quantitative evaluation of cognition-based techniques for assessing user interfaces.
# design and quantitative evaluation of techniques for suggesting optimized interface-design changes.
# an extensible, multimodal software architecture for capturing user traces integrated with pupil-tracking data, auditory recognition, and muscle-activity monitoring.
# specification (language?) of how to define an interface evaluation module and how to integrate it into a larger system.
# (there may be more here, like testing different cognitive models, generating a markup language to represent interfaces, maybe even a unique metric space for interface usability)

--

We propose a framework for interface evaluation and recommendation that integrates behavioral models and design guidelines from both cognitive science and HCI. Our framework behaves like a committee of specialized experts, where each expert provides its own assessment of the interface, given its particular knowledge of HCI or cognitive science. For example, an expert may provide an evaluation based on the GOMS method, Fitts's law, Maeda's design principles, or cognitive models of learning and memory. An aggregator collects all of these assessments and weights the opinions of each expert, and outputs to the developer a merged evaluation score and a weighted set of recommendations.

Systematic methods of estimating human performance with computer interfaces are used only sparsely despite their obvious benefits, the reason being the overhead involved in implementing them. In order to test an interface, both manual coding systems like the GOMS variations and user simulations like those based on ACT-R/PM and EPIC require detailed pseudo-code descriptions of the user workflow with the application interface. Any change to the interface then requires extensive changes to the pseudo-code, a major problem because of the trial-and-error nature of interface design. Updating the models themselves is even more complicated. Even an expert in CPM-GOMS, for example, can't necessarily adapt it to take into account results from new cognitive research.

Our proposal makes automatic interface evaluation easier to use in several ways. First of all, we propose to divide the input to the system into three separate parts, functionality, user traces and interface. By separating the functionality from the interface, even radical interface changes will require updating only that part of the input. The user traces are also defined over the functionality so that they too translate across different interfaces. Second, the parallel modular architecture allows for a lower "entry cost" for using the tool. The system includes a broad array of evaluation modules some of which are very simple and some more complex. The simpler modules use only a subset of the input that a system like GOMS or ACT-R would require. This means that while more input will still lead to better output, interface designers can get minimal evaluations with only minimal information. For example, a visual search module may not require any functionality or user traces in order to determine whether all interface elements are distinct enough to be easy to find. Finally, a parallel modular architecture is much easier to augment with relevant cognitive and design evaluations.

= Background / Related Work =

Each person should add the background related to their specific aims.

* Steven Ellis - Cognitive models of HCI, including GOMS variations and ACT-R
* EJ - Design Guidelines
* Jon - Perception and Action
* Andrew - Multiple task environments
* Gideon - Cognition and dual systems
* Ian - Interface design process
* Trevor - User trace collection methods (especially any eye-tracking, EEG, ... you want to suggest using)

== Cognitive Models ==
I plan to port over most of the background on cognitive models of HCI from the old proposal

Additions will comprise of:
*CPM-GOMS as a bridge from GOMS architecture to the promising procedural optimization of the Model Human Processor
**Context of CPM development, discuss its relation to original GOMS and KLM
***Establish the tasks which were relevant for optimization when CPM was developed and note that its obsolescence may have been unavoidable
**Focus on CPM as the first step in transitioning from descriptive data, provided by mounting efforts in the cognitive sciences realm to discover the nature of task processing and accomplishment, to prescriptive algorithms which can predict an interface’s efficiency and suggest improvements
**CPM’s purpose as an abstraction of cognitive processing – a symbolic representation not designed for accuracy but precision
**CPM’s successful trials, e.g. Ernestine
***Implications of this project include CPM’s ability to accurately estimate processing at a psychomotor level
***Project does suggest limitations, however, when one attempts to examine more complex tasks which involve deeper and more numerous cognitive processes
*ACT-R as an example of a progressive cognitive modeling tool
**A tool clearly built by and for cognitive scientists, and as a result presents a much more accurate view of human processing – helpful for our research
**Built-in automation, which now seems to be a standard feature of cognitive modeling tools
**Still an abstraction of cognitive processing, but makes adaptation to cutting-edge cognitive research findings an integral aspect of its modular structure
**Expand on its focus on multi-tasking, taking what was a huge advance between GOMS and its CPM variation and bringing the simulation several steps closer to approximating the nature of cognition in regards to HCI
**Far more accessible both for researchers and the lay user/designer in its portability to LISP, pre-construction of modules representing cognitive capacities and underlying algorithms modeling paths of cognitive processing

==Design guidelines==
A multitude of rule sets exist for the design of not only interface, but architecture, city planning, and software development. They can range in scale from one primary rule to as many Christopher Alexander's 253 rules for urban environments,<ref>[http://hci.rwth-aachen.de/materials/publications/borchers2000a.pdf Borchers, Jan O. "A Pattern Approach to Interaction Design." 2000.]</ref> which he introduced with the concept design patterns in the 1970s. Study has likewise been conducted on the use of these rules:<ref>http://stl.cs.queensu.ca/~graham/cisc836/lectures/readings/tetzlaff-guidelines.pdf</ref> guidelines are often only partially understood, indistinct to the developer, and "fraught" with potential usability problems given a real-world situation.

===Application to AUE===
And yet, the vast majority of guideline sets, including the most popular rulesets, have been arrived at heuristically. The most successful, such as Raskin's and Schneiderman's, have been forged from years of observation instead of empirical study and experimentation. The problem is similar to the problem of circular logic faced by automated usability evaluations: an automated system is limited in the suggestions it can offer to a set of preprogrammed guidelines which have often not been subjected to rigorous experimentation.<ref>[http://www.eecs.berkeley.edu/Pubs/TechRpts/2000/CSD-00-1105.pdf Ivory, M and Hearst, M. "The State of the Art in Automated Usability Evaluation of User Interfaces." ACM Computing Surveys (CSUR), 2001.]</ref> In the vast majority of existing studies, emphasis has traditionally been placed on either the development of guidelines or the application of existing guidelines to automated evaluation. A mutually-reinforcing development of both simultaneously has not been attempted.

Overlap between rulesets is inevitable and unavoidable. For our purposes of evaluating existing rulesets efficiently, without extracting and analyzing each rule individually, it may be desirable to identify the the overarching ''principles'' or ''philosophy'' (max. 2 or 3) for a given ruleset and determining their quantitative relevance to problems of cognition.

===Popular and seminal examples===
Schneiderman's [http://faculty.washington.edu/jtenenbg/courses/360/f04/sessions/schneidermanGoldenRules.html Eight Golden Rules] date to 1987 and are arguably the most-cited. They are heuristic, but can be somewhat classified by cognitive objective: at least two rules apply primarily to ''repeated use'', versus ''discoverability''. Up to five of Schneiderman's rules emphasize ''predictability'' in the outcomes of operations and ''increased feedback and control'' in the agency of the user. His final rule, paradoxically, removes control from the user by suggesting a reduced short-term memory load, which we can arguably classify as ''simplicity''.

Raskin's [http://www.mprove.de/script/02/raskin/designrules.html Design Rules] are classified into five principles by the author, augmented by definitions and supporting rules. While one principle is primarily aesthetic (a design problem arguably out of the bounds of this proposal) and one is a basic endorsement of testing, the remaining three begin to reflect philosophies similar to Schneiderman's: reliability or ''predictability'', ''simplicity'' or ''efficiency'' (which we can construe as two sides of the same coin), and finally he introduces a concept of ''uninterruptibility''.

Maeda's [http://lawsofsimplicity.com/?cat=5&order=ASC Laws of Simplicity] are fewer, and ostensibly emphasize ''simplicity'' exclusively, although elements of ''use'' as related by Schneiderman's rules and ''efficiency'' as defined by Raskin may be facets of this simplicity. Google's corporate mission statement presents [http://www.google.com/corporate/ux.html Ten Principles], only half of which can be considered true interface guidelines. ''Efficiency'' and ''simplicity'' are cited explicitly, aesthetics are once again noted as crucial, and working within a user's trust is another application of ''predictability''.

===Elements and goals of a guideline set===
Myriad rulesets exist, but variation becomes scarce—it indeed seems possible to parse these common rulesets into overarching principles that can be converted to or associated with quantifiable cognitive properties. For example, it is likely ''simplicity'' has an analogue in the short-term memory retention or visual retention of the user, vis a vis the rule of [http://books.google.com/books?hl=en&lr=&id=j5q0VvOGExYC&oi=fnd&pg=PA357&dq=seven+plus+or+minus+two&ots=prI3PKJBar&sig=vOZnqpnkXKGYWxK6_XlA4I_CRyI Seven, Plus or Minus Two]. ''Predictability'' likewise may have an analogue in Activity Theory, in regards to a user's perceptual expectations for a given action; ''uninterruptibility'' has implications in cognitive task-switching;<ref>[http://portal.acm.org/citation.cfm?id=985692.985715&coll=Portal&dl=ACM&CFID=21136843&CFTOKEN=23841774 Czerwinski, Horvitz, and White. "A Diary Study of Task Switching and Interruptions." Proceedings of the SIGCHI conference on Human factors in computing systems, 2004.]</ref> and so forth.

Within the scope of this proposal, we aim to reduce and refine these philosophies found in seminal rulesets and identify their logical cognitive analogues. By assigning a quantifiable taxonomy to these principles, we will be able to rank and weight them with regard to their real-world applicability, developing a set of "meta-guidelines" and rules for applying them to a given interface in an automated manner. Combined with cognitive models and multi-modal HCI analysis, we seek to develop, in parallel with these guidelines, the interface evaluation system responsible for their application.

== Perception and Action (in progress) ==

*Information Processing Approach
**Advantages
***Formalism eases translation of theory into scripting language
**Disadvantages
***Assumes symbolic representation

*Ecological (Gibsonian) Approach
**Advantages
***Emphasis on bodily and environmental constraints
**Disadvantages
***Lack of formalism hinders translation of theory into scripting language

= Specific Aims and Contributions (to be separated later) =

'''Specific Aims'''
* Incorporate interaction history mechanisms into a set of existing applications.
* Perform user-study evaluation of history-collection techniques.
* Distill a set of cognitive principles/models, and evaluate empirically?
* Build/buy sensing system to include pupil-tracking, muscle-activity monitoring, auditory recognition.
* Design techniques for manual/semi-automated/automated construction of <insert favorite cognitive model here> from interaction histories and sensing data.
* Design system for posterior analysis of interaction history w.r.t. <insert favorite cognitive model here>, evaluating critical path <or equivalent> trajectories.
* Design cognition-based techniques for detecting bottlenecks in critical paths, and offering optimized alternatives.
* Perform quantitative user-study evaluations, collect qualitative feedback from expert interface designers.

'''Contributions'''

* Design-space analysis and quantitative evaluation of cognition-based techniques for assessing user interfaces.
* Design and quantitative evaluation of techniques for suggesting optimized interface-design changes.
* An extensible, multimodal software architecture for capturing user traces integrated with pupil-tracking data, auditory recognition, and muscle-activity monitoring.
* (there may be more here, like testing different cognitive models, generating a markup language to represent interfaces, maybe even a unique metric space for interface usability)

--

See the [http://vrl.cs.brown.edu/wiki/images/b/b1/Flowchart2.pdf flowchart] for a visual overview of our aims.

In order to use this framework, a designer will have to provide:
* Functional specification - what are the possible interactions between the user and the application. This can be thought of as method signatures, with a name (e.g., setVolume), direction (to user or from user) and a list of value types (boolean, number, text, video, ...) for each interaction.
* GUI specification - a mapping of interactions to interface elements (e.g., setVolume is mapped to the grey knob in the bottom left corner with clockwise turning increasing the input number).
* Functional user traces - sequences of representative ways in which the application is used. Instead of writing them, the designer could have users use the application with a trial interface and then use our methods to generalize the user traces beyond the specific interface (The second method is depicted in the diagram). As a form of pre-processing, the system also generates an interaction transition matrix which lists the probability of each type of interaction given the previous interaction.
* Utility function - this is a weighting of various performance metrics (time, cognitive load, fatigue, etc.), where the weighting expresses the importance of a particular dimension to the user. For example, a user at NASA probably cares more about interface accuracy than speed. By passing this information to our committee of experts, we can create interfaces that are tuned to maximize the utility of a particular user type.

Each of the modules can use all of this information or a subset of it. Our approach stresses flexibility and the ability to give more meaningful feedback the more information is provided. After processing the information sent by the system of experts, the aggregator will output:
* An evaluation of the interface. Evaluations are expressed both in terms of the utility function components (i.e. time, fatigue, cognitive load, etc.), and in terms of the overall utility for this interface (as defined by the utility function). These evaluations are given in the form of an efficiency curve, where the utility received on each dimension can change as the user becomes more accustomed to the interface.
* Suggested improvements for the GUI are also output. These suggestions are meant to optimize the utility function that was input to the system. If a user values accuracy over time, interface suggestions will be made accordingly.

== Recording User-Interaction Primitives ==
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

'''Aims'''
* Develop system for logging low-level user-interactions within existing applications. By low-level interactions, we refer to those interactions for which a quantitative, predictive, GOMS-like model may be generated.
* At the same scale of granularity, integrate explicit interface-interactions with multimodal-sensing data. i.e. pupil-tracking, muscle-activity monitoring, auditory recognition, EEG data, and facial and posture-focused video.

'''Contributions'''
* Extensible, multimodal, HCI framework for recording rich interaction-history data in existing applications.

'''Significance'''
* Provide rich history data to serve as the basis for novel quantitative interface evaluation.
* Aid expert HCI and visual designers in traditional design processes.
* Provide data for automated machine-learning strategies applied to interaction.

== Semantic-level Interaction Chunking ==
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

'''Aims'''
* Develop techniques for ''chunking'' low-level interaction primitives into ''semantic-level'' interactions, given an application's functionality and data-context. (And de-chunking? Invertible mapping needed?)
* Perform user-study evaluation to validate chunking methods.

'''Contributions'''
* Design and user-study evaluation of semantic chunking techniques.

'''Significance'''
* Low-level <--> semantic-level mapping allows for cognitive-modeling to be applied at a functionality level, where low-level interaction techniques can be swapped out. This will allow our interface assessment system to make feasible suggestions for more optimal interface design.

== Parallel Framework for Evaluation Modules ==
* Owner: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]]

''This section will describe in more detail the inputs, outputs and architecture that were presented in the introduction.''

==== Module Inputs (Incomplete) ====
* A set of utility dimensions {d1, d2, ...} are defined in the framework. These could be {d1="time", d2="fatigue", ...}
* A set of interaction functions. These specify all of the information that the application wants to give users or get from them. It is not tied to a specific interface. For example, the fact that an applications shows videos would be included here. Whether it displays them embedded, in a pop-up window or fullscreen would not.
* A mapping of interaction functions to interface elements (e.g., buttons, windows, dials,...). Lots of optional information describing visual properties, associated text, physical interactions (e.g., turning the dial clockwise increases the input value) and timing.
* Functional user traces - sequences of interaction functions that represent typical user interactions with the application. Could include a goal hierarchy, in which case the function sequence is at the bottom of the hierarchy.

==== Module Outputs ====
* Every module ouputs at least one of the following:
** An evaluation of the interface
*** This can be on any or all of the utility dimensions, e.g. evaluation={d1=score1, d2=score2, ...}
*** This can alternately be an overall evaluation, ignoring dimensions, e.g. evaluation={score}
**** In this case, the aggregator will treat this as the module giving the same score to all dimensions. Which dimension this evaluator actually predicts well on can be learned by the aggregator over time.
** Recommendation(s) for improving the interface
***This can be a textual description of what changes the designer should make
***This can alternately be a transformation that can automatically be applied to the interface language (without designer intervention)
***In addition to the textual or transformational description of the recommendation, a "change in evaluation" is output to describe how specifically the value will improve the interface
****Recommendation = {description="make this change", Δevaluation={d1=score1, d2=score2, ...}
****Like before, this Δevaluation can cover any number of dimensions, or it can be generic.
***Either a single recommendation or a set of recommendations can be output

==== Aggregator Inputs ====
The aggregator receives as input the outputs of all the modules.

==== Aggregator Outputs ====

Outputs for the aggregator are the same as the outputs for each module. The difference is that the aggregator will consider all the module outputs, and arrive at a merged output based on the past performance of each of the modules.

== Evaluation and Recommendation via Modules ==
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

A "meta-module" called the aggregator will be responsible for assembling and formatting the output of all other modules into a structure that is both extensible and immediately usable, by both an automated designer or a human designer.

===Requirements===
The aggregator's functionality, then, is defined by its '''inputs''', the outputs of the other modules, and the desired '''output''' of the system as a whole, per its position in the architecture. Its purpose is largely formatting and reconciliation of the products of the multitudinous (and extensible) modules. The output of the aggregator must meet several requirements: first, to generate a set of human-readable suggestions for the improvement of the given interface; second, to generate a machine-readable, but also analyzable, evaluation of the various characteristics of the interface and accompanying user traces.

From these specifications, it is logical to assume that a common language or format will be required for the output of individual modules. We propose an XML-based file format, allowing: (1) a section for the standardized identification of problem areas, applicable rules, and proposed improvements, generalized by the individual module and mapped to a single element, or group of elements, in the original interface specification; (2) a section for specification of generalizable "utility" functions, allowing a module to specify how much a measurable quantity of utility is positively or negatively affected by properties of the input interface; (3) new, user-definable sections for evaluations of the given interface not covered by the first two sections. The first two sections are capable of conveying the vast majority of module outputs predicted at this time, but the XML can extensibly allow modules to pass on whatever information may become prominent in the future.

===Specification===
<module id="Fitts-Law">
<interface-elements>
<element>
<desc>submit button</desc>
<problem>
<desc>size</desc>
<suggestion>width *= 2</suggestion>
<suggetsion>height *= 2</suggestions>
<human-suggestion>Increase size relative to other elements</human-suggestion>
</problem>
</element>
</interface-elements>
<utility>
<dimension>
<desc>time</desc>
<value>0:15:35</value>
</dimension>
<dimension>
<desc>frustration</desc>
<value>pulling hair out</value>
</dimension>
<dimension>
<desc>efficiency</desc>
<value>13.2s/KPM task</value>
<value>0.56m/CPM task</value>
</dimension>
</utility>
<tasks>
<task>
<desc>complete form</desc>
</task>
<task>
<desc>lookup SSN</desc>
</task>
<task>
<desc>format phone number</desc>
</task>
</tasks>
</module>

===Logic===
This file provided by each module is then the input for the aggregator. The aggregator's most straightforward function is the compilation of the "problem areas," assembling them and noting problem areas and suggestions that are recommended by more than one module, and weighting them accordingly in its final report. These weightings can begin in an equal state, but the aggregator should be capable of learning iteratively which modules' results are most relevant to the user and update weightings accordingly. This may need to be accomplished with manual tuning, or a machine-learning algorithm capable of determining which modules most often agree with others.

Secondly, the aggregator compiles the utility functions provided by the module specs. This, again, is a summation of similarly-described values from the various modules.

When confronted with user-defined sections of the XML spec, the aggregator is primarily responsible for compiling them and sending them along to the output of the machine. Even if the aggregator does not recognize a section or property of the evaluative spec, if it sees the property reported by more than one module it should be capable of aggregating these intelligently. In future versions of the spec, it should be possible for a module to provide instructions for the aggregator on how to handle unrecognized sections of the XML.

From these compilations, then, the aggregator should be capable of outputting both aggregated human-readable suggestions on interface improvements for a human designer, as well as a comprehensive evaluation of the interface's effectiveness at the given task traces. Again, this is dependent on the specification of the system as a whole, but is likely to include measures and comparisons, graphings of task versus utility, and quantitative measures of an element's effectiveness.

''This section is necessarily defined by the output of the individual modules (which I already expect to be of varied and arbitrary structure) and the desired output of the machine as a whole. It will likely need to be revised heavily after other modules and the "Parellel Framework" section are defined.'' [[User:E J Kalafarski|E J Kalafarski]] 12:34, 24 April 2009 (UTC)

''This section describes the aggregator, which takes the output of multiple independent modules and aggregates the results to provide (1) an evaluation and (2) recommendations for the user interface. We should explain how the aggregator weights the output of different modules (this could be based on historical performance of each module, or perhaps based on E.J.'s cognitive/HCI guidelines).''

== Sample Modules ==

=== CPM-GOMS ===
* Owners: [[User:Steven Ellis | Steven Ellis]]

''This module will provide interface evaluations and suggestions based on a CPM-GOMS model of cognition for the given interface. It will provide a quantitative, predictive, cognition-based parameterization of usability. From empirically collected data, user trajectories through the model (critical paths) will be examined, highlighting bottlenecks within the interface, and offering suggested alterations to the interface to induce more optimal user trajectories.''

I’m hoping to have some input on this section, because it seems to be the crux of the “black box” into which we take the inputs of interface description, user traces, etc. and get our outputs (time, recommendations, etc.). I know at least a few people have pretty strong thoughts on the matter and we ought to discuss the final structure.

That said – my proposal for the module:
*In my opinion the concept of the Model Human Processor (at least as applied in CPM) is outdated – it’s too economic/overly parsimonious in its conception of human activity. I think we need to create a structure which accounts for more realistic conditions of HCI including multitasking, aspects of distributed cognition (and other relevant uses of tools – as far as I can tell CPM doesn’t take into account any sort of productivity aids), executive control processes of attention, etc. ACT-R appears to take steps towards this but we would probably need to look at their algorithms to know for sure.
*Critical paths will continue to play an important role – we should in fact emphasize that part of this tool’s purpose will be a description not only of ways in which the interface should be modified to best fit a critical path, but also ways in which the user’s ought to be instructed in their use of the path. This feedback mechanism could be bidirectional – if the model’s predictions of the user’s goals are incorrect, the critical path determined will also be incorrect and the interface inherently suboptimal. The user could be prompted with a tooltip explaining in brief why and how the interface has changed, along with options to revert, select other configurations (euphemized by goals), and to view a short video detailing how to properly use the interface.
*Call me crazy but, if we assume designers will be willing to code a model of their interfaces into our ACT-R-esque language, could we allow that model to be fairly transparent to the user, who could use a gui to input their goals to find an analogue in the program which would subsequently rearrange its interface to fit the user’s needs? Even if not useful to the users, such dynamic modeling could really help designers (IMO)
*I think the model should do its best to accept models written for ACT-R and whatever other cognitive models there are out there – gives us the best chance of early adoption
*I would particularly appreciate input on the number/complexity/type of inputs we’ll be using, as well as the same qualities for the output.

=== HCI Guidelines ===
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

Schneiderman's Eight Golden Rules and Jakob Nielsen's Ten Heuristics are perhaps the most famous and well-regarded heuristic design guidelines to emerge over the last twenty years. Although the explicit theoretical basis for such heuristics is controversial and not well-explored, the empirical success of these guidelines is established and accepted. This module will parse out up to three or four common (that is, intersecting) principles from these accepted guidelines and apply them to the input interface.

As an example, we identify an analogous principle that appears in Schneiderman ("Reduce short-term memory load")<ref>http://faculty.washington.edu/jtenenbg/courses/360/f04/sessions/schneidermanGoldenRules.html</ref> and Nielsen ("Recognition rather than recall/Minimize the user's memory load")<ref>http://www.useit.com/papers/heuristic/heuristic_list.html</ref>. The input interface is then evaluated for the consideration of the principle, based on an explicit formal description of the interface, such as XAML or XUL. The module attempts to determine how effectively the interface demonstrates the principle. When analyzing an interface for several principles that may be conflicting or opposing in a given context, the module makes use of a hard-coded but iterative (and evolving) weighting of these principles, based on (1) how often they appear in the training set of accepted sets of guidelines, (2) how analogues a heuristic principle is to a cognitive principle in a parallel training set, and (3) how effective the principle's associated suggestion is found to be using a feedback mechanism.

'''Inputs''' 
# A formal description of the interface and its elements (e.g. buttons).
# A formal description of a particular task and the possible paths through a subset of interface elements that permit the user to accomplish that task.

'''Output''' 
Standard XML-formatted file containing problem areas of the input interface, suggestions for each problem area based on principles that were found to have a strong application to a problem element and the problem itself, and a human-readable generated analysis of the element's affinity for the principle. Quantitative outputs will not be possible based on heuristic guidelines, and the "utility" section of this module's output is likely to be blank.

''This section could include an example or two of established design guidelines that could easily be implemented as modules.''

=== Fitts's Law ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

''This module provides an estimate of the required time to complete various tasks that have been decomposed into formalized sequences of interactions with interface elements, and will provide evaluations and recommendations for optimizing the time required to complete those tasks using the interface.''

'''Inputs'''

1. A formal description of the interface and its elements (e.g. buttons).

2. A formal description of a particular task and the possible paths through a subset of interface elements that permit the user to accomplish that task.

3. The physical distances between interface elements along those paths.

4. The width of those elements along the most likely axes of motion.

5. Device (e.g. mouse) characteristics including start/stop time and the inherent speed limitations of the device.

'''Output'''

The module will then use the Shannon formulation of Fitt's Law to compute the average time needed to complete the task along those paths.

=== Affordances ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

''This simple module will provide interface evaluations and recommendations based on a measure of the extent to which the user perceives the relevant affordances of the interface when performing a number of specified tasks.''

'''Inputs'''

Formalized descriptions of...

1. Interface elements

2. Their associated actions

3. The functions of those actions

4. A particular task

5. User traces for that task.

Inputs (1-4) are then used to generate a "user-independent" space of possible functions that the interface is capable of performing with respect to a given task -- what the interface "affords" the user. From this set of possible interactions, our model will then determine the subset of optimal paths for performing a particular task. The user trace (5) is then used to determine what functions actually were performed in the course of a given task of interest and this information is then compared to the optimal path data to determine the extent to which affordances of the interface are present but not perceived.

'''Output'''

The output of this module is a simple ratio of (affordances perceived) / [(relevant affordances present) * (time to complete task)] which provides a quantitative measure of the extent to which the interface is "natural" to use for a particular task.

=== Workflow, Multi-tasking and Interruptions ===
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

There are, at least, two levels at which users work ([http://portal.acm.org/citation.cfm?id=985692.985707&coll=portal&dl=ACM&CFID=20781736&CFTOKEN=83176621 Gonzales, et al., 2004]). Users accomplish individual low-level tasks which are part of larger ''working spheres''; for example, an office worker might send several emails, create several Post-It (TM) note reminders, and then edit a word document, each of these smaller tasks being part of a single larger working sphere of "adding a new section to the website." Thus, it is important to understand this larger workflow context - which often involves extensive levels of multi-tasking, as well as switching between a variety of computing devices and traditional tools, such as notebooks. In this study it was found that the information workers surveyed typically switch individual tasks every 2 minutes and have many simultaneous working spheres which they switch between, on average every 12 minutes. This frenzied pace of switching tasks and switching working spheres suggests that users will not be using a single application or device for a long period of time, and that affordances to support this characteristic pattern of information work are important.

The purpose of this module is to integrate existing work on multi-tasking, interruption and higher-level workflow into a framework which can predict user recovery times from interruptions. Specifically, the goals of this framework will be to:

* Understand the role of the larger workflow context in user interfaces
* Understand the impact of interruptions on user workflow
* Understand how to design software which fits into the larger working spheres in which information work takes place

It is important to point out that because workflow and multi-tasking rely heavily on higher-level brain functioning, it is unrealistic within the scope of this grant to propose a system which can predict user performance given a description of a set of arbitrary software programs. Therefore, we believe this module will function much more in a qualitative role to provide context to the rest of the model. Specifically, our findings related to interruption and multi-tasking will advance the basic research question of "how do you users react to interruptions when using working sets of varying sizes?". This core HCI contribution will help to inform the rest of the outputs of the model in a qualitative manner.

'''Inputs'''

N/A

'''Outputs'''

N/A

=== Working Memory Load ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''This module measures how much information the user needs to retain in memory while interacting with the interface and makes suggestions for improvements.''

* Models
** Beddeley's Model of Working Memory<ref>http://en.wikipedia.org/wiki/Baddeley%27s_model_of_working_memory</ref>
*** Episodic Buffer
** George Miller's "The magic number 7 plus or minus 2"<ref>http://en.wikipedia.org/wiki/The_Magical_Number_Seven,_Plus_or_Minus_Two</ref>
** The magic number 4<ref>Cowan, N. (2001). The magical number 4 in short-term memory: A reconsideration of mental storage capacity. Behavioral and Brain Sciences, 24, 87-185.</ref>
* Chunking<ref>http://en.wikipedia.org/wiki/Chunking_(psychology)</ref>
* Priming<ref>http://en.wikipedia.org/wiki/Priming_(psychology)</ref>
* Subitizing and Counting<ref>http://en.wikipedia.org/wiki/Subitizing</ref>

'''Inputs'''
* Visual Stimuli
* Audio Stimuli
'''Outputs'''
* Remembered percepts
* Half-Life of percepts

=== Automaticity of Interaction ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''Measures how easily the interaction with the interface becomes automatic with experience and makes suggestions for improvements.''

* Learning<ref>http://en.wikipedia.org/wiki/Learning#Mathematical_models_of_learning</ref>
** Logan's instance theory of automatization<ref>http://74.125.95.132/search?q=cache:IZ-Zccsu3SEJ:psych.wisc.edu/ugstudies/psych733/logan_1988.pdf+logan+isntance+teory&cd=1&hl=en&ct=clnk&gl=us&client=firefox-a</ref>
* Fluency
** As meta-cognitive information <ref>http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6VH9-4SM7PFK-4&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=10cd279fa80958981fcc3c06684c09af</ref>
** As a cognitive ''heuristic''<ref>http://en.wikipedia.org/wiki/Fluency_heuristic</ref>

'''Inputs'''
* Interface
* User goals

'''Outputs'''
* Learning curve

== Integration into the Design Process ==
* Owner: [[User:Ian Spector | Ian Spector]]

''This section outlines the process of designing an HCI interface and at what stages our proposal fits in and how.''

= Preliminary Results =

===Workflow, Multi-tasking, and Interruption===

====I. '''Goals'''====

The goals of the preliminary work are to gain qualitative insight into how information workers practice metawork, and to determine whether people might be better-supported with software which facillitates metawork and interruptions. Thus, the preliminary work should investigate, and demonstrate, the need and impact of the core goals of the project.

====II. '''Methodology'''====

Seven information workers, ages 20-38 (5 male, 2 female), were interviewed to determine which methods they use to "stay organized". An initial list of metawork strategies was established from two pilot interviews, and then a final list was compiled. Participants then responded to a series of 17 questions designed to gain insight into their metawork strategies and process. In addition, verbal interviews were conducted to get additional open-ended feedback.

====III. '''Final Results'''====
A histogram of methods people use to "stay organized" in terms of tracking things they need to do (TODOs), appointments and meetings, etc. is shown in the figure below.

[[Image:AcbGraph.jpg]]

In addition to these methods, participants also used a number of other methods, including:

* iCal
* Notes written in xterms
* "Inbox zero" method of email organization
* iGoogle Notepad (for tasks)
* Tag emails as "TODO", "Important", etc.
* Things (Organizer Software)
* Physical items placed to "remind me of things"
* Sometimes arranging windows on desk
* Keeping browser tabs open
* Bookmarking web pages
* Keep programs/files open scrolled to certain locations sometimes with things selected

In addition, three participants said that when interrupted they "rarely" or "very rarely" were able to resume the task they were working on prior to the interruption. Three of the participants said that they would not actively recommend their metawork strategies for other people, and two said that staying organized was "difficult".

Four participants were neutral to the idea of new tools to help them stay organized and three said that they would like to have such a tool/tools.

====IV. '''Discussion'''====
These results qunatiatively support our hypothesis that there is no clearly dominant set of metawork strategies employed by information workers. This highly fragemented landscape is surprising, even though most information workers work in a similar environment - at a desk, on the phone, in meetings - and with the same types of tools - computers, pens, paper, etc. We believe that this suggests that there are complex tradeoffs between these methods and that no single method is sufficient. We therefore believe that users will be better supported with a new set of software-based metawork tools.

= [Criticisms] =
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

''Any criticisms or questions we have regarding the proposal can go here.''

=References=
<references/>

CS295J/Research proposal (draft 2)

2009-04-24T16:28:50Z

Trevor O'Brien: /* Recording User-Interaction Primitives */

= Introduction =
* Owners: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]], [[User: Trevor O'Brien | Trevor]]

'''Propose:''' The design, application and evaluation of a novel, cognition-based, computational framework for assessing interface design and providing automated suggestions to optimize usability.

'''Evaluation Methodology:''' Our techniques will be evaluated quantitatively through a series of user-study trials, as well as qualitatively by a team of expert interface designers.

'''Contributions and Significance:''' We expect this work to make the following contributions:
# design-space analysis and quantitative evaluation of cognition-based techniques for assessing user interfaces.
# design and quantitative evaluation of techniques for suggesting optimized interface-design changes.
# an extensible, multimodal software architecture for capturing user traces integrated with pupil-tracking data, auditory recognition, and muscle-activity monitoring.
# specification (language?) of how to define an interface evaluation module and how to integrate it into a larger system.
# (there may be more here, like testing different cognitive models, generating a markup language to represent interfaces, maybe even a unique metric space for interface usability)

--

We propose a framework for interface evaluation and recommendation that integrates behavioral models and design guidelines from both cognitive science and HCI. Our framework behaves like a committee of specialized experts, where each expert provides its own assessment of the interface, given its particular knowledge of HCI or cognitive science. For example, an expert may provide an evaluation based on the GOMS method, Fitts's law, Maeda's design principles, or cognitive models of learning and memory. An aggregator collects all of these assessments and weights the opinions of each expert, and outputs to the developer a merged evaluation score and a weighted set of recommendations.

Systematic methods of estimating human performance with computer interfaces are used only sparsely despite their obvious benefits, the reason being the overhead involved in implementing them. In order to test an interface, both manual coding systems like the GOMS variations and user simulations like those based on ACT-R/PM and EPIC require detailed pseudo-code descriptions of the user workflow with the application interface. Any change to the interface then requires extensive changes to the pseudo-code, a major problem because of the trial-and-error nature of interface design. Updating the models themselves is even more complicated. Even an expert in CPM-GOMS, for example, can't necessarily adapt it to take into account results from new cognitive research.

Our proposal makes automatic interface evaluation easier to use in several ways. First of all, we propose to divide the input to the system into three separate parts, functionality, user traces and interface. By separating the functionality from the interface, even radical interface changes will require updating only that part of the input. The user traces are also defined over the functionality so that they too translate across different interfaces. Second, the parallel modular architecture allows for a lower "entry cost" for using the tool. The system includes a broad array of evaluation modules some of which are very simple and some more complex. The simpler modules use only a subset of the input that a system like GOMS or ACT-R would require. This means that while more input will still lead to better output, interface designers can get minimal evaluations with only minimal information. For example, a visual search module may not require any functionality or user traces in order to determine whether all interface elements are distinct enough to be easy to find. Finally, a parallel modular architecture is much easier to augment with relevant cognitive and design evaluations.

= Background / Related Work =

Each person should add the background related to their specific aims.

* Steven Ellis - Cognitive models of HCI, including GOMS variations and ACT-R
* EJ - Design Guidelines
* Jon - Perception and Action
* Andrew - Multiple task environments
* Gideon - Cognition and dual systems
* Ian - Interface design process
* Trevor - User trace collection methods (especially any eye-tracking, EEG, ... you want to suggest using)

== Cognitive Models ==
I plan to port over most of the background on cognitive models of HCI from the old proposal

Additions will comprise of:
*CPM-GOMS as a bridge from GOMS architecture to the promising procedural optimization of the Model Human Processor
**Context of CPM development, discuss its relation to original GOMS and KLM
***Establish the tasks which were relevant for optimization when CPM was developed and note that its obsolescence may have been unavoidable
**Focus on CPM as the first step in transitioning from descriptive data, provided by mounting efforts in the cognitive sciences realm to discover the nature of task processing and accomplishment, to prescriptive algorithms which can predict an interface’s efficiency and suggest improvements
**CPM’s purpose as an abstraction of cognitive processing – a symbolic representation not designed for accuracy but precision
**CPM’s successful trials, e.g. Ernestine
***Implications of this project include CPM’s ability to accurately estimate processing at a psychomotor level
***Project does suggest limitations, however, when one attempts to examine more complex tasks which involve deeper and more numerous cognitive processes
*ACT-R as an example of a progressive cognitive modeling tool
**A tool clearly built by and for cognitive scientists, and as a result presents a much more accurate view of human processing – helpful for our research
**Built-in automation, which now seems to be a standard feature of cognitive modeling tools
**Still an abstraction of cognitive processing, but makes adaptation to cutting-edge cognitive research findings an integral aspect of its modular structure
**Expand on its focus on multi-tasking, taking what was a huge advance between GOMS and its CPM variation and bringing the simulation several steps closer to approximating the nature of cognition in regards to HCI
**Far more accessible both for researchers and the lay user/designer in its portability to LISP, pre-construction of modules representing cognitive capacities and underlying algorithms modeling paths of cognitive processing

==Design guidelines==
A multitude of rule sets exist for the design of not only interface, but architecture, city planning, and software development. They can range in scale from one primary rule to as many Christopher Alexander's 253 rules for urban environments,<ref>[http://hci.rwth-aachen.de/materials/publications/borchers2000a.pdf Borchers, Jan O. "A Pattern Approach to Interaction Design." 2000.]</ref> which he introduced with the concept design patterns in the 1970s. Study has likewise been conducted on the use of these rules:<ref>http://stl.cs.queensu.ca/~graham/cisc836/lectures/readings/tetzlaff-guidelines.pdf</ref> guidelines are often only partially understood, indistinct to the developer, and "fraught" with potential usability problems given a real-world situation.

===Application to AUE===
And yet, the vast majority of guideline sets, including the most popular rulesets, have been arrived at heuristically. The most successful, such as Raskin's and Schneiderman's, have been forged from years of observation instead of empirical study and experimentation. The problem is similar to the problem of circular logic faced by automated usability evaluations: an automated system is limited in the suggestions it can offer to a set of preprogrammed guidelines which have often not been subjected to rigorous experimentation.<ref>[http://www.eecs.berkeley.edu/Pubs/TechRpts/2000/CSD-00-1105.pdf Ivory, M and Hearst, M. "The State of the Art in Automated Usability Evaluation of User Interfaces." ACM Computing Surveys (CSUR), 2001.]</ref> In the vast majority of existing studies, emphasis has traditionally been placed on either the development of guidelines or the application of existing guidelines to automated evaluation. A mutually-reinforcing development of both simultaneously has not been attempted.

Overlap between rulesets is inevitable and unavoidable. For our purposes of evaluating existing rulesets efficiently, without extracting and analyzing each rule individually, it may be desirable to identify the the overarching ''principles'' or ''philosophy'' (max. 2 or 3) for a given ruleset and determining their quantitative relevance to problems of cognition.

===Popular and seminal examples===
Schneiderman's [http://faculty.washington.edu/jtenenbg/courses/360/f04/sessions/schneidermanGoldenRules.html Eight Golden Rules] date to 1987 and are arguably the most-cited. They are heuristic, but can be somewhat classified by cognitive objective: at least two rules apply primarily to ''repeated use'', versus ''discoverability''. Up to five of Schneiderman's rules emphasize ''predictability'' in the outcomes of operations and ''increased feedback and control'' in the agency of the user. His final rule, paradoxically, removes control from the user by suggesting a reduced short-term memory load, which we can arguably classify as ''simplicity''.

Raskin's [http://www.mprove.de/script/02/raskin/designrules.html Design Rules] are classified into five principles by the author, augmented by definitions and supporting rules. While one principle is primarily aesthetic (a design problem arguably out of the bounds of this proposal) and one is a basic endorsement of testing, the remaining three begin to reflect philosophies similar to Schneiderman's: reliability or ''predictability'', ''simplicity'' or ''efficiency'' (which we can construe as two sides of the same coin), and finally he introduces a concept of ''uninterruptibility''.

Maeda's [http://lawsofsimplicity.com/?cat=5&order=ASC Laws of Simplicity] are fewer, and ostensibly emphasize ''simplicity'' exclusively, although elements of ''use'' as related by Schneiderman's rules and ''efficiency'' as defined by Raskin may be facets of this simplicity. Google's corporate mission statement presents [http://www.google.com/corporate/ux.html Ten Principles], only half of which can be considered true interface guidelines. ''Efficiency'' and ''simplicity'' are cited explicitly, aesthetics are once again noted as crucial, and working within a user's trust is another application of ''predictability''.

===Elements and goals of a guideline set===
Myriad rulesets exist, but variation becomes scarce—it indeed seems possible to parse these common rulesets into overarching principles that can be converted to or associated with quantifiable cognitive properties. For example, it is likely ''simplicity'' has an analogue in the short-term memory retention or visual retention of the user, vis a vis the rule of [http://books.google.com/books?hl=en&lr=&id=j5q0VvOGExYC&oi=fnd&pg=PA357&dq=seven+plus+or+minus+two&ots=prI3PKJBar&sig=vOZnqpnkXKGYWxK6_XlA4I_CRyI Seven, Plus or Minus Two]. ''Predictability'' likewise may have an analogue in Activity Theory, in regards to a user's perceptual expectations for a given action; ''uninterruptibility'' has implications in cognitive task-switching;<ref>[http://portal.acm.org/citation.cfm?id=985692.985715&coll=Portal&dl=ACM&CFID=21136843&CFTOKEN=23841774 Czerwinski, Horvitz, and White. "A Diary Study of Task Switching and Interruptions." Proceedings of the SIGCHI conference on Human factors in computing systems, 2004.]</ref> and so forth.

Within the scope of this proposal, we aim to reduce and refine these philosophies found in seminal rulesets and identify their logical cognitive analogues. By assigning a quantifiable taxonomy to these principles, we will be able to rank and weight them with regard to their real-world applicability, developing a set of "meta-guidelines" and rules for applying them to a given interface in an automated manner. Combined with cognitive models and multi-modal HCI analysis, we seek to develop, in parallel with these guidelines, the interface evaluation system responsible for their application.

== Perception and Action (in progress) ==

*Information Processing Approach
**Advantages
***Formalism eases translation of theory into scripting language
**Disadvantages
***Assumes symbolic representation

*Ecological (Gibsonian) Approach
**Advantages
***Emphasis on bodily and environmental constraints
**Disadvantages
***Lack of formalism hinders translation of theory into scripting language

= Specific Aims and Contributions (to be separated later) =

'''Specific Aims'''
* Incorporate interaction history mechanisms into a set of existing applications.
* Perform user-study evaluation of history-collection techniques.
* Distill a set of cognitive principles/models, and evaluate empirically?
* Build/buy sensing system to include pupil-tracking, muscle-activity monitoring, auditory recognition.
* Design techniques for manual/semi-automated/automated construction of <insert favorite cognitive model here> from interaction histories and sensing data.
* Design system for posterior analysis of interaction history w.r.t. <insert favorite cognitive model here>, evaluating critical path <or equivalent> trajectories.
* Design cognition-based techniques for detecting bottlenecks in critical paths, and offering optimized alternatives.
* Perform quantitative user-study evaluations, collect qualitative feedback from expert interface designers.

'''Contributions'''

* Design-space analysis and quantitative evaluation of cognition-based techniques for assessing user interfaces.
* Design and quantitative evaluation of techniques for suggesting optimized interface-design changes.
* An extensible, multimodal software architecture for capturing user traces integrated with pupil-tracking data, auditory recognition, and muscle-activity monitoring.
* (there may be more here, like testing different cognitive models, generating a markup language to represent interfaces, maybe even a unique metric space for interface usability)

--

See the [http://vrl.cs.brown.edu/wiki/images/b/b1/Flowchart2.pdf flowchart] for a visual overview of our aims.

In order to use this framework, a designer will have to provide:
* Functional specification - what are the possible interactions between the user and the application. This can be thought of as method signatures, with a name (e.g., setVolume), direction (to user or from user) and a list of value types (boolean, number, text, video, ...) for each interaction.
* GUI specification - a mapping of interactions to interface elements (e.g., setVolume is mapped to the grey knob in the bottom left corner with clockwise turning increasing the input number).
* Functional user traces - sequences of representative ways in which the application is used. Instead of writing them, the designer could have users use the application with a trial interface and then use our methods to generalize the user traces beyond the specific interface (The second method is depicted in the diagram). As a form of pre-processing, the system also generates an interaction transition matrix which lists the probability of each type of interaction given the previous interaction.
* Utility function - this is a weighting of various performance metrics (time, cognitive load, fatigue, etc.), where the weighting expresses the importance of a particular dimension to the user. For example, a user at NASA probably cares more about interface accuracy than speed. By passing this information to our committee of experts, we can create interfaces that are tuned to maximize the utility of a particular user type.

Each of the modules can use all of this information or a subset of it. Our approach stresses flexibility and the ability to give more meaningful feedback the more information is provided. After processing the information sent by the system of experts, the aggregator will output:
* An evaluation of the interface. Evaluations are expressed both in terms of the utility function components (i.e. time, fatigue, cognitive load, etc.), and in terms of the overall utility for this interface (as defined by the utility function). These evaluations are given in the form of an efficiency curve, where the utility received on each dimension can change as the user becomes more accustomed to the interface.
* Suggested improvements for the GUI are also output. These suggestions are meant to optimize the utility function that was input to the system. If a user values accuracy over time, interface suggestions will be made accordingly.

== Recording User-Interaction Primitives ==
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

'''Aims'''
* Develop system for logging low-level user-interactions within existing applications. By low-level interactions, we refer to those interactions for which a quantitative, predictive, GOMS-like model may be generated.
* At the same scale of granularity, integrate explicit interface-interactions with multimodal-sensing data. i.e. pupil-tracking, muscle-activity monitoring, auditory recognition, EEG data, and facial and posture-focused video.

'''Contributions'''
* Extensible, multimodal, HCI framework for recording rich interaction-history data in existing applications.

'''Significance'''
* Provide rich history data to serve as the basis for novel quantitative interface evaluation.
* Aid expert HCI and visual designers in traditional design processes.
* Provide data for automated machine-learning strategies applied to interaction.

== Semantic-level Interaction Chunking ==
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

''The user traces that are collected are tied to a specific interface. In order to use them with different interfaces to the same application, they should be generalized to be based only on the functional description of the application and the user's goal hierarchy. This would abstract away from actions like accessing a menu.''

''In addition to specific user traces, many modules could use a transition probability matrix based on interaction predictions.''

== Parallel Framework for Evaluation Modules ==
* Owner: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]]

''This section will describe in more detail the inputs, outputs and architecture that were presented in the introduction.''

==== Module Inputs (Incomplete) ====
* A set of utility dimensions {d1, d2, ...} are defined in the framework. These could be {d1="time", d2="fatigue", ...}
* A set of interaction functions. These specify all of the information that the application wants to give users or get from them. It is not tied to a specific interface. For example, the fact that an applications shows videos would be included here. Whether it displays them embedded, in a pop-up window or fullscreen would not.
* A mapping of interaction functions to interface elements (e.g., buttons, windows, dials,...). Lots of optional information describing visual properties, associated text, physical interactions (e.g., turning the dial clockwise increases the input value) and timing.
* Functional user traces - sequences of interaction functions that represent typical user interactions with the application. Could include a goal hierarchy, in which case the function sequence is at the bottom of the hierarchy.

==== Module Outputs ====
* Every module ouputs at least one of the following:
** An evaluation of the interface
*** This can be on any or all of the utility dimensions, e.g. evaluation={d1=score1, d2=score2, ...}
*** This can alternately be an overall evaluation, ignoring dimensions, e.g. evaluation={score}
**** In this case, the aggregator will treat this as the module giving the same score to all dimensions. Which dimension this evaluator actually predicts well on can be learned by the aggregator over time.
** Recommendation(s) for improving the interface
***This can be a textual description of what changes the designer should make
***This can alternately be a transformation that can automatically be applied to the interface language (without designer intervention)
***In addition to the textual or transformational description of the recommendation, a "change in evaluation" is output to describe how specifically the value will improve the interface
****Recommendation = {description="make this change", Δevaluation={d1=score1, d2=score2, ...}
****Like before, this Δevaluation can cover any number of dimensions, or it can be generic.
***Either a single recommendation or a set of recommendations can be output

==== Aggregator Inputs ====
The aggregator receives as input the outputs of all the modules.

==== Aggregator Outputs ====

Outputs for the aggregator are the same as the outputs for each module. The difference is that the aggregator will consider all the module outputs, and arrive at a merged output based on the past performance of each of the modules.

== Evaluation and Recommendation via Modules ==
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

A "meta-module" called the aggregator will be responsible for assembling and formatting the output of all other modules into a structure that is both extensible and immediately usable, by both an automated designer or a human designer.

===Requirements===
The aggregator's functionality, then, is defined by its '''inputs''', the outputs of the other modules, and the desired '''output''' of the system as a whole, per its position in the architecture. Its purpose is largely formatting and reconciliation of the products of the multitudinous (and extensible) modules. The output of the aggregator must meet several requirements: first, to generate a set of human-readable suggestions for the improvement of the given interface; second, to generate a machine-readable, but also analyzable, evaluation of the various characteristics of the interface and accompanying user traces.

From these specifications, it is logical to assume that a common language or format will be required for the output of individual modules. We propose an XML-based file format, allowing: (1) a section for the standardized identification of problem areas, applicable rules, and proposed improvements, generalized by the individual module and mapped to a single element, or group of elements, in the original interface specification; (2) a section for specification of generalizable "utility" functions, allowing a module to specify how much a measurable quantity of utility is positively or negatively affected by properties of the input interface; (3) new, user-definable sections for evaluations of the given interface not covered by the first two sections. The first two sections are capable of conveying the vast majority of module outputs predicted at this time, but the XML can extensibly allow modules to pass on whatever information may become prominent in the future.

===Specification===
<module id="Fitts-Law">
<interface-elements>
<element>
<desc>submit button</desc>
<problem>
<desc>size</desc>
<suggestion>width *= 2</suggestion>
<suggetsion>height *= 2</suggestions>
<human-suggestion>Increase size relative to other elements</human-suggestion>
</problem>
</element>
</interface-elements>
<utility>
<dimension>
<desc>time</desc>
<value>0:15:35</value>
</dimension>
<dimension>
<desc>frustration</desc>
<value>pulling hair out</value>
</dimension>
<dimension>
<desc>efficiency</desc>
<value>13.2s/KPM task</value>
<value>0.56m/CPM task</value>
</dimension>
</utility>
<tasks>
<task>
<desc>complete form</desc>
</task>
<task>
<desc>lookup SSN</desc>
</task>
<task>
<desc>format phone number</desc>
</task>
</tasks>
</module>

===Logic===
This file provided by each module is then the input for the aggregator. The aggregator's most straightforward function is the compilation of the "problem areas," assembling them and noting problem areas and suggestions that are recommended by more than one module, and weighting them accordingly in its final report. These weightings can begin in an equal state, but the aggregator should be capable of learning iteratively which modules' results are most relevant to the user and update weightings accordingly. This may need to be accomplished with manual tuning, or a machine-learning algorithm capable of determining which modules most often agree with others.

Secondly, the aggregator compiles the utility functions provided by the module specs. This, again, is a summation of similarly-described values from the various modules.

When confronted with user-defined sections of the XML spec, the aggregator is primarily responsible for compiling them and sending them along to the output of the machine. Even if the aggregator does not recognize a section or property of the evaluative spec, if it sees the property reported by more than one module it should be capable of aggregating these intelligently. In future versions of the spec, it should be possible for a module to provide instructions for the aggregator on how to handle unrecognized sections of the XML.

From these compilations, then, the aggregator should be capable of outputting both aggregated human-readable suggestions on interface improvements for a human designer, as well as a comprehensive evaluation of the interface's effectiveness at the given task traces. Again, this is dependent on the specification of the system as a whole, but is likely to include measures and comparisons, graphings of task versus utility, and quantitative measures of an element's effectiveness.

''This section is necessarily defined by the output of the individual modules (which I already expect to be of varied and arbitrary structure) and the desired output of the machine as a whole. It will likely need to be revised heavily after other modules and the "Parellel Framework" section are defined.'' [[User:E J Kalafarski|E J Kalafarski]] 12:34, 24 April 2009 (UTC)

''This section describes the aggregator, which takes the output of multiple independent modules and aggregates the results to provide (1) an evaluation and (2) recommendations for the user interface. We should explain how the aggregator weights the output of different modules (this could be based on historical performance of each module, or perhaps based on E.J.'s cognitive/HCI guidelines).''

== Sample Modules ==

=== CPM-GOMS ===
* Owners: [[User:Steven Ellis | Steven Ellis]]

''This module will provide interface evaluations and suggestions based on a CPM-GOMS model of cognition for the given interface. It will provide a quantitative, predictive, cognition-based parameterization of usability. From empirically collected data, user trajectories through the model (critical paths) will be examined, highlighting bottlenecks within the interface, and offering suggested alterations to the interface to induce more optimal user trajectories.''

I’m hoping to have some input on this section, because it seems to be the crux of the “black box” into which we take the inputs of interface description, user traces, etc. and get our outputs (time, recommendations, etc.). I know at least a few people have pretty strong thoughts on the matter and we ought to discuss the final structure.

That said – my proposal for the module:
*In my opinion the concept of the Model Human Processor (at least as applied in CPM) is outdated – it’s too economic/overly parsimonious in its conception of human activity. I think we need to create a structure which accounts for more realistic conditions of HCI including multitasking, aspects of distributed cognition (and other relevant uses of tools – as far as I can tell CPM doesn’t take into account any sort of productivity aids), executive control processes of attention, etc. ACT-R appears to take steps towards this but we would probably need to look at their algorithms to know for sure.
*Critical paths will continue to play an important role – we should in fact emphasize that part of this tool’s purpose will be a description not only of ways in which the interface should be modified to best fit a critical path, but also ways in which the user’s ought to be instructed in their use of the path. This feedback mechanism could be bidirectional – if the model’s predictions of the user’s goals are incorrect, the critical path determined will also be incorrect and the interface inherently suboptimal. The user could be prompted with a tooltip explaining in brief why and how the interface has changed, along with options to revert, select other configurations (euphemized by goals), and to view a short video detailing how to properly use the interface.
*Call me crazy but, if we assume designers will be willing to code a model of their interfaces into our ACT-R-esque language, could we allow that model to be fairly transparent to the user, who could use a gui to input their goals to find an analogue in the program which would subsequently rearrange its interface to fit the user’s needs? Even if not useful to the users, such dynamic modeling could really help designers (IMO)
*I think the model should do its best to accept models written for ACT-R and whatever other cognitive models there are out there – gives us the best chance of early adoption
*I would particularly appreciate input on the number/complexity/type of inputs we’ll be using, as well as the same qualities for the output.

=== HCI Guidelines ===
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

Schneiderman's Eight Golden Rules and Jakob Nielsen's Ten Heuristics are perhaps the most famous and well-regarded heuristic design guidelines to emerge over the last twenty years. Although the explicit theoretical basis for such heuristics is controversial and not well-explored, the empirical success of these guidelines is established and accepted. This module will parse out up to three or four common (that is, intersecting) principles from these accepted guidelines and apply them to the input interface.

As an example, we identify an analogous principle that appears in Schneiderman ("Reduce short-term memory load")<ref>http://faculty.washington.edu/jtenenbg/courses/360/f04/sessions/schneidermanGoldenRules.html</ref> and Nielsen ("Recognition rather than recall/Minimize the user's memory load")<ref>http://www.useit.com/papers/heuristic/heuristic_list.html</ref>. The input interface is then evaluated for the consideration of the principle, based on an explicit formal description of the interface, such as XAML or XUL. The module attempts to determine how effectively the interface demonstrates the principle. When analyzing an interface for several principles that may be conflicting or opposing in a given context, the module makes use of a hard-coded but iterative (and evolving) weighting of these principles, based on (1) how often they appear in the training set of accepted sets of guidelines, (2) how analogues a heuristic principle is to a cognitive principle in a parallel training set, and (3) how effective the principle's associated suggestion is found to be using a feedback mechanism.

'''Inputs''' 
# A formal description of the interface and its elements (e.g. buttons).
# A formal description of a particular task and the possible paths through a subset of interface elements that permit the user to accomplish that task.

'''Output''' 
Standard XML-formatted file containing problem areas of the input interface, suggestions for each problem area based on principles that were found to have a strong application to a problem element and the problem itself, and a human-readable generated analysis of the element's affinity for the principle. Quantitative outputs will not be possible based on heuristic guidelines, and the "utility" section of this module's output is likely to be blank.

''This section could include an example or two of established design guidelines that could easily be implemented as modules.''

=== Fitts's Law ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

''This module provides an estimate of the required time to complete various tasks that have been decomposed into formalized sequences of interactions with interface elements, and will provide evaluations and recommendations for optimizing the time required to complete those tasks using the interface.''

'''Inputs'''

1. A formal description of the interface and its elements (e.g. buttons).

2. A formal description of a particular task and the possible paths through a subset of interface elements that permit the user to accomplish that task.

3. The physical distances between interface elements along those paths.

4. The width of those elements along the most likely axes of motion.

5. Device (e.g. mouse) characteristics including start/stop time and the inherent speed limitations of the device.

'''Output'''

The module will then use the Shannon formulation of Fitt's Law to compute the average time needed to complete the task along those paths.

=== Affordances ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

''This simple module will provide interface evaluations and recommendations based on a measure of the extent to which the user perceives the relevant affordances of the interface when performing a number of specified tasks.''

'''Inputs'''

Formalized descriptions of...

1. Interface elements

2. Their associated actions

3. The functions of those actions

4. A particular task

5. User traces for that task.

Inputs (1-4) are then used to generate a "user-independent" space of possible functions that the interface is capable of performing with respect to a given task -- what the interface "affords" the user. From this set of possible interactions, our model will then determine the subset of optimal paths for performing a particular task. The user trace (5) is then used to determine what functions actually were performed in the course of a given task of interest and this information is then compared to the optimal path data to determine the extent to which affordances of the interface are present but not perceived.

'''Output'''

The output of this module is a simple ratio of (affordances perceived) / [(relevant affordances present) * (time to complete task)] which provides a quantitative measure of the extent to which the interface is "natural" to use for a particular task.

=== Workflow, Multi-tasking and Interruptions ===
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

There are, at least, two levels at which users work ([http://portal.acm.org/citation.cfm?id=985692.985707&coll=portal&dl=ACM&CFID=20781736&CFTOKEN=83176621 Gonzales, et al., 2004]). Users accomplish individual low-level tasks which are part of larger ''working spheres''; for example, an office worker might send several emails, create several Post-It (TM) note reminders, and then edit a word document, each of these smaller tasks being part of a single larger working sphere of "adding a new section to the website." Thus, it is important to understand this larger workflow context - which often involves extensive levels of multi-tasking, as well as switching between a variety of computing devices and traditional tools, such as notebooks. In this study it was found that the information workers surveyed typically switch individual tasks every 2 minutes and have many simultaneous working spheres which they switch between, on average every 12 minutes. This frenzied pace of switching tasks and switching working spheres suggests that users will not be using a single application or device for a long period of time, and that affordances to support this characteristic pattern of information work are important.

The purpose of this module is to integrate existing work on multi-tasking, interruption and higher-level workflow into a framework which can predict user recovery times from interruptions. Specifically, the goals of this framework will be to:

* Understand the role of the larger workflow context in user interfaces
* Understand the impact of interruptions on user workflow
* Understand how to design software which fits into the larger working spheres in which information work takes place

It is important to point out that because workflow and multi-tasking rely heavily on higher-level brain functioning, it is unrealistic within the scope of this grant to propose a system which can predict user performance given a description of a set of arbitrary software programs. Therefore, we believe this module will function much more in a qualitative role to provide context to the rest of the model. Specifically, our findings related to interruption and multi-tasking will advance the basic research question of "how do you users react to interruptions when using working sets of varying sizes?". This core HCI contribution will help to inform the rest of the outputs of the model in a qualitative manner.

'''Inputs'''

N/A

'''Outputs'''

N/A

=== Working Memory Load ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''This module measures how much information the user needs to retain in memory while interacting with the interface and makes suggestions for improvements.''

* Models
** Beddeley's Model of Working Memory<ref>http://en.wikipedia.org/wiki/Baddeley%27s_model_of_working_memory</ref>
*** Episodic Buffer
** George Miller's "The magic number 7 plus or minus 2"<ref>http://en.wikipedia.org/wiki/The_Magical_Number_Seven,_Plus_or_Minus_Two</ref>
** The magic number 4<ref>Cowan, N. (2001). The magical number 4 in short-term memory: A reconsideration of mental storage capacity. Behavioral and Brain Sciences, 24, 87-185.</ref>
* Chunking<ref>http://en.wikipedia.org/wiki/Chunking_(psychology)</ref>
* Priming<ref>http://en.wikipedia.org/wiki/Priming_(psychology)</ref>
* Subitizing and Counting<ref>http://en.wikipedia.org/wiki/Subitizing</ref>

'''Inputs'''
* Visual Stimuli
* Audio Stimuli
'''Outputs'''
* Remembered percepts
* Half-Life of percepts

=== Automaticity of Interaction ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''Measures how easily the interaction with the interface becomes automatic with experience and makes suggestions for improvements.''

* Learning<ref>http://en.wikipedia.org/wiki/Learning#Mathematical_models_of_learning</ref>
** Logan's instance theory of automatization<ref>http://74.125.95.132/search?q=cache:IZ-Zccsu3SEJ:psych.wisc.edu/ugstudies/psych733/logan_1988.pdf+logan+isntance+teory&cd=1&hl=en&ct=clnk&gl=us&client=firefox-a</ref>
* Fluency
** As meta-cognitive information <ref>http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6VH9-4SM7PFK-4&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=10cd279fa80958981fcc3c06684c09af</ref>
** As a cognitive ''heuristic''<ref>http://en.wikipedia.org/wiki/Fluency_heuristic</ref>

'''Inputs'''
* Interface
* User goals

'''Outputs'''
* Learning curve

== Integration into the Design Process ==
* Owner: [[User:Ian Spector | Ian Spector]]

''This section outlines the process of designing an HCI interface and at what stages our proposal fits in and how.''

= Preliminary Results =

===Workflow, Multi-tasking, and Interruption===

====I. '''Goals'''====

The goals of the preliminary work are to gain qualitative insight into how information workers practice metawork, and to determine whether people might be better-supported with software which facillitates metawork and interruptions. Thus, the preliminary work should investigate, and demonstrate, the need and impact of the core goals of the project.

====II. '''Methodology'''====

Seven information workers, ages 20-38 (5 male, 2 female), were interviewed to determine which methods they use to "stay organized". An initial list of metawork strategies was established from two pilot interviews, and then a final list was compiled. Participants then responded to a series of 17 questions designed to gain insight into their metawork strategies and process. In addition, verbal interviews were conducted to get additional open-ended feedback.

====III. '''Final Results'''====
A histogram of methods people use to "stay organized" in terms of tracking things they need to do (TODOs), appointments and meetings, etc. is shown in the figure below.

[[Image:AcbGraph.jpg]]

In addition to these methods, participants also used a number of other methods, including:

* iCal
* Notes written in xterms
* "Inbox zero" method of email organization
* iGoogle Notepad (for tasks)
* Tag emails as "TODO", "Important", etc.
* Things (Organizer Software)
* Physical items placed to "remind me of things"
* Sometimes arranging windows on desk
* Keeping browser tabs open
* Bookmarking web pages
* Keep programs/files open scrolled to certain locations sometimes with things selected

In addition, three participants said that when interrupted they "rarely" or "very rarely" were able to resume the task they were working on prior to the interruption. Three of the participants said that they would not actively recommend their metawork strategies for other people, and two said that staying organized was "difficult".

Four participants were neutral to the idea of new tools to help them stay organized and three said that they would like to have such a tool/tools.

====IV. '''Discussion'''====
These results qunatiatively support our hypothesis that there is no clearly dominant set of metawork strategies employed by information workers. This highly fragemented landscape is surprising, even though most information workers work in a similar environment - at a desk, on the phone, in meetings - and with the same types of tools - computers, pens, paper, etc. We believe that this suggests that there are complex tradeoffs between these methods and that no single method is sufficient. We therefore believe that users will be better supported with a new set of software-based metawork tools.

= [Criticisms] =
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

''Any criticisms or questions we have regarding the proposal can go here.''

=References=
<references/>

CS295J/Research proposal (draft 2)

2009-04-24T16:20:26Z

Trevor O'Brien: /* Generalizing User Traces */

= Introduction =
* Owners: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]], [[User: Trevor O'Brien | Trevor]]

'''Propose:''' The design, application and evaluation of a novel, cognition-based, computational framework for assessing interface design and providing automated suggestions to optimize usability.

'''Evaluation Methodology:''' Our techniques will be evaluated quantitatively through a series of user-study trials, as well as qualitatively by a team of expert interface designers.

'''Contributions and Significance:''' We expect this work to make the following contributions:
# design-space analysis and quantitative evaluation of cognition-based techniques for assessing user interfaces.
# design and quantitative evaluation of techniques for suggesting optimized interface-design changes.
# an extensible, multimodal software architecture for capturing user traces integrated with pupil-tracking data, auditory recognition, and muscle-activity monitoring.
# specification (language?) of how to define an interface evaluation module and how to integrate it into a larger system.
# (there may be more here, like testing different cognitive models, generating a markup language to represent interfaces, maybe even a unique metric space for interface usability)

--

We propose a framework for interface evaluation and recommendation that integrates behavioral models and design guidelines from both cognitive science and HCI. Our framework behaves like a committee of specialized experts, where each expert provides its own assessment of the interface, given its particular knowledge of HCI or cognitive science. For example, an expert may provide an evaluation based on the GOMS method, Fitts's law, Maeda's design principles, or cognitive models of learning and memory. An aggregator collects all of these assessments and weights the opinions of each expert, and outputs to the developer a merged evaluation score and a weighted set of recommendations.

Systematic methods of estimating human performance with computer interfaces are used only sparsely despite their obvious benefits, the reason being the overhead involved in implementing them. In order to test an interface, both manual coding systems like the GOMS variations and user simulations like those based on ACT-R/PM and EPIC require detailed pseudo-code descriptions of the user workflow with the application interface. Any change to the interface then requires extensive changes to the pseudo-code, a major problem because of the trial-and-error nature of interface design. Updating the models themselves is even more complicated. Even an expert in CPM-GOMS, for example, can't necessarily adapt it to take into account results from new cognitive research.

Our proposal makes automatic interface evaluation easier to use in several ways. First of all, we propose to divide the input to the system into three separate parts, functionality, user traces and interface. By separating the functionality from the interface, even radical interface changes will require updating only that part of the input. The user traces are also defined over the functionality so that they too translate across different interfaces. Second, the parallel modular architecture allows for a lower "entry cost" for using the tool. The system includes a broad array of evaluation modules some of which are very simple and some more complex. The simpler modules use only a subset of the input that a system like GOMS or ACT-R would require. This means that while more input will still lead to better output, interface designers can get minimal evaluations with only minimal information. For example, a visual search module may not require any functionality or user traces in order to determine whether all interface elements are distinct enough to be easy to find. Finally, a parallel modular architecture is much easier to augment with relevant cognitive and design evaluations.

= Background / Related Work =

Each person should add the background related to their specific aims.

* Steven Ellis - Cognitive models of HCI, including GOMS variations and ACT-R
* EJ - Design Guidelines
* Jon - Perception and Action
* Andrew - Multiple task environments
* Gideon - Cognition and dual systems
* Ian - Interface design process
* Trevor - User trace collection methods (especially any eye-tracking, EEG, ... you want to suggest using)

== Cognitive Models ==
I plan to port over most of the background on cognitive models of HCI from the old proposal

Additions will comprise of:
*CPM-GOMS as a bridge from GOMS architecture to the promising procedural optimization of the Model Human Processor
**Context of CPM development, discuss its relation to original GOMS and KLM
***Establish the tasks which were relevant for optimization when CPM was developed and note that its obsolescence may have been unavoidable
**Focus on CPM as the first step in transitioning from descriptive data, provided by mounting efforts in the cognitive sciences realm to discover the nature of task processing and accomplishment, to prescriptive algorithms which can predict an interface’s efficiency and suggest improvements
**CPM’s purpose as an abstraction of cognitive processing – a symbolic representation not designed for accuracy but precision
**CPM’s successful trials, e.g. Ernestine
***Implications of this project include CPM’s ability to accurately estimate processing at a psychomotor level
***Project does suggest limitations, however, when one attempts to examine more complex tasks which involve deeper and more numerous cognitive processes
*ACT-R as an example of a progressive cognitive modeling tool
**A tool clearly built by and for cognitive scientists, and as a result presents a much more accurate view of human processing – helpful for our research
**Built-in automation, which now seems to be a standard feature of cognitive modeling tools
**Still an abstraction of cognitive processing, but makes adaptation to cutting-edge cognitive research findings an integral aspect of its modular structure
**Expand on its focus on multi-tasking, taking what was a huge advance between GOMS and its CPM variation and bringing the simulation several steps closer to approximating the nature of cognition in regards to HCI
**Far more accessible both for researchers and the lay user/designer in its portability to LISP, pre-construction of modules representing cognitive capacities and underlying algorithms modeling paths of cognitive processing

==Design guidelines==
A multitude of rule sets exist for the design of not only interface, but architecture, city planning, and software development. They can range in scale from one primary rule to as many Christopher Alexander's 253 rules for urban environments,<ref>[http://hci.rwth-aachen.de/materials/publications/borchers2000a.pdf Borchers, Jan O. "A Pattern Approach to Interaction Design." 2000.]</ref> which he introduced with the concept design patterns in the 1970s. Study has likewise been conducted on the use of these rules:<ref>http://stl.cs.queensu.ca/~graham/cisc836/lectures/readings/tetzlaff-guidelines.pdf</ref> guidelines are often only partially understood, indistinct to the developer, and "fraught" with potential usability problems given a real-world situation.

===Application to AUE===
And yet, the vast majority of guideline sets, including the most popular rulesets, have been arrived at heuristically. The most successful, such as Raskin's and Schneiderman's, have been forged from years of observation instead of empirical study and experimentation. The problem is similar to the problem of circular logic faced by automated usability evaluations: an automated system is limited in the suggestions it can offer to a set of preprogrammed guidelines which have often not been subjected to rigorous experimentation.<ref>[http://www.eecs.berkeley.edu/Pubs/TechRpts/2000/CSD-00-1105.pdf Ivory, M and Hearst, M. "The State of the Art in Automated Usability Evaluation of User Interfaces." ACM Computing Surveys (CSUR), 2001.]</ref> In the vast majority of existing studies, emphasis has traditionally been placed on either the development of guidelines or the application of existing guidelines to automated evaluation. A mutually-reinforcing development of both simultaneously has not been attempted.

Overlap between rulesets is inevitable and unavoidable. For our purposes of evaluating existing rulesets efficiently, without extracting and analyzing each rule individually, it may be desirable to identify the the overarching ''principles'' or ''philosophy'' (max. 2 or 3) for a given ruleset and determining their quantitative relevance to problems of cognition.

===Popular and seminal examples===
Schneiderman's [http://faculty.washington.edu/jtenenbg/courses/360/f04/sessions/schneidermanGoldenRules.html Eight Golden Rules] date to 1987 and are arguably the most-cited. They are heuristic, but can be somewhat classified by cognitive objective: at least two rules apply primarily to ''repeated use'', versus ''discoverability''. Up to five of Schneiderman's rules emphasize ''predictability'' in the outcomes of operations and ''increased feedback and control'' in the agency of the user. His final rule, paradoxically, removes control from the user by suggesting a reduced short-term memory load, which we can arguably classify as ''simplicity''.

Raskin's [http://www.mprove.de/script/02/raskin/designrules.html Design Rules] are classified into five principles by the author, augmented by definitions and supporting rules. While one principle is primarily aesthetic (a design problem arguably out of the bounds of this proposal) and one is a basic endorsement of testing, the remaining three begin to reflect philosophies similar to Schneiderman's: reliability or ''predictability'', ''simplicity'' or ''efficiency'' (which we can construe as two sides of the same coin), and finally he introduces a concept of ''uninterruptibility''.

Maeda's [http://lawsofsimplicity.com/?cat=5&order=ASC Laws of Simplicity] are fewer, and ostensibly emphasize ''simplicity'' exclusively, although elements of ''use'' as related by Schneiderman's rules and ''efficiency'' as defined by Raskin may be facets of this simplicity. Google's corporate mission statement presents [http://www.google.com/corporate/ux.html Ten Principles], only half of which can be considered true interface guidelines. ''Efficiency'' and ''simplicity'' are cited explicitly, aesthetics are once again noted as crucial, and working within a user's trust is another application of ''predictability''.

===Elements and goals of a guideline set===
Myriad rulesets exist, but variation becomes scarce—it indeed seems possible to parse these common rulesets into overarching principles that can be converted to or associated with quantifiable cognitive properties. For example, it is likely ''simplicity'' has an analogue in the short-term memory retention or visual retention of the user, vis a vis the rule of [http://books.google.com/books?hl=en&lr=&id=j5q0VvOGExYC&oi=fnd&pg=PA357&dq=seven+plus+or+minus+two&ots=prI3PKJBar&sig=vOZnqpnkXKGYWxK6_XlA4I_CRyI Seven, Plus or Minus Two]. ''Predictability'' likewise may have an analogue in Activity Theory, in regards to a user's perceptual expectations for a given action; ''uninterruptibility'' has implications in cognitive task-switching;<ref>[http://portal.acm.org/citation.cfm?id=985692.985715&coll=Portal&dl=ACM&CFID=21136843&CFTOKEN=23841774 Czerwinski, Horvitz, and White. "A Diary Study of Task Switching and Interruptions." Proceedings of the SIGCHI conference on Human factors in computing systems, 2004.]</ref> and so forth.

Within the scope of this proposal, we aim to reduce and refine these philosophies found in seminal rulesets and identify their logical cognitive analogues. By assigning a quantifiable taxonomy to these principles, we will be able to rank and weight them with regard to their real-world applicability, developing a set of "meta-guidelines" and rules for applying them to a given interface in an automated manner. Combined with cognitive models and multi-modal HCI analysis, we seek to develop, in parallel with these guidelines, the interface evaluation system responsible for their application.

== Perception and Action (in progress) ==

*Information Processing Approach
**Advantages
***Formalism eases translation of theory into scripting language
**Disadvantages
***Assumes symbolic representation

*Ecological (Gibsonian) Approach
**Advantages
***Emphasis on bodily and environmental constraints
**Disadvantages
***Lack of formalism hinders translation of theory into scripting language

= Specific Aims and Contributions (to be separated later) =

'''Specific Aims'''
* Incorporate interaction history mechanisms into a set of existing applications.
* Perform user-study evaluation of history-collection techniques.
* Distill a set of cognitive principles/models, and evaluate empirically?
* Build/buy sensing system to include pupil-tracking, muscle-activity monitoring, auditory recognition.
* Design techniques for manual/semi-automated/automated construction of <insert favorite cognitive model here> from interaction histories and sensing data.
* Design system for posterior analysis of interaction history w.r.t. <insert favorite cognitive model here>, evaluating critical path <or equivalent> trajectories.
* Design cognition-based techniques for detecting bottlenecks in critical paths, and offering optimized alternatives.
* Perform quantitative user-study evaluations, collect qualitative feedback from expert interface designers.

'''Contributions'''

* Design-space analysis and quantitative evaluation of cognition-based techniques for assessing user interfaces.
* Design and quantitative evaluation of techniques for suggesting optimized interface-design changes.
* An extensible, multimodal software architecture for capturing user traces integrated with pupil-tracking data, auditory recognition, and muscle-activity monitoring.
* (there may be more here, like testing different cognitive models, generating a markup language to represent interfaces, maybe even a unique metric space for interface usability)

--

See the [http://vrl.cs.brown.edu/wiki/images/b/b1/Flowchart2.pdf flowchart] for a visual overview of our aims.

In order to use this framework, a designer will have to provide:
* Functional specification - what are the possible interactions between the user and the application. This can be thought of as method signatures, with a name (e.g., setVolume), direction (to user or from user) and a list of value types (boolean, number, text, video, ...) for each interaction.
* GUI specification - a mapping of interactions to interface elements (e.g., setVolume is mapped to the grey knob in the bottom left corner with clockwise turning increasing the input number).
* Functional user traces - sequences of representative ways in which the application is used. Instead of writing them, the designer could have users use the application with a trial interface and then use our methods to generalize the user traces beyond the specific interface (The second method is depicted in the diagram). As a form of pre-processing, the system also generates an interaction transition matrix which lists the probability of each type of interaction given the previous interaction.
* Utility function - this is a weighting of various performance metrics (time, cognitive load, fatigue, etc.), where the weighting expresses the importance of a particular dimension to the user. For example, a user at NASA probably cares more about interface accuracy than speed. By passing this information to our committee of experts, we can create interfaces that are tuned to maximize the utility of a particular user type.

Each of the modules can use all of this information or a subset of it. Our approach stresses flexibility and the ability to give more meaningful feedback the more information is provided. After processing the information sent by the system of experts, the aggregator will output:
* An evaluation of the interface. Evaluations are expressed both in terms of the utility function components (i.e. time, fatigue, cognitive load, etc.), and in terms of the overall utility for this interface (as defined by the utility function). These evaluations are given in the form of an efficiency curve, where the utility received on each dimension can change as the user becomes more accustomed to the interface.
* Suggested improvements for the GUI are also output. These suggestions are meant to optimize the utility function that was input to the system. If a user values accuracy over time, interface suggestions will be made accordingly.

== Recording User-Interaction Primitives ==
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

''Given an interface, our first step is to run users on the interface and log these user interactions. We want to log actions at a sufficiently low level so that a GOMS model can be generated from the data. When possible, we'd also like to log data using additional sensing technologies, such as pupil-tracking, muscle-activity monitoring and auditory recognition; this information will help to analyze the explicit contributions of perception, cognition and motor skills with respect to user performance.''

== Semantic-level Interaction Chunking ==
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

''The user traces that are collected are tied to a specific interface. In order to use them with different interfaces to the same application, they should be generalized to be based only on the functional description of the application and the user's goal hierarchy. This would abstract away from actions like accessing a menu.''

''In addition to specific user traces, many modules could use a transition probability matrix based on interaction predictions.''

== Parallel Framework for Evaluation Modules ==
* Owner: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]]

''This section will describe in more detail the inputs, outputs and architecture that were presented in the introduction.''

==== Module Inputs (Incomplete) ====
* A set of utility dimensions {d1, d2, ...} are defined in the framework. These could be {d1="time", d2="fatigue", ...}
* A set of interaction functions. These specify all of the information that the application wants to give users or get from them. It is not tied to a specific interface. For example, the fact that an applications shows videos would be included here. Whether it displays them embedded, in a pop-up window or fullscreen would not.
* A mapping of interaction functions to interface elements (e.g., buttons, windows, dials,...). Lots of optional information describing visual properties, associated text, physical interactions (e.g., turning the dial clockwise increases the input value) and timing.
* Functional user traces - sequences of interaction functions that represent typical user interactions with the application. Could include a goal hierarchy, in which case the function sequence is at the bottom of the hierarchy.

==== Module Outputs ====
* Every module ouputs at least one of the following:
** An evaluation of the interface
*** This can be on any or all of the utility dimensions, e.g. evaluation={d1=score1, d2=score2, ...}
*** This can alternately be an overall evaluation, ignoring dimensions, e.g. evaluation={score}
**** In this case, the aggregator will treat this as the module giving the same score to all dimensions. Which dimension this evaluator actually predicts well on can be learned by the aggregator over time.
** Recommendation(s) for improving the interface
***This can be a textual description of what changes the designer should make
***This can alternately be a transformation that can automatically be applied to the interface language (without designer intervention)
***In addition to the textual or transformational description of the recommendation, a "change in evaluation" is output to describe how specifically the value will improve the interface
****Recommendation = {description="make this change", Δevaluation={d1=score1, d2=score2, ...}
****Like before, this Δevaluation can cover any number of dimensions, or it can be generic.
***Either a single recommendation or a set of recommendations can be output

==== Aggregator Inputs ====
The aggregator receives as input the outputs of all the modules.

==== Aggregator Outputs ====

Outputs for the aggregator are the same as the outputs for each module. The difference is that the aggregator will consider all the module outputs, and arrive at a merged output based on the past performance of each of the modules.

== Evaluation and Recommendation via Modules ==
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

A "meta-module" called the aggregator will be responsible for assembling and formatting the output of all other modules into a structure that is both extensible and immediately usable, by both an automated designer or a human designer.

===Requirements===
The aggregator's functionality, then, is defined by its '''inputs''', the outputs of the other modules, and the desired '''output''' of the system as a whole, per its position in the architecture. Its purpose is largely formatting and reconciliation of the products of the multitudinous (and extensible) modules. The output of the aggregator must meet several requirements: first, to generate a set of human-readable suggestions for the improvement of the given interface; second, to generate a machine-readable, but also analyzable, evaluation of the various characteristics of the interface and accompanying user traces.

From these specifications, it is logical to assume that a common language or format will be required for the output of individual modules. We propose an XML-based file format, allowing: (1) a section for the standardized identification of problem areas, applicable rules, and proposed improvements, generalized by the individual module and mapped to a single element, or group of elements, in the original interface specification; (2) a section for specification of generalizable "utility" functions, allowing a module to specify how much a measurable quantity of utility is positively or negatively affected by properties of the input interface; (3) new, user-definable sections for evaluations of the given interface not covered by the first two sections. The first two sections are capable of conveying the vast majority of module outputs predicted at this time, but the XML can extensibly allow modules to pass on whatever information may become prominent in the future.

===Specification===
<module id="Fitts-Law">
<interface-elements>
<element>
<desc>submit button</desc>
<problem>
<desc>size</desc>
<suggestion>width *= 2</suggestion>
<suggetsion>height *= 2</suggestions>
<human-suggestion>Increase size relative to other elements</human-suggestion>
</problem>
</element>
</interface-elements>
<utility>
<dimension>
<desc>time</desc>
<value>0:15:35</value>
</dimension>
<dimension>
<desc>frustration</desc>
<value>pulling hair out</value>
</dimension>
<dimension>
<desc>efficiency</desc>
<value>13.2s/KPM task</value>
<value>0.56m/CPM task</value>
</dimension>
</utility>
<tasks>
<task>
<desc>complete form</desc>
</task>
<task>
<desc>lookup SSN</desc>
</task>
<task>
<desc>format phone number</desc>
</task>
</tasks>
</module>

===Logic===
This file provided by each module is then the input for the aggregator. The aggregator's most straightforward function is the compilation of the "problem areas," assembling them and noting problem areas and suggestions that are recommended by more than one module, and weighting them accordingly in its final report. These weightings can begin in an equal state, but the aggregator should be capable of learning iteratively which modules' results are most relevant to the user and update weightings accordingly. This may need to be accomplished with manual tuning, or a machine-learning algorithm capable of determining which modules most often agree with others.

Secondly, the aggregator compiles the utility functions provided by the module specs. This, again, is a summation of similarly-described values from the various modules.

When confronted with user-defined sections of the XML spec, the aggregator is primarily responsible for compiling them and sending them along to the output of the machine. Even if the aggregator does not recognize a section or property of the evaluative spec, if it sees the property reported by more than one module it should be capable of aggregating these intelligently. In future versions of the spec, it should be possible for a module to provide instructions for the aggregator on how to handle unrecognized sections of the XML.

From these compilations, then, the aggregator should be capable of outputting both aggregated human-readable suggestions on interface improvements for a human designer, as well as a comprehensive evaluation of the interface's effectiveness at the given task traces. Again, this is dependent on the specification of the system as a whole, but is likely to include measures and comparisons, graphings of task versus utility, and quantitative measures of an element's effectiveness.

''This section is necessarily defined by the output of the individual modules (which I already expect to be of varied and arbitrary structure) and the desired output of the machine as a whole. It will likely need to be revised heavily after other modules and the "Parellel Framework" section are defined.'' [[User:E J Kalafarski|E J Kalafarski]] 12:34, 24 April 2009 (UTC)

''This section describes the aggregator, which takes the output of multiple independent modules and aggregates the results to provide (1) an evaluation and (2) recommendations for the user interface. We should explain how the aggregator weights the output of different modules (this could be based on historical performance of each module, or perhaps based on E.J.'s cognitive/HCI guidelines).''

== Sample Modules ==

=== CPM-GOMS ===
* Owners: [[User:Steven Ellis | Steven Ellis]]

''This module will provide interface evaluations and suggestions based on a CPM-GOMS model of cognition for the given interface. It will provide a quantitative, predictive, cognition-based parameterization of usability. From empirically collected data, user trajectories through the model (critical paths) will be examined, highlighting bottlenecks within the interface, and offering suggested alterations to the interface to induce more optimal user trajectories.''

I’m hoping to have some input on this section, because it seems to be the crux of the “black box” into which we take the inputs of interface description, user traces, etc. and get our outputs (time, recommendations, etc.). I know at least a few people have pretty strong thoughts on the matter and we ought to discuss the final structure.

That said – my proposal for the module:
*In my opinion the concept of the Model Human Processor (at least as applied in CPM) is outdated – it’s too economic/overly parsimonious in its conception of human activity. I think we need to create a structure which accounts for more realistic conditions of HCI including multitasking, aspects of distributed cognition (and other relevant uses of tools – as far as I can tell CPM doesn’t take into account any sort of productivity aids), executive control processes of attention, etc. ACT-R appears to take steps towards this but we would probably need to look at their algorithms to know for sure.
*Critical paths will continue to play an important role – we should in fact emphasize that part of this tool’s purpose will be a description not only of ways in which the interface should be modified to best fit a critical path, but also ways in which the user’s ought to be instructed in their use of the path. This feedback mechanism could be bidirectional – if the model’s predictions of the user’s goals are incorrect, the critical path determined will also be incorrect and the interface inherently suboptimal. The user could be prompted with a tooltip explaining in brief why and how the interface has changed, along with options to revert, select other configurations (euphemized by goals), and to view a short video detailing how to properly use the interface.
*Call me crazy but, if we assume designers will be willing to code a model of their interfaces into our ACT-R-esque language, could we allow that model to be fairly transparent to the user, who could use a gui to input their goals to find an analogue in the program which would subsequently rearrange its interface to fit the user’s needs? Even if not useful to the users, such dynamic modeling could really help designers (IMO)
*I think the model should do its best to accept models written for ACT-R and whatever other cognitive models there are out there – gives us the best chance of early adoption
*I would particularly appreciate input on the number/complexity/type of inputs we’ll be using, as well as the same qualities for the output.

=== HCI Guidelines ===
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

Schneiderman's Eight Golden Rules and Jakob Nielsen's Ten Heuristics are perhaps the most famous and well-regarded heuristic design guidelines to emerge over the last twenty years. Although the explicit theoretical basis for such heuristics is controversial and not well-explored, the empirical success of these guidelines is established and accepted. This module will parse out up to three or four common (that is, intersecting) principles from these accepted guidelines and apply them to the input interface.

As an example, we identify an analogous principle that appears in Schneiderman ("Reduce short-term memory load")<ref>http://faculty.washington.edu/jtenenbg/courses/360/f04/sessions/schneidermanGoldenRules.html</ref> and Nielsen ("Recognition rather than recall/Minimize the user's memory load")<ref>http://www.useit.com/papers/heuristic/heuristic_list.html</ref>. The input interface is then evaluated for the consideration of the principle, based on an explicit formal description of the interface, such as XAML or XUL. The module attempts to determine how effectively the interface demonstrates the principle. When analyzing an interface for several principles that may be conflicting or opposing in a given context, the module makes use of a hard-coded but iterative (and evolving) weighting of these principles, based on (1) how often they appear in the training set of accepted sets of guidelines, (2) how analogues a heuristic principle is to a cognitive principle in a parallel training set, and (3) how effective the principle's associated suggestion is found to be using a feedback mechanism.

'''Inputs''' 
# A formal description of the interface and its elements (e.g. buttons).
# A formal description of a particular task and the possible paths through a subset of interface elements that permit the user to accomplish that task.

'''Output''' 
Standard XML-formatted file containing problem areas of the input interface, suggestions for each problem area based on principles that were found to have a strong application to a problem element and the problem itself, and a human-readable generated analysis of the element's affinity for the principle. Quantitative outputs will not be possible based on heuristic guidelines, and the "utility" section of this module's output is likely to be blank.

''This section could include an example or two of established design guidelines that could easily be implemented as modules.''

=== Fitts's Law ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

''This module provides an estimate of the required time to complete various tasks that have been decomposed into formalized sequences of interactions with interface elements, and will provide evaluations and recommendations for optimizing the time required to complete those tasks using the interface.''

'''Inputs'''

1. A formal description of the interface and its elements (e.g. buttons).

2. A formal description of a particular task and the possible paths through a subset of interface elements that permit the user to accomplish that task.

3. The physical distances between interface elements along those paths.

4. The width of those elements along the most likely axes of motion.

5. Device (e.g. mouse) characteristics including start/stop time and the inherent speed limitations of the device.

'''Output'''

The module will then use the Shannon formulation of Fitt's Law to compute the average time needed to complete the task along those paths.

=== Affordances ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

''This simple module will provide interface evaluations and recommendations based on a measure of the extent to which the user perceives the relevant affordances of the interface when performing a number of specified tasks.''

'''Inputs'''

Formalized descriptions of...

1. Interface elements

2. Their associated actions

3. The functions of those actions

4. A particular task

5. User traces for that task.

Inputs (1-4) are then used to generate a "user-independent" space of possible functions that the interface is capable of performing with respect to a given task -- what the interface "affords" the user. From this set of possible interactions, our model will then determine the subset of optimal paths for performing a particular task. The user trace (5) is then used to determine what functions actually were performed in the course of a given task of interest and this information is then compared to the optimal path data to determine the extent to which affordances of the interface are present but not perceived.

'''Output'''

The output of this module is a simple ratio of (affordances perceived) / [(relevant affordances present) * (time to complete task)] which provides a quantitative measure of the extent to which the interface is "natural" to use for a particular task.

=== Workflow, Multi-tasking and Interruptions ===
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

There are, at least, two levels at which users work ([http://portal.acm.org/citation.cfm?id=985692.985707&coll=portal&dl=ACM&CFID=20781736&CFTOKEN=83176621 Gonzales, et al., 2004]). Users accomplish individual low-level tasks which are part of larger ''working spheres''; for example, an office worker might send several emails, create several Post-It (TM) note reminders, and then edit a word document, each of these smaller tasks being part of a single larger working sphere of "adding a new section to the website." Thus, it is important to understand this larger workflow context - which often involves extensive levels of multi-tasking, as well as switching between a variety of computing devices and traditional tools, such as notebooks. In this study it was found that the information workers surveyed typically switch individual tasks every 2 minutes and have many simultaneous working spheres which they switch between, on average every 12 minutes. This frenzied pace of switching tasks and switching working spheres suggests that users will not be using a single application or device for a long period of time, and that affordances to support this characteristic pattern of information work are important.

The purpose of this module is to integrate existing work on multi-tasking, interruption and higher-level workflow into a framework which can predict user recovery times from interruptions. Specifically, the goals of this framework will be to:

* Understand the role of the larger workflow context in user interfaces
* Understand the impact of interruptions on user workflow
* Understand how to design software which fits into the larger working spheres in which information work takes place

It is important to point out that because workflow and multi-tasking rely heavily on higher-level brain functioning, it is unrealistic within the scope of this grant to propose a system which can predict user performance given a description of a set of arbitrary software programs. Therefore, we believe this module will function much more in a qualitative role to provide context to the rest of the model. Specifically, our findings related to interruption and multi-tasking will advance the basic research question of "how do you users react to interruptions when using working sets of varying sizes?". This core HCI contribution will help to inform the rest of the outputs of the model in a qualitative manner.

'''Inputs'''

N/A

'''Outputs'''

N/A

=== Working Memory Load ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''This module measures how much information the user needs to retain in memory while interacting with the interface and makes suggestions for improvements.''

* Models
** Beddeley's Model of Working Memory<ref>http://en.wikipedia.org/wiki/Baddeley%27s_model_of_working_memory</ref>
*** Episodic Buffer
** George Miller's "The magic number 7 plus or minus 2"<ref>http://en.wikipedia.org/wiki/The_Magical_Number_Seven,_Plus_or_Minus_Two</ref>
** The magic number 4<ref>Cowan, N. (2001). The magical number 4 in short-term memory: A reconsideration of mental storage capacity. Behavioral and Brain Sciences, 24, 87-185.</ref>
* Chunking<ref>http://en.wikipedia.org/wiki/Chunking_(psychology)</ref>
* Priming<ref>http://en.wikipedia.org/wiki/Priming_(psychology)</ref>
* Subitizing and Counting<ref>http://en.wikipedia.org/wiki/Subitizing</ref>

'''Inputs'''
* Visual Stimuli
* Audio Stimuli
'''Outputs'''
* Remembered percepts
* Half-Life of percepts

=== Automaticity of Interaction ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''Measures how easily the interaction with the interface becomes automatic with experience and makes suggestions for improvements.''

* Learning<ref>http://en.wikipedia.org/wiki/Learning#Mathematical_models_of_learning</ref>
** Logan's instance theory of automatization<ref>http://74.125.95.132/search?q=cache:IZ-Zccsu3SEJ:psych.wisc.edu/ugstudies/psych733/logan_1988.pdf+logan+isntance+teory&cd=1&hl=en&ct=clnk&gl=us&client=firefox-a</ref>
* Fluency
** As meta-cognitive information <ref>http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6VH9-4SM7PFK-4&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=10cd279fa80958981fcc3c06684c09af</ref>
** As a cognitive ''heuristic''<ref>http://en.wikipedia.org/wiki/Fluency_heuristic</ref>

'''Inputs'''
* Interface
* User goals

'''Outputs'''
* Learning curve

== Integration into the Design Process ==
* Owner: [[User:Ian Spector | Ian Spector]]

''This section outlines the process of designing an HCI interface and at what stages our proposal fits in and how.''

= Preliminary Results =

===Workflow, Multi-tasking, and Interruption===

====I. '''Goals'''====

The goals of the preliminary work are to gain qualitative insight into how information workers practice metawork, and to determine whether people might be better-supported with software which facillitates metawork and interruptions. Thus, the preliminary work should investigate, and demonstrate, the need and impact of the core goals of the project.

====II. '''Methodology'''====

Seven information workers, ages 20-38 (5 male, 2 female), were interviewed to determine which methods they use to "stay organized". An initial list of metawork strategies was established from two pilot interviews, and then a final list was compiled. Participants then responded to a series of 17 questions designed to gain insight into their metawork strategies and process. In addition, verbal interviews were conducted to get additional open-ended feedback.

====III. '''Final Results'''====
A histogram of methods people use to "stay organized" in terms of tracking things they need to do (TODOs), appointments and meetings, etc. is shown in the figure below.

[[Image:AcbGraph.jpg]]

In addition to these methods, participants also used a number of other methods, including:

* iCal
* Notes written in xterms
* "Inbox zero" method of email organization
* iGoogle Notepad (for tasks)
* Tag emails as "TODO", "Important", etc.
* Things (Organizer Software)
* Physical items placed to "remind me of things"
* Sometimes arranging windows on desk
* Keeping browser tabs open
* Bookmarking web pages
* Keep programs/files open scrolled to certain locations sometimes with things selected

In addition, three participants said that when interrupted they "rarely" or "very rarely" were able to resume the task they were working on prior to the interruption. Three of the participants said that they would not actively recommend their metawork strategies for other people, and two said that staying organized was "difficult".

Four participants were neutral to the idea of new tools to help them stay organized and three said that they would like to have such a tool/tools.

====IV. '''Discussion'''====
These results qunatiatively support our hypothesis that there is no clearly dominant set of metawork strategies employed by information workers. This highly fragemented landscape is surprising, even though most information workers work in a similar environment - at a desk, on the phone, in meetings - and with the same types of tools - computers, pens, paper, etc. We believe that this suggests that there are complex tradeoffs between these methods and that no single method is sufficient. We therefore believe that users will be better supported with a new set of software-based metawork tools.

= [Criticisms] =
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

''Any criticisms or questions we have regarding the proposal can go here.''

=References=
<references/>

CS295J/Research proposal (draft 2)

2009-04-24T16:19:33Z

Trevor O'Brien: /* Collecting User Traces */

= Introduction =
* Owners: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]], [[User: Trevor O'Brien | Trevor]]

'''Propose:''' The design, application and evaluation of a novel, cognition-based, computational framework for assessing interface design and providing automated suggestions to optimize usability.

'''Evaluation Methodology:''' Our techniques will be evaluated quantitatively through a series of user-study trials, as well as qualitatively by a team of expert interface designers.

'''Contributions and Significance:''' We expect this work to make the following contributions:
# design-space analysis and quantitative evaluation of cognition-based techniques for assessing user interfaces.
# design and quantitative evaluation of techniques for suggesting optimized interface-design changes.
# an extensible, multimodal software architecture for capturing user traces integrated with pupil-tracking data, auditory recognition, and muscle-activity monitoring.
# specification (language?) of how to define an interface evaluation module and how to integrate it into a larger system.
# (there may be more here, like testing different cognitive models, generating a markup language to represent interfaces, maybe even a unique metric space for interface usability)

--

We propose a framework for interface evaluation and recommendation that integrates behavioral models and design guidelines from both cognitive science and HCI. Our framework behaves like a committee of specialized experts, where each expert provides its own assessment of the interface, given its particular knowledge of HCI or cognitive science. For example, an expert may provide an evaluation based on the GOMS method, Fitts's law, Maeda's design principles, or cognitive models of learning and memory. An aggregator collects all of these assessments and weights the opinions of each expert, and outputs to the developer a merged evaluation score and a weighted set of recommendations.

Systematic methods of estimating human performance with computer interfaces are used only sparsely despite their obvious benefits, the reason being the overhead involved in implementing them. In order to test an interface, both manual coding systems like the GOMS variations and user simulations like those based on ACT-R/PM and EPIC require detailed pseudo-code descriptions of the user workflow with the application interface. Any change to the interface then requires extensive changes to the pseudo-code, a major problem because of the trial-and-error nature of interface design. Updating the models themselves is even more complicated. Even an expert in CPM-GOMS, for example, can't necessarily adapt it to take into account results from new cognitive research.

Our proposal makes automatic interface evaluation easier to use in several ways. First of all, we propose to divide the input to the system into three separate parts, functionality, user traces and interface. By separating the functionality from the interface, even radical interface changes will require updating only that part of the input. The user traces are also defined over the functionality so that they too translate across different interfaces. Second, the parallel modular architecture allows for a lower "entry cost" for using the tool. The system includes a broad array of evaluation modules some of which are very simple and some more complex. The simpler modules use only a subset of the input that a system like GOMS or ACT-R would require. This means that while more input will still lead to better output, interface designers can get minimal evaluations with only minimal information. For example, a visual search module may not require any functionality or user traces in order to determine whether all interface elements are distinct enough to be easy to find. Finally, a parallel modular architecture is much easier to augment with relevant cognitive and design evaluations.

= Background / Related Work =

Each person should add the background related to their specific aims.

* Steven Ellis - Cognitive models of HCI, including GOMS variations and ACT-R
* EJ - Design Guidelines
* Jon - Perception and Action
* Andrew - Multiple task environments
* Gideon - Cognition and dual systems
* Ian - Interface design process
* Trevor - User trace collection methods (especially any eye-tracking, EEG, ... you want to suggest using)

== Cognitive Models ==
I plan to port over most of the background on cognitive models of HCI from the old proposal

Additions will comprise of:
*CPM-GOMS as a bridge from GOMS architecture to the promising procedural optimization of the Model Human Processor
**Context of CPM development, discuss its relation to original GOMS and KLM
***Establish the tasks which were relevant for optimization when CPM was developed and note that its obsolescence may have been unavoidable
**Focus on CPM as the first step in transitioning from descriptive data, provided by mounting efforts in the cognitive sciences realm to discover the nature of task processing and accomplishment, to prescriptive algorithms which can predict an interface’s efficiency and suggest improvements
**CPM’s purpose as an abstraction of cognitive processing – a symbolic representation not designed for accuracy but precision
**CPM’s successful trials, e.g. Ernestine
***Implications of this project include CPM’s ability to accurately estimate processing at a psychomotor level
***Project does suggest limitations, however, when one attempts to examine more complex tasks which involve deeper and more numerous cognitive processes
*ACT-R as an example of a progressive cognitive modeling tool
**A tool clearly built by and for cognitive scientists, and as a result presents a much more accurate view of human processing – helpful for our research
**Built-in automation, which now seems to be a standard feature of cognitive modeling tools
**Still an abstraction of cognitive processing, but makes adaptation to cutting-edge cognitive research findings an integral aspect of its modular structure
**Expand on its focus on multi-tasking, taking what was a huge advance between GOMS and its CPM variation and bringing the simulation several steps closer to approximating the nature of cognition in regards to HCI
**Far more accessible both for researchers and the lay user/designer in its portability to LISP, pre-construction of modules representing cognitive capacities and underlying algorithms modeling paths of cognitive processing

==Design guidelines==
A multitude of rule sets exist for the design of not only interface, but architecture, city planning, and software development. They can range in scale from one primary rule to as many Christopher Alexander's 253 rules for urban environments,<ref>[http://hci.rwth-aachen.de/materials/publications/borchers2000a.pdf Borchers, Jan O. "A Pattern Approach to Interaction Design." 2000.]</ref> which he introduced with the concept design patterns in the 1970s. Study has likewise been conducted on the use of these rules:<ref>http://stl.cs.queensu.ca/~graham/cisc836/lectures/readings/tetzlaff-guidelines.pdf</ref> guidelines are often only partially understood, indistinct to the developer, and "fraught" with potential usability problems given a real-world situation.

===Application to AUE===
And yet, the vast majority of guideline sets, including the most popular rulesets, have been arrived at heuristically. The most successful, such as Raskin's and Schneiderman's, have been forged from years of observation instead of empirical study and experimentation. The problem is similar to the problem of circular logic faced by automated usability evaluations: an automated system is limited in the suggestions it can offer to a set of preprogrammed guidelines which have often not been subjected to rigorous experimentation.<ref>[http://www.eecs.berkeley.edu/Pubs/TechRpts/2000/CSD-00-1105.pdf Ivory, M and Hearst, M. "The State of the Art in Automated Usability Evaluation of User Interfaces." ACM Computing Surveys (CSUR), 2001.]</ref> In the vast majority of existing studies, emphasis has traditionally been placed on either the development of guidelines or the application of existing guidelines to automated evaluation. A mutually-reinforcing development of both simultaneously has not been attempted.

Overlap between rulesets is inevitable and unavoidable. For our purposes of evaluating existing rulesets efficiently, without extracting and analyzing each rule individually, it may be desirable to identify the the overarching ''principles'' or ''philosophy'' (max. 2 or 3) for a given ruleset and determining their quantitative relevance to problems of cognition.

===Popular and seminal examples===
Schneiderman's [http://faculty.washington.edu/jtenenbg/courses/360/f04/sessions/schneidermanGoldenRules.html Eight Golden Rules] date to 1987 and are arguably the most-cited. They are heuristic, but can be somewhat classified by cognitive objective: at least two rules apply primarily to ''repeated use'', versus ''discoverability''. Up to five of Schneiderman's rules emphasize ''predictability'' in the outcomes of operations and ''increased feedback and control'' in the agency of the user. His final rule, paradoxically, removes control from the user by suggesting a reduced short-term memory load, which we can arguably classify as ''simplicity''.

Raskin's [http://www.mprove.de/script/02/raskin/designrules.html Design Rules] are classified into five principles by the author, augmented by definitions and supporting rules. While one principle is primarily aesthetic (a design problem arguably out of the bounds of this proposal) and one is a basic endorsement of testing, the remaining three begin to reflect philosophies similar to Schneiderman's: reliability or ''predictability'', ''simplicity'' or ''efficiency'' (which we can construe as two sides of the same coin), and finally he introduces a concept of ''uninterruptibility''.

Maeda's [http://lawsofsimplicity.com/?cat=5&order=ASC Laws of Simplicity] are fewer, and ostensibly emphasize ''simplicity'' exclusively, although elements of ''use'' as related by Schneiderman's rules and ''efficiency'' as defined by Raskin may be facets of this simplicity. Google's corporate mission statement presents [http://www.google.com/corporate/ux.html Ten Principles], only half of which can be considered true interface guidelines. ''Efficiency'' and ''simplicity'' are cited explicitly, aesthetics are once again noted as crucial, and working within a user's trust is another application of ''predictability''.

===Elements and goals of a guideline set===
Myriad rulesets exist, but variation becomes scarce—it indeed seems possible to parse these common rulesets into overarching principles that can be converted to or associated with quantifiable cognitive properties. For example, it is likely ''simplicity'' has an analogue in the short-term memory retention or visual retention of the user, vis a vis the rule of [http://books.google.com/books?hl=en&lr=&id=j5q0VvOGExYC&oi=fnd&pg=PA357&dq=seven+plus+or+minus+two&ots=prI3PKJBar&sig=vOZnqpnkXKGYWxK6_XlA4I_CRyI Seven, Plus or Minus Two]. ''Predictability'' likewise may have an analogue in Activity Theory, in regards to a user's perceptual expectations for a given action; ''uninterruptibility'' has implications in cognitive task-switching;<ref>[http://portal.acm.org/citation.cfm?id=985692.985715&coll=Portal&dl=ACM&CFID=21136843&CFTOKEN=23841774 Czerwinski, Horvitz, and White. "A Diary Study of Task Switching and Interruptions." Proceedings of the SIGCHI conference on Human factors in computing systems, 2004.]</ref> and so forth.

Within the scope of this proposal, we aim to reduce and refine these philosophies found in seminal rulesets and identify their logical cognitive analogues. By assigning a quantifiable taxonomy to these principles, we will be able to rank and weight them with regard to their real-world applicability, developing a set of "meta-guidelines" and rules for applying them to a given interface in an automated manner. Combined with cognitive models and multi-modal HCI analysis, we seek to develop, in parallel with these guidelines, the interface evaluation system responsible for their application.

== Perception and Action (in progress) ==

*Information Processing Approach
**Advantages
***Formalism eases translation of theory into scripting language
**Disadvantages
***Assumes symbolic representation

*Ecological (Gibsonian) Approach
**Advantages
***Emphasis on bodily and environmental constraints
**Disadvantages
***Lack of formalism hinders translation of theory into scripting language

= Specific Aims and Contributions (to be separated later) =

'''Specific Aims'''
* Incorporate interaction history mechanisms into a set of existing applications.
* Perform user-study evaluation of history-collection techniques.
* Distill a set of cognitive principles/models, and evaluate empirically?
* Build/buy sensing system to include pupil-tracking, muscle-activity monitoring, auditory recognition.
* Design techniques for manual/semi-automated/automated construction of <insert favorite cognitive model here> from interaction histories and sensing data.
* Design system for posterior analysis of interaction history w.r.t. <insert favorite cognitive model here>, evaluating critical path <or equivalent> trajectories.
* Design cognition-based techniques for detecting bottlenecks in critical paths, and offering optimized alternatives.
* Perform quantitative user-study evaluations, collect qualitative feedback from expert interface designers.

'''Contributions'''

* Design-space analysis and quantitative evaluation of cognition-based techniques for assessing user interfaces.
* Design and quantitative evaluation of techniques for suggesting optimized interface-design changes.
* An extensible, multimodal software architecture for capturing user traces integrated with pupil-tracking data, auditory recognition, and muscle-activity monitoring.
* (there may be more here, like testing different cognitive models, generating a markup language to represent interfaces, maybe even a unique metric space for interface usability)

--

See the [http://vrl.cs.brown.edu/wiki/images/b/b1/Flowchart2.pdf flowchart] for a visual overview of our aims.

In order to use this framework, a designer will have to provide:
* Functional specification - what are the possible interactions between the user and the application. This can be thought of as method signatures, with a name (e.g., setVolume), direction (to user or from user) and a list of value types (boolean, number, text, video, ...) for each interaction.
* GUI specification - a mapping of interactions to interface elements (e.g., setVolume is mapped to the grey knob in the bottom left corner with clockwise turning increasing the input number).
* Functional user traces - sequences of representative ways in which the application is used. Instead of writing them, the designer could have users use the application with a trial interface and then use our methods to generalize the user traces beyond the specific interface (The second method is depicted in the diagram). As a form of pre-processing, the system also generates an interaction transition matrix which lists the probability of each type of interaction given the previous interaction.
* Utility function - this is a weighting of various performance metrics (time, cognitive load, fatigue, etc.), where the weighting expresses the importance of a particular dimension to the user. For example, a user at NASA probably cares more about interface accuracy than speed. By passing this information to our committee of experts, we can create interfaces that are tuned to maximize the utility of a particular user type.

Each of the modules can use all of this information or a subset of it. Our approach stresses flexibility and the ability to give more meaningful feedback the more information is provided. After processing the information sent by the system of experts, the aggregator will output:
* An evaluation of the interface. Evaluations are expressed both in terms of the utility function components (i.e. time, fatigue, cognitive load, etc.), and in terms of the overall utility for this interface (as defined by the utility function). These evaluations are given in the form of an efficiency curve, where the utility received on each dimension can change as the user becomes more accustomed to the interface.
* Suggested improvements for the GUI are also output. These suggestions are meant to optimize the utility function that was input to the system. If a user values accuracy over time, interface suggestions will be made accordingly.

== Recording User-Interaction Primitives ==
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

''Given an interface, our first step is to run users on the interface and log these user interactions. We want to log actions at a sufficiently low level so that a GOMS model can be generated from the data. When possible, we'd also like to log data using additional sensing technologies, such as pupil-tracking, muscle-activity monitoring and auditory recognition; this information will help to analyze the explicit contributions of perception, cognition and motor skills with respect to user performance.''

== Generalizing User Traces ==
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

''The user traces that are collected are tied to a specific interface. In order to use them with different interfaces to the same application, they should be generalized to be based only on the functional description of the application and the user's goal hierarchy. This would abstract away from actions like accessing a menu.''

''In addition to specific user traces, many modules could use a transition probability matrix based on interaction predictions.''

== Parallel Framework for Evaluation Modules ==
* Owner: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]]

''This section will describe in more detail the inputs, outputs and architecture that were presented in the introduction.''

==== Module Inputs (Incomplete) ====
* A set of utility dimensions {d1, d2, ...} are defined in the framework. These could be {d1="time", d2="fatigue", ...}
* A set of interaction functions. These specify all of the information that the application wants to give users or get from them. It is not tied to a specific interface. For example, the fact that an applications shows videos would be included here. Whether it displays them embedded, in a pop-up window or fullscreen would not.
* A mapping of interaction functions to interface elements (e.g., buttons, windows, dials,...). Lots of optional information describing visual properties, associated text, physical interactions (e.g., turning the dial clockwise increases the input value) and timing.
* Functional user traces - sequences of interaction functions that represent typical user interactions with the application. Could include a goal hierarchy, in which case the function sequence is at the bottom of the hierarchy.

==== Module Outputs ====
* Every module ouputs at least one of the following:
** An evaluation of the interface
*** This can be on any or all of the utility dimensions, e.g. evaluation={d1=score1, d2=score2, ...}
*** This can alternately be an overall evaluation, ignoring dimensions, e.g. evaluation={score}
**** In this case, the aggregator will treat this as the module giving the same score to all dimensions. Which dimension this evaluator actually predicts well on can be learned by the aggregator over time.
** Recommendation(s) for improving the interface
***This can be a textual description of what changes the designer should make
***This can alternately be a transformation that can automatically be applied to the interface language (without designer intervention)
***In addition to the textual or transformational description of the recommendation, a "change in evaluation" is output to describe how specifically the value will improve the interface
****Recommendation = {description="make this change", Δevaluation={d1=score1, d2=score2, ...}
****Like before, this Δevaluation can cover any number of dimensions, or it can be generic.
***Either a single recommendation or a set of recommendations can be output

==== Aggregator Inputs ====
The aggregator receives as input the outputs of all the modules.

==== Aggregator Outputs ====

Outputs for the aggregator are the same as the outputs for each module. The difference is that the aggregator will consider all the module outputs, and arrive at a merged output based on the past performance of each of the modules.

== Evaluation and Recommendation via Modules ==
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

A "meta-module" called the aggregator will be responsible for assembling and formatting the output of all other modules into a structure that is both extensible and immediately usable, by both an automated designer or a human designer.

===Requirements===
The aggregator's functionality, then, is defined by its '''inputs''', the outputs of the other modules, and the desired '''output''' of the system as a whole, per its position in the architecture. Its purpose is largely formatting and reconciliation of the products of the multitudinous (and extensible) modules. The output of the aggregator must meet several requirements: first, to generate a set of human-readable suggestions for the improvement of the given interface; second, to generate a machine-readable, but also analyzable, evaluation of the various characteristics of the interface and accompanying user traces.

From these specifications, it is logical to assume that a common language or format will be required for the output of individual modules. We propose an XML-based file format, allowing: (1) a section for the standardized identification of problem areas, applicable rules, and proposed improvements, generalized by the individual module and mapped to a single element, or group of elements, in the original interface specification; (2) a section for specification of generalizable "utility" functions, allowing a module to specify how much a measurable quantity of utility is positively or negatively affected by properties of the input interface; (3) new, user-definable sections for evaluations of the given interface not covered by the first two sections. The first two sections are capable of conveying the vast majority of module outputs predicted at this time, but the XML can extensibly allow modules to pass on whatever information may become prominent in the future.

===Specification===
<module id="Fitts-Law">
<interface-elements>
<element>
<desc>submit button</desc>
<problem>
<desc>size</desc>
<suggestion>width *= 2</suggestion>
<suggetsion>height *= 2</suggestions>
<human-suggestion>Increase size relative to other elements</human-suggestion>
</problem>
</element>
</interface-elements>
<utility>
<dimension>
<desc>time</desc>
<value>0:15:35</value>
</dimension>
<dimension>
<desc>frustration</desc>
<value>pulling hair out</value>
</dimension>
<dimension>
<desc>efficiency</desc>
<value>13.2s/KPM task</value>
<value>0.56m/CPM task</value>
</dimension>
</utility>
<tasks>
<task>
<desc>complete form</desc>
</task>
<task>
<desc>lookup SSN</desc>
</task>
<task>
<desc>format phone number</desc>
</task>
</tasks>
</module>

===Logic===
This file provided by each module is then the input for the aggregator. The aggregator's most straightforward function is the compilation of the "problem areas," assembling them and noting problem areas and suggestions that are recommended by more than one module, and weighting them accordingly in its final report. These weightings can begin in an equal state, but the aggregator should be capable of learning iteratively which modules' results are most relevant to the user and update weightings accordingly. This may need to be accomplished with manual tuning, or a machine-learning algorithm capable of determining which modules most often agree with others.

Secondly, the aggregator compiles the utility functions provided by the module specs. This, again, is a summation of similarly-described values from the various modules.

When confronted with user-defined sections of the XML spec, the aggregator is primarily responsible for compiling them and sending them along to the output of the machine. Even if the aggregator does not recognize a section or property of the evaluative spec, if it sees the property reported by more than one module it should be capable of aggregating these intelligently. In future versions of the spec, it should be possible for a module to provide instructions for the aggregator on how to handle unrecognized sections of the XML.

From these compilations, then, the aggregator should be capable of outputting both aggregated human-readable suggestions on interface improvements for a human designer, as well as a comprehensive evaluation of the interface's effectiveness at the given task traces. Again, this is dependent on the specification of the system as a whole, but is likely to include measures and comparisons, graphings of task versus utility, and quantitative measures of an element's effectiveness.

''This section is necessarily defined by the output of the individual modules (which I already expect to be of varied and arbitrary structure) and the desired output of the machine as a whole. It will likely need to be revised heavily after other modules and the "Parellel Framework" section are defined.'' [[User:E J Kalafarski|E J Kalafarski]] 12:34, 24 April 2009 (UTC)

''This section describes the aggregator, which takes the output of multiple independent modules and aggregates the results to provide (1) an evaluation and (2) recommendations for the user interface. We should explain how the aggregator weights the output of different modules (this could be based on historical performance of each module, or perhaps based on E.J.'s cognitive/HCI guidelines).''

== Sample Modules ==

=== CPM-GOMS ===
* Owners: [[User:Steven Ellis | Steven Ellis]]

''This module will provide interface evaluations and suggestions based on a CPM-GOMS model of cognition for the given interface. It will provide a quantitative, predictive, cognition-based parameterization of usability. From empirically collected data, user trajectories through the model (critical paths) will be examined, highlighting bottlenecks within the interface, and offering suggested alterations to the interface to induce more optimal user trajectories.''

I’m hoping to have some input on this section, because it seems to be the crux of the “black box” into which we take the inputs of interface description, user traces, etc. and get our outputs (time, recommendations, etc.). I know at least a few people have pretty strong thoughts on the matter and we ought to discuss the final structure.

That said – my proposal for the module:
*In my opinion the concept of the Model Human Processor (at least as applied in CPM) is outdated – it’s too economic/overly parsimonious in its conception of human activity. I think we need to create a structure which accounts for more realistic conditions of HCI including multitasking, aspects of distributed cognition (and other relevant uses of tools – as far as I can tell CPM doesn’t take into account any sort of productivity aids), executive control processes of attention, etc. ACT-R appears to take steps towards this but we would probably need to look at their algorithms to know for sure.
*Critical paths will continue to play an important role – we should in fact emphasize that part of this tool’s purpose will be a description not only of ways in which the interface should be modified to best fit a critical path, but also ways in which the user’s ought to be instructed in their use of the path. This feedback mechanism could be bidirectional – if the model’s predictions of the user’s goals are incorrect, the critical path determined will also be incorrect and the interface inherently suboptimal. The user could be prompted with a tooltip explaining in brief why and how the interface has changed, along with options to revert, select other configurations (euphemized by goals), and to view a short video detailing how to properly use the interface.
*Call me crazy but, if we assume designers will be willing to code a model of their interfaces into our ACT-R-esque language, could we allow that model to be fairly transparent to the user, who could use a gui to input their goals to find an analogue in the program which would subsequently rearrange its interface to fit the user’s needs? Even if not useful to the users, such dynamic modeling could really help designers (IMO)
*I think the model should do its best to accept models written for ACT-R and whatever other cognitive models there are out there – gives us the best chance of early adoption
*I would particularly appreciate input on the number/complexity/type of inputs we’ll be using, as well as the same qualities for the output.

=== HCI Guidelines ===
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

Schneiderman's Eight Golden Rules and Jakob Nielsen's Ten Heuristics are perhaps the most famous and well-regarded heuristic design guidelines to emerge over the last twenty years. Although the explicit theoretical basis for such heuristics is controversial and not well-explored, the empirical success of these guidelines is established and accepted. This module will parse out up to three or four common (that is, intersecting) principles from these accepted guidelines and apply them to the input interface.

As an example, we identify an analogous principle that appears in Schneiderman ("Reduce short-term memory load")<ref>http://faculty.washington.edu/jtenenbg/courses/360/f04/sessions/schneidermanGoldenRules.html</ref> and Nielsen ("Recognition rather than recall/Minimize the user's memory load")<ref>http://www.useit.com/papers/heuristic/heuristic_list.html</ref>. The input interface is then evaluated for the consideration of the principle, based on an explicit formal description of the interface, such as XAML or XUL. The module attempts to determine how effectively the interface demonstrates the principle. When analyzing an interface for several principles that may be conflicting or opposing in a given context, the module makes use of a hard-coded but iterative (and evolving) weighting of these principles, based on (1) how often they appear in the training set of accepted sets of guidelines, (2) how analogues a heuristic principle is to a cognitive principle in a parallel training set, and (3) how effective the principle's associated suggestion is found to be using a feedback mechanism.

'''Inputs''' 
# A formal description of the interface and its elements (e.g. buttons).
# A formal description of a particular task and the possible paths through a subset of interface elements that permit the user to accomplish that task.

'''Output''' 
Standard XML-formatted file containing problem areas of the input interface, suggestions for each problem area based on principles that were found to have a strong application to a problem element and the problem itself, and a human-readable generated analysis of the element's affinity for the principle. Quantitative outputs will not be possible based on heuristic guidelines, and the "utility" section of this module's output is likely to be blank.

''This section could include an example or two of established design guidelines that could easily be implemented as modules.''

=== Fitts's Law ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

''This module provides an estimate of the required time to complete various tasks that have been decomposed into formalized sequences of interactions with interface elements, and will provide evaluations and recommendations for optimizing the time required to complete those tasks using the interface.''

'''Inputs'''

1. A formal description of the interface and its elements (e.g. buttons).

2. A formal description of a particular task and the possible paths through a subset of interface elements that permit the user to accomplish that task.

3. The physical distances between interface elements along those paths.

4. The width of those elements along the most likely axes of motion.

5. Device (e.g. mouse) characteristics including start/stop time and the inherent speed limitations of the device.

'''Output'''

The module will then use the Shannon formulation of Fitt's Law to compute the average time needed to complete the task along those paths.

=== Affordances ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

''This simple module will provide interface evaluations and recommendations based on a measure of the extent to which the user perceives the relevant affordances of the interface when performing a number of specified tasks.''

'''Inputs'''

Formalized descriptions of...

1. Interface elements

2. Their associated actions

3. The functions of those actions

4. A particular task

5. User traces for that task.

Inputs (1-4) are then used to generate a "user-independent" space of possible functions that the interface is capable of performing with respect to a given task -- what the interface "affords" the user. From this set of possible interactions, our model will then determine the subset of optimal paths for performing a particular task. The user trace (5) is then used to determine what functions actually were performed in the course of a given task of interest and this information is then compared to the optimal path data to determine the extent to which affordances of the interface are present but not perceived.

'''Output'''

The output of this module is a simple ratio of (affordances perceived) / [(relevant affordances present) * (time to complete task)] which provides a quantitative measure of the extent to which the interface is "natural" to use for a particular task.

=== Workflow, Multi-tasking and Interruptions ===
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

There are, at least, two levels at which users work ([http://portal.acm.org/citation.cfm?id=985692.985707&coll=portal&dl=ACM&CFID=20781736&CFTOKEN=83176621 Gonzales, et al., 2004]). Users accomplish individual low-level tasks which are part of larger ''working spheres''; for example, an office worker might send several emails, create several Post-It (TM) note reminders, and then edit a word document, each of these smaller tasks being part of a single larger working sphere of "adding a new section to the website." Thus, it is important to understand this larger workflow context - which often involves extensive levels of multi-tasking, as well as switching between a variety of computing devices and traditional tools, such as notebooks. In this study it was found that the information workers surveyed typically switch individual tasks every 2 minutes and have many simultaneous working spheres which they switch between, on average every 12 minutes. This frenzied pace of switching tasks and switching working spheres suggests that users will not be using a single application or device for a long period of time, and that affordances to support this characteristic pattern of information work are important.

The purpose of this module is to integrate existing work on multi-tasking, interruption and higher-level workflow into a framework which can predict user recovery times from interruptions. Specifically, the goals of this framework will be to:

* Understand the role of the larger workflow context in user interfaces
* Understand the impact of interruptions on user workflow
* Understand how to design software which fits into the larger working spheres in which information work takes place

It is important to point out that because workflow and multi-tasking rely heavily on higher-level brain functioning, it is unrealistic within the scope of this grant to propose a system which can predict user performance given a description of a set of arbitrary software programs. Therefore, we believe this module will function much more in a qualitative role to provide context to the rest of the model. Specifically, our findings related to interruption and multi-tasking will advance the basic research question of "how do you users react to interruptions when using working sets of varying sizes?". This core HCI contribution will help to inform the rest of the outputs of the model in a qualitative manner.

'''Inputs'''

N/A

'''Outputs'''

N/A

=== Working Memory Load ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''This module measures how much information the user needs to retain in memory while interacting with the interface and makes suggestions for improvements.''

* Models
** Beddeley's Model of Working Memory<ref>http://en.wikipedia.org/wiki/Baddeley%27s_model_of_working_memory</ref>
*** Episodic Buffer
** George Miller's "The magic number 7 plus or minus 2"<ref>http://en.wikipedia.org/wiki/The_Magical_Number_Seven,_Plus_or_Minus_Two</ref>
** The magic number 4<ref>Cowan, N. (2001). The magical number 4 in short-term memory: A reconsideration of mental storage capacity. Behavioral and Brain Sciences, 24, 87-185.</ref>
* Chunking<ref>http://en.wikipedia.org/wiki/Chunking_(psychology)</ref>
* Priming<ref>http://en.wikipedia.org/wiki/Priming_(psychology)</ref>
* Subitizing and Counting<ref>http://en.wikipedia.org/wiki/Subitizing</ref>

'''Inputs'''
* Visual Stimuli
* Audio Stimuli
'''Outputs'''
* Remembered percepts
* Half-Life of percepts

=== Automaticity of Interaction ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''Measures how easily the interaction with the interface becomes automatic with experience and makes suggestions for improvements.''

* Learning<ref>http://en.wikipedia.org/wiki/Learning#Mathematical_models_of_learning</ref>
** Logan's instance theory of automatization<ref>http://74.125.95.132/search?q=cache:IZ-Zccsu3SEJ:psych.wisc.edu/ugstudies/psych733/logan_1988.pdf+logan+isntance+teory&cd=1&hl=en&ct=clnk&gl=us&client=firefox-a</ref>
* Fluency
** As meta-cognitive information <ref>http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6VH9-4SM7PFK-4&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=10cd279fa80958981fcc3c06684c09af</ref>
** As a cognitive ''heuristic''<ref>http://en.wikipedia.org/wiki/Fluency_heuristic</ref>

'''Inputs'''
* Interface
* User goals

'''Outputs'''
* Learning curve

== Integration into the Design Process ==
* Owner: [[User:Ian Spector | Ian Spector]]

''This section outlines the process of designing an HCI interface and at what stages our proposal fits in and how.''

= Preliminary Results =

===Workflow, Multi-tasking, and Interruption===

====I. '''Goals'''====

The goals of the preliminary work are to gain qualitative insight into how information workers practice metawork, and to determine whether people might be better-supported with software which facillitates metawork and interruptions. Thus, the preliminary work should investigate, and demonstrate, the need and impact of the core goals of the project.

====II. '''Methodology'''====

Seven information workers, ages 20-38 (5 male, 2 female), were interviewed to determine which methods they use to "stay organized". An initial list of metawork strategies was established from two pilot interviews, and then a final list was compiled. Participants then responded to a series of 17 questions designed to gain insight into their metawork strategies and process. In addition, verbal interviews were conducted to get additional open-ended feedback.

====III. '''Final Results'''====
A histogram of methods people use to "stay organized" in terms of tracking things they need to do (TODOs), appointments and meetings, etc. is shown in the figure below.

[[Image:AcbGraph.jpg]]

In addition to these methods, participants also used a number of other methods, including:

* iCal
* Notes written in xterms
* "Inbox zero" method of email organization
* iGoogle Notepad (for tasks)
* Tag emails as "TODO", "Important", etc.
* Things (Organizer Software)
* Physical items placed to "remind me of things"
* Sometimes arranging windows on desk
* Keeping browser tabs open
* Bookmarking web pages
* Keep programs/files open scrolled to certain locations sometimes with things selected

In addition, three participants said that when interrupted they "rarely" or "very rarely" were able to resume the task they were working on prior to the interruption. Three of the participants said that they would not actively recommend their metawork strategies for other people, and two said that staying organized was "difficult".

Four participants were neutral to the idea of new tools to help them stay organized and three said that they would like to have such a tool/tools.

====IV. '''Discussion'''====
These results qunatiatively support our hypothesis that there is no clearly dominant set of metawork strategies employed by information workers. This highly fragemented landscape is surprising, even though most information workers work in a similar environment - at a desk, on the phone, in meetings - and with the same types of tools - computers, pens, paper, etc. We believe that this suggests that there are complex tradeoffs between these methods and that no single method is sufficient. We therefore believe that users will be better supported with a new set of software-based metawork tools.

= [Criticisms] =
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

''Any criticisms or questions we have regarding the proposal can go here.''

=References=
<references/>

CS295J/Research proposal (draft 2)

2009-04-23T21:41:48Z

Trevor O'Brien: /* Specific Aims and Contributions (to be separated later) */

CS295J/Research proposal (draft 2)

2009-04-23T21:38:16Z

Trevor O'Brien: /* Specific Aims and Contributions (to be separated later) */

CS295J/Research proposal (draft 2)

2009-04-23T21:31:33Z

Trevor O'Brien: /* Introduction */

CS295J/Contributions for class 12

2009-04-23T20:31:32Z

Trevor O'Brien: /* Contributions */

== A predictive, fully-integrated model of user workflow which encompasses low-level tasks, working spheres, communication chains, interruptions and multi-tasking.==
Owner: Andrew Bragdon
#Traditionally, software design and usability testing is focused on low-level task performance. However, prior work (Gonzales, et al.) provides strong empirical evidence that users also work at a higher, ''working sphere'' level. Su, et al., develops a predictive model of task switching based on communication chains. Our model will specifically identify and predict key aspects of higher level information work behaviors, such as task switching. We will conduct initial exploratory studies to test specific instances of this high-level hypothesis. We will then use the refined model to identify specific predictions for the outcome of a formal, ecologically valid study involving a complex, non-trivial application.
# Impact: Current information work systems are almost always designed around the individual task, and because they do not take into account the larger workflow context, these systems arguably do not properly model the way users work. By establishing a predictive model for user workflow based on individual task items, larger goal-oriented working spheres, multi-tasking behavior, and communication chains, developers will be able to design computing systems that properly model users and thus significantly increase worker productivity in the United States and around the world.
# 3-week Feasibility Study: To ascertain the feasibility of this project we will conduct an initial pilot test to investigate the core idea: a predictive model of user workflow. We will spend 1 week studying the real workflow of several people through job shadowing. We will then create two systems designed to help a user accomplish some simple information work task. One system will be designed to take larger workflow into account (experimental group), while one will not (control group). In a synthetic environment, participants will perform a controlled series of tasks while receiving interruptions at controlled times. If the two groups perform roughly the same, then we will need to reassess this avenue of research. However, if the two groups perform differrently then our pilot test will have lent support to our approach and core hypothesis.
# Risks/Costs: Risk will play an important factor in this research, and thus a core goal of our research agenda will be to manage this risk. The most effective way to do this will be to compartmentalize the risk by conducting empirical investigations - which will form the basis for the model - into the separate areas: low-level tasks, working spheres, communication chains, interruptions and multi-tasking in parallel. While one experiment may become bogged down in details, the others will be able to advance sufficiently to contribute to a strong core model, even if one or two facets encounter setbacks during the course of the research agenda. The primary cost drivers will be the preliminary empirical evaluations, the final system implementation, and the final experiments which will be designed to support the original hypothesis. The cost will span student support, both Ph.D. and Master's students, as well as full-time research staff. Projected cost: $1.5 million over three years.

== A mixed-initiative system for interface design ==
Owner: Eric

=== Summary ===
Note: click here for [http://vrl.cs.brown.edu/wiki/images/b/b4/Flowchart.pdf flowchart].

We propose a framework for interface evaluation and recommendation that integrates behavioral models and design guidelines from both cognitive science and HCI. Our framework behaves like a committee of specialized experts, where each expert provides its own assessment of the interface, given its particular knowledge of HCI or cognitive science. For example, an expert may provide an evaluation based on the GOMS method, Fitts's law, Maeda's design principles, or cognitive models of learning and memory. An aggregator collects all of these assessments and weights the opinions of each expert based on past accuracy, and outputs to the developer a merged evaluation score and a weighted set of recommendations.

Different users have different abilities and interface preferences. For example, a user at NASA probably cares more about interface accuracy than speed. By passing this information to our committee of experts, we can create interfaces that are tuned to maximize the utility of a particular user type.

We evaluate our framework through a series of user studies. Interfaces passed to our committee of experts receive evaluation scores on a number of different dimensions, such as time, accuracy, and ease of use for novices versus experts. We can compare these predicted scores to the actual scores observed in user studies to evaluate performance. The aggregator can retroactively weight the experts' opinions to determine which weighting would have given the best predictions of user behavior for the given interface, and observe whether that weighting generalizes to other interface evaluations.

==== Inputs ====
* The task the user is trying to accomplish
* The GUI he/she is using to perform this task
* The utility a user gets for values of different performance metrics (time, cognitive load, fatigue, etc.)
* The predicted and/or actual trace of a user using this GUI

==== Outputs ====
* An evaluation of the GUI, in terms of the individual metric values (i.e. time, cognitive load, etc.), and the overall utility for this as expressed by the utility function.
* Suggested improvements for the GUI, in two forms:
** Immediate transformations that can be automatically applied to the GUI
** Higher level suggestions/guidelines that would have to be made by a developer

=== Contributions ===
* The creation of a language for ''abstractly representing user interfaces'' in terms of the layout of graphical components and the functional relationships between these components.
* A system for ''generating interaction histories'' within user interfaces to facilitate individual and collaborative scientific discovery, and to enable researchers to more easily document and analyze user behavior.
* A system that takes user traces and creates a GOMS model that decomposes user actions into various cognitive, perceptual, and motor control tasks.
* The development of other evaluation methods using various cognitive/HCI models and guidelines.
* A design tool that can provide a designer with recommendations for interface improvements. These recommendations can be made for a specific type of user or for the average user, as expressed by a utility function.

== A Unified Model of Cognitive Perception and Processing in HCI ==
Steven

*We plan to develop a unified model of the conscious and subconscious mental processes which direct human behavior in interaction with digital interfaces. We will draw upon existing theoretical conceptions of these processes, including distributed cognition, activity theory, etc. This model will be empirically validated in a manner similar to that used in assessing interface efficiency via CPM-GOMS.

*Impact: This contribution will provide a standard model of mental processes upon which effective tools for interface evaluation can be built and by which they can be justified. Such a model would eliminate the need for costly wide-ranging testing of new interface suggestions.

== A Quantitative, Cognition-Based Framework for Analyzing and Predicting User Performance ==
Owner: [[Trevor|Trevor O'Brien]]

=== Summary ===

Established guidelines for designing human computer interfaces are based on experience, intuition and introspection. Because there is no integrated theoretical foundation, many rule-sets have emerged despite the absence of comparative evaluations. We propose to develop a theoretical foundation for interface design, drawing on recent advances in cognitive science – the study of how people think, perceive and interact with the world. We will distill a broad range of principles and computational models of cognition that are relevant to interface design and use them to compare and unify existing guidelines. To validate our theoretical foundation, we will use our findings to develop a quantitative mechanism for assessing interface designs, identifying interface elements that are detrimental to user performance, and suggesting effective alternatives. Results from this system will be explored over a set of case studies, and the quantitative assessments output by this system will be compared to actual user performance.

Within this framework, we will introduce novel techniques for collecting and filtering user interaction histories with respect to user goals. In addition, we propose to integrate data from multiple sensing technologies -- including pupil-tracking, muscle-activity monitoring and auditory recognition -- with these interaction histories to analyze the explicit contributions of perception, cognition and motor skills with respect to user performance. This data will be used to define a CPM-GOMS model of cognition for a given interface, providing a quantitative, predictive, cognition-based parameterization of usability. From empirically collected data, user trajectories through the model (critical paths) will be examined with respect to the cognitive principles obtained from our theoretical findings, highlighting bottlenecks within the interface, and offering suggested alterations to the interface to induce more optimal user trajectories.

=== Contributions ===

* Design and user-study evaluation of novel techniques for collecting and filtering user traces with respect to user goals.
* Extensible, low-cost architecture for integrating pupil-tracking, muscle-activity monitoring, and auditory recognition with user traces in existing applications.
* System for isolating cognitive, perceptual, and motor tasks from an interface design to generate CPM_GOMS models for analysis.
* Design and quantitative evaluation of semi-automated techniques for extracting critical paths from an existing CPM_GOMS model.
* Novel algorithm for analyzing empirically-collected critical paths and suggesting optimized paths based on established research in cognitive science.

=== Specific Aims ===

* Incorporate interaction history mechanisms into a set of existing applications.
* Perform user-study evaluation of history-collection techniques.
* Distill a set of cognitive principles/models, and evaluate empirically?
* Build/buy sensing system to include pupil-tracking, muscle-activity monitoring, auditory recognition.
* Design techniques for manual/semi-automated/automated construction of CPM-GOMS model from interaction histories and sensing data.
* Design system for posterior analysis of interaction history w.r.t. CPM_GOMS model, evaluating critical path trajectories.
* Design cognition-based techniques for detecting bottlenecks in critical paths, and offering optimized alternatives.

=== Input ===

* User traces: interaction histories coupled with contextual information about the interface itself and the application to which it belongs.
* User-sensing data: head-tracking data, pupil-tracking data, muscle-activity monitoring, auditory recognition.
* High-level user goals: set of overarching user intents.
* Set of cognitive guidelines/heuristics (used to analyze and optimize critical paths)

=== Output ===

* CPM-GOMS-style breakdown of perceptual, motor and cognitive tasks.
* Empirically observed critical path trajectories.
* Cognition-based breakdown of user trajectories. Highlights bottlenecks, offers suggestions to improve bottlenecks. (parallelization?)

=== Significance ===

* Evaluate utility of cognitive principles in HCI.
* Provide a quantitative, cognition-based mechanism for task analysis.
* Software system/architecture that allows other researchers and developers to analyze interface design effectively.
* In the case of scientific applications, aid in the adoption of tools by domain scientists.

==A Unified Framework of Cognition in Analysis and Prediction in HCI==
Owner: [[User:E J Kalafarski|E J Kalafarski]] 14:25, 17 April 2009 (UTC)

===Summary===
Established guidelines for designing human-computer interfaces are based on experience, intuition and introspection. Because there is no integrated theoretical foundation, many rulesets have emerged despite the absence of comparative evaluation. We propose a theoretical foundation for interface design, drawing on recent advances in cognitive science—the study of how people think, perceive and interact with the world.

We will distill a broad range of principles and computational models of cognition that are relevant to interface design and use them to compare and unify existing guidelines. To validate our theoretical foundation, we will use our findings to develop a quantitative mechanism for assessing interface designs, identifying interface elements that are detrimental to user performance, and suggesting effective alternatives. Results from this system will be explored over a set of case studies, and the quantitative assessments output by this system will be compared to actual user performance.

Cognition, in this broad sense, cannot be limited to internal processing. A central focus of our work will be to broaden the range of cognitive theories that are used in HCI design, and we will explore the effects of improvement on all "links" in the interaction "chain"—cognition, perception, and motor skills, the CPM of the so-called CPM-GOMs evaluation method. Few low-level theories of perception and action, such as Fitts's law, have garnered general acceptance in the HCI community due to their simple, quantitative nature, and widespread applicability. Our aim is to produce similar predictive models that apply to lower levels of perception as well as higher levels of cognition, including vision, learning, memory, attention, reasoning and task management.

We will focus on generating an extensible, generalizable framework that can be applied to a broad range of interface design challenges. Interfaces exist everywhere in the modern world, and much research has accumulated regarding how people manage multiple tasks, how machines distribute tasks and work, and how devices communicate and anticipate user action effective. The promise of such a foundation is rich, and the potential benefits game-changing. We intend to do no less than provide new methodology for the examination of the human-computer system as a unified cognitive entity. [[User:E J Kalafarski|E J Kalafarski]] 14:58, 17 April 2009 (UTC)

====Input====
* Interface specification (within, to begin with, narrow and realistic parameters)
* User traces, including user-sensing data: interaction histories coupled with contextual information about the interface itself and the application to which it belongs.
* High-level user goals and intent
* Set of cognitive guidelines/heuristics (used to analyze and optimize critical paths)

====Output====
* CPM-GOMS-style breakdown of perceptual, motor and cognitive tasks.
* Empirically observed critical path trajectories.
* Cognition-based breakdown of user trajectories. Highlights bottlenecks, offers suggestions to improve bottlenecks.

=== Contributions ===
* The creation of a language for ''abstractly representing user interfaces'' in terms of the layout of graphical components and the functional relationships between these components.
* A system for ''generating interaction histories'' within user interfaces to facilitate individual and collaborative scientific discovery, and to enable researchers to more easily document and analyze user behavior.
* A design tool that can provide a designer with recommendations for interface improvements, based on a ''unified matrix'' of cognitive principles and heuristic design guidelines.

===Specific Aims===
* Incorporate interaction history mechanisms into a set of existing applications.
* Perform user-study evaluation of history-collection techniques.
* Distill a set of cognitive principles/models, and evaluate empirically?
* Build/buy sensing system to include pupil-tracking, muscle-activity monitoring, auditory recognition.
* Design techniques for manual/semi-automated/automated construction of CPM-GOMS model from interaction histories and sensing data.
* Design system for posterior analysis of interaction history w.r.t. CPM_GOMS model, evaluating critical path trajectories.
* Design cognition-based techniques for detecting bottlenecks in critical paths, and offering optimized alternatives.

===Significance===
* Merges disparate-but-related fields of usability and cognition.
* Provide a quantitative, cognition-based mechanism for task analysis.
* Software system/architecture that allows other researchers and developers to analyze interface design effectively.
* Revise and provide theoretical foundations for industry-standard human usability guidelines.

==="Big World" Goals===
* Broader understanding of basic, reliable usability principles and more prevalent application of them.
* Efficient, profitable, and compatible usability application methods in business.
* A set of usability principles that supersedes Human-Computer Interaction and is applicable in all varieties of product development.

== Interface Evaluation Using Parallel Cognitive Models (Adam and Jon) ==

=== Summary ===

Existing approaches for using cognitive models to evaluate user interfaces are based on extensive cognitive architectures that incorporate that integrate much of cognitive theory. The disadvantage of this approach is that modifying and using these models is resource intensive

We propose a distributed approach whereby simple cognitive models of specific behaviors each evaluate an interface independently, and those scores are combined to give a general evaluation. This approach is more flexible, and permits easier modification and use at multiple levels of detail.

=== Input ===

*Functions that the interface performs
*Interface description
*User traces

=== Sample Modules ===

*Exogenous attention
**Question: To what extent does the interface draw your attention to what you need?
**Need Interface display; user traces.

*Automaticity
**Question: How often is executive attention needed?
**Need: Predictability given previous actions and information displayed

*Fitts Law
**Question: Time to move pointer from one operation location to the next?
**Need a physical arrangement of elements and transition matrix (user traces?)

*Memory Load
**Question: How many things are remembered simultaneously?
**Need Interval (average?) between presentation and use. User traces.

*Affordances
**Question: Are interactions with the interface “natural”?
**Elements, their associated actions, and their functions

*Interruptions
**Question: How long for interruption recovery?
**Need User traces and predictability given information displayed

=== Output ===

*Efficiency curve

CS295J/Contributions for class 12

2009-04-17T13:48:42Z

Trevor O'Brien: /* Input */

CS295J/Contributions for class 12

2009-04-17T13:45:46Z

Trevor O'Brien: /* A Quantitative, Cognition-Based Framework for Analyzing and Predicting User Performance */

CS295J/Contributions for class 12

2009-04-17T01:16:41Z

Trevor O'Brien:

CS295J/Assignments

2009-04-15T15:50:47Z

Trevor O'Brien: /* Possible Tasks (also consider those from last week -- add here if you pick one) */

== Assignment 11 (out April 10, due April 17, 2009) ==

=== Part A (due Tuesday at noon) ===
First, revise any contribution that you added to the proposal to reflect our converging view of what we are proposing. Given the evolution of the proposal since the contributions were originally created, if you wish to start from a fresh set of contributions, you can outline them [[CS295J/Contributions for class 12 |here]].

Own one or more of the possible tasks below (enough to spend 8-10 hours on). Edit this page to indicate your ownership and to describe enough details of what you'll do for the group to avoid duplication. It's ok, for exmaple, for two folks to try using or extending CPM-GOMS with different extensions or a different application; indicating what your extension/application is, though would be helpful. It's also fine to add tasks to the list.

=== Part B (due class time) ===

Revise any parts of the preliminary results that you have added. That may include adding things that you would like to do, taking out pieces that don't fit into our worldview any longer, or revising things. The "CPM-GOMS for gmail" project presented today, for example, should go into the preliminary work section to establish viability of CPM-GOMS in the context of a more modern application.

Complete owned task(s).

=== Possible Tasks (also consider those from last week -- add here if you pick one) ===

* Possible additions to the proposed research
** Experiments with EEG to determine if they can inform modeling
** Maybe similar experiments with muscle sensing or body motion
* Possible preliminary experiments
** Try some EEG
** Try some low-cost muscle tracking
** Try to decode some captured events
** Try to get some pupil-tracking thing going and sync'ed with other events
** Try some light-weight JavaScript-based mouse tracking (will see how far I can get in 10 hours) ([[User:E J Kalafarski|E J Kalafarski]] 15:16, 14 April 2009 (UTC))
* Other
** Get "Human information processor model" concept into related work; ernestine is canonical example, but other more recent work too
** Revise proposal introduction again, emphasis on our new converging view ([[User:E J Kalafarski|E J Kalafarski]] 15:13, 14 April 2009 (UTC), [[User:Trevor|Trevor O'Brien]])
** Will revise and expand "big world" significance and contributions, based on the output/architecture we discussed last week ([[User:E J Kalafarski|E J Kalafarski]] 15:15, 14 April 2009 (UTC), [[User:Trevor|Trevor O'Brien]])
* CPM-GOMS
** Extend the cognition element in the CPM-GOMS model to account for dual-process theories of reasoning. (OWNER - Gideon)
** Extend the model by incorporating an 'Affect' output, instead of just relying on time as the only dependent variable. (SUGGESTED BY - Gideon; Ian (I like this idea))

== Assignment 10 (out April 3, due April 10, 2009) ==

=== Part A (due Tuesday at noon) ===
Own one or more of the possible tasks below (enough to spend 8-10 hours on). Edit this page to indicate your ownership and to describe enough details of what you'll do for the group to avoid duplication. It's ok, for exmaple, for two folks to try using or extending CPM-GOMS with different extensions or a different application; indicating what your extension/application is, though would be helpful. It's also fine to add tasks to the list.

=== Part B (due class time) ===

Complete owned task(s).

=== Possible Tasks ===

* read CPM-GOMS and consider as theme ('''[[User:E J Kalafarski|E J Kalafarski]]''', '''Eric''', '''Jon''', '''Gideon''', '''[[User:Trevor O'Brien|Trevor]]''', '''[[User:Steven Ellis|Steven]]''', '''Ian''')
* converge proposal (ID weaknesses & fix or propose fixes)
** make contributions and aims agree ('''[[User:Trevor O'Brien|Trevor]]''')
** make contributions and aims compelling and significant
*** intro acceptable (merge the two best) ('''[[User:E J Kalafarski|E J Kalafarski]]''': will rewrite intro, merging the two best "in progress" intros, with an emphasis on the project's interdisciplinary aspect)
*** improves world, lots of people, in big ways, long time ('''[[User:E J Kalafarski|E J Kalafarski]]''': will brainstorm and add to "big picture" contributions, but someone else should brainstorm this as well, '''Eric''')
*** extends knowledge, increased productivity (designers and users) ('''Eric''')
*** bg & significance section consistent with contribs/aims ('''Ian''')
* try CPM-GOMS, maybe w/ 1 extension ('''Gideon''', '''[[User:Steven Ellis|Steven]]''')
* what's input and output (architecture)? ('''[[User:E J Kalafarski|E J Kalafarski]]''': will attempt to come up with an architecture that accommodates all seven proposed modules and presents a simple, useful final product, ''Jon: Will brainstorm b/c I've been wondering about this for a long time'', '''Adam''', '''[[User:Trevor O'Brien|Trevor]]''')
* Gajos paper ('''[[User:E J Kalafarski|E J Kalafarski]]''': will read, consider architecture method, '''Adam''', '''[[User:Trevor O'Brien|Trevor]]''')

== Assignment 9 (out March 20, due April 3, 2009) ==

Part A is what we talked about in class -- a revision of the introduction to re-evaluate the big picture. Part B is the original assignment. While I've asked for both below, it may not be realistic given the time available. Please make sure to finish A by class time and to do as much of B as you can.

=== Part A, due noon Tuesday March 31 ===

Refine your introduction to be consistent with the theme of bringing knowledge of cognitive and perceptual modeling to bear in a principled way on human-computer interface design. Evaluate each of the suggested contributions for their impact, their relevance to the theme, and their cost. Triage as appropriate to achieve the best overall proposal given the 5 years with 5 people scope.

* [[CS295J/Proposal intros from class 9|Link to Intros]]
* [[CS295J/Triage for class 10|Link to Triages]]

=== Part B, due at class time ===

Complete your 30 hour preliminary work project and writeup in the proposal. It should explicitly support the contributions of the proposal. Consider this a hard, externally imposed deadline, i.e., you have to have something as finished as possible that could be reviewed by an outside reviewer. Trim scope, if necessary, but finish!

Pick another gap in the proposal and plug it. Any section that you have "owned" should also be "final" and reviewable.

====Gaps====
* <s>Significance (expand further with unified description of relatable components)</s>
* Specific contributions > Multi-modal HCI (question marks, needs content)
* Background > Workflow analysis > Interface improvements (incomplete)
* Research plan (we may have to cut this section, it is woefully underdeveloped)

Read [[CS295J/Literature to read for class 10|"must reads"]] -- add one if you want.

== Assignment 8 (out March 13, due March 20, 2009) ==
=== Part A, due Tuesday noon ===
* Outline a coherent 250 word summary to a coherent proposal [[CS295J/Proposal intros from class 9|here]]
* Select a gap to own
** '''Andrew Bragdon''': Metawork Support Tool proposal; will examine integrating this into the main proposal vs. making a separate proposal.
** '''EJ''': "Significance/intellectual merit" section is currently bare, I can take a stab at that. I believe this can/needs to incorporate a gap Trevor identified, "mapping between individual contributions and centralized theme of the proposal;" I'll try to start to illustrate the relationships between our individual projects. [[User:E J Kalafarski|E J Kalafarski]] 12:20, 17 March 2009 (UTC)
**'''Jon''': The "models of perception" section needs to be revised and expanded. I've changed "Gibsonianism" to "The Ecological Approach to Perception" and I've added a section on "The Computational Approach to Perception"; I will update these for Friday.
**'''Gideon''': More up-to-date background section on distributed cognition. I know Jon is doing this for the other areas, but I feel that this is very important.
** '''Trevor''': I'll take a pass through the ''Specific Aims'' section, which currently lacks specificity. --- [[User:Trevor O'Brien|Trevor O'Brien]] 15:29, 17 March 2009 (UTC)
**'''Eric''': I'll flesh out the ''Workflow Analysis'' section. What's there could be cleaned up, and more could be added, particularly on learning from interaction histories.
**'''Steven''': There's currently no background (slash anything) on the topic of embodied models of cognition, I'll fill in some background on that.
* suggest 0.5 "must read" [[CS295J/Literature to read for class 9|papers]]

=== Part B, due in class ===
* Finish writing the coherent 250 word summary to a coherent proposal [[CS295J/Proposal intros from class 9|still here]]
* Fill your gap
* read "must read"s for discussion w.r.t. proposal relevance
* get to another week's worth of results in your preliminary results -- make sure to be consistent with your summary!
* be prepared to tell us about those new results in class, tied to the intellectual claims

== Assignment 7 (out March 6, due March 13, 2009) ==
=== Part A, due Tuesday noon ===
* review [[CS295J/Research proposal|proposal]] for
** intellectual contribution (1-2 paragraphs)
** major gaps (bullet list) (don't duplicate gaps already listed)
** review goes [[CS295J/Proposal reviews from class 8|here]]
** feel free to improve any part of the proposal rather than criticizing it, if you wish :-)
* suggest 0.5 "must read" [[CS295J/Literature to read for class 8|papers]]

=== Part B, due in class ===
* read "must read"s for discussion w.r.t. proposal relevance
* get to another week's worth of results in your preliminary results
* be prepared to tell us about those new results in class, tied to the intellectual claims

== Assignment 6 (out February 27, due March 6, 2009) ==

* If you are not familiar with NIH proposals, skim through [http://vis.cs.brown.edu/docs/pdf/bib/Laidlaw-2005-DA2.pdf this active proposal] to get a sense of the sections. The [http://vis.cs.brown.edu/docs/pdf/bib/NIH-2001-PHS.pdf guide to nih proposals] may also be helpful.
* Create some preliminary results
** Start the work you proposed in your poster for assignment 4
** Start the work someone else proposed :-)
** Create a more detailed critique-enabled workflow of a scientific user
** Working in pairs is fine; preferably, pair members should have different backgrounds
* Write them up as a subsection of the preliminary results section of the proposal.
** By Wednesday 9am have an outline of the section you will produce
*** This should be in past tense, as it will be when the work is done.
*** At the top level, it should state how the preliminary results enable the overall multi-year proposal or how they demonstrate feasibility of some questionable or risky part.
*** Check out the NIH proposal for examples, albeit in a different domain.
** By class have as much filled in from that outline as possible
* Add any "must reads" to the [[CS295J/Literature to read for class 6]] page. You are "expected" to add 1/2 of a reference.
* Read and be prepared to discuss the "must reads".
** Put in fictional placeholders for parts that are not done by Friday.
* By Wednesday 5pm e-mail constructive comments about at least 1/2 of the outlines to the entire class
* Success criteria for this assignment
*# Proposal section will demonstrate a feasibility or prerequisite for interesting research
*# Results by class are complete and concrete enough that they are interesting, even though some parts may be missing

== Assignment 5 (out February 20, due February 27, 2009) ==
# Videotape a 15-20 minute interactive session with a user doing a visually challenging task for which they are at least an advanced beginner (ie, they don't have to look up what to do, but they have not yet internalized the operations and made them subconscious). Analyze the session and report your observations and conclusions.

== Assignment 4 (out February 13, due February 20, 2009) ==

# Be prepared to decide on a subset of applications to critique. Flesh out at least one potential application in [[CS295J/Application Critiques]], including a specific workflow. Modify any others you want, particularly in terms of arguments for or against proceeding with them. These should be completed by Thursday noon.
#* Some possibilities: Google scholar, Mathematica, Tableau, Google notebook, Matlab, paper, media wiki, Ensight Gold, AVS, VisTrails. Note that a given piece of software could be represented more than once with a different workflow.
#* Possible criteria: main purpose is analysis, perhaps scientific; interative; scientific; amenable to cognition-driven improvemennts; interesting; fun
# Bring your ranking of the subset of (application+workflow)s we should critique
# Be prepared to decide on the [[CS295J/Model elements]] of cognition we will emulate to critique each application. Revise some part of this list so that it can be used as a concrete basis for evaluation.
# Add any "must reads" to the [[CS295J/Literature to read for class 5]] page. You are "expected" to add 1/3 of a reference.
# Read the "must reads".

== Assignment 3 (out February 6, due February 13) ==
* Flesh out and support a contribution within the proposal
** Add any new references to the literature section.
** Many of our readings date from 8+ years ago; check to make sure your contribution hasn't already been done.
** If any of the new references are "must reads", add to the [[CS295J/Literature to read for class 4]] (2/13/09) page. You are "expected" to add 1/2 of a reference.
** Estimate the impact of the contribution.
** Estimate the risk and costs of the contribution.
** Propose a 3-week (30 hour) result that you could create to demonstrate feasibility.
** Bring a printout of your contribution concept to class. It should be legible from 2 meters away, so use big text. You can hand-write it on posterboard or paste together printouts.
* Bring a list of holes in the proposal -- e.g., "background section needs something on workflow capture."

== Assignment 2 (out January 30, 2009) ==
* Refine literature short summaries to include the relationship to our project.
* Draft by tuesday noon a subsection in the background section of the [[CS295J/Research proposal]], making sure to include citations to the relevant materials in the literature page. Add any new references to the literature page.
* Read and comment/edit by Thursday noon all background sections
* Revise your background section by Friday noon (so David can print before class)

* Add to or refine one or more specific contribution in the [[CS295J/Research proposal]]; each contribution must have a list of ways it can be demonstrated. Some will become part of the preliminary results, others will be parts of the future work that will be proposed.
* Add to or refine one or more specific aim in the [[CS295J/Research proposal]]; make it consistent with the contribution you added.

* Identify the one additional most important paper for us to read this week also by Tuesday noon; be prepared to summarize relevance in 2 minutes in class
* Read those "most important" papers for class discussion.

* Be prepared to summarize to the class your contributions to the background, contributions, and aims sections.
* If there is preliminary work that will help to make decisions about contributions and aims, please get started on it (and be ready to report on what you'd like to do or what you have done).

== Assignment 1 (out January 23, 2009) ==
* spend 10 hours adding to any part of the wiki you think is relevant
* by Monday noon add any potential readings. If you've got a tentative summary evaluation, go ahead and add it. It's ok to edit folks summary evaluations, but try to make the result more accurate or precise without losing information.
* by Wednesday noon finish with any summary evaluation and also identify at least one as-relevant-as-possible reading as yours. Put your name on that entry in the reading list as the "owner" so that there are no duplicates.
* by Wednesday 5pm -- select 2 additional relevant readings that are owned and that you will read by Friday and be prepared to discuss. Put your name as a "discussant" in the reading list; there should be a max of two discussants per reading.
* by Friday class -- author a summary description, less than 250 words, in the wiki of how the reading you own relates to our project. Be prepared to describe, in two minutes, how your reading relates to the project. Also be prepared for everyone in class to discuss after your description. You may bring notes for yourself, but no slides. The wiki page for your reading will be displayed while you talk.
* by Friday class -- read and be prepared to discuss the other two readings you choose.
* by Friday class -- make one more wiki page titled "'''<Last-Name> week 1'''" with a list of the keys for the citations you added, the readings you summarized, the reading you presented, the two readings you were a discussant on, and any other readings you did in detail.
* Let me know if you have any kind of problems. You should be spending right around 10 hours -- if that's a problem, let's talk.
* The [[../How Tos|How Tos]] page has some tips. Edit or add as you find others.

CS295J/Literature to read for class 11

2009-04-10T13:49:53Z

Trevor O'Brien:

* [[Image:Defining Insight for Visual Analytics.pdf|Defining Insight for Visual Analytics]]: Perhaps an alternative to time as the unit of interface value.

* [http://www.cs.washington.edu/homes/kgajos/papers/kgajos-chi08-supple.pdf Improving the Performance of Motor-Impaired Users with Automatically-Generated, Ability-Based Interfaces] Gajos-2008-IPM

* [http://www.cs.washington.edu/homes/kgajos/papers/kgajos-nectar08.pdf Decision-Theoretic User Interface Generation] Gajos-2008-DTU

* [http://portal.acm.org/citation.cfm?id=1461832&dl=&coll= Project ernestine: validating a GOMS analysis for predicting and explaining real-world task performance]

CS295J/Literature to read for class 11

2009-04-10T13:49:02Z

Trevor O'Brien:

CS295J/Assignments

2009-04-07T15:53:32Z

Trevor O'Brien: /* Possible Tasks */

== Assignment 10 (out April 3, due April 10, 2009) ==

=== Part A (due Tuesday at noon) ===
Own one or more of the possible tasks below (enough to spend 8-10 hours on). Edit this page to indicate your ownership and to describe enough details of what you'll do for the group to avoid duplication. It's ok, for exmaple, for two folks to try using or extending CPM-GOMS with different extensions or a different application; indicating what your extension/application is, though would be helpful. It's also fine to add tasks to the list.

=== Part B (due class time) ===

Complete owned task(s).

=== Possible Tasks ===

* read CPM-GOMS and consider as theme ('''[[User:E J Kalafarski|E J Kalafarski]]''', '''Eric''', '''Jon''', '''Gideon''', '''[[User:Trevor O'Brien|Trevor]]''')
* converge proposal (ID weaknesses & fix or propose fixes)
** make contributions and aims agree ('''[[User:Trevor O'Brien|Trevor]]''')
** make contributions and aims compelling and significant
*** intro acceptable (merge the two best) ('''[[User:E J Kalafarski|E J Kalafarski]]''': will rewrite intro, merging the two best "in progress" intros, with an emphasis on the project's interdisciplinary aspect)
*** improves world, lots of people, in big ways, long time ('''[[User:E J Kalafarski|E J Kalafarski]]''': will brainstorm and add to "big picture" contributions, but someone else should brainstorm this as well, '''Eric''')
*** extends knowledge, increased productivity (designers and users) ('''Eric''')
*** bg & significance section consistent with contribs/aims
* try CPM-GOMS, maybe w/ 1 extension ('''Gideon''')
* what's input and output (architecture)? ('''[[User:E J Kalafarski|E J Kalafarski]]''': will attempt to come up with an architecture that accommodates all seven proposed modules and presents a simple, useful final product, ''Jon: Will brainstorm b/c I've been wondering about this for a long time'', '''Adam''', '''[[User:Trevor O'Brien|Trevor]]''')
* Gajos paper ('''[[User:E J Kalafarski|E J Kalafarski]]''': will read, consider architecture method, '''Adam''', '''[[User:Trevor O'Brien|Trevor]]''')

== Assignment 9 (out March 20, due April 3, 2009) ==

Part A is what we talked about in class -- a revision of the introduction to re-evaluate the big picture. Part B is the original assignment. While I've asked for both below, it may not be realistic given the time available. Please make sure to finish A by class time and to do as much of B as you can.

=== Part A, due noon Tuesday March 31 ===

Refine your introduction to be consistent with the theme of bringing knowledge of cognitive and perceptual modeling to bear in a principled way on human-computer interface design. Evaluate each of the suggested contributions for their impact, their relevance to the theme, and their cost. Triage as appropriate to achieve the best overall proposal given the 5 years with 5 people scope.

* [[CS295J/Proposal intros from class 9|Link to Intros]]
* [[CS295J/Triage for class 10|Link to Triages]]

=== Part B, due at class time ===

Complete your 30 hour preliminary work project and writeup in the proposal. It should explicitly support the contributions of the proposal. Consider this a hard, externally imposed deadline, i.e., you have to have something as finished as possible that could be reviewed by an outside reviewer. Trim scope, if necessary, but finish!

Pick another gap in the proposal and plug it. Any section that you have "owned" should also be "final" and reviewable.

====Gaps====
* <s>Significance (expand further with unified description of relatable components)</s>
* Specific contributions > Multi-modal HCI (question marks, needs content)
* Background > Workflow analysis > Interface improvements (incomplete)
* Research plan (we may have to cut this section, it is woefully underdeveloped)

Read [[CS295J/Literature to read for class 10|"must reads"]] -- add one if you want.

== Assignment 8 (out March 13, due March 20, 2009) ==
=== Part A, due Tuesday noon ===
* Outline a coherent 250 word summary to a coherent proposal [[CS295J/Proposal intros from class 9|here]]
* Select a gap to own
** '''Andrew Bragdon''': Metawork Support Tool proposal; will examine integrating this into the main proposal vs. making a separate proposal.
** '''EJ''': "Significance/intellectual merit" section is currently bare, I can take a stab at that. I believe this can/needs to incorporate a gap Trevor identified, "mapping between individual contributions and centralized theme of the proposal;" I'll try to start to illustrate the relationships between our individual projects. [[User:E J Kalafarski|E J Kalafarski]] 12:20, 17 March 2009 (UTC)
**'''Jon''': The "models of perception" section needs to be revised and expanded. I've changed "Gibsonianism" to "The Ecological Approach to Perception" and I've added a section on "The Computational Approach to Perception"; I will update these for Friday.
**'''Gideon''': More up-to-date background section on distributed cognition. I know Jon is doing this for the other areas, but I feel that this is very important.
** '''Trevor''': I'll take a pass through the ''Specific Aims'' section, which currently lacks specificity. --- [[User:Trevor O'Brien|Trevor O'Brien]] 15:29, 17 March 2009 (UTC)
**'''Eric''': I'll flesh out the ''Workflow Analysis'' section. What's there could be cleaned up, and more could be added, particularly on learning from interaction histories.
**'''Steven''': There's currently no background (slash anything) on the topic of embodied models of cognition, I'll fill in some background on that.
* suggest 0.5 "must read" [[CS295J/Literature to read for class 9|papers]]

=== Part B, due in class ===
* Finish writing the coherent 250 word summary to a coherent proposal [[CS295J/Proposal intros from class 9|still here]]
* Fill your gap
* read "must read"s for discussion w.r.t. proposal relevance
* get to another week's worth of results in your preliminary results -- make sure to be consistent with your summary!
* be prepared to tell us about those new results in class, tied to the intellectual claims

== Assignment 7 (out March 6, due March 13, 2009) ==
=== Part A, due Tuesday noon ===
* review [[CS295J/Research proposal|proposal]] for
** intellectual contribution (1-2 paragraphs)
** major gaps (bullet list) (don't duplicate gaps already listed)
** review goes [[CS295J/Proposal reviews from class 8|here]]
** feel free to improve any part of the proposal rather than criticizing it, if you wish :-)
* suggest 0.5 "must read" [[CS295J/Literature to read for class 8|papers]]

=== Part B, due in class ===
* read "must read"s for discussion w.r.t. proposal relevance
* get to another week's worth of results in your preliminary results
* be prepared to tell us about those new results in class, tied to the intellectual claims

== Assignment 6 (out February 27, due March 6, 2009) ==

* If you are not familiar with NIH proposals, skim through [http://vis.cs.brown.edu/docs/pdf/bib/Laidlaw-2005-DA2.pdf this active proposal] to get a sense of the sections. The [http://vis.cs.brown.edu/docs/pdf/bib/NIH-2001-PHS.pdf guide to nih proposals] may also be helpful.
* Create some preliminary results
** Start the work you proposed in your poster for assignment 4
** Start the work someone else proposed :-)
** Create a more detailed critique-enabled workflow of a scientific user
** Working in pairs is fine; preferably, pair members should have different backgrounds
* Write them up as a subsection of the preliminary results section of the proposal.
** By Wednesday 9am have an outline of the section you will produce
*** This should be in past tense, as it will be when the work is done.
*** At the top level, it should state how the preliminary results enable the overall multi-year proposal or how they demonstrate feasibility of some questionable or risky part.
*** Check out the NIH proposal for examples, albeit in a different domain.
** By class have as much filled in from that outline as possible
* Add any "must reads" to the [[CS295J/Literature to read for class 6]] page. You are "expected" to add 1/2 of a reference.
* Read and be prepared to discuss the "must reads".
** Put in fictional placeholders for parts that are not done by Friday.
* By Wednesday 5pm e-mail constructive comments about at least 1/2 of the outlines to the entire class
* Success criteria for this assignment
*# Proposal section will demonstrate a feasibility or prerequisite for interesting research
*# Results by class are complete and concrete enough that they are interesting, even though some parts may be missing

== Assignment 5 (out February 20, due February 27, 2009) ==
# Videotape a 15-20 minute interactive session with a user doing a visually challenging task for which they are at least an advanced beginner (ie, they don't have to look up what to do, but they have not yet internalized the operations and made them subconscious). Analyze the session and report your observations and conclusions.

== Assignment 4 (out February 13, due February 20, 2009) ==

# Be prepared to decide on a subset of applications to critique. Flesh out at least one potential application in [[CS295J/Application Critiques]], including a specific workflow. Modify any others you want, particularly in terms of arguments for or against proceeding with them. These should be completed by Thursday noon.
#* Some possibilities: Google scholar, Mathematica, Tableau, Google notebook, Matlab, paper, media wiki, Ensight Gold, AVS, VisTrails. Note that a given piece of software could be represented more than once with a different workflow.
#* Possible criteria: main purpose is analysis, perhaps scientific; interative; scientific; amenable to cognition-driven improvemennts; interesting; fun
# Bring your ranking of the subset of (application+workflow)s we should critique
# Be prepared to decide on the [[CS295J/Model elements]] of cognition we will emulate to critique each application. Revise some part of this list so that it can be used as a concrete basis for evaluation.
# Add any "must reads" to the [[CS295J/Literature to read for class 5]] page. You are "expected" to add 1/3 of a reference.
# Read the "must reads".

== Assignment 3 (out February 6, due February 13) ==
* Flesh out and support a contribution within the proposal
** Add any new references to the literature section.
** Many of our readings date from 8+ years ago; check to make sure your contribution hasn't already been done.
** If any of the new references are "must reads", add to the [[CS295J/Literature to read for class 4]] (2/13/09) page. You are "expected" to add 1/2 of a reference.
** Estimate the impact of the contribution.
** Estimate the risk and costs of the contribution.
** Propose a 3-week (30 hour) result that you could create to demonstrate feasibility.
** Bring a printout of your contribution concept to class. It should be legible from 2 meters away, so use big text. You can hand-write it on posterboard or paste together printouts.
* Bring a list of holes in the proposal -- e.g., "background section needs something on workflow capture."

== Assignment 2 (out January 30, 2009) ==
* Refine literature short summaries to include the relationship to our project.
* Draft by tuesday noon a subsection in the background section of the [[CS295J/Research proposal]], making sure to include citations to the relevant materials in the literature page. Add any new references to the literature page.
* Read and comment/edit by Thursday noon all background sections
* Revise your background section by Friday noon (so David can print before class)

* Add to or refine one or more specific contribution in the [[CS295J/Research proposal]]; each contribution must have a list of ways it can be demonstrated. Some will become part of the preliminary results, others will be parts of the future work that will be proposed.
* Add to or refine one or more specific aim in the [[CS295J/Research proposal]]; make it consistent with the contribution you added.

* Identify the one additional most important paper for us to read this week also by Tuesday noon; be prepared to summarize relevance in 2 minutes in class
* Read those "most important" papers for class discussion.

* Be prepared to summarize to the class your contributions to the background, contributions, and aims sections.
* If there is preliminary work that will help to make decisions about contributions and aims, please get started on it (and be ready to report on what you'd like to do or what you have done).

== Assignment 1 (out January 23, 2009) ==
* spend 10 hours adding to any part of the wiki you think is relevant
* by Monday noon add any potential readings. If you've got a tentative summary evaluation, go ahead and add it. It's ok to edit folks summary evaluations, but try to make the result more accurate or precise without losing information.
* by Wednesday noon finish with any summary evaluation and also identify at least one as-relevant-as-possible reading as yours. Put your name on that entry in the reading list as the "owner" so that there are no duplicates.
* by Wednesday 5pm -- select 2 additional relevant readings that are owned and that you will read by Friday and be prepared to discuss. Put your name as a "discussant" in the reading list; there should be a max of two discussants per reading.
* by Friday class -- author a summary description, less than 250 words, in the wiki of how the reading you own relates to our project. Be prepared to describe, in two minutes, how your reading relates to the project. Also be prepared for everyone in class to discuss after your description. You may bring notes for yourself, but no slides. The wiki page for your reading will be displayed while you talk.
* by Friday class -- read and be prepared to discuss the other two readings you choose.
* by Friday class -- make one more wiki page titled "'''<Last-Name> week 1'''" with a list of the keys for the citations you added, the readings you summarized, the reading you presented, the two readings you were a discussant on, and any other readings you did in detail.
* Let me know if you have any kind of problems. You should be spending right around 10 hours -- if that's a problem, let's talk.
* The [[../How Tos|How Tos]] page has some tips. Edit or add as you find others.

CS295J/Research proposal

2009-03-20T16:26:05Z

Trevor O'Brien: /* Specific Aims */

==Project summary==

We propose to integrate theories of cognition, models of perception, rules of design, and concepts from the discipline of human-computer interaction to develop a predictive model of user performance in interacting with computer software for visual and analytical work. Our proposed model comprises a set of computational elements representing components of human cognition, memory, or perception. The collective abilities and limitations of these elements can be used to provide feedback on the likely efficacy of user interaction techniques.

The choice of human computational elements will be guided by several models or theories of cognition and perception, including Gestalt, Distributed, Gibson, ???(where pathway, when pathway)???, ???working-memory???, ..., and ???. The list of elements will be extensible. The framework coupling them will allow for experimental predictions of utility of user interfaces that can be verified against human performance.

Coupling the system with users will involve a data capture mechanism for collecting the communications between a user interface and a user. These will be primarily event based, and will include a new low-cost camera-based eye-tracking system

During early development, existing interfaces will be evaluated manually to characterize their

(we need some way to specify interaction techniques...)

==Specific Contributions==
# A model of human cognitive and perceptual abilities when using computers
## Impact: Such a model would allow us to predict human performance with interfaces. Validation of the model would allow us to more rapidly converge on ideal interfaces while simultaneously ruling out sub-optimal ones.
## 3-Week Feasibility Study: (1) Distill (perhaps from review articles) the major findings in all of the relevant subfields into a set of principles that will be grouped to form appropriate model components. Some of the most relevant subfields include: memory, attention, visual perception, psychoacoustics, task switching, categorization, event perception, haptics (ANY OTHERS?). (2) Use these to devise a simple predictive model of human interaction. This model could simply consist of the set of design principles but should allow some form of quantitative scoring/evaluation of interfaces. (3) Develop a small set (5-10) of candidate GUIs that are designed to help the user accomplish the same overarching task(s) (e.g. importing, analyzing, and flexibly graphing data in Matlab). (4) Test/validate the model by comparing predicted performance to actual performance with the GUIs.
## Risks/Costs: There are always potential risks to any human subjects that might participate in testing the model which might necessitate IRB approval. Costs might include: (1) paying for any necessary hardware and software for developing, displaying, and testing the GUIs, and (2) paying for human subjects.
# Something about design rules collected and merged
## Something comparing these collected rules to a baseline (establishing their value)
## ???
# A predictive, fully-integrated model of user workflow which encompasses low-level tasks, working spheres, communication chains, interruptions and multi-tasking. (OWNER: Andrew Bragdon)
##Traditionally, software design and usability testing is focused on low-level task performance. However, prior work (Gonzales, et al.) provides strong empirical evidence that users also work at a higher, ''working sphere'' level. Su, et al., develops a predictive model of task switching based on communication chains. Our model will specifically identify and predict key aspects of higher level information work behaviors, such as task switching. We will conduct initial exploratory studies to test specific instances of this high-level hypothesis. We will then use the refined model to identify specific predictions for the outcome of a formal, ecologically valid study involving a complex, non-trivial application.
## Impact: To truly design computing systems which are designed around the way users work, we must understand ''how'' users work. To do this, we need to establish a predictive model of user workflow that encompasses multiple levels of workflow: individual task items, larger goal-oriented working spheres, multi-tasking behavior, and communication chains. Current information work systems are almost always designed around the lowest level of workflow, the individual task, and do not take into account the larger workflow context. Fundamentally, a predictive model would allow us to design computing systems which significantly increase worker productivity in the United States and around the world, by designing these systems around the way people work.
## Risk and Costs: Risk will play an important factor in this research, and thus a core goal of our research agenda will be to manage this risk. The most effective way to do this will be to compartmentalize the risk by conducting empirical investigations - which will form the basis for the model - into the separate areas: low-level tasks, working spheres, communication chains, interruptions and multi-tasking in parallel. While one experiment may become bogged down in details, the others will be able to advance sufficiently to contribute to a strong core model, even if one or two facets encounter setbacks during the course of the research agenda. The primary cost drivers will be the preliminary empirical evaluations, the final system implementation, and the final experiments which will be designed to support the original hypothesis. The cost will span student support, both Ph.D. and Master's students, as well as full-time research staff. Projected cost: $1.5 million over three years.
## 3-week Feasibility Study: To ascertain the feasibility of this project we will conduct an initial pilot test to investigate the core idea: a predictive model of user workflow. We will spend 1 week studying the real workflow of several people through job shadowing. We will then create two systems designed to help a user accomplish some simple information work task. One system will be designed to take larger workflow into account (experimental group), while one will not (control group). In a synthetic environment, participants will perform a controlled series of tasks while receiving interruptions at controlled times. If the two groups perform roughly the same, then we will need to reassess this avenue of research. However, if the two groups perform differrently then our pilot test will have lent support to our approach and core hypothesis.
# A low-overhead mechanism for capturing event-based interactions between a user and a computer, including web-cam based eye tracking. (should we buy or find out about borrowing use of pupil tracker?) ''Should we include other methods of interaction here? Audio recognition seems to be the lowest cost. It would seem that a system that took into account head-tracking, audio, and fingertip or some other high-DOF input would provide a very strong foundation for a multi-modal HCI system. It may be more interesting to create a software toolkit that allows for synchronized usage of those inputs than a low-cost hardware setup for pupil-tracking. I agree pupil-tracking is useful, but developing something in-house may not be the strongest contribution we can make with our time.'' (Trevor)
## Accuracy study of eye tracking (2 cameras? double as an input device?)
## ???
# A classification of standard design guidelines into overarching principles and measurement of the relation of each design principle with quantifiable cognitive principles. (Owners: EJ and Adam)
## A ranking/weighting of the relevance of each cognitive principle to each design principle.
## Specific rules for designing/assessing interfaces with respect to leveraging each cognitive principle for its most closely related design principles.
# A systematic method to determine task distribution based on psychological principles. (Owner - [http://vrl.cs.brown.edu/wiki/CS295J/Class_Members%27_Pages/Gideon Gideon])
## The theory of distributed cognition (Clark-1998-TEM) is a well-suited basis for constructing a human-computer interaction framework (Hollan-2000-DCF). However, a systematic method of determining which tasks should be distributed to which agent in a distributed environment has yet to be clearly defined. I propose that the cognitive psychology literature is ripe with empirical evidence on task performance variables. Dual-process theories (Evans-2003-ITM) usually agree on the nature of high-level cognitive operations which are used in human reasoning. It is also argued that low-level processing, which is based on perceptual similarity, contiguity, and association, is determined by a set of autonomous subsystems. In general, one might say that humans excel at these low-level functions in relation to traditional von-neuman architectures (and even current neural-networks), but recently, cognitive science has been less focused on research on high-level reasoning in humans as the evidence is showing that we rarely engage in such demanding cognitive operations. Therefore, we believe that an optimal configuration for distributed cognition amongst people and computers will take advantage of these specialties and deficiencies, and distribute tasks accordingly. Our research project will contribute a set of guidelines, or heuristics, to allow engineers to effectively determine which tasks should be assigned to which entities.
## The impact of this contribution will be a systematic approach to interface design. Engineers will finally have a well-documented standard about how to determine what operations humans should be responsible for, and which should be off-loaded to the computer.
## There is no risk for this contribution. Costs associated will be an extensive search of cognitive psychology literature for the past 25 years or so. Research on memory, reasoning, perception and more will be required in order to conduct a complete and accurate assessment.
## In the course of 30 hours, we may perform a hand-analysis of the empirical results in psychology in order to develop a set of approximately 10 guidelines or rules.
# A method for collecting data on user performance in cognitive, perceptual, and motor-control tasks that requires less monetary cost, allows for a greater number of samples, and measures user improvements over time. (Owner - Eric)
##In order to reach a model of human cognitive and perceptual abilities when using computers, experimental analysis of human performance on these tasks will likely be necessary. User studies can often aid in this analysis, but they require much money, much time, and are subject to user fatigue. Alternately, we propose a web-based method for evaluating user performance in perceptual, motor-control, and cognitive tasks. The idea is to take a task would normally be measured through user studies in a laboratory and map this task into a simple online game. Somewhat similar work has been done by Popovic et. al. at the University of Washington, in that they took the task of folding proteins and mapped it into an online game ( http://www.economist.com/displaystory.cfm?story_id=11326188 ) with much success. We will analyze the value of this method by comparing it to similar tasks performed in laboratory experiments, both in terms of user performance and deployment costs.
##By converting the task into a simple game, we hope to reduce the problem of user fatigue. Additionally, if the game is played on a social networking site, we are able to track basic information of users who perform the tasks and, more importantly, can identify returning users. Thus, we can track not only a user's performance, but also how they improve at a given task over time.
##There are no clear risks involved with this study. Potential costs would be those required for development of each experiment and for web hosting.
##As a prototype, we can select one particular task to map into a simple online game. To check for feasibility, we need to ensure that the results we get from our proposed method are similar to the results found in laboratory settings. There are two possible effects we must test for: bias in results due to the mapping into a game, and bias in results due to the sample of subjects or any change caused by the web-based component. A simple test would be to first test users in a lab using traditional methods as a baseline, and then see how performance differs from that in laboratory tests using the game-based mapping. This will determine if the game appropriately measures the given task. Next, an online version of the game can be introduced, and performance can be compared with the laboratory settings. If performance is similar in all of these tests, we have found a method for measuring low-level tasks that allows for many samples and minimal cost.

===Usability/cognition matrix===
A classification of standard design guidelines into overarching principles and measurement of the relation of each design principle with quantifiable cognitive principles.
# A ranking/weighting of the relevance of each cognitive principle to each design principle.
# Specific rules for designing/assessing interfaces with respect to leveraging each cognitive principle for its most closely related design principles.

==Specific Aims==

=== Primary ===
# Filter and merge existing HCI guidelines into a comprehensive collection of rules, with specific emphasis on those guidelines that derive from cognitive principles.
# Formulate a set of cognitive models to evaluate HCI guidelines quantitatively, with respect to cognition, drawing from research in Cognitive Science and Perceptual Pyschology.
# Formulate a set of performance/usability models to evaluate HCI guidelines quantitatively, with respect to user performance, drawing from research in Human-Computer Interaction and Ubiquitous Computing.
# Analyze correlations between cognition, perception and usability, validating (?) Cognitive Science's utility in a broad HCI context.
# Through isolated experiments, determine correlations between individual cognitive/perceptual processes and usability.
# Integrate findings into a quantitative system for evaluating HCI design with respect to performance, determining design decisions that are detrimental to user performance, and offering sets of suggestions for improvement.
# Integrate findings into a real-time system that may be plugged into an existing interactive application to enhance usability.
## This quantitative system will do so by taking into account:
### User goals, as determined via interaction history.
### Performance following interruptions or shifts in user-goals (working spheres).
### Collaborative usability through sharing interaction.

=== Other Application Aims ===
# Develop a scoring system for interfaces to evaluate the degree to which all changes and causal relations are tracked by motion cues that are contiguous in time and/or space.
# Accurately assess computational and psychological costs for tasks and subtasks. To do this, we will develop two non-trivial prototype systems; a conventional control system and a novel system which is based on our model of task switching. We will use our model to make specific predictions about relative task performance and user affect responses, and then test these predictions empirically in a formal study.
# Collect ratings of the design and cognitive principles on a range of interfaces and use this data to generate a relevance matrix.

====Usability/cognition matrix====
* Extract general design principles that are common across multiple sets of guidelines.
* Extract quantifiable cognitive principles that are relevant to HCI.
* Collect ratings of the design and cognitive principles on a range of interfaces and use this data to generate a relevance matrix.
* Generate specific rules for designing/assessing interfaces with regard to highly correlated pairs of design and cognitive principles.
* Investigate the possibility of using cognitive simulations to generate such assessments. [[User:E J Kalafarski|E J Kalafarski]] 16:46, 13 March 2009 (UTC)

==Background==

===Models of cognition===
There are several models of cognition, ranging from fundamental aspects of neurological processing to extremely high-level psychological analysis. Three main theories seem to have become recognized as the most helpful in conceptualizing the actual process of HCI. These models all agree that one cannot accurately analyze HCI by viewing the user without context, but the extent and nature of this context varies greatly.

'''[http://en.wikipedia.org/wiki/Activity_Theory Activity Theory]''', developed in the early 20th century by Russian psychologists S.L. Rubinstein and A.N. Leontiev, posits the existence of four discrete aspects of human-computer interaction. The "Subject" is the human interacting with the item, who possesses an "Object" (e.g. a goal) which they hope to accomplish by using a tool. The Subject conceptualizes the realization of the Object via an "Action", which may be as simple or complex as is necessary. The Action is made up of one or more "Operations", the most fundamental level of interaction including typing, clicking, etc.

A key concept in Activity Theory is that of the artifact, which mediates all interaction. The computer itself need not be the only artifact in HCI - others include all sorts of signs, algorithmic methods, instruments, etc.

A longer synopsis of Activity Theory may be found at [http://mcs.open.ac.uk/yr258/act_theory/ this website].

'''The [http://en.wikipedia.org/wiki/Situated_cognition Situated Action Model]''' focuses on emergent behavior, emphasizing the subjective aspect of human-computer interaction and the therefore-necessary allowance for a wide variety of users. This model proposes the least amount of contextual interaction, and seems to maintain that the interactive experience is determined entirely by the user's ability to use the system in question. While limiting, this concept of usability can be very informative when designing for less tech-savvy users.

'''[http://en.wikipedia.org/wiki/Distributed_cognition Distributed Cognition]''' proposes that the computer (or, as in Activity Theory, any other artifact) can be used and ought to be thought of as an extension of the mental processing of the human. This is not to say that the two are of equal or even comparable cognitive abilities, but that each has unique strengths and that recognition of and planning around these relative advantages can lead to increased efficiency and effectiveness. The rotation of blocks in Tetris serves as a perfect example of this sort of cognitive symbiosis.

====Embodied Models of Cognition====
This field of applied research straddles the divide between cognitive modeling and automated user evaluation. Embodied models are so named because they employ sensory and mechanic modules in order to simulate the process of a user's interaction with a given interface. Such models have much promise for advancing the field of automated user evaluation, but literature thus reviewed suggests that current models leave much to be desired.

Models rely on a combination of data gleaned from the interface itself and information and instructions delivered by an evaluation expert. Simulated eyes and ears can interpret stimuli from the interface, but these stimuli may have to be predefined by the expert to varying degrees, depending on the specific model and the complexity of the stimuli. Simulated hands can deliver input under the constraints of well established laws of user performance, as well as with estimated levels of error.

The imperfect nature of these interactions is both a boon and a burden. A new, interesting perspective can be gained, but it may turn out to irrelevant when considering the behavior of true humans (aka a false positive). An error may be logged in the design of a certain pane, but the appearance may in fact be highly salient for human users (aka a false negative). Minimizing these errors is a difficult task, and it may prove to be quite a long time before embodied models can surpass other forms of automated evaluation.

Embodied models do offer a few undeniable benefits, however. They can evaluate interfaces at an early level which would otherwise not be fiscally justifiable. They can also continue to evaluate interfaces at the designer's leisure - at any hour and for any duration. While these tasks may also be accomplished by other automated evaluation tools, embodied models require preparatory minimal effort on the part of the designer.

For several reasons however, not the least of which is the difficulty in defining the detail level at which an interface ought to operate, embodied models are unlikely to become substitutes for human subjects in the near future.

(Steven)

====Workflow Context====
There are, at least, two levels at which users work ([http://portal.acm.org/citation.cfm?id=985692.985707&coll=portal&dl=ACM&CFID=20781736&CFTOKEN=83176621 Gonzales, et al., 2004]). Users accomplish individual low-level tasks which are part of larger ''working spheres''; for example, an office worker might send several emails, create several Post-It (TM) note reminders, and then edit a word document, each of these smaller tasks being part of a single larger working sphere of "adding a new section to the website." Thus, it is important to understand this larger workflow context - which often involves extensive levels of multi-tasking, as well as switching between a variety of computing devices and traditional tools, such as notebooks. In this study it was found that the information workers surveyed typically switch individual tasks every 2 minutes and have many simultaneous working spheres which they switch between, on average every 12 minutes. This frenzied pace of switching tasks and switching working spheres suggests that users will not be using a single application or device for a long period of time, and that affordances to support this characteristic pattern of information work are important.

Czerwinski, et al. conducted a [http://portal.acm.org/citation.cfm?id=985692.985715&coll=Portal&dl=ACM&CFID=21136843&CFTOKEN=23841774 diary study] of task switching and interruptions of users in 2004. This study showed that task complexity, task duration, length of absence, and number of interruptions all affected the users' own perceived diffculty of switching tasks. [http://delivery.acm.org/10.1145/1250000/1240730/p677-iqbal.pdf?key1=1240730&key2=4525483321&coll=Portal&dl=ACM&CFID=21136843&CFTOKEN=23841774 Iqbal, et al.] studied task disruption and recovery in a field study, and found that users often visited several applications as a result of an alert, such as a new email notification, and that 27% of task suspensions resulted in 2 hours or more of disruption. Users in the study said that losing context was a significant problem in switching tasks, and led in part to the length of some of these disruptions. This work hints at the importance of providing cues to users to maintain and regain lost context during task switching.

(Andrew Bragdon - OWNER)

The problem of task switching is exacerbated when some tasks are more routine than others. When a person intends to switch from a routine task to a novel task at some later time, they often forget the context of the original task ([http://web.ebscohost.com/ehost/pdf?vid=1&hid=7&sid=54ec1e22-3df2-462c-b484-7a7c052c2173%40SRCSM1 Aarts et al., 1999]). Also, if both tasks are done in the same context, with the same tools or with the same materials, people have difficulty inhibiting the routine task while doing the novel task (Stroop, 1935). This inhibition also makes switching back to the routine task slower (Allport et al., 1994). All of these problems can be alleviated to some degree by salient cues in the environment. Enacting the intention to switch becomes easier when there is a salient reminder at the appropriate time (McDaniel and Einstein, 1993) and associating different environmental cues with different goals can automatically trigger appropriate behavior ([http://web.ebscohost.com/ehost/pdf?vid=1&hid=4&sid=68f77032-f093-4139-a833-760d2217b513%40sessionmgr9 Aarts and Dijksterhuis, 2003]).

(Adam)

(Edited by Andrew)

====Qunatitative Models: Fitts's law, Steering Law====
Fitts's law and the steering law are examples of quantiative models that predict user performance when using certain types of user interfaces. In addition to these classic models, [http://delivery.acm.org/10.1145/1250000/1240850/p1495-cao.pdf?key1=1240850&key2=9904483321&coll=GUIDE&dl=GUIDE&CFID=20565232&CFTOKEN=69614228 Cao and Zhai] developed and validated a quantitative model of human performance of pen stroke gestures in 2007. [http://tlaloc.sfsu.edu/~lank/research/appearing/FSS604LankE.pdf Lank and Saund] utilized a model which used curvature to predict the speed of a pen as it moved across a surface to help disambiguate target selection intent.

In addition, quantitative models are often tested against new interfaces to verify that they hold. For example, [http://portal.acm.org/citation.cfm?id=1054972.1055012&coll=GUIDE&dl=ACM&CFID=20565232&CFTOKEN=69614228 Grossman et al.] verified that their Bubble Cursor approach to enlarging effective pointing target sizes obeyed Fitts's law for actual distance traveled.

In addition to formal models, machine learning techniques have been applied to modeling user interaction as well. For example, [http://delivery.acm.org/10.1145/1250000/1240669/p271-hurst.pdf?key1=1240669&key2=6465483321&coll=GUIDE&dl=GUIDE&CFID=20565232&CFTOKEN=69614228 Hurst, et al.], used a learning classifier, trained on low-level mouse and keyboard usage patterns, to identify novice and expert users dynamically with accuracies as high as 91%. This classifier was then used to provide different information and feedback to the user as appropriate.

(Andrew Bragdon - OWNER)

====Distributed cognition====

Distributed cognition is a theory in which thoughts take place in and outside of the brain. Human's have a great ability to use tools and to incorporate their environments into their sphere of thinking. Clark puts it nicely in [[http://vrl.cs.brown.edu/wiki/index.php?title=CS295J/Literature#Cognition Clark-1994-TEM]].

Therefore, optimal configurations when considering HCI design will treat the brain, person, interface, and computer as a holistic system comprising cognitions.

In practical terms, the issue at hand for our proposal is how to best maximize utility by distributing the cognitive tasks at hand to different components of the whole system. Simply, which tasks can we off-load to the computer to do for us, faster and more accurately? What tasks should we purposely leave the computer out of?

Typically, those tasks most eligible to be off-loaded are the ones which we perform poorly on. Conveniently, the tasks which computers perform poorly on we excel at. Here are a few examples:

*Computer's Area of Expertise: number crunching, memory, logical reasoning, precise
*Human's Area of Expertise: Associative thought, real-world knowledge, social behavior, alogical reasoning, imprecise

Using this division of cognitive labor allows us to optimize task work flows. Ignoring it puts strain and bottlenecks at either the computer, the human, or the interface. The field of HCI is full of examples of failures which can be attributed to not recognizing which tasks should be handled by which sub-system.

As a heuristic to divide thinking, one might turn to the dual-process theory literature [http://vrl.cs.brown.edu/wiki/CS295J/Literature Evans-2003-ITM]. What is most often called System 1 is what human's are good at, while System 2 tasks are what computers do well.

(Owner - [http://vrl.cs.brown.edu/wiki/CS295J/Class_Members%27_Pages/Gideon Gideon])

====Information Processing Approach to Cognition====
The dominant approach in Cognitive Science is called information processing. It sees cognition as a system that takes in information from the environment, forms mental representations and manipulates those representations in order to create the information needed to achieve its goals. This approach includes three levels of analysis originally proposed by Marr (will cite):
# Computational - What are the goals of a process or representation? What are the inputs and desired outputs required of a system which performs a task? Models at this level of analysis are often considered normative models, because any agent wanting to perform the task should conform to them. Rational agent models of decision making, for example, belong at this level of analysis.
# Process/Algorithmic - What are the processes or algorithms involved in how humans perform the task? This is the most common level of analysis as it focuses on the mental representations, manipulations and computational faculties involved in actual human processing. Algorithmic descriptions of human capabilities and limitations, such as working memory size, belong at this level of analysis.
# Implementation - How are the processes and algorithms be actual biological computation? Dopamine theories of reward learning, for example, belong at this level of analysis.
The information processing approach is often contrasted with the distributed cognition approach. Its advantage is that it finds general mechanisms that are valid across many different contexts and situations. Its disadvantage is that it can have difficulty explaining the rich interactions between people and their environment.

In considering users as information processors, interfaces should take into account people's computational limitations on short term memory, learning and vision as well as the algorithms and representations that they use to process information and pursue goals.

(Adam)

===Models of perception===

====The Ecological Approach to Perception====
(relevant stuff about Gibson's theory)
We will build on top of this theory/model by ... .

Gibsonianism, named after James J. Gibson and more commonly referred to as ecological psychology, is an epistemological direct realist theory of perception and action. It is often contrasted to information processing and cognitivist approaches which generally assume that perception is a constructive process operating on impoverished sense-data inputs (e.g. photoreceptor activity) to generate representations of the the world with added structure and meaning (e.g. a mental or neural "picture" of a chair), ecological psychology treats perception as direct, non-inferential, unmediated (by retinal images or mental representations) epistemic contact with behaviorally-relevant features of the environment (Warren, 2005). The possibilities for action that the environment offers a given animal are taken to be specified by the co-perception of self (e.g. proprioception) and information available in structured energy distributions (e.g. the optic array of light arriving at the eyes), and these possibilities for action constitute the affordances of the environment with respect that animal (Gibson, 1986).

The notion of affordance has proven to be a very useful concept in HCI, however, some of its original meaning has been lost in translation. For example, it is worth noting that the original definition of affordances emphasizes possibilities for action and not their relative likelihoods. For example, for most humans, laptop computer screens afford puncturing with Swiss Army knives, however, it is unlikely that a user will attempt to retrieve an electronic coupon by carving it out of their monitor. This example illustrates that interfaces often afford a class of actions that are undesirable from the perspective of both the designer and the user.

===Design guidelines===

A multitude of rule sets exist for the design of not only interface, but architecture, city planning, and software development. They can range in scale from one primary rule to as many Christopher Alexander's 253 rules for urban environments,<ref>http://hci.rwth-aachen.de/materials/publications/borchers2000a.pdf</ref> which he introduced with the concept design patterns in the 1970s. Study has likewise been conducted on the use of these rules:<ref>http://stl.cs.queensu.ca/~graham/cisc836/lectures/readings/tetzlaff-guidelines.pdf</ref> guidelines are often only partially understood, indistinct to the developer, and "fraught" with potential usability problems given a real-world situation.

====Application to AUE====

And yet, the vast majority of guideline sets, including the most popular rulesets, have been arrived at heuristically. The most successful, such as Raskin's and Schneiderman's, have been forged from years of observation instead of empirical study and experimentation. The problem is similar to the problem of circular logic faced by automated usability evaluations: an automated system is limited in the suggestions it can offer to a set of preprogrammed guidelines which have often not been subjected to rigorous experimentation.<ref>http://www.eecs.berkeley.edu/Pubs/TechRpts/2000/CSD-00-1105.pdf</ref> In the vast majority of existing studies, emphasis has traditionally been placed on either the development of guidelines or the application of existing guidelines to automated evaluation. A mutually-reinforcing development of both simultaneously has not been attempted.

Overlap between rulesets is inevitable and unavoidable. For our purposes of evaluating existing rulesets efficiently, without extracting and analyzing each rule individually, it may be desirable to identify the the overarching ''principles'' or ''philosophy'' (max. 2 or 3) for a given ruleset and determining their quantitative relevance to problems of cognition.

====Popular and seminal examples====
Schneiderman's [http://faculty.washington.edu/jtenenbg/courses/360/f04/sessions/schneidermanGoldenRules.html Eight Golden Rules] date to 1987 and are arguably the most-cited. They are heuristic, but can be somewhat classified by cognitive objective: at least two rules apply primarily to ''repeated use'', versus ''discoverability''. Up to five of Schneiderman's rules emphasize ''predictability'' in the outcomes of operations and ''increased feedback and control'' in the agency of the user. His final rule, paradoxically, removes control from the user by suggesting a reduced short-term memory load, which we can arguably classify as ''simplicity''.

Raskin's [http://www.mprove.de/script/02/raskin/designrules.html Design Rules] are classified into five principles by the author, augmented by definitions and supporting rules. While one principle is primarily aesthetic (a design problem arguably out of the bounds of this proposal) and one is a basic endorsement of testing, the remaining three begin to reflect philosophies similar to Schneiderman's: reliability or ''predictability'', ''simplicity'' or ''efficiency'' (which we can construe as two sides of the same coin), and finally he introduces a concept of ''uninterruptibility''.

Maeda's [http://lawsofsimplicity.com/?cat=5&order=ASC Laws of Simplicity] are fewer, and ostensibly emphasize ''simplicity'' exclusively, although elements of ''use'' as related by Schneiderman's rules and ''efficiency'' as defined by Raskin may be facets of this simplicity. Google's corporate mission statement presents [http://www.google.com/corporate/ux.html Ten Principles], only half of which can be considered true interface guidelines. ''Efficiency'' and ''simplicity'' are cited explicitly, aesthetics are once again noted as crucial, and working within a user's trust is another application of ''predictability''.

====Elements and goals of a guideline set====

Myriad rulesets exist, but variation becomes scarce—it indeed seems possible to parse these common rulesets into overarching principles that can be converted to or associated with quantifiable cognitive properties. For example, it is likely ''simplicity'' has an analogue in the short-term memory retention or visual retention of the user, vis a vis the rule of [http://books.google.com/books?hl=en&lr=&id=j5q0VvOGExYC&oi=fnd&pg=PA357&dq=seven+plus+or+minus+two&ots=prI3PKJBar&sig=vOZnqpnkXKGYWxK6_XlA4I_CRyI Seven, Plus or Minus Two]. ''Predictability'' likewise may have an analogue in Activity Theory, in regards to a user's perceptual expectations for a given action; ''uninterruptibility'' has implications in cognitive task-switching;<ref>http://portal.acm.org/citation.cfm?id=985692.985715&coll=Portal&dl=ACM&CFID=21136843&CFTOKEN=23841774</ref> and so forth.

Within the scope of this proposal, we aim to reduce and refine these philosophies found in seminal rulesets and identify their logical cognitive analogues. By assigning a quantifiable taxonomy to these principles, we will be able to rank and weight them with regard to their real-world applicability, developing a set of "meta-guidelines" and rules for applying them to a given interface in an automated manner. Combined with cognitive models and multi-modal HCI analysis, we seek to develop, in parallel with these guidelines, the interface evaluation system responsible for their application. [[User:E J Kalafarski|E J Kalafarski]] 15:21, 6 February 2009 (UTC)

:I. Introduction, applications of guidelines
::A. Application to automated usability evaluations (AUE)
:II. Popular and seminal examples
::A. Shneiderman
::B. Google
::C. Maeda
::D. Existing international standards
:III. Elements of guideline sets, relationship to design patterns
:IV. Goals for potentially developing a guideline set within the scope of this proposal

===User interface evaluations===

====Interaction capture====

[http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=4376144 Yi et. al.] have performed a survey of the visualization literature and categorized different types of interactions that users were faced with. They are as follows:
# Select: mark something as interesting
# Explore: show me something else
# Reconfigure: show me a different arrangement
# Encode: show me a different representation
# Abstract/Elaborate: show me more or less detail
# Filter: show me something conditionally
# Connect: show me related items

Different GUI components may be able to perform the same type of interaction. We would like to categorize GUI components or patterns that are used bring about these interactions. We then have a library of components we can use to complete a given task. The goal is to create components for a given interaction that can minimize cost to the user. Because the cost of a component is likely dependent on the other components used, the goal of the designer might be to choose a combination of components that minimizes this cost. To do this, we need a way to measure costs, which is discussed in the next section.

====Cost-based analyses====
In [http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=4658124 A Framework of Interaction Costs in Information Visualization], Lam performs a survey of 32 user studies and classifies several types of costs that can be used for qualitative interface evaluation. The classification scheme is based on Donald Norman's [http://en.wikipedia.org/wiki/Seven_stages_of_action Seven Stages of Action] from his book, [http://www.amazon.com/Design-Everyday-Things-Donald-Norman/dp/0385267746 The Design of Everyday Things] ([http://www.networksplus.net/tracyj/everydaythings.pdf summary]).

# '''Decision costs:''' How does user performance decrease when there is an overwhelming amount of data to observe or too many possible actions to take.
# '''System-power costs:''' How does the user translate a high-level goal into a sequence of allowable actions by the interface?
# '''Multiple input mode costs:''' Cost of providing an action selection system that is not unified, for example, if there is one button that does two different things, depending on context.
# '''Physical-motion costs:''' Physical cost to the user to interact with the interface, for example, measuring mouse movement costs with Fitts' Law.
# '''Visual-cluttering costs:''' Cost due to unwanted visual distractions, such as a mouse hovering pop-up occluding part of the screen.
# '''View- and State-change costs:''' when the user causes the interface to change views, this new view should be consistent with the old one, in that it should meet the users expectations of where things should be in the new view, based on his knowledge of the old one.

==== Evaluation in practice ====
User interfaces are usually evaluated in practice using two methods: ''usability inspection methods'', where a programmer or one or more experts evaluates the interface through inspection; or ''usability testing'', where empirical tests are performed with some group of naive human users. Some usability inspection methods include [http://en.wikipedia.org/wiki/Cognitive_walkthrough Cognitive walkthrough], [http://en.wikipedia.org/wiki/Heuristic_evaluation Heuristic evaluation], and [http://en.wikipedia.org/wiki/Pluralistic_walkthrough Pluralistic walkthrough]. While these inspection methods do not using naive human subjects, the details of the methods might be useful in helping to formalize what interactions are made between a user and an interface, and what each interactions' costs are for a given design.

[http://portal.acm.org/citation.cfm?id=108862 Jeffries et. al.] provide a real-world comparison between two of the usability inspection methods (heuristic evaluation and cognitive walkthrough), the usability testing method, as well as following some published software guidelines for interface design.

=== Multimodal HCI ===

Continued advancements in several signal processing techniques have given rise to a multitude of mechanisms that allow for rich, multimodal, human-computer interaction. These include systems for head-tracking, eye- or pupil-tracking, fingertip tracking, recognition of speech, and detection of electrical impulses in the brain, among others [http://vr.kjist.ac.kr/~dhong/website/paperworks/hci2002coursePapers/April24/Sharma98.pdf Sharma-1998-TMH]. With ever-increasing computing power, integrating these systems in real-time applications has become a plausible endeavor.

==== Head-tracking ====
:In virtual, stereoscopic environments, head-tracking has been exploited with great success to create an immersive effect, allowing a user to move freely and naturally while dynamically updating the user’s viewpoint in a visual environment. Head-tracking has been employed in non-immersive settings as well, though careful consideration must be paid to account for unintended movements by the user, which may result in distracting visual effects.

==== Pupil-tracking ====
:Pupil-tracking has been studied a great deal in the field of Cognitive Science ... (need some examples here from CogSci). In the HCI community, pupil-tracking has traditionally been used for posterior analysis of interface designs, and is particularly prevalent in web interface design. An alternative utility of pupil-tracking is to employ it in real-time as an actual mode of interaction. This has been examined in relatively few cases (citations), where typically the eyes are used to control a cursor onscreen. Like head-tracking, implementations of pupil-tracking must be conscious of unintended eye-movements, which are incredibly frequent.

==== Fingertip-tracking, Gestural recognition ====
:Fingertip tracking and gestural recognition of the hands are the subjects of much research in the HCI community, particularly in the Virtual Environment and Augmented Reality disciplines. Less implicit than head or pupil-tracking, gestural recognition of the hands may draw upon the wealth of precedents readily observed in natural human interactions. As sensing technologies become less obtrusive and more robust, this method of interaction has the potential to become quite effective.

==== Speech Recognition ====
:Speech recognition is becoming much better, though effective implementation of its desired effects is non-trivial in many applications. (More on this later).

==== Brain Activity Detection ====
:The use of electroencephelograms (EEGs) in HCI is quite recent, and with limited degrees of freedom, few robust interfaces have been designed around it. Some recent advances in the pragmatic use of EEGs in HCI research can be seen in [http://portal.acm.org/citation.cfm?id=1357054.1357187&coll=Portal&dl=GUIDE&CFID=21136843&CFTOKEN=23841774 Grimes et al.] The possibility of using brain function to interface with a machine is cause for great excitement in the HCI community, and further advances in non-invasive techniques for accessing brain function may allow teleo-HCI to become a reality.

In sum, the synchronized usage of these modes of interaction make it possible to architect an HCI system capable of sensing and interpreting many of the mechanisms humans use to transmit information among one another. (Trevor)

=== Workflow analysis ===

Research in workflow and interaction analysis remains relatively sparse, though its utility would appear to be many-fold. Tools for such analysis have the potential to facilitate data navigation, provide search mechanisms, and allow for more efficient collaborative discovery. In addition, awareness and caching of interaction histories readily allows for explanatory presentations of results, and has the potential to provide training data for machine learning mechanisms.

VisTrails is an optimized workflow system developed at the Sci Institute at the University of Utah, and implemented within their VTK visualization package. The primary purpose of the system is to increase performance when working with multiple visualizations simultaneously. This is accomplished by storing low-level workflow processes to reduce computational redundancy. Three papers on VisTrails can be found here: [http://www.cs.brown.edu/people/trevor/Papers/Callahan-2006-MED.pdf Callahan-2006-MED], [http://www.cs.brown.edu/people/trevor/Papers/Callahan-2006-VVM.pdf Callahan-2006-VVM], [http://www.cs.brown.edu/people/trevor/Papers/Bavoil-2005-VEI.pdf Bavoil-2005-VEI]

Jeff Heer of Stanford (formerly Berkley) has presented work on using Graphical Interaction Histories within the Tableau InfoVis application. Though geared toward two-dimensional visualizations with clearly defined events, his work offers some very useful design guidelines for working with interaction histories, including some evaluation from the deployment of his techniques within Tableau. His paper from Infovis '08 can be seen here: [http://www.cs.brown.edu/people/trevor/Papers/Heer-2008-GraphicalHistories.pdf Heer-2008-GraphicalHistories]

If you want to check out some of Trevor's work having to do with using interaction histories in 3D, time-varying scientific visualizations, his preliminary work that was presented at Vis '08 can be seen here: [http://www.cs.brown.edu/people/trevor/trevor_iweb/Publications_files/obrien-2008-visDemo.pdf Abstract], [http://www.cs.brown.edu/people/trevor/trevor_iweb/Publications_files/obrien-2008-visPoster.pdf Poster]

Optimizing workflows that have been captured -- Tovi?

Does ethnography fit in here?

* [http://viscog.cs.drexel.edu/courses/cs530-w06/readings/Horvitz-UAI98.pdf The Lumiere Project: Bayesian User Modeling for Inferring the Goals and Needs of Software Users]
* [http://www.cs.washington.edu/homes/pedrod/papers/kdd02a.pdf Relational Markov Models and their Application to Adaptive Web Navigation]
:Introduces the notion of Relational Markov Models (RMMs), a construct that extends traditional Markov Models by imposing a relational hierarchy on the state space. This abstraction allows for learning and inference on very large state spaces with only sparse training data. These methods are evaluated with respect to Adaptive Web Interfaces, but I believe the general RMM idea can be applied in many other HCI settings.

* [http://www.cs.brown.edu/people/trevor/Papers/Heer-2008-GraphicalHistories.pdf Graphical Histories for Visualization: Supporting Analysis, Communication, and Evaluation]
: Work by Jeff Heer of Stanford (formerly Berkley) on using Graphical Interaction Histories within the Tableau InfoVis application. This is a great recent example of "workflow analysis" that we've been discussing in class. Though geared toward two-dimensional visualizations with clearly defined events, his work offers some very useful design guidelines for working with interaction histories, including evaluations from the deployment of his techniques within Tableau. (Trevor)

* [http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.68.5619&rep=rep1&type=pdf A Machine Learning Approach to Web Personalization]
* [http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.56.7874&rep=rep1&type=pdf Learning Dynamic Bayesian Networks]
* [http://www.informatik.uni-freiburg.de/~ml/teaching/ws03/pll/rmms.ppt Relational Hidden Markov Models (powerpoint)]
* [http://www.springerlink.com/content/h181k064m40711k6/fulltext.pdf COLLAGEN: A Collaboration Manager for Software Interface Agents]
* [http://www.cs.utexas.edu/~ear/CogSci.pdf User Modeling via Stereotypes]
* [http://siona.udea.edu.co/~jfduitam/thesis/projects/ELM-ART/userModelingAndAdaptiveNavigation.pdf User Modeling and Adaptive Navigation Support in WWW-Based Tutoring Systems]
* [http://www.um.org/ User Modeling Inc.]
* [http://delivery.acm.org/10.1145/340000/336374/p220-dieberger.pdf?key1=336374&key2=5275657321&coll=GUIDE&dl=GUIDE&CFID=27551349&CFTOKEN=46488930 Visualizing Interaction History on a Collaborative Web Server]
* [http://delivery.acm.org/10.1145/650000/642673/p353-kaptelinin.pdf?key1=642673&key2=0375657321&coll=GUIDE&dl=GUIDE&CFID=27551320&CFTOKEN=62077475 UMEA: Translating Interaction Histories into Project Contexts]

==Significance==

While attempts have been made in the past to apply cognitive theory to the task of developing human-computer interfaces, there remains much work to be done. No standard and widespread model for cognitive interaction with a device exists. The roles of perception and cognition, while examined and studied independently, are often at odds with empirical and successful design guidelines in practice. Methods of study and evaluation, such as eye-tracking and workflow analysis, are still governed primarily by the needs at the end of the development process, with no quantitative model capable of influencing efficiency and consistency in the field.

We demonstrate in wide-ranging preliminary work that cognitive theory has a tangible and valuable role in all the stages of interface design and evaluation: models of distributed cognition can exert useful influence on the design of interfaces and the guidelines that govern it; algorithmic workflow analysis can lead to new interaction methods, including predictive options; a model of human perception can greatly enhance the usefulness of multimodal user study techniques; a better understanding of why classical strategies work will bring us closer to the "holy grail" of automated interface evaluation and recommendation. We can bring the field further down many of the only partially-explored avenues of the field in the years ahead.

The disparate elements of our proposed works are the building blocks of a cohesive theoretical understanding of the role cognition is capable of playing. As a demonstration of the feasibility of a more complete, multi-year investigation, these elements have been specifically selected to show the opportunities of integration throughout the interface development process. We construct an architecture of this preliminary work, showing the relationships of these layers and how they construct a foundation for further substantial work. [[User:E J Kalafarski|E J Kalafarski]] 14:41, 20 March 2009 (UTC)

(diagram)

==Preliminary results==

=== Gideon ===

'''Preliminary Psychological Measures Show the Need for a Standardized Division of Cognitive Labor Across Humans and Computers'''

=====Week 1:=====

A meta-analysis of a subset of the cognitive psychology and human-computer interaction literature presents evidence that interactions between humans and computers can be improved by taking into account the cognitive resources required for different types of tasks. It is well known that humans and computers excel at different types of tasks, but the field has not made an explicit effort to standardize a set of guidelines that interface designers may use when developing computer systems. For people and computers to function together in an optimized, complementary fashion, we still need a systematic way of distributing tasks amongst them.

It is often the case that what computers excel at, humans have difficulty with (e.g., elaborate arithmetic). While, the opposite is also true (e.g., some forms of pattern-recognition). After a preliminary search of the literature, we've explored common tasks in software today that have neglected consideration of this performance dichotomy. Designers have not been appropriately addressed these gaps in the computer industry due to a lack of multidisciplinary research. Our meta-analysis presents data that supports our view on two different tasks: 2D shape rotation and Go.

=====Week 2=====

To support our meta-analysis, we conducted two experiments to verify our results. They are outlined below.

Experiment 1:

It has been shown that computers consistently perform better when having to rotate a 2D shape in performing a decision task. In our paradigm, both a computer and people were given an image of an object, and asked to choose from 4 other objects the only one which was a 2D rotation of the primed object. Computers are both more accurate (100% correctness) and faster in their decisions.

Experiment 2:

We have shown that in a classification task, people consistently perform better than state-of-the-art trained computer systems when judging board positions in the game Go.

Our preliminary studies clearly demonstrate that interactive systems benefit from consistent and rule-based task distribution guidelines. Furthermore, the literature is rich with other types of tasks which await our systematic exploitation. Upon further study, the community will benefit from a tested and systematic approach for designing improved human-computer interfaces.

=====Week 3=====

Our previous findings have supported our claims, but this week we have extended our concept beyond perception-based tasks. Here, we have created a custom task in which participants asked to complete logical syllogisms are given three different versions of software to assist them in their task.

Version 1: Three static black squares on a white screen.
Version 2: Three movable black circles which cannot touch on a screen.
Version 3: Three transparent labeled movable circles which may collide on a screen.

Examples like the following were given:

No nutritional things are inexpensive
Some vitamin tablets are inexpensive
Therefore, some vitamin tablets are not nutritional

Participants using version 3 performed better than using any other version. This is evidence of cognitive offloading of what working memory would be having to store and manipulate being performed (better) by machine.

=== Steven ===

Preliminary Results:

Redone Week 1 Results

*Problem: The current state of cognitive HCI theory is both deeply conflicted and antiquated. We have carefully examined the existing literature and began developing a more modern, applicable, and empirically proven conceptualization of the interaction itself.

Preliminary Work:
*A literature review of the leading cognitive HCI theories, with special attention paid to activity theory and subjective realities.
::The development of a common vocabulary with which to describe the components of each of these theories, to facilitate both understanding and to discern applicability to the current digital environment. This lexicon is then used to define the intuitive merits of each approach and isolate them as contributing factors for the eventual theory. This segment required 30 hours of research.
*Task-oriented user studies are conducted to distill the users’ progression of goal development and execution. Users are asked to describe their actions when viewing a recording at a later session, and these descriptions are analyzed to compile a list of working adjectives which users might use to describe their actions (in contrast with those used by developers). A factor analysis produces the most salient terms, which are linked to those derived in the 30 hour project.
::These studies are also used as the proof-of-concept of the development of a descriptive model against which the prescriptive theory will be measured.
Week 2 Results
*We have developed a skeleton of the hybrid cognitive HCI theory, along with descriptions of the sources of each component and relevant citations. This skeleton has been compared to the descriptive data gained from the above trials, as well as additional user trials based upon the insight gained from these original comparisons. The skeleton presents a foundation upon which we can begin to describe not merely the stages in a user’s task conception and completion but the more pertinent transition between these stages. We have thus begun to correlate semantic data from user trials with the theory’s stages and created a framework in which to present empirical evidence of the theory’s viability.

===Trevor and Eric===

''Interaction Histories''

'''Project idea:''' Generating interaction histories within scientific visualization applications to facilitate individual and collaborative scientific discovery.

'''Preliminary Work'''
# A software infrastructure for caching and filtering interaction events has been developed within an existing scientific application aimed at exploring animal kinematic data captured via high-speed x-ray and CT.
# Methods for visualizing, editing, and sharing interaction histories have been designed and implemented.
# Methods for annotating and querying interactions have been implemented.

'''Preliminary Work from the Future'''
# Software model extended to two other interactive visualizations -- a flow visualization application, and a protein visualization application.
# User study to examine techniques for automatic history generation
## Automatic creation v. semi-automatic creation v. manual creation
# Timed-task pilot study performed to validate utility of interaction history techniques
## Task performance with histories v. without histories

'''General Outline of Tasks'''
# Capture user interaction history
# Predict user interactions, given interaction history
## Use a relational markov model?
# Modify the UI, given predicted user interactions
# Evaluate this modified UI
## Compare performance with and without UI modifications
## Evaluate performance when predicted interactions are incorrect x% of the time

===EJ===
'''Problem:''' There currently exists no metric for evaluating interfaces that attempts to reconcile popular and successful heuristic design guidelines with cognitive theory.

'''Preliminary Work''' 
I. A mapping of empirically-effective heuristic design guidelines to fundamental cognitive principles.
:*A set of "common" design guidelines is arrived at through survey of popular and effective heuristic guidelines in use.
:*Proposed common design principles:
Discoverability
Flexibility
Appropriate visual presentation
Predictability
Consistency
Simplicity
Memory load reduction
Feedback
Task match
User control
Efficiency
:*Proposed cognitive principles:
Affordance
Visual cue
Cognitive load
Chunking
Activity
Actability
II. A weighting or priority for each of these analogues.
:*These can begin as binary estimations based on empirical evidence.
:*Through experimentation, these values should converge to discrete priority values for each analogue, allowing a ranking of analogues.
III. A system for applying these analogues and respective priority to the evaluation of an interface.
:*This can occur manually or in an automated fashion.
:*In this step (or possibly in a separate step), analogues should be assigned a ''suggestion'' or potential correction to provide in the event of "failure" of a particular test by a given interface.

===EJ and Jon===

'''Photoshop Study'''

We demonstrate that the efficiency of performing subtasks in Photoshop can be predicted by a simple model of human perceptual and cognitive abilities. In particular, we show that several of the tools commonly used to perform basic operations in Photoshop often violate the user's expectations of how those tools should work or where those tools ought to be located within the interface. These violations can be categorized as follows: (1) unintuitive relationships between adjustments to common tool parameters and their perceptual results (e.g. adjusting the Magic Wand tool's "tolerance" setting often leads to unintended selections), (2) inefficient means of adjusting tool parameters (e.g. adjusting the "tolerance" setting by clicking, typing a number, hitting enter, observing the results, and iterating this process until the desired perceptual effect is achieved), (3) mismatches between the user's expectations for the names and locations of tools (or menu items) and their actual names and locations (e.g. resizing a picture via the "transform" menu item), (4) the availability of a tool in multiple locations imposes a cognitive load on the user when searching for that tool and these various contexts influence the user's expectations about the effects of using that tool, (5) [what else?], etc.

===Jon===

We have developed a model of human cognitive and perceptual abilities that allows us to predict human performance and thereby converge on ideal interfaces while simultaneously ruling out sub-optimal ones. The model consists of a set of design principles combined with an extensive catalog of human perceptual and cognitive constraints on interface design. The effectiveness of this model has been demonstrated by using the combined set of principles to assign scores to a set of GUIs designed to help a user accomplish the same overarching task, and comparing those scores with actual user performance.

===Eric===

'''Project idea:''' Introduce a method for collecting data on user performance in cognitive, perceptual, and motor-control tasks that requires less monetary cost, allows for a greater number of samples, and measures user improvements over time.

'''Preliminary Work'''
Create a simple "brain training"-style game in which users must perform a simple cognitive task. As an example, perhaps the task is to manipulate shape1 into shape2, given a simple set of operators. Each of the users actions (mouse movements, button clicks, etc.) will be documented along with the state of the game at that time. By varying the interface for different users, we can see how it affects performance in terms of cognitive and low-level tasks.

Preliminary tests will first measure user performance in a laboratory setting. We will run subjects on two different interfaces and compare the differences in performance. We will then perform this same test in an online setting, and will evaluate how performance differs in this case, which is more subject to user interruptions and noisy data. If performance is similar in all of these tests, we have found a method for measuring low-level tasks that allows for many samples and minimal cost. If performance differs greatly, it may be that the amount of noise introduced by users playing in a casual setting may make the project infeasible.

===Andrew Bragdon===

====I. '''Goals'''====

A. Develop a qualitative theory for predicting user performance with and without automatic meta-work tools for saving and resuming context.

B. Formative study should inspire 30-hour feasibility study

C. 30-hour feasibility study should give a high-level indication into the relative merit of such an approach

====II. '''Formative Study'''====

A. '''Description of methodology/experimental procedure'''

1. In a highly-controlled task environment, understand how an integrated model comprising current theories in perception and HCI can predict/explain task performance

2. Users trained in the task extensively to control for learning (learning aspect of task could be investigated in future study)

B. '''Results'''

1. Users favored continuous rotation over static view or meditative pauses.

2. Hand gesticulation seemed to be used as a method of validation in stereo views.

3. Several verbal comments regarding occlusion suggest drawbacks to the tube view.

4. Rotating is beneficial and perhaps necessary for this task.

5. No references to external, offscreen information. In fact, very rarely did participants glance away from the screen.

6. Our assessment of a typical session beginning with a new dataset:

1. Data loads, split second decision to begin rotating.
2. Continuous rotation until proper viewpoint determined.
3. Rocking back and forth interaction about the optimal viewpoint. (Thinking?)
4. [In stereo views, participants were noted tilting their heads fairly consistently.]

C. '''Discussion'''

1. Some of the users' strategies can be explained by current theories in perception

D. '''Conclusion'''

1. Modeling low-level performance accurately does not seem to take into account for higher level workflow processes.

====III. '''30-hour study'''====

A. Now that we have investigated highly controlled task performance, explore larger workflow context

B. Perform study examining how interruptions affect user performance, and explore what coping strategies are used by users in such an environment, and explore whether a simple tool - working sets - helps improve performance by allowing developers to save, and load, and switch between window and window state configurations

1. Software developers (a good example of a challenging, and creative type of information work) will receive task requests by email; notifications will appear on their screen in real time

2. Each email will have differrent priorities (e.g., low, high, emergency)

3. Participants will be asked to manage priorities effectively to accomplish the tasks given

4. Once they begin working, we will "interrupt" them at controlled times with new task requests of differrent priorities

5. We will analyze their actions for coping strategies, metawork, and working spheres to try to understand how the larger workflow context is affected by interruptions

6. Goal will be to run 4-6 people, '''would like feedback on this'''. Can control for experience by recruiting experienced developers and students from the general population of Brown University.

7. Note on progress so far: I "ran" myself and found that switching tasks incurred a huge cost in returning to what I was doing. I think that tools for aiding in switching between working sets will significantly benefit developers in particular, and information workers in general. Some coping strategies I used: writing things on paper, typing notes into Notepad, and keeping previously used tabs open. Sometimes it would get too chaotic and I would need to close all open windows and reset from my notes. Overall, I would say that task switching/pausing/resuming in Visual Studio (test application) is not well supported.

===Adam Darlow===

I. '''Goals'''

A. Evaluate the interactions between various cognitive principles and design principles. There are three basic relations:

1. The cognitive principle is the motivation behind the design principle.
2. The cognitive principle suggests a method for achieving the design principle.
3. The cognitive principle and design principle are unrelated. (Hopefully few)

B. Make design rules which are suggested by the combination of a cognitive principle and a design principle.

II. '''Description of methodology/experimental procedure'''

A. Collecting commonly accepted design principles from the literature on interface design and well established cognitive principles from the cognitive psychology literature and constructing a matrix which crosses them. Most squares in the matrix should suggest specific design rules.

III. '''Results'''

As a preliminary effort, I have chosen the following two cognitive principles and three design principles:

Cognitive principles

#C1. People derive complex associations and causal interpretations from temporal correlations and patterns.

#C2. People have limited working memory (7 +- 2), but each space can hold a chunk of related information.

Design Principles (from Maeda (TBD link))

#D1. Achieve simplicity through thoughtful reduction.

#D2. Organization makes a system of many appear fewer.

#D3. Knowledge makes everything simpler.

The resulting matrix entries are as follows:

#C1 + D1. Remove extraneous correlations. Things shouldn't consistently and apparently change or happen in conjunction unless they are actually related and their relation is important to the user.

#C1 + D2. Use temporal and spatial contiguity to help users organize and group multiple events meaningfully.

#C1 + D3. Use temporal correlations to effectively teach the important causal relations inherent in the interface.

#C2 + D1. Reduce the interface such that a user has to be aware of no more than 5 items simultaneously.

#C2 + D2. Groups of semantically related items can for many purposes be treated as a single item.

#C2 + D3. Teach users how things are related so that it can be chunked.

V. '''Conclusion'''

VI. '''Future Directions'''

To expand the matrix and evaluate the resulting design rules.

===Usability/cognition matrix===
We have demonstrated the feasibility of mapping and quantifying the relationships between established design principles and a preliminary set of cognitive principles. By assigning specific rules for the evaluation and development of interfaces to these relations, we've begun the construction of a larger and more concrete set of design guidelines. Although the work here is based on manually-generated user ratings, we describe how the continuing development of this framework would incorporate more advanced and automated data collection. We've provided the foundations for continuing work towards a unified method of systematic, and in some cases automated, interface evaluation.

==Research plan==

We can speculate here about the details of a longer-term research plan, but it may not be necessary to actually flesh out this part of the "proposal". There does need to be enough to define what the overall proposed work is, but that may show up in earlier sections.

===Usability/cognition matrix===
We plan to select or create a varied set of interfaces for the same or similar tasks and ask users questions relating independently to both the chosen usability principles and cognitive principles, with the intent of using a machine learning algorithm to estimate the correlations between each cognitive principle and each design principle. For each highly-correlated pair, we will extract a design rule or suggestion based on both principles. [[User:E J Kalafarski|E J Kalafarski]] 16:54, 13 March 2009 (UTC)

====Cognitive principles====
* 7 plus-or-minus-two/cognitive load
* visual pop-out
* temporal and spacial contiguity
* queued recall
* affordance

====Design principles====
* appropriate visual presentation
* memory-load reduction
* consistency
* control
* discoverability

==References==
<references />

CS295J/Proposal intros from class 9

2009-03-17T15:45:42Z

Trevor O'Brien: /* Collaborative */

==Collaborative==

Established guidelines for designing human computer interfaces are based on experience, intuition and introspection. Because there is no integrated theoretical foundation, many rulesets have emerged with no intuitive way to compare or unify them. We propose to develop a theoretical foundation for interface design, drawing on recent advances in cognitive science—the study of how people think, perceive and interact with the world. We will distill a broad range of principles and computational models of cognition that are relevant to interface design and use them to compare and unify existing guidelines. Where possible, we will use computational models to enable richer automatic interface assessment than is currently available.

A large part of our project will be to broaden the range of cognitive theories that are used in HCI design. Only a few low level theories of perception and action, such as Fitts's law, have garnered general acceptance in the HCI community because they are simple, make quantitative predictions and apply without modification to a broad range of tasks and interfaces. Our aim is to produce similar predictive models that apply to these lower levels of perception as well as higher levels of cognition, including higher-level vision, learning, memory, attention and task management.

Much of our work will focus on how cognitive principles can enable interface design to go beyond the focus of the functionality of the individual application [a bit vague]. Much research has accumulated regarding how people manage multiple tasks and we will apply it to principles of how an interface should be designed with not only its own purpose in mind but such that it both helps maintain focus in a multi-tasking environment and minimizes the cost of switching to other tasks or applications in the same working sphere. The newer approach of distributed cognition also provides a different perspective by examining the human-computer system as a unified cognitive entity. We will extract and test principles from this literature on how to ensure that the human part of the system is only responsible for those parts of the task for which it is more capable than the computer.

===Collaborators so far===
* Adam
* [[User:E J Kalafarski|E J Kalafarski]] 15:25, 17 March 2009 (UTC)

==Alternative Collaborative (In progress...)==
Established guidelines for designing human computer interfaces are based on experience, intuition and introspection. Because there is no integrated theoretical foundation, many rule-sets have emerged despite the absence of comparative evaluations. We propose to develop a theoretical foundation for interface design, drawing on recent advances in cognitive science -- the study of how people think, perceive and interact with the world. We will distill a broad range of principles and computational models of cognition that are relevant to interface design and use them to compare and unify existing guidelines. To validate our theoretical foundation, we will use our findings to develop a quantitative mechanism for assessing interface designs, identifying interface elements that are detrimental to user performance, and suggesting effective alternatives. Results from this system will be explored over a set of case studies.

A central focus of our work will be to broaden the range of cognitive theories that are used in HCI design. Few low level theories of perception and action, such as Fitts's law, have garnered general acceptance in the HCI community because due to their simple, quantitative nature, and wide-spread applicability. Our aim is to produce similar predictive models that apply to lower levels of perception as well as higher levels of cognition, including higher-level vision, learning, memory, attention and task management.

We will focus on generating extensible, generalizable models of cognition that can be applied to a broad range of interface design challenges. Much research has accumulated regarding how people manage multiple tasks, and we will apply it to principles of how an interface should be designed with not only its own purpose in mind but such that it both helps maintain focus in a multi-tasking environment and minimizes the cost of switching to other tasks or applications in the same working sphere. The newer approach of distributed cognition also provides a different perspective by examining the human-computer system as a unified cognitive entity.

===Collaborators so far===
* Adam
* Trevor

==Gideon==

*Creating a model of HCI based on distributed cognition
**Our model (and its many levels)
**Consideration of cognitive theory
**Contributions
***Framework
***Quantitative Models
**End-Product